Marketing Effectiveness Double Jeopardy v.2

The DMA Intelligent Marketing Databank has now been updated with 2021 data and covers insight from over 1,000 campaigns. Read about some of the key insights derived from the latest batch of data in WARC.

What does effective marketing measurement look like? Ian Gibbs, founder of Data Stories Consulting and data expert for the Data & Marketing Association (DMA UK), reveals all.

You are what you measure. And if all you are measuring are ‘vanity metrics’ then that’s what your marketing efforts are always destined to be: mere vanity projects that have no real-world impact on your brand or your business.

If we don’t have true visibility on campaign effectiveness, then the logical conclusion can only be that we’re simply flying blind with our marketing spend, without any real clue as to what is working and what isn’t. This leaves the impact of marketing something akin to a game of roulette with no real scientific basis on which to assess the drivers and inhibitors of campaign success.

Now in its second year, the DMA’s Intelligent Marketing Databank contains half a decade’s worth of effectiveness data, covering over 1000 campaigns from a range of sectors and media channels. Crucially, the databank is designed to answer two fundamental questions: A) How successful are marketers’ efforts at generating campaign impact? and B) How successful are marketers at measuring the effectiveness of their marketing efforts?

This is where double jeopardy comes in to play as the industry needs to get to grips with two fundamental challenges posed by both questions, namely:

  1. Marketers continue to distort their measurement plans by placing an over-reliance on campaign delivery metrics (like reach and impressions) and vanity metrics (like clicks and open rates) in post-campaign evaluation (41% of the measures being used by marketers relate to campaign delivery metrics), rather than true measures of impact relating to response, brand and business effects.
  2. Marketing effectiveness is on the decline. After an initial effectiveness boost in the early pandemic phase of 2020, the number of effects generated per campaign declined by 23% in 2021. This picture is true even when less meaningful campaign delivery metrics have been excluded from the analysis – i.e. when focusing on measures of short-term response, brand building, or overall business effectiveness.

What’s causing the decline in effectiveness?

While there are various explanations for the decline in effectiveness observed in 2021, a handful of overriding drivers can be identified.

While the number of brand effects per campaign (e.g. brand awareness or brand consideration) has increased marginally year-on-year, this has been outweighed by a sharp decline in response effectiveness – i.e. the ability of brands to drive short-term sales, acquisitions, sign-ups and the various types of metrics usually associated with “performance” marketing. Response effectiveness actually improved in 2020 as advertisers did more with less and did what they could to sustain their businesses through the challenges of the early pandemic phase. A year later, however, and it has not been possible to sustain this type of performance in the short term.

Pre-pandemic, 44% of campaigns ran in a short-term time frame (i.e. up to three months in duration). This shot up to 53% in 2020 as businesses focused on short-term survival and stayed this way in 2021. Short-term campaigns accumulate fewer effects than their medium or long-term counterparts and, without a rebalancing to longer-term activity, overall effectiveness has been on the wane in 2021.

In 2021, the 45% of campaigns that a had a pure customer acquisition objective was a record low in the five years’ worth of effectiveness data in the databank. While customer retention is a vital tool in the marketing mix, a balanced approach to retention and acquisition is key to long-term business growth, with acquisition campaigns overall recording more effects than retention-only campaigns.

The final driver of effectiveness decline is harder to prove but is a debate worth having. As an industry, if we’re not getting measurement right then are we optimising future campaigns as effectively as we could be? In the spirit of test and learn, campaign evaluation should not just be a box-ticking exercise. Alongside standard media planning inputs, performance marketing targets, and, frankly, the unquantifiable experience of marketers, campaign evaluation should be a necessary input into future campaign plans rather than just a simple justification of past spend. If we believe in the mantra that “the more you measure, the more you grow”, poor campaign measurement can only have a long-term negative impact on overall marketing effectiveness.

What does good measurement look like?

Currently accounting for 41% of all effectiveness measures used, marketers need to steer away from using simple campaign delivery metrics and vanity metrics when evaluating campaign success. Some of these metrics are useful in the campaign planning phase and are sometimes useful when optimising between different pieces of digital creative. However, as true measures of campaign success they have little meaning.

Instead, marketers must evaluate campaigns according to stated campaign objectives (which may sound obvious, but for many it is easier said than done). Response objectives (such as sales, sign-ups, acquisitions and leads) currently account for 36% of all measures identified in the databank. Brand objectives (such as awareness, consideration and purchase intent) make up 17% of measures, and business measures (such as profit growth, shareholder value and market share) 6% of measures. In the future, 100% of measures should be focused on these three core groups of metrics.

Most notably, the number of business metrics reported in 2021 has declined. Business metrics transcend the day-to-day language of the marketing team. They speak the language of the boardroom and bridge the gap between CMO and CEO. At a time when consumer budgets are being squeezed, eliciting campaign response is harder to come by. Marketing budgets are under more pressure than ever, so the language of business metrics is going to be vital for marketers to get to grips with if they are to unlock more budget.

You are what you measure, so make sure you are measuring for business success, not vanity.

Launching the Intelligent Marketing Databank 2022

Register now for the DMA’s Making Measurement Meaningful event on the 28th April!

We’ll be launching the latest findings from the Intelligent Marketing Databank, which now contains effectiveness data from over 1,000 campaigns.

Brand and Response, Retention and Acquisition, Multi-channel, Attention, How Small Brands Grow…. we’re going to be covering a lot of ground and to cap it all this is all going to be face-to-face. See you there!

Removing ‘vanity’ metrics from marketing

This article first appeared in Research Live:

Are we condemned to repeat the mistakes of generations past? When it comes to measuring marketing effectiveness is does sometimes feel that way.

When I started my career in media research more than 15 years ago, I was shown a pie chart. It was a very simple pie chart with just a tiny sliver highlighted to show what proportion of ads are typically clicked on – ‘0.01% click-through rate (CTR)’ said the chart title. But what about the impact of the other 99.99% who saw your ad but didn’t click?

A neat summary of the challenge of using a relatively meaningless digital vanity metric to measure campaign performance, but sadly a conversation that is as common today as it was over a decade ago. The problem was most recently highlighted in the Meaningful Marketing Measurement 2021 report produced by the Data & Marketing Association (DMA) and Data Stories Consulting.

The report features insight drawn from the newly created DMA Intelligent Marketing Databank, built on the foundations of more than 850 DMA awards entries. While the report offers insights into the drivers of marketing campaign effectiveness, of more relevance to the research, data and insight community is the analysis of the measurement of campaign effectiveness.

There were 167 methods of articulating campaign effectiveness identified across the 850 campaigns – in itself this is an issue if you work for an organisation that is trying to develop a common language around marketing effectiveness.

Of greater concern, is that just 60% of the measures identified have real value to marketers and measurement practitioners. They are concerned with response measures such as sales, conversions and acquisitions; brand measures such as awareness, consideration and purchase intent; and broader business effects such as profitability, market share growth and shareholder value.

The remaining 40% sadly tell us nothing of true campaign impact. They relate to media planning metrics such as reach, frequency and impression delivery, and digital vanity metrics such as clicks, likes and shares.

Media planning metrics have considerable value at the start of the planning process and are of course useful metrics in the media auditing process – giving marketers some assurance that they actually got what they paid for. As measures of campaign impact they are insufficient.

Digital vanity metrics might have some value as measures by which display, social and video campaigns can be optimised while in-flight. But who cares if 10 million people shared your ad if it can’t be linked back to some sort of positive commercial outcome for your organisation?

The problem tends to be worse for brand campaigns than direct response campaigns. More than half of brand campaigns are measured using meaningless campaign delivery and digital vanity metrics, compared with nearly 30% of direct response campaigns. Arguably, the use of a CTR to measure the emotional reaction that brand campaigns attempt to trigger in consumers does more disservice to brand building efforts than it does to direct response, yet still the problem persists.

Talking to peers in the industry, there are a number of drivers of this tendency towards mismeasurement:

  • Digital ads have produced a ubiquity of campaign data. Marketers are measuring performance using the most easily available data, rather than the most relevant data.
  • The effort required to access the most relevant data isn’t seen to be worth the reward. Time-poor marketers and measurement practitioners are gripped by inertia and it is simply easier to continue doing what has always been done.
  • Measuring brand campaigns is seen as an expensive and time consuming effort in an industry overly-obsessed with delivering short-term results.
  • Perhaps some marketers simply don’t want this level of scrutiny on their campaigns? Perhaps it’s easier to paint the most flattering picture of campaign performance when there aren’t robust data sources around to call it in to question?

This may all sound a bit doom and gloom, but it shouldn’t be. The mismeasurement of campaign effectiveness should be seen as an opportunity. We shouldn’t be afraid of the scrutiny measurement places on campaign performance.

The most successful measurement frameworks plug campaign effectiveness data into a continuous feedback loop of ongoing marketing performance improvement, helping marketers to better sing about their achievements rather than be afraid of their failures.

The research, data and insight industry needs to better equip marketers with the tools they need to measure campaign effectiveness at all budget levels, and challenge the perceptions that robust measurement is out of reach for all but the largest brands. At the same time, marketers must become comfortable with assigning some of their campaign budget to robust measurement programs. Sacrificing a bit of reach for a lot of certainty is a trade-off worth making.

Media owners and platforms, who are guilty of providing meaningless campaign delivery and digital vanity metrics in post-campaign reports, also need to sharpen up their measurement offerings to marketers. It’s not too far-fetched to link back an obsession with CTRs to nearly everything that’s wrong with the digital ecosystem, from ad fraud and the explosion of low quality click-bait style content, to the lack of sustainable ad models for premium digital publishers.

Get measurement right, and a lot of positive benefits for the entire marketing ecosystem should follow. The aspiration has to be to show consumers fewer, better ads.

That aspiration all starts with good measurement. Let’s get rid of the 40% of meaningless campaign delivery and digital vanity metrics in our measurement plans and focus on what really matters.

Ian Gibbs is founder of Data Stories Consulting

Listen to Data Stories on the Data Stories podcast (nominative determinism in action)

From the Audiense website:

Data Stories: Leaders at Work is a weekly podcast brought to you by Audiense. Hosted by Rahul Jerome, founder of, the series captures personal anecdotes and career highlights from some of the most talented and brightest minds in the research and insights industry.

On this episode of Data Stories: Leaders at Work, I sit down with Ian Gibbs, founder of Data Stories Consulting – an independent data consultancy focused on advertising, publishing, digital, and media.

Ian first discusses his background and the journey to his current position. He mentions that business, economics, and statistics always came naturally to him, so he studied business economics in university. Ian admits that though he enjoyed the statistical, numbers side to economics, he gravitated more towards the marketing, theory side. After university, he was unsure how to utilise this degree; however, a tech market analyst position fell in his lap. Ian explains that this was the first stepping stone in his career.

Ian shares his experience as he transitioned from the agency side to the client-side and the aspects he enjoyed on the client-side at The Guardian. At the time of his hire and most of his tenure at The Guardian, Ian explains that there were profound structural changes occurring within the company due to the rise in technological advances. For example, the shift from printed articles to articles featured online was pivotal. Most of his work involved the strategic application of research to reshape the business. Overall, he was appreciative of being part of the company while they underwent these drastic changes.

Currently, Ian divides his time between joint industry currency for mail and running Data Stories Consulting. His main focus is to work with other businesses across the advertising ecosystem to help with various measuring challenges. The main project at the moment is DMA intelligent marketing data bank. Ian explains that the decision to have his own consulting business was a gentle push for him – his partner encouraged him to pursue this endeavor, coupled with an opportunity to leave his prior company. Though this decision was risky, Ian admits that the success he has already had with his company was well worth the risk – for Ian, the transition from The Guardian to an independent agency was a key moment in his career.

An essential piece Ian wished he had known before he started his career was simply the fact that his current position even existed. He says that better planning then may have helped him even further – he would be more developed in his expertise and have established even more connections that have aided in his current success.

Ian shares a few pieces of advice for prospective freelance researchers. First, he mentions that possessing a certain skill level is vital – there is a skill shortage in his particular field, and he is hopeful that future people within the sector will be adept in the field. Next, he highlights the importance of having a few potential clients lined up before opening a business – starting from square one and proving yourself will be more difficult than having clients who already trust you and are willing to use your service. Lastly, Ian emphasizes the idea of networking and how crucial using those connections will aid in the success of a business. Using these connections well will help build yourself and will provide an opportunity for your brand to develop effectively. Ian also cautions the listener that broadcasting yourself in a broad sense is not efficient. He suggests that it is imperative to be specific and skilled in your field to be more marketable – putting in the extra effort will go a long way.

Ian provides a couple of books and a few people that have inspired him. First, he shares two books that piqued his interest: ‘Weapons of Math Destruction’ and ‘The Average is Always Wrong’. Ian also discusses the work of various people within the field that has inspired him. He mentions that he would be remiss to not highlight his boss, Andre, at The Guardian who helped him develop and progress in the company.

When asked about certain skills that are essential for the industry, Ian lists a few that he believes are important:

  • Have an understanding that data and insights have two distinct roles: internal versus external
    • Strategic insights – shape future strategy and path
    • Marketing – convey important topics and issues to the world
  • Develop soft skills
    • Constantly possess a curiosity and passion in your field
  • If you are interested in the client-side, make sure you own what you produce
    • Elevate your standing around your projects
    • Don’t let others take credit for your work

Ian also provides advice to his younger self. He mentions that if you are not enjoying the aspects of your job, do not be afraid to move – it is easier to change jobs at the beginning of a career. He also stresses the importance of being flexible so that more opportunities may open for you.

Lastly, Ian’s personal vision for the next three to five years is to progress his current collaboration – intelligent marketing data bank in conjunction with the DMA. He hopes this will be a credible source of effectiveness data; however, Ian realises that the success of this collaboration will take time. Additionally, he feels the need to take more responsibility in the ethics in the work involving media and advertising measurements – quality and effectiveness need to be stressed.

The full version of the podcast with Ian Gibbs can be listened here:

Launching the DMA Intelligent Marketing Databank

The DMA Intelligent Marketing Databank is coming! This is a very exciting collaboration between Data Stories Consulting and the DMA, harnessing campaign effectiveness insight from over 850 DMA award entries.
Join us for our launch event on the 21st July as we give an honest assessment on how the industry measures campaign impact; reveal the scale of the pandemic effectiveness boom; and offer insight in to the key drivers of campaign effectiveness over the last four years.

Sign up to the launch webinar here.

You can plan it, but can you measure it?

In 24 hours time I’ll be hosting a measurement panel debate with these very clever people for Facebook Select: Facebook’s key trade engagement program attended by over 150 senior agency and advertiser clients. Very much looking forward to hearing what Jennie Roper, Shazia Ginai, Georgia Protopapa and Nicolas Arrive have to say!

The Drivers of Brand Effectiveness

As always, I’ve enjoyed being given the keys to On Device Research‘s ever expanding ad effectiveness database once again. In a major new study analysing over 550 campaigns we’ve been able to shine a light on the top three media planning drivers of brand effectivenes. Video is up there, as are multi-channel campaigns… but there’s good news for premium publishers too, whose rates probably don’t reflect the massive impact they can have on brand building KPIs.

The full report – Planning for Brand Effectiveness – can be downloaded here:

The life of Mail in lockdown…

As JICMAIL’s Director of Data Leadership it’s been fascinating to study what effect lockdown has had on the nation’s mail interaction rates. It’s been a positive double-whammy for advertisers who have stayed active with Direct Mail and Door Drops in Q2 2020: increased ad impression delivery through increased interaction, plus a higher share of of the door mat.

Find out more here.