Drinking from the big ‘data hosepipe’

Today’s marketers have access to more consumer data than ever. But they need to learn to use the ‘data hydrant’. Mark Earls argues that it is important to be less precise – to draw bigger conclusions from large datasets by spotting important patterns, rather than smaller ones at lower levels of granularity

We live in a world of ‘big data’. It must be true: McKinsey Quarterly, The Economist and even Campaign magazine all agree. All of these fine publications have published articles in recent months exhorting managers to get jiggy with what they call the ‘data hosepipe’.

Thanks to the proliferation of interconnected devices and machines (what the geeks call ‘the internet of things’) and the explosion of interconnected people (aka ‘the social web’), there’s now a constant flow of data coming our way about people and things and people and brands and people and people. A constant and unrelenting torrent of data that somebody – somehow – in your organisation needs to get to grips with and to sweat for insight (or everyone does, if you believe McKinsey).

All the experts agree that the hosepipe is on (even if no one can quite nail the actual gobsmacking size of the aperture or the speed of flow). The International Data Corporation (IDC) suggests that there are now more bytes of consumer data than all of the grains of sand on all of the beaches in all of the world. The Economist, by contrast, points out that Tesco alone generates 1.5 billion bytes of customer data a month.

Whatever the specifics, this should be good news for marketers: we have more and more information about the people who should be our greatest concern – ‘consumers’ – which is excellent news. The truth, however, is that in many organisations marketing runs the risk of being overwhelmed by consumer data. We are discovering you can have too much of a supposedly good thing.

getting to grips with data How times change. Some of you will remember (as I do) doing manual copies of Nielsen sales data and plotting graphs manually to align these across categories, with promotional activity or with the tracking study. Many of you will remember the excitement in the early days of marketing dashboards as we tried to knit together half a dozen disparate datasets. Now it’s quite common to have four or five times as many datasets to synthesise. In addition to the sales figures and longitudinal quant studies we used to have, we now have call centre records, customer service and customer satisfaction numbers as well as real-time transaction data. And still we complain about what’s missing.

That’s without considering the never-ending source of amusement and distraction that is social media: you merely have to flick a switch to be washed away by the conversations, the griping and the enthusing, the gossip and the speculation that real people are having in real time about our services and products (and those of our competitors). And, of course, there are any number of different companies offering to read the social web for you – many of them excellent and pioneering.

The truth is it’s genuinely hard – however good your dashboard – to make sense of so much data; it’s genuinely difficult to keep abreast of so many datasets moving so fast; genuinely bewildering to try to work out what to do with all this data; and it’s nigh on impossible to set strategy in the face of so much information.

"The truth is that in many organisations, marketing runs the risk of being overwhelmed by consumer data. We are discovering you can have too much of a supposedly good thing"

Just think of Wall Street: how has more data helped traders and their employers get better at predicting the future? Or avoiding the biases that the behavioural economists describe – loss aversion, framing etc. Traders can’t possibly compute the amount of information that’s coming onto their screens in real time (which is probably why the more successful traders are also heavy users of instant messaging platforms which help them keep in touch with the reactions of their peers to new information – a sort of ‘phone a friend’ option for the City).

How do you cope? I’ve noticed a number of different kinds of responses to the hosepipe from the marketers I meet. Some have always preferred the ostrich approach to being hosed by data: rather than up their engagement with this rich nutrient liquor, this kind of marketer is actually using less data than ever before and instead falling back on instinct and ‘best practice’ (ie what other companies do). Others – as in The Lord of the Rings – defer to one mighty measure (‘one… to hold them and in the darkness bind them’) that will run everything else.

Fans of the (otherwise excellent) Net Promoter Score are a good example of this. For many users, it comes to so dominate the foreground of marketing thinking that few other measurements are paid much attention. Another such treasure is Millward Brown’s Awareness Index, which is perhaps more useful beyond its intended use (the measurement of advertising effectiveness), for example as a means of setting individual and agency performance targets.

"The more precisely we can read something, the more we obsess with micro changes in a data line and thus the more signal becomes noise. Too much monitoring of social media falls into this particular trap – more noise than signal"

At the same time, there is the opposite tendency: inappropriate precision. The more precisely we can read something, the more we obsess with micro changes in a data line and thus the more signal becomes noise. Too much monitoring of social media falls into this particular trap – more noise than signal. Old-school tracking studies also tend to end up in the same space – what fun we’ve had arguing about two points up or down.

Related to this tendency is the love of micro-segmentations that direct marketers and database software sales teams have. Recently, I joked with a US audience about having seen a 30-plus consumer segmentation solution; one plucky person confessed to having worked with a 300-plus segment solution. Because we can, we do; having access to more granulation must be good, mustn’t it?

There are two kinds of problems with this. First, there’s the seductive nature of infinite detail. In his short story, On Exactitude in Science, Jorge Luis Borges describes a civilisation in which map making became so all-consuming that the maps ended up being bigger and more detailed than the landscape being mapped. Eventually, the maps were abandoned and scraps left fluttering in the breeze. The point is this: just because we can analyse and reanalyse in incredible detail doesn’t mean we should.

Second, it turns out that coarse granularity is better than fine granularity when you are trying to build a strategy.

One person who learned about these kinds of dangers – and did so the hard way – is Hilary Clinton. During her fight for the Democratic Presidential nomination back in 2008, her favourite pollster, Mark Penn, is supposed to have kept providing more and more nuanced insights about smaller and smaller segments, each of which would require a bespoke policy offer from Senator Clinton. At the same time, the Obama team were finding ways to unite larger and larger groups with shared insights.

Less not more precision So what I’m proposing instead for marketers who are struggling to deal with the data hosepipe is this: less precision, not more; bigger insights from big patterns, not smaller and smaller ones. Insights that help us make big strategic decisions, rather than merely guiding our tactical ones.

The key to this is simple. Rather than crunch a big dataset into tiny pieces, learn to spot the patterns that tell big important stories. One pioneer of this is a personal hero of mine – the late and great Andrew Ehrenberg. He insisted on rounding up numbers in any analysis to stop decimal points distracting the eyes and minds of those reading it with small fluttering variations and instead draw their attention to the similarities between brands or across markets. Equally, his Dirichlet model was – and remains – a really beautiful pattern that a number of fmcg markets exhibit.

The kinds of patterns I have in mind have roots in the diffusion science of the 1950s and 1960s, similar to those of Ehrenberg, but they are simpler to recognise and easier to draw, too. The most obvious one here is the long-tail distribution (made famous by TED founder Chris Anderson in his book of the same name).

A long tail is a plot of popularity ranking versus sales in which a handful of items in the market enjoy a majority of the market’s share; conversely, most of the items in the market (the tail) have little or no market share. Lots of markets create long tails (hence the fact that Anderson’s book was a book rather than just an article), most famously what the academics call ‘culture markets’ such as music downloads, book sales and fashion items, alcoholic drinks, dog breeds, baby names and so on.

why is this a pattern worth spotting? Put simply, the presence of a long-tail pattern in a market is a strong indication that that market is shaped by social influence rather than independent choice: that consumer purchase decisions in it are fundamentally social rather than individual.

If your product really is bought independently, then the traditional kind of messaging (persuasive product-led propositions) and the kind of media considerations and vehicles (reach and cost) will be fine; but if it’s socially chosen, then you’re going to have to work social influence and social channels much more, particularly in the modern world. It’s not that product excellence doesn’t count, it’s just that what people say to each other and see each other doing is much more so.

Think how beer and alcohol advertising have changed in recent years (to involve social techniques and media), reflecting the fact that booze is a fundamentally social choice – as in ‘I’ll have what she’s having’. This kind of analysis makes it clearer which side of this divide you need to be on with your comms.

Perhaps the significance of the different implications of each is best crystallised in the kind of market research approach you might want to use in each case. So, if you’re sure your market is built on independent choice, then by all means ask people about the product and how it makes them feel (bearing in mind, of course, the fact the respondents are much less reliable witnesses about their own lives to themselves, and to us).

"If your product really is bought independently, then the traditional kind of messaging will be fine; but if it’s socially chosen, you’re going to have to work social influence and social channels more"

If, by contrast, your market is socially shaped, then your research approach should focus on how consumers are connected to each other and how willing and able they are to copy from each other, and how you might go about changing these things. These are not things that traditional research methodologies and researchers are necessarily very good at, so different kinds of research techniques such as network analysis and ethnography become useful here.

The point being that this kind of pattern spotting that makes it easier to know the kind of thing you’re dealing with – the kind of behaviour you’re trying to change – has got to be job number one for any marketing strategy, hasn’t it? Not everything is a special case This kind of general question ‘what kind of thing are we dealing with?’ has never been an easy one for marketers and their advisers. Now, as much as ever, we feel compelled to treat every case as a special one; every instance as unique. From the marketer’s perspective it must be flattering to be told by your agency or other advisors that your brand is like no other but is it helpful to talk and think in this way?

Equally, while it’s somehow reassuring to think of the marketing challenge you’re facing as uniquely tricky as opposed to generally so – it makes the person meeting that challenge seem somewhat more special, more heroic even. But does it really help us solve the problems better? In particular, does it encourage us to apply knowledge, experience and skills from elsewhere? All too often, in my experience, this ‘special case’ leads us to ignore what we already know as marketers.

We follow the same kind of thinking when we demand ‘category experience’ or ‘category-specific expertise’ from our advisors and teams – as if the learnings of a particular category are only relevant within it; as if different categories of product were bought by entirely different kinds of species or different kinds of cultures.

The fact that the kinds of patterns I’ve described exist in our data tells us that there are important underlying communalities between behaviour in different markets. They show the big important and similar things that you need to know to build the basics of your strategy. Pattern spotting allows us to identify important similarities between things (as opposed to small and insignificant differences).

Patterns help you see the signal for the noise: the stuff that matters. And seeing what matters is what strategy needs to be built on.

Conclusions Today’s marketers are awash with consumer data – we have more than any previous generation of marketers have ever had – which could help us understand our consumers. Yet more data actually seems to mean less. For many of us, being exposed to the hosepipe of data makes it harder to engage with data in the same way. Some of us default to the ostrich position, others seek a silver bullet that we hope will make analysing everything else less necessary. And the temptation to boil the ocean doesn’t help us much either – digging deeper and deeper, becoming more and more precise.

Of course, precision is important and useful when planning implementation but what’s needed for strategy is exactly the opposite: it is the ability to spot patterns in the data. Patterns signal important and significant phenomena below the surface of a market. It’s not all of the answer, but getting these things right, and doing so rapidly, makes it possible to build strong and clear strategies and create time and space to sweat the detail of implementation.

I’ll Have What She’s Having – Mapping Social Behavior’ by Alex Bentley, Mark Earls and Mike O’Brien was published by MIT Press on 3 November 2011.

Mark Earls is HERDmeister at HERD

[email protected]