More data is being created today than that seen from the dawn of man to 2003. In fact, the amount of data we now have at our fingertips doubles in size every 18 months. It has exploded at such a rate that data, or ‘big data’ as it’s now known, is profoundly changing the way businesses are innovating and developing their next generation of products and services. But unless brand owners understand how to analyse and interpret big data, they may be confusing a spurious result with a fabulous insight.

Big data, at least to most marketers, looks like a dashboard. It’s simply presented data in neatly crafted reporting systems – basic visualisations of KPIs, vital indicators and trends data – to enable quick interpretation and clear storytelling.

Big data versus traditional analysis

Big data finds patterns in that myriad complex mass of data. These are patterns that shed light into previously murky corners of our brand or product experience to tell us things we’ve never previously known.

Traditional business intelligence software summarises data using statistics to look for patterns and hierarchies on a mission-critical basis. It’s run by analysts and IT experts and is an expensive, high-quality resource to some companies or briefed out to specialist research agencies.

By way of contrast, big data uses any data available and doesn’t try to cleanly integrate the data sets, instead using different analysis methods, such as ‘Magnetic Agile Deep’ (MAD) analysis.

Big data doesn’t rely on a long-range, carefully-designed analysis plan, but literally looks for patterns, albeit for a purpose. It allows brand owners to see both the wood and the trees.

Unconstrained data analysis leads to paradigm shifts in innovation, which is great news for businesses investing in this area.

But, brands be warned: sometimes all is not what it may seem.

The problem with meaningless noise

Dr Michael Shermer, founder of the Skeptics Society and widely published scientist, coined the word ‘patternicity’ in 2008 as “the tendency to find meaningful patterns in meaningless noise”.  Shermer argues that we are hardwired to seek patterns because we have evolved as ‘belief-engines’: “pattern-recognition machines that connect the dots and create meaning out of the patterns that we think we see in nature” (Scientific American, 2008). Statisticians call these pattern-seeking errors ‘Type I’ errors – believing something is true when it’s not.

And this is the Achilles heel of big data. The more variables included in an analysis, the disproportionately higher the number of spurious results. Unless you understand the nature of the data and the complexities of the possible interactions you may think you are looking at a fabulous insight but in fact you are looking at a spurious result.

So, the real skill is in the interpretation of the analysis to identify the ‘true’, meaningful patterns.

Research no longer the enemy of creativity

As companies are realising how big data is impacting on innovations and changing the way they do business, many have begun to look for people who can make sense of data – people who enjoy data, analysis, rigour and sampling and method. Data analysts, statisticians and researchers are all now working at the pointy end of insights to drive better innovation throughout the business cycle.

And just as it seems that the rise of the nerd is inexorable, so big data is shifting the gravity in the world of innovation from creativity to analysis.

If you don’t believe me, listen to this guy: “If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis.” That’s Professor Hal Varian, UC Berkeley, chief economist at Google.

Big data is useless without good analysts and the impact of this embrace of what were previously market researchers has been profound. Advertising agencies, for one, are galloping to employ experts in big data.

Meanwhile, the research profession has been undergoing a transformation, first to ‘insights’, and, via planning, to strategy. It now seems that big data has made the very data itself king. From process (research), to benefit (insights), it’s now back to the future with big data (data and analysis).

Research as a discipline has moved from mitigating risk and being the enemy of creativity, to empowering innovation, to being the innovation in the creative industry.

Data analysts have become the new creators.

Implications for marketing and advertising

McKinsey tell us that big data has “the potential to drive a radical transformation in research, innovation, and marketing”. The UK’s largest retailer, Tesco, is a great example of this. Tesco collects the data on all of its customers’ transactions through its loyalty card and then analyses this data to identify new business opportunities across pricing, promotions, and place.

The retailer was also one of the first to recognise that big data is useless without great analysts. Following the joint founding of Tesco Clubcard in 1995, the retailer wholly acquired data analytics pioneers DunnHumby in 2010. Tesco has built its success on analysing big data, with the volume of data doubling 11 times over since Tesco started. No wonder they bought DunnHumby.

Further, in a clear sign for the future of marketing, big data and advertising, in 2012 Tesco bought US-based word-of-mouth marketing agency, Buzz Agent, for US$60 million. Today, the bricks and mortar retailer holds all the data on its customers and has the ability to analyse this big data to inform and guide innovation at record speed.

Tesco now also owns a consumer media channel through which it can directly communicate specific messages to targeted consumer segments. And given that word-of-mouth marketing agencies don’t make ads, it’s looking like Tesco will be less reliant on traditional advertising campaigns to drive its future success.

The end of Mad Men?

So, big data is either a pretty dashboard of well-presented, easy-to-interpret data sets or else a very well-presented, clever analytical process, which drives paradigm-shifting innovations.

I suspect that in the rush to embrace big data we may still be seeing many more pretty front-end dashboards than we will Tesco Clubcards. This is why ‘big data’ is such a ‘big thing’ in the world of advertising.

Everyone recognises how badly so much quantitative data has been historically presented and how good it is to see data-driven success stories. But great analysis is actually the key to big data, rather than the data itself.

It might be anathema to the creative department but the nerds are taking over.

Move over Mad Men. Make way for MAD data.