Building better surveys – a comprehensive guide to getting the truth
Data is valuable. It’s intel, it’s knowledge, it’s power. But the willingness of the public to tell the truth in surveys is increasingly in question. Dipesh Soneji on the truth.
This article was sponsored by Ruby Cha Cha to let readers know about its strategic market research services »
Surveys are the most useful way to measure the wide spread opinions of a target market. Their rigour and audience breadth is not something that can be achieved by other means like focus groups or online communities.
The trust issues with survey data are driven by several factors. Think about how wrong political polling has been for major elections and referendums. Take Brexit – even on the day of the referendum, some polling organisations were predicting a ‘remain’ result. Of course, hindsight is 20/20, but imagine the embarrassment felt by those in charge of calling the wrong result.
Marketers cannot afford to call the wrong shots based on poor data, as the repercussions can cost millions of dollars.
The barriers to truth in surveys
As researchers we don’t always make it easy on survey participants – sometimes we bore, sometimes we ask the impossible and sometimes we confuse. Factor in human nature and this leads to several barriers to getting to the truth:
- Right answers, wrong people: findings from surveys must represent the profile of the target audience. For example, if a survey was designed to reflect the views of Australians aged 18 and over, yet it achieved a gender mix of 70% to 30%, it would be presenting a false truth of how the country feels.
- It’s a chore: long surveys are a burden and people rush their way through, paying less attention to the questions as they progress.
- Right questions, wrong options: if a question doesn’t provide the most relevant answer options the truth is never uncovered.
- The ‘think so’: for some questions, people don’t mean to lie. They believe they are answering correctly – such in cases of as remembering past events or predicting future actions. In the words of George Costanza, “It’s not a lie… if you believe it.” But these types of questions can lead to levels of ‘overclaim’ and ‘underclaim’.
- There’s ego: people want to build their self-worth and appear to be better than they are, leading to overclaim.
- The Open Self: people lean in to their Open Self – (the public self which is known by others) leading to answers they think are more socially desirable.
- The Hidden Self: people don’t always want to reveal their Hidden Self (the self others don’t know about) even with the anonymity of online surveys.
But none of this means that you shouldn’t use survey data.
Survey data is a powerful asset as it models your market through needs-based segmentations, provides rigorous competitor analysis and categorically tells you which NPD concepts should be pursued, and which should be canned.
Eight ways to getting better answers out of surveys
It’s the role of researchers to reduce the potential for inaccurate answers and to get the truth from people. Key ways to achieve this include:
Results of market research can be misleading if the profile doesn’t accurately represent the target audience. When it comes to being representative, it’s crucial for a marketer to brief an agency on what matters the most. The marketing plan or previous research may demonstrate that it is about getting ‘the classics’ right: correct age, gender and location proportions. However, it could be that the right proportion of segments or ethnic backgrounds or household make-up are what matter most to driving the category.
Yes, people are being incentivised to take part in surveys, but we still have to respect their time and attention by being focused in our surveys. Aim to collect just the data you need to answer your brief, so the person isn’t overburdened. Manage the person’s time expectations up front and keep your surveys as short as possible – anything superfluous is superfluous.
Make it easier for people to answer questions by reducing the potential for confusion. You don’t have to use formalised ‘survey speak’ wording (largely developed more than 50 years ago!). Don’t use marketing jargon – reword so it matches how real people speak; make scales relevant – do they need to be so granular?
In quantitative surveys there are few opportunities for people to articulate their opinions and thoughts. If we want to allow people to tell us the truth, it’s crucial to include the best possible answer options. The best way to do this is to walk in their shoes and use qualitative research upfront to help design the most relevant answer options.
5. Be realistic
Sometimes survey questions ask the impossible and then the results are taken as gospel. Don’t ask about events which took place a long time ago or for information that requires very small details and is likely to have been forgotten.
Build logic checks into surveys – these can be used to see if people are paying attention (a certain option must be selected to continue, for example) and identify contradictory responses. Deleting any unreliable people from the dataset gets you cleaner data and closer to the truth.
If you’re a marketer that captures behavioural data on your customer base, you can use this to calibrate the results of a survey. For example: if the customer database captures the actual order frequency and the survey captures claimed order frequency, the two values should be compared to understand the potential level of overclaim or underclaim in the survey. Crucially, this can then help calibrate the rest of the survey results, such as predicting actual NPD purchase intent using known behaviours.
There is real skill involved in getting the right interpretation of a dataset to pull out the truth. This is achieved by understanding the true context of each fact and incorporating this in its interpretation. Statistics without context have no meaning, they are simply numbers.
Marketers have many sources of reliable information which can be used to build strategy. Surveys are essential for segmenting audiences and understanding drivers and market forces. The key to getting to the truth, though, lies in a fine balancing act of study, design and interpretation. This delivers reliable consumer data – to help marketers make better decisions, to measure ROI of new initiatives and to talk to trade. The culmination of this ensures their brand has the best chance of winning.
Dipesh Soneji is director of quantitative research at Ruby Cha Cha, a strategic market research consultancy.