83% of people make this mistake
This article is part of a series by Katie Harris, principal at Zebra Connections (if you missed the other parts, start here).
I’m going to do a ‘before and after’ exercise on a sentence I pulled from a qualitative research report that I was recently asked to read.
This is the before sentence:
“Of the four groups, 83% of people said they liked the design”.
Now, at this point, some of you will already know exactly what I’m talking about: you can go and play. For those who don’t please keep reading.
There are many things wrong with the sentence above masquerading as a qualitative research finding. Here are three points to start:
1. Using percentages
Using percentages, in a qualitative context, is rather meaningless. While at first blush, a grand 83% looks pretty good, what does it really mean?
It means that, assuming four groups of eight participants, 26.6 research participants, screened to fit a particular profile, and willing to attend a particular research group, said they liked the design.
That’s a very small, very skewed sample. It’s hardly likely to be even vaguely representative of the entire market (in a statistically sound sense). But that’s only the beginning…
2. It’s not controlled
To get a good, clean read on any particular issue in a quantitative survey, the way the questions are ordered, and the way the questions are asked, are critical factors.
For all intents and purposes, and as much as possible, the survey should be administered in a relatively controlled environment. Even rotating the order of questions is controlled. This ensures reliability (being able to replicate the findings) and therefore, some confidence in the results.
In contrast, to get a good read in a qualitative study (we don’t necessarily go for clean in qual), we need to dance around a bit. Cover the floor. A good qualitative facilitator will bounce around, jump ahead, reverse, turn corners, step to the side… you might even see a grand jeté.
The point here is that the context within which the questions are asked, i.e. the discussion group, will vary wildly for each group. In effect, it will be confounded by all sorts of, well, confounding variables; not least, the discussion itself.
The fact that 83% of people said they liked the design means absolutely zip without understanding the discussion that came before.
3. It’s open to misinterpretation
The third, and possibly most worrying point, is the potential for misinterpretation.
The most obvious in this case is making the assumption that 83% of people, per se, liked the design. And then using this ‘finding’ to make big, important and expensive decisions, like changing the design.
This is the after sentence:
“Positive feedback for the design was based on factors X, Y and Z.”
Note its relative helpfulness (you know what factors are driving the positive response) and above all, note its glaring lack of inappropriate percentages!