This article is part of a series by Katie Harris, principal at Zebra Connections (if you missed the other parts, start here).
Social media monitors (SMMs) trawl the web to find mentions of your brand, or whatever it is that youre interested in monitoring. There are many SMM products and services available: some free, some that you pay for.
Here’s a very basic example: www.whostalkin.com
If you type in the name of a brand or topic of interest, you’ll get an idea of the kind of information SMMs return.
Depending on the level of sophistication built into the SMM you use, you can refine your search with key words, run analytics etc.
There’s a lot of hype around SMMs. Not surprising really. The idea of getting feedback on the cyber-buzz around your brand, product or service is timely and sounds quite marvelous!
Kind of. Until you think about it a bit more. Which I have. And wearing my qualitative researcher’s hat, SMMs fail in two important ways:
- Sample definition
What constitutes a SMM sample?
In a nutshell, a SMM sample comprises the searchable/findable content sourced from various online channels. Thats as precise as you can get really. The truth is, you just cant know whos represented (or not) within that content.
For example, SMMs can’t identify and screen out marketers who may be posting from domains that havent been identified as such. This means that in many cases, SMMs can’t distinguish between content generated by marketing folk and content generated by non-marketing folk.
And lets face it, quite a lot (most?) of the brand chatter out there is actually generated, nurtured and sent bouncing around the interwebs by marketing folk. People like us. The kind of people we try very hard to screen out of market research samples.
It’s also worth noting that SMMs can’t automatically, distinguish between content generated by core customers, infrequent customers or non-customers. This means that all customer/brand relationship variations are automatically given the same share of voice and weight in the analysis.
Another factor to consider is that the sample will be skewed. And while a sample skew, in itself, is not necessarily a problem, its certainly a problem when you don’t know how its skewed. Which is the case here.
Without being able to define the sample, and without knowing how the sample is skewed, theres no foundation or context for meaningful content analysis.
Sentiment is the very essence of what were trying to understand through market research. And this is something that SMMs dont gauge very well.
Although automated sentiment analysis is often sold with the SMM package, there are two things about it that trouble me:
There seems to be considerable scope for error in the labelling.
For example, how would automated sentiment analysis label a statement such as F&*#ing brilliant!?
Depending on the context, this statement could be:
- Dripping with irony
- An exclamation of genuine excitement and joy
- A description of a high-wattage light bulb
So, would it be labelled as negative, positive or neutral?
Notably, some SMMs claim to be contextually savvy, and that they can identify positive, negative or neutral sentiment with 90% accuracy (is 90% good enough?).
BUT (note caps), even with 90% accuracy, these labels are still seriously wanting. They dont provide me with information thats of much use – if any – because theyre too vague.
It’s the finer points of sentiment; the despair, frustration, excitement, boredom, curiosity etc. underlying the positive or negative sentiment labels that I’m interested in. This is the level of sentiment I need if I’m to understand what’s going on with any effect. And to get to this level of sentiment, I really need to dig a bit deeper.
Where’s the gold?
I need to dig deeper, but where do I begin? Back to my earlier point about SMM sample definition and skews; I dont know where the real gold (vs. fools gold) lies.
Without spending the time and effort to sort through each and every buzzversation (easily in the thousands), I cant distinguish between content of import and that of little consequence. I just dont know where to drill deeper in a meaningful, robust kind of way.
So its virtually back to square one.
SMMs are an exciting idea and Im itching to find a way to use them.
From a PR or customer service point of view, I imagine theyre worth their weight in (real) gold.
But in my qualitative market research business, I cant (yet) use them with either confidence, or pragmatic effect.
Sentiment aside, the sample scope/limitations and the unknown skews preclude the output from forming anything approximating a solid foundation for analysis.
Bit of an issue for me.