An open letter to CMOS: why you shouldn’t trust net promoter scores

Marketers should be suspicious of anyone claiming to solve a complex problem with a simple solution, write Dr Dave Stewart and Prof Steve Worthington, and the ever-popular net promoter score sinks of simplicity.

Dear CMOs,

As chief marketing officer (CMO), you should know, perhaps better than your C-suite colleagues, to be suspicious of any consultant claiming to have solved a complex problem with a simple solution.

Simple solutions are naturally attractive because they are easily applied to what in reality are often very complex and intractable management situations. But what is often overlooked is that a simple solution to a complex problem is usually the result of a number of simplifying assumptions which are hidden from view. Consequently, any manager who relies on their intuition and the herd instinct when assessing a consultant’s simple solution to a complex problem is unlikely to be aware of its underlying assumptions, let alone question their veracity.

However once a new management approach is in place, it can be difficult to dislodge, no matter how perverse the consequences. By the time any disparity between a consultant’s claims and reality becomes fully apparent, management has usually invested significant time and resources embedding the approach in the organisational culture. Moreover, reputations of senior managers are often bound up with its success, making its removal problematic.

So it is with the net promoter score (NPS), introduced to the business world by Bain and Company’s, Fred Reichheld in a 2003 article in the Harvard Business Review, which was soon followed by his 2006 book, The Ultimate Question. Like a modern day Don Quixote, Reichheld is a crusader on a quest to cure CEOs of their addiction to “bad profits”, which he describes as “profits earned at the expense of customer relationships”. He warns that bad profits create detractors that “strangle a company’s growth”, but by delighting customers we turn them into promoters – “not only increasing their own purchases but also providing enthusiastic referrals” generating “good profits” and growth. But Reichheld’s quest does not end there. The real treasure is what he describes as the “Missing link between the Golden Rule, loyalty and true growth.” Thankfully, after a “long and arduous quest” our hero discovers the missing link, christening it the net promoter score.

This rhetorical style pervades his writing; one could be cynical and view it merely as a device for disguising the shallowness in his claims, none of which are supported by any credible evidence.

So what then are Reichheld’s claims?

Claim #1: There are only three types of advocacy behaviours exhibited by customers – that of a promoter, passive and detractor.

Reichheld intones that this is a “fundamental perspective”, but whose perspective this is can only be guessed at. If this is so fundamental, what happens to the customer who gives a ‘mixed review’? They are not the ecstatic promoter, the whingeing detractor “trapped in a bad relationship”, nor the mute passive. There may be many such customers. Perhaps you yourself are one of them?

Claim #2: Customers reveal their behaviour when they respond to one question on a zero to 10 scale. To the question, “How likely is it that you would recommend this company to a friend or colleague?” Promoters give a nine to 10 rating, passives a rating of seven to eight, and a response below seven unmasks a detractor.

Reichheld does not provide any evidence in support of this claim. Instead he pronounces that customer responses “link so closely to their behaviors that it is a practical surrogate for what they will do”. But if this were true, why then would he choose to withhold the supporting evidence?

Fortunately, a 2007 article in an academic journal, Managing Service Quality, does provide objective evidence for us to examine in part the veracity of this claim. The article gives the results of a market research study of over 8000 US customers across retail banking, mass-merchant retailing and internet service providers (ISPs). Of particular interest are customer responses to the recommend question and whether they actually recommended the organisation over the following year.

The table below shows the results for these three industries.

table

For example, in retail banking over a half of ‘promoters’ do not behave like promoters, and over a quarter of ‘passives’ behave like promoters! On this basis alone, the evidence is clear: Reichheld’s “fundamental perspective” is fundamentally flawed.

While this study does not explore negative comments being made by an organisation’s customers, there is no reason to expect that the results for ‘detractors’ would be any better. Why? Because any customer who does not intend to recommend an organisation could just as well be a ‘passive’.

The next two claims concern NPS which Reichheld defines as the difference between the percentages of promoters and detractors.

Claim #3: Increasing NPS drives organisational growth.

Reichheld’s ‘evidence’ in support of this claim is given in appendix A of his book, comprising charts of NPS by growth for selected organisations across half a dozen industries. While the charts show positive relationships between growth and NPS, they do not demonstrate that NPS drives growth. In one chart, for example, it shows growth over 1999-2002 versus NPS for 2001-2002. The difference in the two time periods is important. Essentially this suggests that growth drives NPS, which is the opposite of Reichheld’s claim, making his argument entirely specious. This point was documented in the academic literature in 2007 (Journal of Marketing).

Implicit claims #4: The value of a promoter to the organisation is equal and opposite to the value of a detractor, and a passive customer has zero value to the organisation.

We have discussed this at length in a recent article on the problems with the net promoter score.

READ: The problems with net promoter scores: how to better measure customer advocacy »

While Reichheld does not explicitly make these claims in his book, nonetheless they are implicit in the NPS calculation. We don’t intend to repeat the full text of the earlier article here. Suffice to say that there is no evidence to support these claims. Quite the contrary, as a worked example in his book contradicts both claims.

NPS – a case of collective delusion

Many in the corporate world like NPS because it is so simple. Everyone ‘gets it’, from the board all the way down to frontline staff. But it is the simplicity of NPS that hides a dark side which we have sought to reveal.

Despite the fact that we have exposed NPS to be a failed metric based on unsupported claims and false assumptions, we do not expect business leaders to suddenly ‘see the light’. As explained at the outset, there is too much at stake here to suddenly reverse earlier decisions. But we do think that after more than a decade since its release, it is time for business leaders to fully investigate whether NPS is providing the strategic direction they expected. We believe that you can play an important role in bringing this to the attention of your C-suite colleagues and in so doing, begin the search for a more credible alternative to NPS. For example, perhaps instead of looking in the rear view mirror at how our customers have perceived us, we should instead look ahead through the windscreen at how far our customers think we can take them with us as an organisation. What is our ‘growth’ score? How many customers are prepared to accompany us as an organisation on a ‘growth’ trajectory?

So to conclude, in a recent documentary, famed magician and sceptic, James Randi remarks: “No matter how smart or how well-educated you are, you can be deceived.” Almost paradoxically, with NPS the deception is hidden in its simplicity.

Yours sincerely,

Dr Dave Stewart, Marketing Decision Analysis, Melbourne

Professor Steve Worthington, Swinburne University, Melbourne

 

 

 

  • Maurice FitzGerald

    Quite willing to join a discussion directly with the authors of this article. While Steward and Worthington make some valid points, their main arguments are “fundamentally flawed”. I will explain why… [Note that what I write below, after the paragraph explaining why I care, is different from what I just posted in reply to the other similar article by the same authors.]

    First a note on why I care: I am VP of Customer Experience for HP Software and HP has 14 years of experience of asking the “recommendation” question to customers in double-blind benchmark surveys executed for us by Harris, Neilsen, IPSOS and Qualtrics over the years. The NPS trends compared to competition have been superb predictors of revenue and market share trends over that period. In my most recent analysis of eight years of data for the main software business, NPS continues to show its predictive value for license and professional services sales in particular. As Reichheld & Markey state, it is not perfect for all businesses, and as expected, does not have predictive value for support revenue. Customer Experience Scores should be better at that and we are implementing now.

    Let’s consider the claims about the claims:

    Claim #1 – only three categories of behavior

    Reicheld and Markey do not claim at any point that customer behavior changes totally as soon as you change category, just that they are useful indicators. I imaging an experimental psychologist could come up with 50 categories. What the authors claim, correctly, is that these are useful ways of thinking about customer behavior. In a recent discussion with Rob Markey, he mentioned, for example, that it is useful to think of Detractors in two sub-categories: customers who hate you, and those who are severely disappointed in you right now, and want you to improve. Note that the table in the Claim 2 section supports what I am saying here: behavior changes, just not radically.

    Claim #2: Customers reveal their behaviour when they respond to one question on a zero to 10 scale.

    I can’t distinguish between this one and Claim #1. My response is the same.

    Claim #3: Increasing NPS drives organisational growth

    It does indeed, when the metric used is a double-blind benchmark comparison to competitors. By “drives” I mean precedes. This has been shown repeatedly across most industries. Steward and Worthington provide no counter-proof, just a reference to a pair of graphs in an obsolete version of the book. These graphs are not in v2.0, and I no longer have my copy of the first version. I suspect a typo, assuming they actually exist. This lack of rigor from academics is surprising and particularly disappointing. It is the most important single fact about NPS and has been dismissed without any research of any type.

    Implicit claims #4: The value of a promoter to the organisation is equal and opposite to the value of a detractor, and a passive customer has zero value to the organisation.

    Frankly, this is where the Worthington and Steward claim to have mind-reading abilities that are not backed up by fact. They suggest that the Dell example they refer to is “inadvertent proof” that the “implicit claim” is incorrect. What is incorrect is their mind-reading. The Dell example proves that no such claim is made implicitly or explicitly in the book. I suggest the authors read v2.0 which has now been out for four years. This makes it more clear that the value of the customers in different categories is simply different in a quite predictable way, on average. Here in HP, and giving the example of services business, we simply find that the gross margins are highest for Promoters, less for Passives and less again for Detractors. All three have positive contribution margins. There is no suggestion that Detractors have the opposite value to Promoters or that Passives have no value. This “claim” exists entirely in the authors’ imagination.

    [The following paragraphs are also in my reply to the other version of the same article, so you do not need to read further if you have read what I wrote there.]

    – The article authors as well as some of the replies here mix up transactional NPS scores and benchmarks. At least one reply seems to refer to NPS being applied to call center people. NPS is not a good metric for call centers as it is primarily an overall brand metric. Ideally, you should use Customer Effort Score in call centers and for “quote to cash” transactions.

    – There are different types of NPS surveys. The Telstra example shows that the authors have not understood the difference between a transactional survey and a doubleblind benchmark survey.

    – The main issue with NPS surveys is simple: It is difficult to work out what do actually do with the responses. The method uses open text answers to one or two questions that follow the rating question. Until now, there has been no software that correctly analyzes and groups responses in an actionable way, without human bias. I expect to release software in the public domain this week that does it better than anything I have seen. I will put a message here when I have done so.