as a phd candidate in marketing who has spent considerable time in NP and WOM, i will admit i have not explored the stats behind the "buzz". I have even started using it for my product. I hope this research receives attention from business leaders.

With regard to the results of our research, my co-authors and I believe that are two key findings:

1) We did not find Net Promoter to be a good predictor of growth at all when attempting to replicate Reichheld's methodology. We find this quite strange given the overwhelming evidence Reichheld reports regarding Net Promoter's linkage to growth.

2) The most troubling finding from our research, however, is that we found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, we were able to replicate a subset of Reichheld’s reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI. It is virtually impossible to imagine a scenario other than research bias as the cause of this finding.

This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.

Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.

Net Promoter “Believers” typically focus on “testimonials” from users or “qualifications” regarding the usefulness of the metric in response to evidence contradicting their earlier assertions and beliefs. Unfortunately, as “researchers” this is the equivalent of ignoring the 900-pound gorilla in the room; the research upon which Net Promoter is based is the sole reason for its existence. If it is biased, then it is disqualified—period.

It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld's reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]

While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased. This issue is so overridingly important to very core of what we as researchers (academic and practitioner) do and stand for that it must be addressed. Our credibility in science and in practice is based upon honest and fair evaluations of data...if this is contaminated and unchallenged, then there is no reason to believe anything we say.

Tim Keiningham

Keiningham provides a compelling commentary. Clearly, unbiased research and credibility are fundamental considerations in all published work. Instead of contributing to a "bandwagon effect" (e.g., "our firm will adopt Net Promoter because several leading firms have"), firms should carefully read the evidence presented in this article.
Look forward to hear comments from JM readers on this blog thread.

When it comes to loyalty, many researchers have adopted Reichheld’s assertions indiscriminately.
Who doesn’t know (or has used) the often-cited claim that the cost of customer acquisition is five times higher than the cost of retaining an existing customer (Reichheld and Sasser 1990, Harvard Business Review)? Being both an academic and a practitioner, I believe that many managers have based their strategic decisions on that particular assertion without criticizing or validating whether the “five times” was justified in their specific application (cf. not only do companies or industries differ substantially; nor are all customers equally profitable or equally costly to serve).
In a similar vein, many companies have adopted Reichheld’s Net Promoter metric as being the “single most reliable indicator of a company’s ability to grow” without considering themselves the (potential / better) role of other measures such as customer satisfaction or retention.

While many scientific papers end with: “one should test the generalizability of our research findings by adopting similar studies” as a limitation or viable opportunity for further research, Keiningham and colleagues (2007) have suited the action to the word. In their research, they presume to question the superiority of the Net Promoter metric against other often-used satisfaction/loyalty metrics (such as overall satisfaction, repurchase intentions, etc). They offer a longitudinal, peer-reviewed, and cross-industry re-examination of the Net Promoter assertions by using data from the Norwegian Customer Satisfaction Barometer (NCSB).

The findings reported by Keiningham and colleagues (2007) are of major importance for both researchers and managers.
Not solely because they do not find support in favor of the Net Promoter metric (and hence, do not reconcile with Reichheld findings). Indirectly, their research also contains an important message for managers: always be very critical, and take nothing for granted. In this case, they alert to be careful when preferring the Net Promoter metric to other metrics.
From an academic point of view, I believe that this research will open a new debate about the suitability of Net Promoter metric and what should be the best predictor(s) of several firm performance outcomes (such as company’s growth rate) on the short and long run.
I am looking forward to knowing/reading Reichheld’s response, as well as other research studies within this domain. For the near future, it will become crucial to gain more insight, and to reach a general agreement on the role of the Net Promoter metric… certainly because many companies continue using the Net Promoter metric.

Bart Larivière.

In some ways it’s easy to see why business managers were drawn to the NPS like moths to a flame. In a world of increasing complexity it’s reassuring to think that you simply need to trust your intuition, thin slice and only worry about One Number Your Need To Grow.

At the same time, the apparent link between advocacy and growth made the NPS a blessing for anyone trying to convince companies to adopt more bottom-up approaches to marketing and other business innovation … it’s such a simple tool.

As such, I hope that Tim (Keiningham) and colleagues, and those that have been doing research that supports Reichheld’s research, find some synthesis here. It would be a great shame if the baby was thrown out with the bathwater.

In the meantime, I thought you might all be interested in a podcast I’ve recently carried out with independent research consultant and social/consumer psychologist Dr Alain Samson at the London School of Economics. Alain was co-author of the Advocacy Drives Growth (Customer Advocacy Drives UK Business Growth). He discusses his research findings about the Net Promoter Score’s strengths and weakness, and looks at how research into advocacy and co-creation could be aligned. You can hear the interview here:

http://www.connectedmarketing.org/2007/07/16/series-2-alain-samson-interview-july-2007/

Justin Kirby, co-editor/-author, Connected Marketing

Timothy Keiningham and colleagues (2007) did an extraordinary job of challenging the research rigor of Net Promoter metric initiated by Reichheld’s (2003).

These authors are acutely aware of the fact that the Net Promoter metric, though very popular in practice, lacks scientific scrutiny—a fact that brings unexpected risks to firms because “...managers have adopted the Net Promoter metric…on the basis of the belief that solid science underpins the findings...” As academic researchers, it is our responsibility to assure that rigorous research methodology supports any conclusions drawn about the input-output relationships of interest to businesses. Thus, the research question investigated by Keiningham and colleagues (2007) is profoundly important to Net Promoter metric users and advocates.

The investigation by Keiningham and colleagues (2007) is quite objective and comprehensive. Their findings are credible as reported. I personally like their thorough discussion on the possible causes for differences between what they found and what Reichheld (2003) and Satmetrix (2004) report. Such comprehensive discussion demonstrates the unbiasedness of the authors’ research undertaking in questioning the results of Reichheld (2003) and Satmetrix (2004).

Lastly, I agree with Siva K. Balasubramanian that the authors should be applauded for the way they managed to compare ACSI and Net Promoter scores, given that the latter was not readily available. The authors figured out a smart way to “assess” the data of Reichheld (2003). Their comparison results are so astonishing that firms cannot afford to ignore them. I cannot wait to see comments from Reichheld in response to the significant correlation results between performance growth data and ACSI as reported by Keiningham and colleagues (2007), as well as other published works as noted.

After reading the paper and Tim's remarks above, there is little that I can add to the discussion surrounding the uselessness of the NPS. Beyond it's evident lack of common sense (using an average of customer recommendations to forecast business growth is one step up from using sacrificial chickens as a forecasting method), a careful read of Reichheld's book should have raised some flags in the minds of those with some mathematical knowledge (e.g. passing correlation for causation, resorting to post hocs, etc.) and prevented the adoption of the NPS.
All in all, this is one more study that stresses the inadequacy of NPS as a business metric, and in face of mounting criticism and now of solid research, it is time Reichheld reconsider his proposal of NPS a valid metric.
I've been posting for sometime about the NPS at http://adelino.typepad.com

As a practitioner, the simplicity of having one single variable to focus on is clearly appealing. This excellent research highlights the danger in accepting a seemingly wonder tool without proper scrutiny: we might end up channeling resources to initiatives that add little (if any) to shareholder value.

The evidence points to a clear need to critically assess the real value of Net Promoter as an indicator of future growth.

I agree with most comments here, so forgive me for being lazy and simply directing you to my blog entry on the discussion: http://www.kaizo.net/blogs/2007/07/the_debate_about_nps_goes_on.html

Logrolling is an amazing process to watch in action. A noted loyalty guru, publishes an article in a respected journal, and then expands the thinking in a highly promoted business book from a noted publisher. The most senior executives at the likes of GE, T-Mobile, Intuit, American Express, and others join in the celebration. It feels simple and easy and no one seems to get too hung-up in the factual underpinning. Soon the legend grows and the analysis is publicly claimed in C-level circles to be based upon research in “4,000” companies (rather than the actual “400” initially considered by Reichheld, and the “over 50” upon which his analysis was actually conducted). It’s time someone slowed things down long enough to take “a look under the hood”. Thanks Keiningham, Cooil, Andresassen and Askoy for your work.

Maybe the idea of taking the number of bottom-six box responses and subtracting them from the top-two box responses is a good way to handle missing data and give better attention to customers in pain. Let’s hold on to that idea. But based on any number of studies that I have seen reviewed I’m not surprised that “willingness to recommend” is no better a predictor of growth than “overall satisfaction”, or “intention to repurchase”.

Let’s also inject a bit of common sense into the discussion. First, if you only need to ask one question (even if it was really such a great predictor) then besides putting that number on the executive scorecard, what good is it? Does it provide any diagnostics? Does it help establish priorities for action that will delver the greatest return? Can I develop a plan of action based on the results? So can that be called meaningful research?

Secondly, how can likelihood to recommend be a universally valid question? In some categories it surely would work well. Just ask a new mother about a brand of diapers, or a golfer about their newest titanium driver and you will surely get an earful and a clear measure of recommendation. But for most of the public we either don’t carry such emotion for most of the products we use, or it’s just not our nature to take a step out on the ledge to “recommend” anything. So how can it possibly be the only question you need to ask?

Very interesting critique, I will make sure to pass this along to my colleagues in Pru.

There are, I think, three issues here:

1) NPS has very little actionability regarding insight into growth opportunities/improvement priorities, at least in part because of multiple flaws and deficiencies in the technique. The research Tim Keiningham co-authored, as well as other academic studies, clearly identify these challenges and address their consequences.

2) There has been a 'holy grail' tendency for some organizations to accept the single number concept with little question, either disregarding or not allowing more actionable customer loyalty behavior research results to be applied or circulated. Recommendation is one desired outcome of loyalty predispositions, but there is a great deal more for companies to know about the experiences they create and customers' resulting strategic and tactical behavior.

3) The proclivity of many companies to be passive users of customer data in the first place (as covered in my 2005 book, One Customer, Divisible) puts into question how NPS data are used. Delivering on a value promise requires insight down to the individual customer level as well as understanding the impact of customer touch point processes, often at a very granular and systemic mannner.

While there's little argument that the single metric appeal of NPS, and all the dialogue around the concept, has increased the visibility of CEM (customer experience management), the statistical shortfalls, willingness of some to accept the premise without reservation, and sustainable value and application of results are, IMHO, doing more harm than good to those of us whose mission is to help companies understand customer commitment and advocacy behavior.

Research done at Maritz Research also raises questions about the usefulness of Net Promoter. You can read the Maritz White Paper on this at:
http://www.maritzresearch.com/pdf/measure.pdf

For any of your readers interested in learning about Net Promoter in the real world, please check out the Net Promoter Conference blogs (from the London and New York events). Multiple bloggers have written up key takeaways from ~ a dozen sessions on the successes of companies who presented such as T-Mobile, LEGO, HSBC, Schwab, Intuit, Philips, GE, T-Mobile, IBM, and more.

Here's the link to the most recent conference blog:
http://netpromoter.typepad.com/npc_london_2007/

Some posts in this string call to mind blogs by new Net Promoter blogger, Jeanne Bliss, author of Chief Customer Officer. It's one thing to know your Net Promoter Score, and another thing to do something about it. Here are links to those blogs:

Mining the Gold: Listening Hard to “Detractors”
http://netpromoter.typepad.com/jeanne_bliss/2007/07/mining-the-gold.html

We Know Our Net Promoter Score. Now What?
http://netpromoter.typepad.com/jeanne_bliss/2007/05/coming_soon.html

Value Speaks Louder than Words

Jock Lawrie, Alonso Matta, Ken Roberts – Forethought Research Australia

The NPS folly is not as new to industry as it may be to academia as heralded by this important JM article. Not all private assessments find their way into the public domain. Ken Roberts drafted the exact question from which the NPS was based in 1997. Our firm has 10 years of longitudinal data assessing the performance of the exact construct as a predictor of market share. The following is an extract of that assessment released to clients in 2006.

For full text including statistical analysis - http://www.forethought.com.au/docs/thinktank/FORETHOUGHT-ValueSpeaksLouderThanWords.pdf

Executive Summary

Management has recently been attracted to the notion that advocacy in the form of a Net Promoter Score (NPS) provides insight into market success and according to some is the only marketing metric management needs to focus on.

For example, the cover story in the September 2006 Australian edition of CFO magazine features an interview with Trevor Schauenberg, Vice President of GE Capital Solutions Australia espousing the insight provided by the NPS as a managerial metric.

‘At a recent strategic planning session we put all of our business units on a matrix showing two years of growth rates and discovered that the units with the highest growth rates had NSP scores of 50 per cent and above. Conversely, the business units that scored 30 per cent or lower had very low growth rates.’

What Schauenberg liked most was the correlation between advocacy and revenue. However, correlations do not imply causality nor do they suggest one variable is a leading indicator for another. What Schauenberg and executives like him are discovering is the well documented power of relative positive word of mouth on growth rates in particular for intangible services.

Buyers do not suddenly advocate; first comes a management strategy; then execution which includes crafted communication, followed by customer and hopefully non-customer experience and then positive word of mouth. Positive word-of-mouth adds critical salience to marketing claims which aids in building brand momentum. The process is not always exactly in that order but it’s difficult to imagine word-.of-mouth occurring ahead of management strategy. What is proven in this client briefing is that ideally share gaining strategy is based on value (price and quality) to the buyer.

Our analysis based on more than ten years of data is clear that the NPS is statistically insignificant in explaining changes in market share and that another measures specifically, value is significant in explaining changes in market share.

Once a brand has built momentum then advocacy can provide insight into the velocity of that momentum and indeed be correlated with relative revenue growth but what troubles most executives is understanding which price and non-price drivers ignite changes in market share.

If management crave affirmation about past decisions then advocacy is perhaps an appropriate single measure. If instead management want insight into the next customer led discontinuity likely to drive changes in market share then the NPS is a most curious choice of metric.

Additionally we find:

• Positive company advocates are highly likely to have experienced superior value. That is, advocacy too is driven by value,

• When a competitor is offering superior value advocacy is by no means a good predictor of future purchase behavior. Indeed, based on our data sets across multiple industries around one in ten customers who give 9 and 10 on the advocacy question intend to move their business elsewhere,

• The rule-of-thumb score classes proposed by Reichheld (promoters are those respondents who give a likelihood of recommendation of 9 or 10 while the detractors give 6 or less) are not supported statistically, mask important changes and potentially mislead management that there is negative NPS when this may not be the case,

• Upon measuring the NPS, it is unclear what can be done to improve advocacy without measuring anything else. It is important to know why people are speaking (be it positively or negatively) about a company, if they are speaking about it at all. In other words, the client needs to identify the drivers of advocacy. If that type of driver analysis is to be undertaken we strongly recommend value as the dependent variable.

Hopefully the article by Keiningham, Cooil, Andreassen & Aksoy (2007) serves as a wake-up call to industry. The wide-spread use of Net Promoter causes one to wonder whether the “powers that be” at these corporations ever took a basic statistics course. One of the basic tenets taught is the superiority of multi-item measures. I applaud the authors (as others in this blog have done) for their creative assessment of seemingly unavailable data. While statistics can be found to support almost any position, Keiningham and his colleagues (2007) appear to present an unbiased analysis with a foundation for truly improving business decisions, rather than “selling” an idea (or a book for that matter). I look forward to comments and an explanation by Reichheld.

Keiningham, Cooil, Andreassen, and Aksoy show the outrageous claims attributed to Net Promoter for what they are: false. And their proof that Net Promoter was not superior to the ACSI despite Reichheld claiming that the ACSI had a 0.00 correlation to growth would be hilarious were it not for the fact that it clearly demonstrates research bias on the part of Reichheld.

Taken together with another paper that these authors have written on the customer-level claims of Net Promoter (in the current issue of Managing Service Quality) the entire research conducted by Reichheld in support of Net Promoter does not hold up to scrutiny.

Reichheld claimed to have a epiphany after many years of satisfaction/loyalty research. Asking customers if they were satisfied was useless, instead we must measure willingness to recommend.

But anyone who has used satisfaction and likelihood to recommend measures in the same survey knows they typically have an extremely high correlation. Which is hardly surprising.

How then can he reject one measure, and promote the other as the perfect metric ?

It is wonderful to see the interest that our research has garnered. Thanks very much to everyone who contributed their insights.

We have also been extremely pleased at the attention that this research has received in the press. One such article appears in the current issue of COLLOQUY magazine:
http://www.colloquy.com/article_view.asp?uid=4030

COLLOQUY starkly contrasts the debate as one of "faith versus science," stating:
"In other words, NPS is a concept so simple that even lunk-headed CEOs get it, and they’re finally paying attention to us—so why mess with it? For NPS promoters, faith trumps science. Like proponents of Intelligent Design, they respond to facts with platitudes."

Fred Reichheld, inventor of the Net Promoter metric, has chosen "not to invest time responding" to our findings.

Also, we thought you might like to know that Managing Service Quality has just released another paper by the authors that investigates other aspects of the original Net Promoter research, entitled, "The Value of Different Customer Satisfaction and Loyalty Metrics in Predicting Customer Retention, Recommendation, and Share-of-Wallet."

This research examines different customer satisfaction and loyalty metrics and tests their relationship to customer loyalty behaviors. The goal was to test the robustness of the customer-level analysis conducted by Reichheld and Satmetrix, which served as the foundation of their Net Promoter research. Contrary to Reichheld's assertions, the results indicate that recommend intention alone will not suffice as a single predictor of customers' future loyalty behaviors. Use of multiple indicators instead of a single predictor model performs significantly better in predicting customer recommendations and retention.

Managing Service Quality is making the paper available for free for download at the following URL:
http://tinyurl.com/2aq3aj

These findings, taken in conjunction with the findings of our research reported in the Journal of Marketing regarding the relationship between Net Promoter and growth call into question the robustness of the entire analysis reported by Reichheld.

Thank you very much for your interest in this research.

- Tim Keiningham

The discussion on this blog already raises the most pressing and important points about this research. The idea of challenging conventional wisdom (or fads, or what have you) is a truly important role for academia. As schools become more concerned with revenue and fundraising, I at times have my concerns that articles that make such challenges will be less welcome. I thus truly applaud the publication of this piece in a top journal, and appreciate the commentary that it has inspired.

I feel the importance of this piece from this perspective cannot be overstated, and I encourage others to use this paper for educational purposes about challenging popular wisdom. Academic advances and the business-popular press often reach very different audiences, and likewise often yield very different conclusions. While certainly academia can be criticized for its incrementalism, many of the “suggestions” from the business press become accepted without critical evaluation. Perhaps put most succinctly by Samuel Clemmons: "What gets us into trouble is not what we don't know, it's what we know for sure that just ain't so." Research like the work by Keiningham and co-authors play an important role in challenging conventional accepted wisdom.

On the other hand, academic cannot sit back and take on the role of purely critiquing conventional wisdom. Measures like Net Promoter become popular because of real business needs, and either the lack of suggestions from academic sources or a lack of communication from academic sources on appropriate and feasible solutions for practice. And so while Keiningham et al.’s work make an important contribution, it presents a need for more research aimed at developing better measures to assist in practice. I thus fully agree with the final point made in the paper: that there is a need for rigorously and scientifically tested measures that can predict business outcomes. But there must also be more work building bridges between academia and practice, so that research can help develop scientifically appropriate solutions in a way that can be implemented.

It’s been interesting to read the debate regarding the paper published in the Journal of Marketing. While at first we, Satmetrix, have been listening to the conversation, we feel it’s time to speak up and provide our perspective. If you really evaluate the overall discussion you will find that the general sentiment regarding this debate is quite positive for Net Promoter and the companies that are applying it to their business. The comment below from Justin Kirby is particularly insightful:

"At the same time, the apparent link between advocacy and growth made the NPS a blessing for anyone trying to convince companies to adopt more bottom-up approaches to marketing and other business innovation … it’s such a simple tool. As such, I hope that Tim (Keiningham) and colleagues, and those that have been doing research that supports Reichheld’s research, find some synthesis here. It would be a great shame if the baby was thrown out with the bathwater."

Justin, we share that desire. Many companies are using Net Promoter as a rallying cry to create customer centric organizations. The fact that it is simple and actionable at all levels in the organization creates a common framework for evaluating the loyalty of customers across the business. We also have been gathering numerous examples of companies applying this success - just check out www.netpromoter.com and see for yourself. Much of this is a result of the two highly successful, sold out Net Promoter conferences sponsored by Satmetrix earlier this year.

As for the research conducted by Satmetrix in the development of Net Promoter Score, we are confident in the quality of our research and take exception to the allegation of a research bias. In the Journal of Marketing paper the authors state: “… although we discuss the possibility of research bias in the data analytics used to support Net Promoter, this has not been proved.” Later the authors go on to say, “Nevertheless, we would expect that a serious, longer-term, longitudinal study would show that changes in satisfaction/loyalty metrics are important predictors of relative changes in revenue within firms.” This is exactly what our research has proven and the foundation of the link between NPS and growth.

For further information on the research conducted by Satmetrix, please visit one of the following:
Dr. Laura Brooks latest blog posting at http://netpromoter.typepad.com/laura_brooks/2007/09/building-value-.html#more
The Satmetrix white paper describing the research at: http://www.satmetrix.com/resources/download_white_paper.php?pdf=netpromoterWPfinal.pdf
Or research conducted by the London School of Economics at:
http://www.lse.ac.uk/collections/pressAndInformationOffice/PDF/AdvocacyDrivesGrowth_5-9-05.pdf

As for the comparison with ACSI, the Journal of Marketing paper shows Net Promoter is at least equivalent to ACSI in correlating growth. That says that a simple metric driven by a single question is at least as appropriate at predicting growth as a more complex algorithm driven by multiple questions. NPS has the added benefit of being simple and actionable. Business leaders can easily understand NPS enabling them to drive customer-centric organizations. It’s actionable in that you can nurture promoters, address detractors and move the passives. It is also a powerful metric in understanding both the increased growth achieved through positive referrals, as well as the lost potential for growth caused by the effects of negative word-of-mouth.

While NPS offers an alternative to traditional approaches to measuring and improving customer loyalty, business leaders should see it as a compliment in their research portfolio. Its simplicity offers them a vehicle for driving culture change; this is good for the company and good for their customers. Getting information into the hands of front line employees allows change to happen one customer at a time. The game is won not in collecting the score, but in changing the score. Net Promoter programs deployed by Satmetrix are based on both research and experience in improving customer loyalty. We have found Net Promoter to provide an excellent foundation for getting business leaders to focus on what matters most to customers to improve loyalty and drive growth.

It’s time to work together to help the business community evaluate their customer loyalty and take action on that information. That’s where the real opportunity lies.

Thank you,
Deborah Eastman
CMO
Satmetrix
www.satmetrix.com

Ms. Eastman is correct in acknowledging that we need to help the business community evaluate their customer loyalty and take action on that information. She is also correct that Net Promoter’s simplicity made it a rallying cry for organizations to support loyalty initiatives within their organizations. Unfortunately, this is not the issue. The issue is the integrity of the research reported in our most prestigious journals.

First, Ms. Eastman correctly notes that we state in our paper, “although we discuss the possibility of research bias in the data analytics used to support Net Promoter, this has not been proved.” We follow that statement with, “Only an independent, objective validation of the data used to support Reichheld’s (2003) and Satmetrix’s (2004) findings could conclusively prove such a bias.” This is a function of the appropriate stated limitations of scientific research (i.e., the standards are at their absolute highest); we would need to have 100% certainty … not even 99% certainty would be acceptable for us not to state this as a limitation.

Second, we are very clearly discussing the “reported” research by Mr. Reichheld in his Harvard Business Review article, his book, The Ultimate Question, and his presentations. In each of these cases, Mr. Reichheld has been very specific about the “superiority” of Net Promoter in linking to growth, and the inadequacy of the ACSI (and other satisfaction measures) in linking to growth. How the research was conducted, and how it was reported by Mr. Reichheld may differ, but thus far we have not seen any reports distinguishing the two.

Third, Mr. Reichheld was very specific that a Bain team examined the relationship between the ACSI and growth, and they found a 0.00 correlation. As a result, the statement that Net Promoter is “at least equivalent to ACSI in correlating growth” is not good enough, unless the argument is that both the ACSI and Net Promoter have a 0.00 correlation to growth. Furthermore, our examination used exemplars reported by Mr. Reichheld of the relationship between Net Promoter and growth, therefore one would presume that the deck was stacked in favor of Net Promoter for this part of our investigation. We sincerely implore Ms. Eastman to explain how we could not support Net Promoter’s superiority to the ACSI when replicating Net Promoter data reported in The Ultimate Question.

The ACSI has faced a withering critique from Mr. Reichheld since December 2003; therefore fairness and truth require an answer. It is unacceptable for Mr. Reichheld and his supporters to portray themselves as victims when we call attention to the results of our scientific research given that Mr. Reichheld made direct attacks on the ACSI (and by inference, the firms that use it as a source of competitive differentiation, CFI International and Foresee Results) … it is the pot calling the kettle black.

Ms. Eastman appears to believe that our issue is with managers’ use Net Promoter. That is not the case. Our overriding issue is the honest reporting of research in our most prestigious journals. The Harvard Business Review is one of only forty (40) Financial Times scientific journals (which means the research reported there will have tremendous impact on the work of management scientists). Our obligation as scientific researchers DEMANDS that we call attention to research reported in these journals that defies logic, and points to potential bias. And although we would prefer otherwise, since it was our research that uncovered this issue, the responsibility of calling attention in this case falls on my co-authors and me. The reported superiority of Net Promoter to the ACSI, particularly given the reported 0.00 correlation between the ACSI and growth, is far too strong evidence of research bias for us to ignore. Therefore, we sincerely want a reasonable answer to this issue so that we can move forward and jointly focus on improving customers’ experiences, thereby in our own small way, making the world a better place.

Sincerely,

Tim Keiningham

Tim, thanks for clarification. For more information you can read Dr. Laura Brooks’ recent blog posting http://netpromoter.typepad.com/laura_brooks/2007/09/building-value-.html about our original research and ongoing experience with NPS. As for the ACSI, Satmetrix has not done any specific research to evaluate the relative value of ACSI vs. NPS. We have found NPS to track consistently with composite metrics such as our Customer Loyalty Index (CLI) and have a preference for NPS as we have found it to be easier to adopt and more understandable/actionable by business users.

Deborah Eastman
CMO,Satmetrix

P.S. I prefer Deb or Deborah, Ms. Eastman makes me sound so old =)

Ms. Eastman’s letter in defense of the Net Promoter research conducted by Satmetrix is quite interesting. First, she states, “As for the research conducted by Satmetrix in the development of Net Promoter Score, we are confident in the quality of our research and take exception to the allegation of a research bias” … unfortunately, she offers no explanation as to why the superiority of Net Promoter in linking to growth espoused by Reichheld in his articles, speeches and book cannot be replicated. Furthermore, she completely ignores the direct attacks on the ACSI as being inferior, even reporting it has a 0.00 correlation to growth. The argument that Net Promoter is “is at least equivalent to ACSI in correlating growth” is absolutely ridiculous. Reichheld claims that his research shows that the ACSI HAS ABSOLUTELY NO CORRELATION TO GROWTH! Even if the measures were equivalent, which they are not, “equivalence” equates to bias in Reichheld’s reported research. Given the “confidence in the quality” of the research and the “taking exception” to the allegation of research bias, an answer to this issue must be given; otherwise, as Shakespeare would say, “methinks thou doest protest too much.”

The truth is, however, that Net Promoter and the ACSI are NOT equivalent. The linkage of the ACSI to firm performance is well document in numerous peer-reviewed scientific studies (many of which have appeared in the Journal of Marketing). We have yet to see any serious scientific study on Net Promoter reported in a leading scientific journal, with the exception of the paper by Reichheld reported in the December 2003 issue of the Harvard Business Review.

Finally, it is also very interesting that Net Promoter is no longer “the one number you need to grow,” to quote Reichheld. Now, Ms. Eastman claims Net Promoter is “a compliment in [a manager’s] research portfolio.” Perhaps that explains why despite co-inventing Net Promoter, Satmetrix now has its own Customer Loyalty Index (CLI) to measure customer loyalty. It appears that Satmetrix recognizes that there really is more than one number you need to grow.

I am a firm believer that customer loyalty is a leading indicator of financial success of companies. Customer loyalty can be measure through objective indices (e.g., defection rates, and repurchase rates) as well as subjective measures (e.g., likelihood to switch, likelihood to purchase again, and likelihood to recommend). “Customer loyalty”, measured via customer surveys, is like any other psychological construct; it is an unobservable entity that requires some observable indicators (items) that bring it into the measurable world. As was alluded to in an earlier posting above, customers’ ratings loyalty survey items are highly correlated.

I recently conducted a customer loyalty study for the PC industry using a variety of commonly used loyalty items. The inter-correlations of these four items ranged from .73 to .86 (quite high). Additionally, a factor analysis of these loyalty items (overall satisfaction, likelihood to choose again, likelihood to recommend, likelihood to purchase again) revealed a clear one-factor solution. Finally, the reliability (Cronbach’s alpha) of this four-item loyalty index was .94. This pattern of results suggests that these four items measure ONE underlying construct. The conclusion is that these four items are simply observable indicators of the single construct we call “customer loyalty.” It is no wonder that Keiningham and his colleagues found that both satisfaction and likelihood to recommend questions showed similar correlations to growth.

To say that one item is better than another in predicting a given criterion (in this case, growth) is suggesting that some survey items are measuring something quite different than the other survey items. Empirical data show that this is not the case.

Before researchers or practitioners argue about the relative merits of single loyalty items, we should start with the application of sound psychological measurement principles in the creation of reliable and valid measures of customer loyalty. A brief summary of the results of the PC study can be found on my blog (www.businessoverbroadway.squarespace.com).

Thanks,

Bob E. Hayes, Ph.D.
Business Over Broadway
www.businessoverbroadway.com

Deborah -- Thank you very much for your clarification. Based upon this information, it appears that the issue of bias rests solely on shoulders of Mr. Reichheld. Given his strident attacks on other metrics, particularly the ACSI, without evidence to support these conclusions from the research conducted by Satmetrix, a response is warranted. -- Tim Keiningham

This is a brilliant case study of a more general phenomenon which we had already identified - the conflict of interest that is rife within management academics. For more, see Merlin Stone, Neil Woodcock and Michael Starkey, The Role of Academics and Analysts in Misleading Business - the case of Customer Relationship Management, Journal of Database Marketing and Customer Strategy Management, Vol 11, No 2, pp 121-132 2003. A doctoral student of mine has followed up this work with a study which shows that nearly all that the most well-known academics have written about CRM is inconsistent and poor quality. We are looking forward to publishing this study, once he has received his doctorate - and assuming the academics who examine him can face the music!

Arya,
For clarification on your comments see my earlier response to both ASCI and the Satmetrix CLI. Our Customer Loyalty Index predates Net Promoter. Some of our client use the index and others use Net Promoter. Both see results in improving loyalty and linking to growth.

How can anyone argue with Bob's comments above that customer loyalty links to financial success? It is just logical.
Deborah

Having read the comments from Ms. Eastman, the blog by Dr. Brooks, and the white paper describing the Net Promoter research, I have a few observations.

First, the analytic rigor reported is ridiculously naïve, and in no way justifies the claims of Net Promoter being “the single most reliable indicator of a company’s ability to grow” (Netpromoter.com). The research claims that pre-post surveys of customers demonstrates that recommend intention is the best predictor of customers’ future loyalty behaviors. Unfortunately, the research conveniently leaves out any tables showing these relationships so that we would understand the strength of these relationships, the differences among the variables under investigation, and the actual way loyalty behaviors were coded for analysis.

There is no way that this research would meet the standards of serious scientific inquiry. It would never find its way into the pages of a top tier journal, and certainly not the Journal of Marketing.

It appears that the Satmetrix research most closely resembles the study conducted by Keiningham, Cooil, Aksoy, Andreassen, and Weiner in Managing Service Quality designed to test the customer-level claims regarding Net Promoter, "The Value of Different Customer Satisfaction and Loyalty Metrics in Predicting Customer Retention, Recommendation, and Share-of-Wallet." In this research, Keiningham et. al. clearly show that the strength of the relationships between the variables under investigation and various loyalty behaviors ranges from modest (explaining less than 20% of variance) to nil. It is difficult to imagine that the results for Satmetrix differed in any material way. Hardly the finding of a holy grail metric. Worse still, the most important loyalty behavior in terms of firm financial performance, share of wallet, showed virtually no connection to these attitudinal metrics.

Second, Satmetrix clearly knew that the attacks on other metrics, particularly the ACSI, by Mr. Reichheld could not be supported by their research. Unfortunately, they conveniently said nothing while using Mr. Reichheld’s book, HBR article, and speeches, all of which made these outrageous claims, to promote their economic interest. Is it acceptable behavior to allow your partner to attack another so long as you personally didn’t pull the gun?

Third, Ms. Eastman ends by saying “How can anyone argue with Bob's comments above that customer loyalty links to financial success? It is just logical.” Actually, this is a simple platitude designed to say what we all “want” to believe. The truth, however, is far more complex. As the work of Reinartz and Kumar has clearly demonstrated, this just isn’t so. The fact is that most customers aren’t profitable in most industries. Yet unprofitable customers can also be quite loyal. In making loyalty a viable business strategy, the goal is “profitable” loyalty, not just loyalty … and Net Promoter does absolutely nothing to help businesses achieve that aim.

In their articles and whitepaper, Satmetrix and Reichheld said that the "recommend" loyalty question was, of all questions tested, most highly correlated with future consumer behavior. It is based on individual behavior, not corporate performance. And although an earlier post is correct in noting that substantial statistical evidence is absent from Satmetrix' whitepaper, some evidence is presented supporting the statement.

But then a leap is made in both logic and research. That is, if a correlation to consumer retention and referral exists, then a correlation should be found in company financials.

There is a telling phrase in Laura Brooks article to the effect that the correlation exists, if the right financial indicators are used. And here we may find the secret or the flaw in all the research relevant to this debate.

The JM article gives short shrift to discussing the financial data used in the reported study. In that article, in this blog, and in Satmetrix' whitepaper (save for the comment above) it is taken for granted that "revenue" and "profit" are obviously defined numbers, and easily comparable year over year and firm to firm. This is not the case.

In our experience publicly available financial numbers are not correlated with anything. They are not suitable research data because, within legal limits, they are manipulated to meet the needs of various stakeholders. For financial data, the authors of the JM article used publicly available company reports.

The only financial data we can imagine being connected to net promoter would be that rarely seen by the public and only sometimes available to managers. Managerial accounting tends to report sales and the direct cost of inputs used to produce sales. Each unit of a company can have its own P/L statement and these usually vary widely in their results. Public accounting is characterized by allocations, aggregations of many units, and tax, debt, and other administrative expenses. Revenue and expense deferrals and recognition render public financial data unsuitable for empirical research. There is no way public financial data could correlate to net promoter.

Unfortunately Satmetrix' whitepaper also earns poor marks for explaining the sources and types of financial data they used to develop their impressive results.

Intuitively it seems that net promoter could be linked to Unit performance when only direct inputs and production are analyzed. Satmetrix may be suggesting they have access to these inside data and hopefully they will make this clear -- what do they mean by "revenue?"

Richard,

Your belief that publicly available revenue data was not used by Reichheld and colleagues (and hence our results are not comparable) would make sense were it not for one obvious point in the analysis reported by Reichheld—the scatter diagrams used for Net Promoter reflect industries (e.g., airlines, personal computers, life insurance, Internet service providers, etc.). The notion that Reichheld had access to what you yourself refer to as “financial data [that] … would be … rarely seen by the public and only sometimes available to managers” for each firm in the industries under investigation is beyond implausible.

Additionally, your belief that attitudinal measures should not link to publicly available firm financial information has been proven otherwise by numerous researchers. For example, the Journal of Marketing has three papers linking the ACSI to firm financial performance using publicly reported financial information (i.e., Anderson, Fornell, and Mazvancheryl 2004; Fornell, Mithas, Morgenson, and Krishan 2006; Gruca and Rego 2005).

Therefore, while we agree that the research reported by Reichheld and Satmetrix “earns poor marks for explaining the sources and types of financial data they used to develop their impressive results,” we do not believe it plausible that it is the source of the financial information that is the cause of the difference in the results my co-authors and I report in our paper, and those reported by Reichheld.

Tim Keiningham

Tim Keiningham,

I am in agreement with you that Reichheld's results should be questioned and regarding the financial connection are at least overstatements. I did not write that I believe Reichheld was not using publicly available financials, but that Laura Brooks of Satmetrix seems to imply that they did have access to the "right" financials.

My point is that, notwithstanding your citations which I admit I have not read, experience has shown me that public financials do not reliably show a connection between consumer attitudes and behavior and revenue/profit.

And why couldn't this be a reasonable argument?

There's too much noise in public financials. They are the products of management's decisions on revenue and expense recognition, mergers and acquitions, tax and debt, and aggregations and consolidations over many diverse business units.

It is not at all implausible that Satmetrix has managerial financial reports. Many consulting firms have access to this type of information.

I'd like Satmetrix to tell us exactly what financial data was used.

Bruce Cooil, Tor Wallin Andreassen, Lerzan Aksoy and I wish to thank all of you for your interest in this research, and want to especially thank those who contributed to this dialogue.

For those who are interested, the following URL links to a videotaped presentation I made in Toronto regarding this research.
http://www.ipsosloyalty.com/knowledgecenter/event-070501.cfm

Again, thank you very much for your interest in this research.

- Tim Keiningham