Many readers seem to enjoy my qualitative versus quantitative research post. I take this to mean that designers are hungry insight that beyond the requisite (and useless) customer satisfaction survey.
I’m not a huge fan of customer satisfaction surveys because they are usually 100% reliable but 0% valid; they tell you nothing (but consistently tell you nothing). Witness, for example, the Foresee customer satisfaction survey. This survey is designed to give pop up as a user leaves a Web site. They are asked a variety of customized questions and then a variety of demographic information. Foresee tallies these results regularly and even ranks your Web site (or company in general) in comparison to your competitors.
What does a designer learn from this? Almost nothing.
Why? Several reasons.
- Consumers have “satisfaction” fatigue: consumers are surveyed to death these days. We are all familiar with the Likert scale “strongly agree” to “strongly disagree.” Few survey researchers actually rework their surveys for validity. It’s called “acquiescence bias” where people tend to just answer the same way repeatedly. Survey researchers who know better use reverse-scoring techniques; but most don’t. These surveys, therefore, result in a questionable assertion that they are actually valid representations of how consumers actually feel.
- Satisfaction surveys breed incremental “metricism”: surveys tell you nothing new: designers need to innovate their products, Web sites, and images. Satisfaction surveys tell you nothing you don’t already know. What’s more, they may actually inhibit creativity because they draw attention to minute changes that may be due to chance alone. Once organizations become regular consumers of satisfaction surveys, even small improvements become cause for celebration — even if they don’t reflect real improvements (see number 1).
- Surveys provide numbers, not detail: designers need thick description to make their designs truly evocative of lived experiences. Satisfaction surveys are simply stripped down representations of how people feel (or more accurately, how they say they feel). Designers need richer information to spark creative solutions.
- Customer satisfaction is a poor predictor of looming competition: Imagine a company that had consistently high customer satisfaction scores. Imagine also that this company falls victim to “incremental metricism,” and fails to see a competitor’s new, better designed product on the horizon. This competitive product would never appear in customer satisfaction surveys. It’s possible for customers to be “satisfied,” only to have them lured away by an innovative, better design.
Instead of customer satisfaction surveys, I recommend designers pore over free trend-spotting data, like the Pew Internet and American Life Project, which is a comprehensive and rigorous survey of current attitudes and beliefs. It’s harder to pull out insights from data that don’t look like they’re relevant, but the return is so much better than from a tired, staid customer satisfaction survey.
Categories: Research Methods · customer satisfaction · product design · quantitative research
{ 1 trackback }
{ 2 comments… read them below or add one }
I’d respectfully add that such surveys don’t provide any point of reference. They aren’t normed. Is 3.5 on a five point scale a high score? What does ’satsified’ mean?
(And this, mea culpa, from someone who has written his fair share of ‘evaluation forms’.)
Supplement Customer Satisfaction with Quali to make the exploratory big learnings, and move with incremental improvements via quanti?