Harvard Business Review has a great post about making effective surveys. Rob Markey is a partner with Bain and Company, the consultancy, and huge proponent of the “Net Promoter Score.” He tells us that he himself ignores email invitations to surveys routinely, even though he’s in the business of surveys.

Response rates below 40% in consumer businesses or 60% in commercial environments indicate a problem. And if your response rates aren’t increasing, your customers are telling you that you don’t respect their time.

That sounds a bit familiar to us.  Customer surveys (or employee surveys, or stakeholder surveys) all too often are exercises merely in “data collection” and not meaningful engagement with customer feedback.  The challenge with quantitative surveys is that it’s relatively easy to treat customers as data points. In face-to-face interviews, it’s relatively impossible to ignore the real concerns of real people, particularly if you have a senior executive witnessing the interview first hand. But in surveys? Every customer becomes a data point.

If your response rates are falling, that is a finding. If you cannot get your surveys shorter than 10 minutes, that is also an indicator that you are placing your company’s needs for insight above your customers’ needs for free time.

Markey has a few great suggestions, including make sure you reach out to customers later and show them what you’ve done with their feedback. We also suggest a few more steps:

• Pilot test your survey with real customers. This needs to be built into the timeline, but it will be worth it. You need to know if your questions actually get to the heart of what customers believe. And you’ll know if you’re asking too much from them.
• Avoid using “off the shelf” charts from Survey Monkey or Excel. Senior executives see hundreds of these charts every week. Be inventive and create your own charts.
• Tell a story. Just like with qualitative research, you need a narrative arc embedded in your report. Make it evocative. Make it human.
• Integrate qualitative findings: mixed methods are hard to do well, but they are the gold standard in good insight. Weave direct quotes from interviews or focus groups into the charts.
• Share the story with customers. This will be difficult if your results are not as positive as you’d hoped. But authenticity is a value that consumers value in companies. Be brave and face your truth. Your customers will respect you for it.

Roger Martin has a great post on Harvard Business Review that summarizes how ethnographic research differs from quantitative surveys.

Martin writes:

Qualitative, and especially observational or ethnographic, research enables us to delve much more deeply into the relationship between our firm and its product/service and the customer. Because we aren’t obsessed about adding all the responses together for ‘rigorous quantitative analysis’, we can let the customer use his own voice/words/vocabulary.

This sounds a lot like the notion of “verstehen,” which refers to the deep understanding that comes from interpretive, qualitative research.

Quantitative research has its place; how else could we measure improvement if not through counting instances or events? Yet we often forget that quantitative data is primarily designed to summarize findings quickly. This is why it’s so popular but also why it’s inadequate to describe many experiences.

I like to us a football game metaphor to describe the real difference between qualitative and quantitative research. Let’s say that the Steelers beat the Patriots 49-15. What would you know about that game? Simply that the Steelers had won.

Would you really know where the turning point in the game came? Would you know about the significance of a mid-game interception? Or perhaps the critical sacking of the Patriots’ quarterback? No, you’d know nothing of the ebb and flow of the game, critical mistakes and successes, or even how the Patriots might feel about their loss. They might actually feel vindicated if their defensive line held tough against the Steelers for 3 out of 4 quarters.

Statistics are a great way of quickly conveying how a group of events, people, or things are similar and different. Mode, median and mean measure “central tendency,” and standard deviation and inter-quartile range tell you “dispersion.” With these two types of measures, you can tell me how similar people are when they choose orange juice, how different they are when they rent cars or attend movies. But you cannot tell me what “more pulp,” means to people, why a “subcompact” car turns off some people, or what people perceive the word “blockbuster” to actually mean.

In short, ethnographic research can clarify all of these deep, nuanced details that quantitative data skates over or takes for granted. Do you want to know how many people attended a “summer blockbuster?” Then by all means, count them. But if you want to know what kind of movie people believe a “blockbuster” to be, then you need to do in-depth ethnographic work.

Few would disagree that fundamental economic change is upon us. Business models are crumbling daily. From the auto industry to the banking industry, it is clear that old ways of doing things are no longer working. The market research industry is just as vulnerable to this shift, yet, like the auto industry before it, it is hardly aware of how deeply its business model is threatened.

The Long Disruption

The market research industry is built for the 20^th Century mass production model, which is rapidly disappearing. The “mass audience” is gone and a fragmented diverse populace has taken its place. This new “audience” defies the easy aggregation of summary statistics on which market research so often relies.  Chris Anderson of Wired figured this out long ago with his book The Long Tail.

The Economic Disruption of The Long Tail -- Anderson, Wired Magazine

He argued that technology lowered the cost of providing services to ever-smaller niches of people, making it possible to sell profitably goods and services that were once too specialized.

This technological shift also means the end of “appointment television.” Digital video recorders allow individuals to time shift their programming to suite them, and not the program executives at television networks.

The Birth (And Death) of Market Research

What does this all have to do with market research? Full-service market research firms are built for the blockbuster era, not for the time of the long tail.

Market research was heavily influenced by the school of “applied sociology,” lead by Paul Lazarsfeld. While at Columbia, Lazarsfeld pioneered many statistical techniques we use today, including the cross tabulation (Babbie and Benaquisto 2002) and the Lazarsfeld-Stanton Analyzer, a machine that records audience reaction to programming in real time (Mattlerart 1996).

The Lazarsfeld-Stanton Analyzer summarizing "the public"

CNN used a variant of this machine for the recent State of The Union address, showing real-time reactions from Democrats in blue, Republicans in red, and Independents in yellow.

The Lazarsfeld brand of insight is based on a fundamental assumption: that the “average” means something. An entry-level statistics course will teach you that average is dragged up or down by extreme values, and the long tail is nothing if not a collection of many extreme values. There is nothing meaningful about knowing that the “average American” rented 30 digital movies a month if, in fact, there were many thousands of Americans who rented none and a many tiny segments that rented somewhere between zero and 40 movies. The “average” is meaningless in this example, yet this ham-fisted approach to summarizing “the public” is what the market research industry is built upon.

Design Research for The Long Tail

Market researchers may argue that with proper segmentation, you can understand every niche within the long tail. This may be true, but to truly understand the diversity between people, your task is not simply to “summarize” the audience, but to delve deeply into the dynamics of what makes them different.

This is why design research is a better fit for today’s long-tail economic model. Context matters. Design research is all about understanding the context because it is rooted in qualitative methodologies, and ethnography in particular. Designers solve contextual problems. The award-winning Braille watch, for example, allows its users to check the time surreptitiously and quickly, something that is both polite and useful. A Lazarsfeld approach would not uncover the social subtleties of checking one’s watch, and certainly could not uncover the specific needs of the blind.

The Braille Watch by David Chavez

Dan Formosa details this limitation of market research in his insightful article in Interactions magazine. He argues that market research should focus on consumer response — after a product is designed. Design research, on the other hand, is about evaluating a product as it is being developed. I would go further; design research is about knowing what to build as well as evaluating the prototype.

Design research uncovers how long-tail niches develop and what differentiates them. It is not the equivalent to “market segments” because it provides specific direction on how to apply research findings. What are the dynamics of renting a movie? What motivates the “heavy renter”? What is it about her television or home that supports heavy renting? You cannot know the answer to these questions by simply providing a laundry list of demographic characteristics and psychographic survey results. You must know the context in which the long tail emerges.

Some may say that good quality market research would not make these kinds of mistakes. And they are right. Highly skilled social scientists are method-agnostic; they choose the right method for the right research question. However, full-service market research firms have become the GM of the industry — they keep building Hummers instead of Priuses. Focus groups don’t uncover contextual nuances, but they’re cheap and profitable. Surveys don’t get to the heart of why a product doesn’t work. Design research, using an ethnographic approach, provides “thick description” of the entire phenomenon of renting movies.

This is where market research cannot go. And this is where market research will fail, unless it rejects the “build another Hummer” mentality.

References

Babbie, E. and L. Benaquisto (2002). Fundamentals of Social Research. Scarborough, Thomson Nelson.

Mattlerart, A. (1996). The Invention of Communication. Minneapolis, University of Minnesota Press.

DT has a great post over at Design Sojourn that discusses Six Sigma methodology and how it relates to design. He cites Tim Brown at IDEO who argues that Six Sigma is essentially Newtonian, while design thinking is quantum. In his own design work, DT expressed doubts about using Six Sigma:

After studying the Six Sigma process, I point blank said: �There was no way any of my designers are going to be judged on the quality and success of a design based on how many sketches or iterations we did before we deliver it.�

Both Brown and DT cite Sara Beckman, who recently discussed the topic in the New York Times. Beckman reviews how Six Sigma focuses on incremental improvements, while design and design thinking focuses on big changes. For those of you who aren’t familiar with Six Sigma, it’s a method pioneered by Motorola, which aims to reduce the number of errors to 3 in one million. The “six sigma” refers to six standard deviations. The number of errors should be at the extreme end of the normal curve, or between + or – 3 standard deviations, represented by the Greek symbol sigma.

I argue that design is more complementary to the “interpretivist” paradigm of qualitative research while Six Sigma is positivist. Interpretivists don’t believe the world is a static place. They see reality as being continuously created by you, me and other social actors. There is no such thing as “The Truth” in interpretivist approaches, just different versions of the truth. Typical methods of interpretivists are ethnography, in-depth interviewing and discourse analysis. Positivist research, on the other hand, assumes that reality is static. Positivists believe that “The Truth,” is out there to be discovered. Typical methods would include quantitative surveys.

Designers should focus on interpretivist methods, therefore. They should uncover different versions of the truth using observation and interviewing, as well as deep reflection on symbols and their meanings. Surveys and other quantitative methods are more Six Sigma in that they can measure improvement over time. Designers ought to consider measuring improvement, but starting with qualitative approaches is best.

In an interesting study, researchers at UBC have found that customers express higher satisfaction when they’re served by white men than by women or people of colour — even when their behaviour is exactly the same. Marketing professor Karl Aquino expressed surprise at the findings, as he told The Globe and Mail

�We had thought there would be some bias going on in the sense of people who were males or whites would be rated more positively,� Mr. Aquino said

�But we didn’t anticipate that for performing the same behaviours, the women and minorities would actually be rated lower,� he said of the study to be published in the Academy of Management Journal.

This study should not be surprising at all.

What this study demonstrates is what Raymond Breton calls the “symbolic order”; we unconsciously place white men at the top of our social hierarchy. We do this in multiple ways, including placing art, culture and ideas at the top of an invisible ladder. Public Enemy sums it up nicely in “Fight the Power”:

None of my heroes don’t appear on no stamp

We know that people have largely unconscious reactions of sexism and racism, oftentimes without even realizing it. It is likely that these unconscious ideas bleed into marketing research easily, especially when such studies are quantitative in nature, and therefore lack the thick description or deep probing offered by qualitative approaches.

This finding has wide-reaching implications. First, when companies use customer satisfaction surveys, they must be aware of the inherent inaccuracy of these surveys. You may believe you’re accurately measuring actual satisfaction, but this study shows that frequently, we don’t measure any such thing. Secondly, such surveys are often used to award bonuses or even job security. As we know in academia, student evaluations are frequently what stands between a scholar and a full-time position. If we know that customer satisfaction is driven by factors other than actual performance, then we are likely to be unwittingly simply rewarding membership in a dominant group.

Read the entire story on The Globe. It’s worth a think.

Those of you out there who’ve tried it know: recruiting research participants is HARD. Here are a few insights from the research to help you with better recuitment.

1. Personalized contact with respondents, followed by pre-contact and aggressive follow-up phone calls *: Don’t count on a form letter, email or random tweet to do the job. Capitalize on your personal relationship with that person. If you don’t have a personal relationship, ensure that you use the person’s name, and for God’s sake, spell it correctly!

   Once you’ve made initial contact, you are not done. Not by a long shot. Make sure you speak to the person (you can do this through IM or email if you’d like) to give them more information. They’re now interested. Don’t stop! One more step!

   Follow up 1 week after initial contact. Assuage any fears they may have. Answer any questions honestly. And above all, be available for more information.

2. External researchers with social capital are best**: University-based researchers have been shown to have the best participation rates, but you don’t have to be a professor.� Researcher Sister Marie Augusta Neal of Emmanuel College achieved a near perfect response rate because of her close ties to the respondents and their communities. The lesson here is, if you hire a consultant, make sure they’re trusted. Even better if they personally know the people to be recruited.
3. Monetary incentives have no effect, unless money is offered �no strings attached�***: Little known fact: the best way to use a monetary incentive is to offer it, up front, with absolutely no strings attached. The “free” money makes people feel more indebted socially. Evidence of this effect can be found in the book Freakonomics. Researchers found that daycare centres that levied late penalties on tardy parents actually had more of a late-pickup problem than those that levied no fine. Why? Because the parents reduced their relationship to the daycare as a mere transaction. Use the “gift economy” approach and ensure a feeling of indebtedness. My personal favourite is a coupon for a single iTunes song at $.99. It is cheap but appears to have great value. Offer it, up front, and then ask for participation

*� Cook, C., F. Heath, and R. Thompson. 2000. “A Meta-analysis of Response Rates in Web or Internet-based Surveys.” Educational and Psychological Measurement 60:821-836.

** Rogelberg, S., A. Luong, M. Sederburg, and D. Cristol. 2000. “Employee Attitude Surveys: Examining the Attitudes of Noncompliant Employees.” Journal of Applied Psychology 85:284-293.

***Hager, M., S. Wilson, T. Pollak, and P. Rooney. 2003. “Response Rates for Mail Surveys of Nonprofit Organizations: A Review and Empirical Test.” Nonprofit and Voluntary Sector Quarterly 32:252-267. Singer, E. (2006) Introduction: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, 70, 637-645

Folks,

Below is a (very!) brief overview of online surveys. This slideshow, via slideshare, is intended for people in the Web design industry. IAs, designers, media planners, strategists, usability researchers, and producers will learn if they should, in fact, do a survey.