The Essence of Interaction Design Research: A Call for Consistency

This post is reproduced from the original Interactions magazine article

It started with an innocent query to the IxDA listserv. Someone was sure they had read an article in Interactions magazine once but could not find it again: Wasn’t there something written sometime by someone about something like sample size in usability research? asked an expectant interaction designer. Woe is the hapless interaction designer who is unprepared for the firestorm that follows the dreaded “sample size” question. 106 replies later, and not only was the answer clearly left unanswered but worse, it left many scratching their heads in genuine confusion: what is the essence of interaction design research? Is it data-driven and “scientific”? Is it exploratory and qualitative? No consensus was reached. Again.

What's your sample size again?

This schizophrenia is both a blessing and a curse. One the one hand, an interaction designer has the freedom to assemble her research program like an artist assembling an installation: whatever inspires her can indeed find a place in the final result. Yet, such a lack of standards leads to a distinct lack of consistency and expertise. If interaction design research is whatever you want it to be, what is to stop other occupations “colonizing” what ought to be the purview of the interaction research? See, for example, Dan Formosa’s article in this year’s January-February issue of Interactions, lamenting the intrusion of market research into the design field. When there are no standards, there is freedom.

As Sartre said, we are “condemned to be free,” meaning when there are no pre-defined codes of conduct, then we must tragically, wonderfully, horribly, create ourselves. The confusion over the essence of interaction design research is us, thrashing about as we desperately create ourselves.

You are condemned to be free: courtesy of http://artoftheprank.com/author/mibanez/

In this article, I explain how this lack of standardization affects the practice of interaction design research. In particular, I note that the dreaded “sample size” debate is actually indicative of a larger issue of theoretical training. I call on interaction designers to embrace standardization – not blindly, but with eyes wide open – for the benefit of the interaction design research and for the profession itself.

The Long and Winding Road

Most people stumble into interaction design. Unlike a profession such as medicine, for example, interaction design has a distinctively ill-defined apprenticeship. The proliferation of interaction design job titles demonstrates this lack of definition. A lack of standardization is liberating to many but has the unintended consequence of undermining the interaction designer’s autonomy. To become an, accountant, professor or engineer, individuals must meet compulsory standards, pass examinations, and prove their mastery of the profession’s “canon” of knowledge in order to practice it. Jobs that require a “canon” are typically called a “profession” instead of a mere “occupation.”

Indeed, a “profession” is not simply a job requiring skill. A profession differs from an occupation in that its members exercise exclusive control over a specific body of knowledge (Friedman, 2000, Greenwood, 1957, Larson, 1977). A profession must therefore have a clearly defined certification process, which in turn allows its members to exercise a sort of monopoly over the work itself. If a doctor is fired from a hospital, she continues to be a doctor. No hospital administrator can remove her ability to write prescriptions, for example. Only her peers can remove or grant this ability. Her peers have decided she has met the minimum acceptable standards to write prescriptions and practice medicine; the hospital administrator’s opinion is irrelevant. The power of the professional, then, is inextricably bound up with her knowledge and training.

The “community of practice” is no substitute for a profession. It is merely the poor man’s version of a profession; it refers to the informal knowledge sharing sessions of Xerox technicians, who bully each other instead of fighting for higher wages or more autonomy (Seely Brown and Duguid, 1991).  As with copier repair, there is no body of knowledge that is collectively recognized as comprising “interaction design,” much less “interaction design research.” In their 2006 survey, the IA Institute found 48% of self-identified information architects had no formal training, and almost 3% of those surveyed “weren’t sure” (!) if they had formal education (Information Architecture Institute, 2006).  It is for this reason that there is much confusion about what interaction design research should really look like. No accountant questions how to gather data for creating a cash-flow statement. Certainly, there may be debate about the “right” method, and perhaps there are several schools of thought to which individual accountants tend to subscribe. But in the end, there is no debate that a cash flow statement has X, Y, and Z and if it has A, B, and C, then it is not a cash flow statement, but a balance sheet.

Interaction designers have no such luxury. What exactly constitutes an “interaction”? Where does interaction design end and aesthetic design begin? These questions may seem overly theoretical to some, and indeed, they are theoretical. But it is this very line of questioning that defines the professionalization process. What constitutes a dentist over a dental hygienist? Dentists and dental professors themselves defined that difference – for their own benefit (Adams, 2003). Practitioners of a discipline must delineate the theoretical confines of a discipline (and the requisite knowledge that must be mastered to claim expertise in that discipline) in order to claim occupational autonomy. Simply forming a “community of practice” and gathering for “shop talk” is not sufficient. Xerox technicians have not successfully created a monopoly of knowledge over photocopiers, neither have they created a strong lobby for occupational control.

The HCI Connection

This is not to say that interaction design is completely bereft of an intellectual tradition. The IA Institute’s industry survey did find that, of the information architects that were formally trained, 40% of them had training in Library Science and another 12% in Human Computer Interaction. This suggests there is, at least, a significant number of practitioners (at least those identifying as “information architects”) with similar training. The HCI and Library Science disciplines inculcate their students with a distinctively quantitative approach to research. The November 2009 annual meeting for the American Society for Information Science and Technology (ASIS&T) included a full-day pre-conference workshop “infometrics” and “scientometrics,” which trained participants on a multitude of quantitative methods. The ASIS&T also maintains several “special interest groups” or SIGs that are specifically targeted around metrics, measurement and quantitative methods. Not one SIG specializes in “design” or “qualitative” methods. The Computer Human Interaction (CHI) SIG in ASIS&T professes interest in “online users and their behavior,” and not the symbolic, interpretive or otherwise cultural aspects of the online experience.

One Small Question: What is reality?

Underneath this focus on metrics and “behavior” is a set of implicit: assumptions within the HCI/Information Science tradition. This assumption cuts to the heart of the “sample size” debate: what is the nature of the world and what is the best way to research it? Most researchers subscribe, at least in part, to two established schools of methodological thought: quantitative and qualitative. While they may never be “purely” quantitative or qualitative in their research approaches, researchers tend to subscribe to the overall tenets of their school. The archetypical or “ideal type” quantitative researcher may not actually exist, but describing her methodological approach elucidates unspoken assumptions many researchers may have.

The archetypical quantitative researcher first starts with the assumption that the world is a “real” place that exists independently of human beings (Bryman, 2006). In other words, quantitative research has an objectivist ontology, one which assumes reality is an objective thing that can be researched. Accordingly, the ideal-type quantitative researcher also assumes that the scientific method is the best way to discover this reality, and that a researcher does not affect or shape the outcomes of the research, if appropriate steps to avoid “bias” are taken. On the whole, this approach means looking for the most “typical” occurrence, one which has a necessarily statistical description (Alasuutari, 1995).

Table 1: Qualitative Versus Quantitative Research Paradigms

By contrast, the archetypical qualitative researcher assumes the world is not an objective reality but something that is constructed by us humans, every moment of everyday (Denzin and Lincoln, 2000). Such a researcher considers how humans “make sense” of the world as having primary importance, so his methods are typically aimed at uncovering or “unriddling” this sensemaking process (Alasuutari, 1995). Numerical representations of the “typical” occurrence are irrelevant in this view because there is no typical occurrence.

One can see how “scientific” approaches to interaction design research evolved, therefore, from the objectivist, positivist research paradigm. In this paradigm, it makes sense to count and to find the “average.” And of course in order to do so, one must count sufficient numbers to make it statistically valid. But if one adopts the assumption that there is no such thing as “typical,” that how we make sense of language, for example, tells us how to build Web sites, then it is a logical choice to reject “sample size” as important. The process of sensemaking is more important to the constructivist, interpretivist researcher.

The Design Connection

It is unclear how many self-identified “interaction designers” would reject, wholesale, the title or description of “information architect.” Herein lies the problem. To reject “information architecture” in favor of “interaction design,” is actually to reject the positivist tradition of information “science” in favor of “design.” This is a significant turn.

Design spans both art and science, making its ontological and epistemological position unclear. Design requires both the “logical character of the scientific approach and the intuitive and artistic dimensions of the creative effort.” It spans both deductive and adductive logic; it is the “the process of creation and decision-making” (Borja De Mozota, 2003). Interaction designers draw on both the “science” of decision-making but also the art of creativity.

Is it any wonder then, where our collective schizophrenia comes from? We are fraught with existential angst by the very label of the occupation. We are not entirely sure if we are information scientists or if we are artists. We create our own professional identities as a bricolage, choosing pieces that suit us and rejecting those that don’t. If there are no standards, there is freedom.

Freedom Ain’t Free: A Call To Action

Interaction designers may feel blessed to draw on the scientific tradition for one research project, and on the interpretivist tradition for another. This may feel liberating. But it has its cost.

Professionals command higher pay, status, and autonomy precisely because they have agreed to subscribe to a canon of collected knowledge. They accept that they must prove their familiarity with, say, contracts law even though they do not plan to use it and could easily do without it, thank you very much. Professionals do endure such “irrelevant” learning because they recognize the benefits of having their occupation controlled, even somewhat, by their peers. They enjoy greater freedom at work (Greenwood, 1957). They have higher salaries (Larson, 1977). They can even withstand the slings and arrows of globalization and maintain their professional autonomy (Faulconbridge and Muzio, 2008). Professionals know that by sacrificing a little, they get a lot.

So a call to action. Interaction designers: now is the time to define the theoretical boundaries of your knowledge. What exactly constitutes an “interaction” and how exactly might one “design” it? What is the difference between an interaction designer and an information architect? What, by extension constitutes interaction design research? And finally, for once and for all, does an interaction designer need to care about sample size?

These questions must be answered. We must answer them. I’m not suggesting that interaction designers drop everything and begin furiously debating in the pages of academic journals. Rather, I am suggesting that design educators begin instilling clear and defined canons of knowledge in their students, that practitioners begin adopting (gasp!) standards when hiring, and that collectively, we pursue a consensus.

I present two illustrative examples of professionalization: engineering and medicine. Engineers structured their occupation and thereby collect some benefit, but physicians gained exclusive rights over key aspects of their practice, making their professionalization process much more successful. David Noble traces the professionalization of the engineer in his fascinating history America By Design (Noble, 1979).

It was businessmen, not university-based researchers, which lead the drive to professionalized engineering, resulting in engineers becoming “company men” instead of independent practitioners. Engineers successfully controlled entry into the profession but oftentimes rely on engineering employers for a professional identity. By contrast, physicians professionalized their occupation as a group of independent practitioners. Indeed, it is physicians that all other professions look to emulate (Ritzer and Walczak, 1988). While there have been many recent changes that limit physician autonomy (O’Connor and Lanning, 1992), physicians continue to maintain a near monopoly over the legal ability to prescribe drugs (in the United States, nurse practitioners can prescribe some drugs).

The lesson from these two professions is first to ensure practitioners, not companies, drive professionalization. The Interaction Design Association and the Information Architecture Institute are great starts in this direction. But secondly, interaction designers must gain exclusive control over a certain body of knowledge. For example, interaction designers may seek to “own” accessibility-compliant Web site design. Interaction designers may end up with several schools of thought, which is perfectly acceptable (there are, after all, Jungian psychiatrists as well as Freudians). But at the very least, we will never waste another single pixel on the dreaded “sample size” question!

About the Author

Sam Ladner is a sociologist with an interest in the design of technology and its effect on organizations. She mixes private-sector consulting work with academic research and teaching. Using a range of methods including interviewing, observation and ethnography, she consults on digital product design, organizational change, and the social aspects of technological innovation. She holds a PhD in sociology from York University. She currently works for her own firm as consultant and principal with Copernicus Consulting Group and frequently partners with design firms.

References

ADAMS, T. (2003) Feminization of Professions: The Case of Women In Dentistry. Canadian Review of Sociology and Anthropology, 40.

ALASUUTARI, P. (1995) Researching Culture: Qualitative Methods and Cultural Studies, Thousand Oaks, Sage.

BORJA DE MOZOTA, B. (2003) Design Management: Using Design to Build Value and Corporate Innovation, New York, All Worth Press.

BRYMAN, A. (2006) Integrating Quantitative and Qualitative Research: How Is It Done? Qualitative Research, 6, 97-113.

DENZIN, N. & LINCOLN, Y. (2000) Introduction: The Discipline and Practice of Qualitative Research. IN DENZIN, N. & LINCOLN, Y. (Eds.) Handbook of Qualitative Research. 2nd ed. Thousand Oaks, Sage.

FAULCONBRIDGE, J. & MUZIO, D. (2008) Organizational professionalism in globalizing law firms. Work, Employment and Society, 22, 7-25.

FRIEDMAN, M. (2000) Autonomy, Social Disruption and Women. IN MACKENZIE, C. & STOLJAR, N. (Eds.) Relational Autonomy: Feminist Perspectives on Autonomy, Agency, and the Social Self. Oxford, Oxford University Press.

GREENWOOD, E. (1957) Attributes of a profession. Social Work, 2, 44-55.

INFORMATION ARCHITECTURE INSTITUTE (2006) Information Architecture For The World Wide Web Industry Survey. Seattle, Information Architecture Institute.

LARSON, M. (1977) The Rise of Professionalism, Berkeley, University of California Press.

NOBLE, D. F. (1979) America by design : science, technology, and the rise of corporate capitalism, New York, Oxford University Press.

O’CONNOR, S. & LANNING, J. (1992) The End of Autonomy? Reflections of the Postprofessional Physician. Health Care Management Review, 17, 63-73.

RITZER, G. & WALCZAK, D. (1988) Rationalization and the Deprofessionalization of Physicians. Social Forces, 67, 1-22.

SEELY BROWN, J. & DUGUID, P. (1991) Organizational Learning and Communities of Practice: toward a unified view of working, learning and innovation. Organization Science, 2, 40-57.

“© ACM, (2009). This is the author’s version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in PUBLICATION, XVII.2 – March / April, 2010,

Gladwell sparked quite the social media flare when he claimed recently that social media pundits greatly overestimate social media’s ability to effect social change. He compared the famous civil-rights era lunch-counter sit-ins to the revolutionary activity in contemporary Iran and found good, old-fashioned face-to-face relationships were a more effective mobilization tool than the new-fangled Twitter and Facebook.

Malcom Gladwell for The New Yorker

But Gladwell is wrong.

He makes the classic mistake of arguing that a particular technology may (or may not) lead to a particular result. In the real, messy, social world, X technology is not guaranteed to lead to Y results. Nor is X technology guaranteed NOT to lead to Y results. Gladwell commits the same sin as those of social media pundits he so blithely condemns. Namely, Gladwell is a technological determinist with a poor grasp of actual social interaction.

Sociologists, by contrast, recognize the social world is complex and full of exceptions. Their contribution to the phenomena of social change is far more nuanced than Gladwell suggets.

He correctly asserts that the lunch-counter sit-ins were both more effective and required more commitment from its participants. He cites Golnaz Esfandiari who shrewdly noted that Iranian tweets were written almost exclusively in English, not Farsi. The “Twitter revolution” was actually Westerners in Western places, scrolling through the tweets tagged “#iranelection.”

But he goes on to argue that activism in the Facebook Age ain’t what it used to be,

Fifty years after one of the most extraordinary episodes of social upheaval in American history, we seem to have forgotten what activism is.

Social media help us organize life but they don’t make change, he argues:

The instruments of social media are well suited to making the existing social order more efficient. They are not a natural enemy of the status quo.

Sociologists of technology have long ago found that technology has indeterminant results, which can only be understood by examining the social context in which the technology is introduced. Technology researchers Pinch and Bjiker called this “interpretive flexibility” which suggests that technologies are used differently in different social contexts.

Bicycle Mower

Pinch and Bijker  introduced this idea decades ago, yet Gladwell has no grasp on these decades of socio-technical analysis.

Instead, he uses a single social theory, specifically, Granovetter’s social network theory of “weak ties” versus “strong ties.” Gladwell argues that social change requires a great deal of “strong ties,” which existed in the south at the time of the civil rights movements.

Yet he does no comparison to other examples of social change, such as the ‘60s counter-culture in general, which cannot be demonstrably be traced to a disproportionate amount of strong ties. Instead, social scientists have attributed the mass youth uprisings of the ‘60s in North America and especially France to a high number of young people with “post bourgeois” values.

Gladwell also simplistically characterizes effective organization as necessarily “hierarchical.” But sociologists know that social institutions can either enable or constrain weak or strong ties. Sociologist Mario Luis Small, for example, has shown that how organizations connect people matters, and conceivably matters more than the technology they use. Small found that daycare centres in New York City encouraged the formation of new ties between parents when they had frequent, non-competitive and regular interactions, such as meeting to plan field trips or daycare holiday parties.

Researchers have also found recently that mobile technology use actually strengthens strong ties, depending on the context in which they are used.

The lunch-counter sit-ins succeeded because they had the requisite organizational structure already in place, not because these organizations were “hierarchical” as Gladwell argues. Hierarchies are not required to create social change, as much as Lenin’s “vanguard elite” might want us to believe it.

And these lunch-counter protesters did have technology that they used to incite more activism, notably, the telephone and the newsletters and newspapers. Gladwell skates over this fact in his zeal to condemn social media.

Maybe we should cut Gladwell some slack. He is, after all, a great synthesizer and a storyteller. He succeeds in popularizing ideas that have gained little attention outside academic circles. And granted, his New Yorker post was a brief post, not a treatise.

But fans of Gladwell take note: his ideas are usually not his. Worse, he often fails to apply them in the same thoughtful, nuanced ways their originators had intended.

UPDATE: Gladwell has incited a fair number of awesome ripostes including @leighh ’s very similar post and this awesome one from Anil Dash.

“But how many people did you talk to?” If you’ve ever done qualitative research, you’ve heard that question at least once. And the first time? You were flummoxed. In 3 short minutes, you can be assured that will never happen again.

Folks, qualitative research does not worry about numbers of people; it worries about deep understanding. Weber called this “verstehen.” (Come to think of it, most German people call it that too. Coincidence?). Geertz called it “thick description.” It’s about knowing — really knowing — the phenomenon you’re researching. You’ve lived, breathed, and slept this thing, this social occurrence, this…this…part of everyday life. You know it inside and out.

Courtesy of daniel_blue on Flickr

You know when it’s typical, when it’s unusual, what kinds of people� do this thing, and how. You know why someone would never do this thing, and when they would but just lie about it. In short, you’ve transcended merely noticing this phenomenon. Now, you’re ready to give a 1-hour lecture on it, complete with illustrative examples.

Now if that thing is, say, kitchen use, then stand back! You’re not an Iron Chef, you are a Platinum Chef! You have spent hours inside kitchens of all shapes and sizes. You know how people love them, how they hate them, when they’re ashamed of them and when (very rarely) they destroy them. You can tell casual observers it is “simplistic” to think of how many people have gas stoves. No, you tell them, it’s not about how many people, it’s about WHY they have gas stoves! It’s about what happens when you finally buy a gas stove! It’s about….so much more than how many.

Welcome to the world of verstehen. When you have verstehen, you can perhaps count how many people have gas stoves. Sure, you could determine that more men than women have them. Maybe you could find out that more of them were built between 1970 and 80 than 1990 and 2000. But what good is that number? What does it even mean?

When you’re designing, you must know what the gas stove means. You must know what it means to transform your kitchen into one that can and should host a gas stove. You must know why a person would be “ashamed” to have a gas stove (are they ashamed of their new wealth? do they come from a long line of safety-conscious firefighters?). You must know more than “how many.”

So the next time someone asks you, “how many people did you talk to?”, you can answer them with an hour-long treatise about why that doesn’t matter. You can tell them you are going to blow them away with the thick description of what this thing means to people. You are going to tell them you know more about this thing than anyone who ever lived, and then, dammit, you’re gonna design something so fantastic, so amazing that they too will be screaming in German. You have verstehen!

See my discussion about sampling methods in qual and quant research for more insight into the reasons why “how many” is irrelevant in qualitative research.

Those of you out there who’ve tried it know: recruiting research participants is HARD. Here are a few insights from the research to help you with better recuitment.

1. Personalized contact with respondents, followed by pre-contact and aggressive follow-up phone calls *: Don’t count on a form letter, email or random tweet to do the job. Capitalize on your personal relationship with that person. If you don’t have a personal relationship, ensure that you use the person’s name, and for God’s sake, spell it correctly!

   Once you’ve made initial contact, you are not done. Not by a long shot. Make sure you speak to the person (you can do this through IM or email if you’d like) to give them more information. They’re now interested. Don’t stop! One more step!

   Follow up 1 week after initial contact. Assuage any fears they may have. Answer any questions honestly. And above all, be available for more information.

2. External researchers with social capital are best**: University-based researchers have been shown to have the best participation rates, but you don’t have to be a professor.� Researcher Sister Marie Augusta Neal of Emmanuel College achieved a near perfect response rate because of her close ties to the respondents and their communities. The lesson here is, if you hire a consultant, make sure they’re trusted. Even better if they personally know the people to be recruited.
3. Monetary incentives have no effect, unless money is offered �no strings attached�***: Little known fact: the best way to use a monetary incentive is to offer it, up front, with absolutely no strings attached. The “free” money makes people feel more indebted socially. Evidence of this effect can be found in the book Freakonomics. Researchers found that daycare centres that levied late penalties on tardy parents actually had more of a late-pickup problem than those that levied no fine. Why? Because the parents reduced their relationship to the daycare as a mere transaction. Use the “gift economy” approach and ensure a feeling of indebtedness. My personal favourite is a coupon for a single iTunes song at $.99. It is cheap but appears to have great value. Offer it, up front, and then ask for participation

*� Cook, C., F. Heath, and R. Thompson. 2000. “A Meta-analysis of Response Rates in Web or Internet-based Surveys.” Educational and Psychological Measurement 60:821-836.

** Rogelberg, S., A. Luong, M. Sederburg, and D. Cristol. 2000. “Employee Attitude Surveys: Examining the Attitudes of Noncompliant Employees.” Journal of Applied Psychology 85:284-293.

***Hager, M., S. Wilson, T. Pollak, and P. Rooney. 2003. “Response Rates for Mail Surveys of Nonprofit Organizations: A Review and Empirical Test.” Nonprofit and Voluntary Sector Quarterly 32:252-267. Singer, E. (2006) Introduction: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, 70, 637-645

Many designers are self-taught, intuitive consumers of research who can translate insights into great designs. But few are trained in the arcane art of research itself. For that reason, many designers don’t know the finer differences between qual and quant research and end up using their respective results inappropriately.

Quantitative research is based on the assumption that random events are predictable, and if you compare your results to pure random results, you can discern distinctive, meaningful patterns about the social world.

Random events are relatively MORE predictable if you have more of them. Imagine if you flipped a coin 20 times. How many heads would you get? Now if you flipped it 20,000 times? You’re more likely to get an even 50/50 split — which is what most people would predict. If you got a 65/35 split with 2o flips, okay, could happen. But with 20,000 flips? No way. Something else is going on.

Translate that to design research by looking at gender, for example. Let’s say you have 20 people, 10 men and 10 women. 65% of the women choose one design, while only 35% of the men do. Is this a meaningful pattern? Impossible to say — you only have 20 people. Now if you had 200 people (100 men and 100 women) and 65 of the women chose one design, chances are you have a meaningful pattern.

This is why sample size matters in quantitative research. But, little known fact, sample size is COMPLETELY IRRELEVANT in qualitative research. Why?

Qualitative research assumes that people have meaningful experiences that can be interpreted. Notice how there’s nothing in there about “prediction” or “randomness.” People have experiences. Researchers discern what these experiences signify. That’s it. Sample size is not only irrelevant, it actually gets in the way of important insight.

Consider the case study, for example. Few people would say case studies are useless. We can learn a great deal about a single design case, where it went wrong and where it went right. The problem comes when you try to predict future events based on this single event.

If you abandon the need for prediction, then sample size never matters. You can always derive insight about design problems from even a single case. Designers that attempt to predict “success” of a single design change, for example, should test that change, repeatedly, with a probability sample.