Photo courtesy of Danny Hope
Figure 1.1
The Appeal of Lab Research
Part of the appeal of lab-based user research was that it provided a seemingly scientific basis for making decisions by using observational data, instead of someone’s error-prone gut instincts. Stakeholders appreciated the firm protocol and apparent reliability of properly managed lab research. Lots of user research practitioners continue to perform lab research just because it’s what people have been doing for a long time.
Market Research vs. User Experience Research
Let’s make something clear. Focus groups are practically synonymous with user research in most people’s minds, and focus groups belong to the world of market research. But there’s a huge difference between market research and user experience (UX) research. Market research is much more common and comprises the lion’s share of research spending; UX research comprises just a fraction of that (see Figure 1.2).
Figure 1.2
However, this book is about user experience research, not market research. The main difference between the two fields is that market research focuses on opinions and preferences, whereas UX research focuses on behaviors. The distinction can be confusing, especially since a lot of online consumer research companies try to convince you they give you insight into “what your customers are doing on your Web site,” when they’re really just providing opinions.
A market research study might have goals like these:
“Determine how users respond to our branding.”
“Identify different segments’ color preferences for the homepage.”
“See if users like our new mascot.”
“Determine what users enjoy most and least about our site.”
While the goals of a UX study, on the other hand, would sound more like these:
“Can anyone actually use my interface?”
“Determine where users make errors in completing a purchase.”
“See whether users can successfully create a playlist.”
“Understand why users aren’t logging in.”
“See how users mentally organize different product categories.”
It’s important to keep in mind that market research is pretty useless over small sample sizes. Opinions can vary widely across demographics and location, are very sensitive to the phrasing of the research questions, and can change fairly quickly. Behavior, on the other hand, is fairly consistent across demographics and location for many tasks, and most usability flaws in a given software interface can be uncovered in a moderated study by a much smaller number of users.
How small? That’s a contentious question. UX luminary Jakob Nielsen (in)famously claimed that having five users was enough to uncover 80% of usability flaws in an interface, but others like Jared Spool insist that the number depends on factors such as user segmentation, risks associated with the errors, task complexity, and so on. At any rate, our point is that a moderated UX study usually requires much fewer people than a market research study.
Put it this way: ask 10 people what they think about how well a door is designed, and their comments might not overlap at all. One blames the condition of the hinges, another talks about the weight of the door, another complains about the color of the doorframe, and so on. But if you observe 10 people walking through the door, and the first two accidentally try to push when they ought to pull, then you’ve found your design flaw right there.
So, to put it all together: whether you go with market research or UX research depends on what you’re trying to find out. This book is about UX research, so it’s focused on user behaviors rather than opinions.
Further reading about the sample size question
Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084–3088). Boca Raton, FL: CRC Press.
Lewis, J. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. The International Journal of Human–Computer Interaction 13(4), 445–479.
Lindgaard, G., and Chattratichart, J. (2007). Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, CA, USA, April 28–May 03, 2007). CHI ’07. ACM, New York, NY, 1415–142
Is Lab Research Dead?
Heck no. Lab and remote research share the same broad purpose: to understand how people interact and behave with the thing you’ve made (from here on, let’s just call it “the interface”). There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with security, equipment, or the type of interaction you want to have with your research participants. More generally, lab research is appropriate when you need a high degree of control over some aspect of the session, such as the following situations.
Info security. Security is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you must have your users sign a nondisclosure form.
Inability to use screen sharing. You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions apply only to moderated studies, for which you need to see what’s on your users’ screens.
The need for special equipment. Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly, which is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.
The importance of seeing the user’s body. Some kinds of research will require you to study certain things about the user that are difficult