A review of
Beyond the Usability Lab: Conducting Large-Scale Online User Experience Studies
By Bill Albert, Tom Tullis, Donna Tedesco.
Morgan Kaufmann, 2010

In 2008, UXMagazine devoted an issue to remote usability testing guest edited by Tomer Sharon. As managing editor at the time, I confess to having had a rather skeptical view of the value of remote testing.

One of the articles that made me re-think my attitude was a case study comparing the relative usability of two sets of information about the Apollo program, one provided by NASA, and one by Wikipedia. That case study was written by Tom Tullis, one of the authors of Beyond the Usability Lab, which takes the themes introduced in that issue (Vol 7, Issue 3, 2008) and explores them in detail.

Traditional usability testing has been focused on small sets of users, and the book does not take issue with this approach. Indeed Bill Albert said to me in an interview,  “…if the goal is really just to identify usability issues, I think I’d fall in line with a lot of other people, saying six to eight users is plenty to identify the significant usability issues with a particular design.” (User Experience podcast, episode 55, www.uxpod.com).

What the 2010 book concerns is the fact that web technology “enables us to move beyond the lab and efficiently conduct user experience studies with a much larger sample of users.”

The book begins with an introduction describing what the authors mean by “online usability studies,” including a description of when such studies are appropriate, what one can expect to achieve, and the strengths and limitations. Remote studies are good for comparing designs, for collecting detailed and scalable usability metrics, and for exploring design issues in the users’ own environment, with all its attendant complexities. On the other hand, there are many instances (such as identifying the major usability issues with early prototypes) when other methods are preferred.

The book is well-structured for the practitioner. After the introduction, the following three chapters explore planning, designing, piloting, and launching the study. The hands-on approach is reminiscent of Rubin’s (and now Chisnell’s) classic Handbook of Usability Testing, in that it contains sufficient detail to enable a practitioner to engage the method with a degree of confidence.

Chapters 5 and 6 discuss data preparation, analysis, and presentation.  Chapter 7 provides good in-depth analysis of specific tools (Loop11, RelevantView, UserZoom, and WebEffective) that can be used to conduct remote studies, as well as advice on choosing the appropriate tool for your own study. Chapter 8 discusses discount methods, including “totally homegrown techniques for the adventurous.”

Chapter 9 presents seven case studies of remote research conducted with between 24-300 users with a range of tools.

Throughout the book, specific examples illustrate concepts and methods. The authors provide detailed instructions for using Microsoft Excel to calculate appropriate averages and confidence intervals. There is also advice on dealing with data gathered from open-ended questions (when simple numerical analysis is not adequate).

The authors describe how to identify and deal with data from “flat-liners”—participants who complete studies as quickly as possible to obtain the associated incentive.

It’s a real pleasure to encounter a book that not only takes the reader on a journey through the rich possibilities of technique, but does so in a manner that is clear, readable, and accessible. I was particularly pleased with the simple explanations of statistical techniques, which are so often presented as incomprehensible.

If you’re interested in any of the following questions, you can look to this book for practical and effective answers:

• Should you conduct a between-subjects or within-subject study?
• What variables do you need to consider?
• How can you deal with outliers?
• How can you calculate and display confidence intervals?

The book does not shy away from the difficulties involved in conducting remote research. For example, if you want click-stream data, it may be necessary to have participants install or allow a plug-in, which may mean you can’t test with so-called novice users.

If I were to complain, it would be about the need for a chapter specifically on conducting studies on mobile devices—an area that is ripe for a similarly detailed “how-to” guide.

Whether you’ve conducted remote studies in the past and want to extend your capability and knowledge, or you are a complete newcomer, this excellent book is a necessary companion on your journey from the lab into the world outside. You will refer to it often, and it will alert you to opportunities and dangers. What more could you ask of a book?

Topics: Usability Testing, Remote Testing, UX Methods
Published in: June, 2011 in Design for Social Change