Having clients actively engaged increases buy-in.

Remote user testing is primarily employed  when speed,   convenience, or international testing is called for. However, there are practical qualitative benefits to using remote methodologies, as well. In fact, the remote aspect of the process makes it suited to Wizard of Oz-style moderating techniques, in which a single trained moderator can hide a great deal of complex, behind-the-scenes setup, which can be invaluable for capturing user insights.

One new opportunity is the possibility of multiple people involved in the moderating process —in particular, clients or stakeholders who are invested in the outcome of the user research, but who are not themselves trained for moderating. No longer do clients have to stand behind a “two-way mirror,” passively observing research sessions; remote method­ologies make it possible for them to easily contribute to the process, providing richer results along with greater client buy-in and understanding of the project. This article addresses the benefits, tools, methodologies, and logistics of involving clients in research.

Tools

To get the most out of client participation, clients should be able to see and hear every­thing that the moderator can during the session. In its most basic form, this simply requires clients to sit alongside the moderator and listen in on the session, passing notes to the moderator whenever appropriate. However, there are many ways to use various remote research tools to streamline this process.
For remote web research, the tool choice is an important consideration. The UserVue screen-sharing tool is distinguished by its automated recording functionality, as well as its support for observer chat and video mark­ing for later analysis. However, there are many screen sharing tools with basic built-in observer support, including GoToMeeting,NetMeeting, and Adobe Connect. The audio can be shared using any two-line phone with a conference call feature; the moderator can speak to the participant on one line, and clients can listen in on the other. It’s important that their line remains muted, so participants don’t hear them.

Since the moderator will clearly be unable to speak to anyone other than the participant, typically some form of online chat or instant messaging is used to communicate with clients.

Benefits

This approach provides many advantages:

• TRANSPARENCY
  Having clients not only observe —but engage—with the research as it is conducted can make them more invested in the outcome of the research, and make the findings easier to under­stand and present. Instead of seeing only the product of the research, participating clients can draw upon recollections of the original sessions to place the findings in a richer context. This helps solidify the buy-in and confidence of stakeholders who are unfamiliar with user research and have misgivings about the soundness and validity of the methods. When the mechanisms of the research and the fac­tors underlying the findings are clear, the value of the research becomes more apparent (to the extent that the research is actually valuable)
• GETTING THE RIGHT PARTICIPANTS
  If recruiting for the study is also conducted live (that is, recruits are collected and screened as the study is conducted), clients can help the moderator select their ideal recruits. This allows clients to plan and account for demographic contingencies. For instance, consider a study of a computer hardware retailer in which the moderator has interviewed a reseller of that compa­ny’s products. Resellers are clearly a special-use case of that website, and their input may be valuable, but not necessarily indicative of broader usage of the site. If that was not addressed in the recruiting screener, and another otherwise-qualified reseller is recruited, the client can be on hand to approve or reject the recruit as they deem fit. In this way, clients may be assured that the recruiting sample is not skewed in any misleading way.
• ADDRESSING THE RIGHT QUESTIONS
  Much research is based on goals and objectives specified prior to the research. Whilethis is adequate when the scope of the research is extremely targeted and specific (such as the usability of a particular widget), broader research projects (for example, “How do users respond to our site?”) often benefit from more flexible approaches. If a session gets bogged down in a particular aspect that is of lower priority to the clients, clients can indicate to the moderator that they can move on to a different topic. Even more useful is the notion of emerging topics: clients can identify new and unforeseen areas of interest as they arise during the session, allowing for a flexible and dynamic scope of research. Of course, it’s up to the moderator’s best judgment to keep the scope of the project manageable—clients may identify issues that overreach the domain of the research, and the moderator must balance what is useful to uncover and what constraints the research schedule can accommodate.
• EXPERTISE AND ASSISTANCE
  Since practically any number of people can listen in on a research session, it’s possible to have clients of many different backgrounds and departments participating in the session — for example, a designer, an engineer, a customer service representative, and a CEO can all listen and supply any neces­sary assistance to the moderator. Expert clients can help decode or provide helpful context to complex user comments, espe­cially if the project requires domain-specific knowledge to understand technical jargon. If a prototype or beta model is what’s being tested, those involved in building the prototype can assist the moderator with any malfunctions that arise and see for themselves if the prototype properly reflects the behavior of the finished design.
• NOTE-TAKING AND VIDEO ANNOTATION
  When clients are not actively making requests, any notes they care to take can provide valuable insight into what aspects of the session clients pay the most attention to. If the screen-sharing tool provides live video annotation functionality (as UserVue does), clients can even leave markers for the moderator to analyze later.

Preparing Clients

Participating clients should be properly briefed in advance of the research to make their contributions as useful and non-disruptive as possible.

To minimize questions to the moderator, provide clients with a document they can refer to should they have any questions about their participation or the course of the research. The document should include clear instructions about how to set up screen-shar­ing tools, how to chat or mark videos in UserVue, how to properly contact the modera­tor for requests via IM or chat, and so on. Generally, limiting the number of clients com­municating directly with the moderator to one or two helps keep the requests manageable.

Clients should also be given expectations about what constraints the moderator will be working under during the session. Since the moderator will be taking notes and speaking to the user, it may not be possible to respond immediately to clients’ questions. Style is another constraint: the moderator must keep the tasks natural and the language neutral, so client requests may not be acted upon right away, or precisely as requested. For exam­ple, a client may ask, “Ask the user what he thinks of the navigation bar,” but the moderator may choose to get at the issue in a more natural and less leading manner with a series of questions: “Where would you go to look for laptops?” or “How would you get back to the main page?”

As mentioned above, clients may occa­sionally ask the moderator to address issues that lie far outside of the scope of the research or would be unduly complex to tackle within the allotted session time. Clients should be encouraged in advance to indicate impor­tant emerging topics to the moderator, but to generally adhere to the scope of the research. It may also be wise to politely request that clients defer to the moderator’s judgment with regard to balancing emerging topics with established research goals.

Managing Clients During Testing

If clients are adequately prepared in the above manner, minimal management will be required during the study. However, the mod­erator should not assume that clients will follow the preparations to the letter and should be prepared to actively manage clients during the study.
A common issue is dealing with clients who make too many demands of the moderator to reasonably act upon within the allotted time. It is important to promptly acknowledge all client requests to reassure them that the moderator is listening, but it is not necessary to promise to act upon every request. In order to avoid appearing rushed or irritated with client requests, moderators should use polite, concise, and unambiguously friendly language to acknowledge them: “Understood. I’ll see what I can do! ” Moderators should also ask for clarifi­cation if the client request was unclear: “Not sure I quite understand, can you explain? Thanks!” Ultimately, if there are too many requests to handle, the moderator should have a short discussion with the clients at the end of the session, acknowledging it: “Sorry I couldn’t get to all of your requests—it was important to leave time to achieve all of our study goals. I’ll definitely try to handle as many of your requests as I can, so feel free to keep sending them along!” The aim is to make clients feel comfortable communicating with the moderator, but also that they understand if the moderator cannot comply with every request.

There may also be the problem of clients who don’t seem to be paying enough atten­tion to the session, or who do not ask any
questions at all. If clients are silent for a long time (ten minutes), it’s good practice to ask clients if there’s anything they’re curious about or would like to have the moderator probe further on. If clients respond no, then the moderator can simply continue; if clients remain silent, then the moderator can follow up with a brief assessment of the session to encourage more client discussion: “This user seems to have a lot of trouble logging in. I’ll try to find out more about that.”

Finally, since moderators are responsible for gathering the feedback, they should also avoid soliciting advice or over-relying on clients for what to tell a participant; for example, “What should I ask the user now?”  Client requests should be regarded as more of a tool for refin­ing the research objectives, not a primary source of guidance for the moderator. The primary guidance, of course, should come from documents based on the research objectives, such as the moderator script.

After Testing

It can be very productive to hold very short debriefing sessions after each session to let clients fully elaborate on things they may have said during the study. When all testing is complete, clients may participate in brainstorming sessions, contributing insights they may have gathered as outside observers. This is also a good opportunity for moderators to identify from clients’ interpretations any biases they may have about the research. For example, they may take a user’s comment that “the login process was okay” to mean that there are no problems with the login process, whereas a trained moderator may have identified numerous behavioral issues that contradict that perception. Moderators can then be prepared to address this bias during the drafting of the report. Finally, when the findings are presented, it’s nice to take time to acknowledge the clients’ contributions to the project, specifically calling out instances where they helped. Your ongoing efforts will benefit from having the assistance of clients who are active and interested and who have gained practical experience in user research.

Topics: Usability Testing, Remote Testing, UX Methods
Published in: September, 2008 in Remote Testing