We’ve all been there. The product team needs data to make a decision, the user research team is fully engaged on other projects, and there’s no budget or time to hire a consultant. So, the product team has to decide between no research or doing user research themselves.

Everyone on the product team should know the basics of research for two reasons. First, if others on the product team can conduct basic research sessions themselves, then the internal research team can focus on more strategic or methodologically difficult research projects. Second, those non-researchers who can do the basics themselves become better consumers of research. They better understand the challenges of finding appropriate participants, designing a good study, and collecting unbiased data.

Possibly most beneficial to the research team, non-researchers who can do some basic studies themselves better understand the labor and time involved, and as a result have more realistic expectations of the research team. This article is a primer that user researchers can provide to others in their organization that will teach them the basics.

How to Get User Feedback Without a Dedicated Researcher

In a nutshell, you need to:

• Define your goals and decide on an approach
• Recruit and screen participants
• Gather data
• Analyze the data
• Share it with the team
• Make decisions

Forming Your Goals and Deciding on an Approach

Setting research goals is the step we see skipped most often by non-researchers. How do you set your goals?

• Start with the end in mind. Ask yourself, “What decision do I need to make?” Work backwards to figure out what information you need in order to make that decision.
• Focus on objectives, not questions. Often people start with “What do I want to know?” but that will result in a long list of questions that may not have any real impact if you answer them. For each question, ask yourself, “What will I do with this information?” If it won’t put you closer to a product decision, leave it out.
• Write down your goals and decisions to be made. You WILL forget, so make sure these are recorded. This also makes it easier to get the team to align on a strategy and to write a report.

Once you’ve identified your goals, now you need to decide how you’re going to address them. In other words, if your questions can’t be answered with a Google search (which may actually be what your team needs), you need to decide which research method you are going to use.

A detailed explanation of methods is beyond the scope of this article, but there are many tools out there to help you decide (such as Laura Klein’s UX for Lean Startups).

We’ll use the overly simple distinction:

• Why or how question qualitative approach. If you want to know why users do (or would be interested in) a thing or how they interact with something, then go with qualitative research. Common methods include focus groups, interviews, and usability tests. These can be in person or remote (for instance, by telephone or video conference) depending on who you want to recruit and what resources you have (renting a research facility, for example, will require a budget).
• How many or how much question survey. If you want to know how many or how much, then go with quantitative research. Common methods include surveys, A/B testing, web analytics, and summative (benchmark) studies.

In this article, we’ll focus on one-on-one interviews (often called in-depth interviews or IDIs) and usability studies because they are the most foundational types of studies that a novice should be able to learn quickly and conduct independently with little guidance from the research team. There are, however, a number of situations when you should bring in help from experts (see Figure 1), including when you know you need any method other than an interview or a usability study. In those cases, you should talk to your in-house researchers or seek the help of an expert; methods like focus groups and surveys are always more complicated than they seem.

Figure 1. You may need to call the experts when logistics get complex, your users are specialized, or your question requires an advanced research approach to answer.

Recruiting & Screening Participants

You can do this yourself or hire an agency that specializes in recruiting research participants. Either way, the steps are the same:

• Write a screening questionnaire
• Decide on what, if any, incentive to offer
• Find participants who meet your screening criteria
• Schedule the participants and send them reminders to help ensure they show up
• Send a thank-you note and incentive quickly after the session (if it wasn’t in person)

If you hire a recruiting firm, you can skip these last three steps, as the research agency will take care of them for you.

Who to recruit as participants

Who you recruit will depend on current goals for your product or designs. Depending on the complexity of the product, you likely have multiple types of target customers. It’s usually helpful to think about the situations in which your product or feature will be used, and then recruit people who are in those situations regularly.

In addition, there are other characteristics you should consider, depending on your research goals:

User Profiles. Your company may have personas, or at least marketing segments, defined. If so, use them in your recruiting. For example, is your product targeted at a certain age group or other broad group such as “moms” or “students?” Is your product targeted at business users at a certain company size, such as start-ups or enterprises? Do users’ businesses fall within a certain vertical, such as “financial services”?

Knowledge. Are you trying to attract new customers or make the initial experience very intuitive? If so, you’ll likely want to recruit people for your study who have never used your product before.

Are you trying to increase your current customers’ usage of your product or change their product mix? Then you’ll want to recruit current customers with at least a moderate amount of experience with your product.

Are you designing features that make users more efficient at repetitive or frequent tasks? If so, then you’ll want to recruit heavy users of the features being designed.

Role. People play roles in both their personal and professional lives. For example, if your product is an ecommerce website, you’ll usually need to recruit people who are the purchase decision makers as well as purchase influencers. However, if your product is a piece of software or a web app, you’ll usually be less interested in the purchase role and more interested in those who are the actual hands-on end users.

How to screen

You will need a screening questionnaire even if you just interview your own customers. (This is a short list of questions you ask prospective participants to make sure they are your target users.) When screening participants, the most important questions to ask concern role and knowledge. You may also want to make sure that you get good representation across some common demographic variables, such as age and gender for consumer products, or industry, company size, and job tasks for business products.

When writing your screening questions, keep in mind that you want to avoid giving away the “right” answer to the potential participant. Avoid yes/no questions and use multiple choice questions wherever possible.

Where to find participants

Finding participants differs depending on whether or not you are recruiting current customers and have access to a customer list.

Working With a List. If the study calls for current customers, you may be able to simply pull a list of customer contacts narrowed down by any criteria for which you already have data (for example, company size or how much the customer spends with your company).

Before reaching out to current customers you will also want to check with, or at least inform, anyone in your company who has a relationship with the customer, such as a sales rep or a customer support contact. This is both a professional courtesy and a safeguard against reaching out to customers who shouldn’t be contacted—like those who are currently so dissatisfied that they are on the brink of leaving. Make sure you screen this list against your company’s “do not call” list.

Working Without a List. If you don’t have a list of customers or if the study calls for non-customers, there are a number of options to pursue, but the process can be labor-intensive. You may be able to use a site intercept tool such as Ethnio to invite website visitors or app users to participate in your study. You may also be able to reach both customers and non-customers through social media, such as Facebook, LinkedIn, and Twitter.

If you have a social media team, be sure to reach out to them for guidance and assistance. There may often be other online sources available, such as forums for people who fit your profile. There may also be in-person Meetups, clubs, organizations, conferences, or user groups that can be a source for recruiting. Sometimes, these types of groups will expect you to “sponsor” their meeting in the form of refreshments in exchange for being allowed to recruit for your study.

Write a Discussion Guide

Whether you’re conducting an interview or a usability study, you must write a discussion guide to make sure you cover the key things you’re looking to understand during the session.

Why write a discussion guide? Because you’re human. The act of writing the guide (and sharing it with your team members for review) gets your questions outside your head and helps you better understand the problem. It also allows you to have a repeatable and consistent structure to the sessions. By keeping the sessions largely consistent across participants, you can detect patterns in what participants say and do.

For both interviews and usability studies, the session should be ordered in an hourglass format, that is, general/broad to specific/narrow to general/broad.

Introduction

• In both interviews and usability studies, introduce yourself and ask the participant to “tell me a little about yourself.”
• Explain “what to expect from the session.”
• Ask for participants’ approval to record the session and discuss confidentiality (you may need them to sign a non-disclosure agreement (NDA) or a Consent to Record).
• You probably want to start each session with a few background questions about the participant to provide context for the participants’ answers and behaviors during the session. For example, if you were building an app to help people buy movie tickets, perhaps you’d want to know what kinds of movies they like best, how often they go to the movies, and with whom they go to the movies. These initial questions help the participant warm up and get used to the environment of a research session, and allow you to build rapport with the participant.

General Usage

• In both interviews and usability studies, move to general questions about how they usually use similar products or conduct usual activities with regard to your product category. For example, if you were building an app to help people buy movie tickets, you might ask questions like, “Tell me about the last time you went to the movies. [Pause for answer.] [If they didn’t spontaneously tell you] Tell me about how you got your tickets that night.”

Specific Usage and Feedback

• In an interview, ask specific questions about your product or concepts that tie in with what you wanted to learn in your research.
• In a usability study, ask people to complete typical tasks using your product or prototype.
  • Scenarios – When writing tasks for a usability study, you’ll want to have an introductory scenario that sets the stage. You may have one such scenario that sets the stage for all tasks or you may have multiple scenarios. These scenarios typically include motivation. In the movie ticket app example, it might be something like, “You and your friends Bob and Jane have decided to go out this Saturday evening.” Next, have the participant complete tasks such as find a movie and buy tickets. Later in the session, you might set up future tasks with the scenario, “Bob has realized he can’t make it after all.” This might be followed by tasks such as return Bob’s ticket or find another friend to transfer Bob’s ticket to.
  • Tasks – Write your tasks so that they are commands that represent real user goals, such as, “Find a movie that starts around 7:00 p.m. this Saturday evening.” Tasks should not tell participants how to do a task, such as “Click the ‘movies’ button.”
  • Think Aloud & Follow-up Questions – You should ask your participants to “think aloud” as they complete tasks. In addition, prepare a short set of probing or follow-up questions for each task to help you understand what needs to be improved. A typical question after a task is, “What do you think about how that worked?”

Wrap-up

• At the end of interviews and usability studies, ask more general questions, like overall feedback on the product/prototype or concept, blue-sky brainstorming about how things could be improved in the future, and final thoughts. A good question to end sessions with is, “Is there anything I didn’t ask you that I should have?”

The guide is there to help structure the session. Keep in mind it really is a guide—not a script. You won’t necessarily ask every question exactly as written. You should aim to keep things conversational. If a participant answers a question you were planning to ask later, you probably won’t need to ask that question again. Every participant may not complete every task. Some take longer or are more verbose in answering questions. Let them go where their own thoughts take them, so long as they are not going too far off-topic, keeping in mind your goals for research and the time available for each session.

A good time to revisit goals is after you’ve written the first draft of your discussion guide. Check each question and task against your original goals. Prioritize them based on how important each is to achieving your goals. Then, review your draft with the rest of the product team to make sure you are all aligned on the goals and that they will be achieved.

Running a Session

Legal Considerations

If you’re planning to record a session, you must get the participant’s consent. You should have a policy on whether the consent needs to be in writing or whether it can simply be verbal. If you’ll be using any information, designs, ideas, mock-ups, or concepts that are not publicly available, you’ll need to get an NDA in place between you and the participant. If you don’t already have standard forms for these, you’ll need to work with your legal department to have forms drawn up. You’ll also want to check with your company’s legal department to see if they have any policies about where and for how long the recording consent and NDA documents must be stored.

Logistics

It’s often helpful to have at least two people: a moderator and a note-taker. It’s difficult for one person to do both of these jobs well at the same time.

Decide on your dress code. Some teams believe that it’s important to blend in with the participants, and thus, try to dress like they do. Some teams believe it’s important to project a professional image and therefore have a more professional dress code. This is something you’ll need to decide upon and make sure everyone follows. You don’t want your moderator showing up in a suit and your note-taker showing up in jeans and a t-shirt.

If you’re able to video and audio record, do so. There are many smartphone apps that let you easily create audio recordings that will suffice for interviews. If you can, though, it is preferable to make video recordings. It’s more engaging for the other members of the product team to see video clips than to just read quotes in your final report.

Whether you’re conducting your sessions in-person or remotely, the easiest way to record video sessions is to use one of the many web-based video conferencing solutions such as WebEx, GoToMeeting, or Zoom. In addition to recording, these tools will allow the product team to observe the sessions in real-time and also allow for remote or in-person participants. For in-person recording, invest in a decent omnidirectional microphone or an external webcam with an omnidirectional microphone; otherwise, your recordings may only pick up one person clearly.

Note Taking

Taking good notes for research is a little different from taking notes in class or a meeting. One way to facilitate good note taking is to prepare a note-taking template for both data collection and analysis. This type of template should have your discussion guide questions pre-populated so you can easily fill in responses. You must be prepared to jump around in your note-taking template because participants may jump around in their responses.

A different way to take notes is to construct a coding system that allows you to write more free-form notes, but quickly tag the notes in real-time. For example, you could have codes that represent tasks (#T1, #T2), interview questions (#I), comments from participants (#C), observations of what participants did (#O), great quotes (#Q), great video (#V), and your own interpretations of insights or findings (#F). This keeps your notes in chronological order, but allows you to quickly scan and search them during analysis.

Remember, in a usability study, you want to take notes not only on what participants say, but also on what they do during the tasks. It’s just as important to observe behavior as to listen to their verbal responses.

Make sure your notes include:

• Participant’s name
• Session time
• Task or question asked
• As complete as possible statement (verbatim is best)
• What the respondent was doing at the time

Asking questions during the session

What makes a question “good?” What gets you usable, accurate data? Asking questions during a session is not like having a normal conversation, so you may be uncomfortable at first. Practice will help. When probing and following up on the questions in your discussion guide, keep these tips in mind:

• Questions should ask about only one thing: “What do you notice first?” NOT “What’s the first thing you notice on this page and where would you go next?”
• Questions should be open-ended—not easily answered with yes or no.
• Avoid leading questions: “Is this easy?” Instead, ask, “What do you think about this?”
• Ask questions you think you know the answer to. You might learn something new or realize your basic assumptions were wrong.
• Ask questions that seem obvious. Don’t assume you understand what the participants “mean” without them actually saying it.
• Don’t provide an answer within a question: “Are you trying to find your order history?” Instead, ask, “What are you trying to do here?”
• Wait for an answer. This is harder than it sounds. If the participant doesn’t answer immediately, keep waiting. Do not offer an answer or ask another question. Remember, this is not a normal conversation; you do not need to avoid silence. Let them think and answer.
• Ask follow up questions: “Why?,” “Tell me more about that,” “Why do you say that?,” “Help me understand what you’re thinking.”
• Do not correct answers. If participants misunderstand your product, ask questions to figure out what is causing the confusion. Do not tell them they are wrong.

Analyzing Data

Now that you have those nicely recorded notes, analysis should flow from your observations.

In the case of interviews, printing your notes out and pulling out a highlighter and/or sticky notes is useful. If you didn’t organize your notes by discussion guide question or task, you can take the time to do that now.

Review your key themes and note any patterns, focusing on behaviors displayed, issues encountered, or comments made by multiple participants. Look for common themes across participants.

If something is interesting but is only supported by a comment or observation by a single person, then note it. It’s something that may need further investigation, but not likely something that can be used to immediately affect your design.

Sharing Your Findings

Make sure to document what you did: your study design and methodology, the details about who you recruited, and when you conducted the sessions. If you’re testing prototypes or concepts, you’ll also want to note the version or sprint because these tend to evolve quickly over time. Naturally, the bulk of your report should document findings and insights that came out of the analysis of your observations. Documenting both what you did and what you found are important so that future team members understand your rationale and don’t retread the same ground.

Making Decisions

Now is the time to decide what changes to make to a product or experience design based on feedback from users. The real discipline comes in waiting until research is complete and the findings are shared and digested. It’s tempting for product team members observing research sessions to see one participant struggle or hear one participant make a suggestion and then jump into “fixing it.” You don’t want to do this; if only 1 out of 10 participants had an issue with something, that means 9 out of 10 didn’t have an issue. You want to make product and design decisions based on those 9 out of 10 participants, not on the 1 participant who was different from the rest. Be sure to prioritize what you’re going to do and record those things in your product backlog or put them on the roadmap for future releases.

Now you have a process for defining your goals, recruiting and screening participants, gathering and analyzing data, sharing findings with the team, and making decisions based on those findings. These basics should get you well on your way to conducting straight-forward interviews and usability studies with only minimal assistance from your research team. Sometimes you will still need to bring in an expert, but now you’ll likely find you can do a lot on your own.

Topics: User Research, Usability Testing, UX Methods
Published in: September, 2016 in Innovation

Comments are closed.