As practitioners, we spend much effort designing and testing products and services within the confines of our offices and labs when we know that a rich user experience lies outside. We need more research “in the wild,” where peo­ple use the very interfaces we take so much time to design, test, iterate, and develop.

Now, what if we could conduct studies “in the wild” using a robust platform that collects data from the users, their actions and environ­ment, and even handles compensation, recording, and event and task scheduling? And what if this tool was not an additional research device that must be carried around, but a device that users have with them wherever they go— their own mobile phone?

User Experience is More Than “Walk-Up-And-Use”

Traditional usability testing involves task-based research in the lab where designs can be tested, iterated, and validated. This methodology is ideally suited to assess usability in a highly tactical manner where spe­cific design questions can be answered, and is critical to the user-centered design process as it ensures that the core features are usable. However, the specificity of usability testing is also a limitation because it mostly focuses on the “walk-up-and-use” user experience. Exposure to the product is limited, and participants generally don’t interact with products they actually own. While this method is very useful for design itera­tion, it lacks external validity and does not lead to an understanding of how the product is used in the real world. Ultimately, true product usability, usage, and use­fulness should be determined in a natural environment over time.

“Remember When You…” Methods Don’t Quite Capture It Either

The data collected to describe the real-world user experience are often obtained through surveys or focus groups. While these methods are beneficial in the early-stage assessment of feature importance, pricing, or intent to purchase, results often tend to be high-level and less helpful when specific design decisions need to be made. Even when these methods are focused on design questions, the collected data are largely retrospective. Asking users to reflect on a task completed in the past is not as effective as asking the same questions during or immediately following the completion of the task. So, we need research methods that can tap experiences as they happen.

Getting Out in the Field Can Be a Challenge

Ethnography, or observing behaviors in a natural environment, avoids retrospection and potential confabulation associated with “remember when you…,” leading to higher validity of the collected data. However, the study logistics can be complex, and fieldwork and data analysis time-consuming.

Longitudinal Research is a Major Endeavor

Traditionally, longitudinal research involves data collection over time. Instead of tapping the user experience just once, this method allows for repeated testing on the same set of users over multiple sessions. It can assess learning based on how the user adapts to and uses the product during critical periods of the product’s lifecycle.

Longitudinal research is also compelling because it provides information on how the user experience can change over time. Users can learn how to use the product more efficiently, can lose interest in the features that they initially found appealing, or find another use for the product. These findings give us insight into product usage and applicability to user needs. The methodology can also be influential to design as it fills in the “post-walk-up-and-use” gap that is left open follow­ing typical usability testing.

The longitudinal approach, however, is used infrequently, despite the advantages it offers. At CHI 2007 (ACM’s conference on Computer-Human Interaction), a new special interest group on longitu­dinal research was formed. Out of the fifty-plus participants, less than a quarter had conducted a longitudinal study in the past. At CHI 2008, the group discussed possible reasons for the dearth of this type of research. The main reasons revolved around complex logistics, time-consuming data analysis, and the per­ception that a longitudinal study is a major endeavor.

Experience Sampling Can Tap Motivation

Traditional usability testing captures the “walk-up-and-use” aspect of the user experience, ethnography helps us understand the user experience in the real world, and longitudinal research provides insight into learning and usage over time. However, we still lack infor­mation on the drivers for usage. Influencers that impact usage of a product include motivation, fun, and emotion.

This is where longitudinal research using experience sampling methods can help. Experience sampling refers to in situ research where a phenomenon is examined in the place and time it occurs. A common example is a “pager study” where the researcher “pages” participants at different times and prompts them to provide information via a diary at that very moment. These data, while time-consuming to extract and analyze, provide deep insight into user activity, motiva­tion, and other cognitive and social dimensions. They can identify patterns of use, context that drives usage, and motivators.

Leveraging Advances in Mobile Technology

We need to be able to capture experiences in the places and at the times they occur, but in a more efficient manner, while retaining the tactical and rigorous elements of in-lab research. Mobile technology has advanced to a point where researchers can do more than simply page users and ask them to make a diary entry. We can now expand upon longitudinal and experience sampling techniques to better solicit, monitor, and receive data on user interactions at given points in time. And the mobile device itself can be much more than a conduit between the user and researcher.

Setting Triggers to Capture Specific Experiences

An interesting aspect of this technology is that the user’s own actions can trigger data collection. User phone activity is automatically and remotely monitored. When an event of interest occurs (for example, the user accessed a particular place on the phone or physically
entered a building), a prompt requesting data on the experience is sent to the user. In an attempt to avoid interruptions, the system tries to predict when the experience is finished or when the user is “free.” Thus, the research program actively solicits feedback on the experi­ence at a time when the user is more likely to be available.

In the example shown in Figure 1, a person uses a navigation appli­cation on a phone to find directions. If the application is of interest to the research, the user’s actions are recorded and sent to the research lab. When the application is closed and the phone has been idle for a period of time, the user is asked questions regarding his/her experience.

 Improving Task-Based User Testing

Using this new technology, we can conduct large sample usability studies of phone functions and features (such as adding a new contact to the Address Book) with the user’s own device and in his/her natural environment. Mobile service providers have data on how often and when users make calls or send text messages, but their vision does not extend much fur­ther. They do not know if the user called John by using speed dial, by selecting his number from the Address Book, or by entering the number. They do not know if the user even entered John into his or her contacts.

In the example shown in Figure 2, a researcher sends a prompt to the user to complete a specific task on their music phone (for example, create a playlist). The user completes the task and then answers questions about the experience (ease of use, frequency of use, preferences, satisfaction, etc.). All data are sent back to the researcher, including data from the device itself to assess if the task was successfully completed and how it was completed (for example, screens and clicks).

Assuming that novices had difficulty with this task during in-lab testing, we could retest it on owners of the phone after a few months of ownership. This would help us determine whether or not they were able to learn the functionality over time.

User Feedback — More Than Just Text

Free-form user feedback is now a reality with QWERTY keyboards on mobile phones. Anyone who observes today’s youth texting incredi­bly quickly can envision the tremendous data collection benefit over handwritten diary input that requires transcription and review. The mobile device can be leveraged to open up novel forms of data col­lection never before possible without specialized equipment. For example, the phone can display questions and solicit feedback in the form of text, single or multiple selection, or slider manipulation. It could present images for the user to provide context for questions. Imagine the depth of experience that could be captured if the user could respond by speaking their responses, taking a picture, or recording a video of what they are doing. The capabilities of a mobile device as a research tool create a wealth of opportunities.

Capturing Non-Phone Experiences

We could also use the phone as a data collection method to capture non-phone experiences. Consider a pharmaceutical scenario to study drug treatment adherence and patient experience when using medication. The phone can prompt the user to take their medication and provide feedback on their current symptoms. In the case of a medical device such as a lancet or injector, the user could provide feedback on the interaction with the device and the level of pain they experienced when using it.

Another example could be the setup of a home theater product. It is not practical to test in the lab the myriad of home theater configura­tions that exist in homes today. Practically, lab testing can only test a few common configurations, but we know that many problems can occur when the user attempts to install a new entertainment component into the nest of wires and components that make up his or her home entertainment system. Using mobile technology, hundreds of home environments can be tested to assess setup difficulties. Figure 3 depicts a user installing a new HD receiver. The user can use his or her mobile device to report on the usefulness and usability of the instruc­tion manual. The user can be prompted to use the camera to take pictures of their home system configuration as well as document any problems encountered during setup. The ability to collect data on hun­dreds of users using this technique is compelling.

Next Steps

As practitioners, we should strive to constantly improve our user experience research techniques with the goal to uncover how users really use products. Several methods using mobile technology have been used in the past. These include solutions using SMS messaging to solicit and collect data or even questionnaires on old Palm Pilots. However, these methods involve manual work by the researcher (such as sending SMS prompts). The need for a more robust system where research parameters can be managed automatically is clear. The tech­nology described in this article can make research easier through device- and server-powered research solutions that can be configured to reside on today’s mobile phones.

We have developed a new mobile technology platform called LEOtrace® Mobile that would make all this possible. It is currently avail­able on devices running Windows Mobile 6 with a release schedule that includes Symbian, Linux/Java, and other operating systems in late 2008 and 2009. The platform is in beta testing with users now and it is suffi­ciently advanced to automatically trigger prompts to the user based on events such as the start and close of applications, when applications move to the foreground or background, when a call is initiated, received, and ended, when a text message or email is received and sent, or even when a specific website is visited or when the user walks into a particular GPS location (for example, a shopping mall).

The potential of remotely capturing user experiences as they occur is a more ecologically valid approach than many of the traditional tech­niques we have been using. The method easily scales to “large N” studies and makes research more efficient. When coupled with objective user behavior data collected from the device itself, the user’s subjective feedback can be analyzed to more effectively improve product design and must affect a positive change in the user experience.

Topics: Usability Testing, Mobile
Published in: September, 2008 in Remote Testing