Keeping a diary is not just a hobby undertaken by teenage girls trying to make sense of life and love. Diary methods have been widely used in medical research where patients are asked to keep a diary during a clinical trial or disease treatment. In Human-Computer Interaction, a diary study is a qualitative technique for collecting data on what users have done or experienced. Much like a travel journal contains descriptions of the traveler’s experiences, a UX diary contains descriptions of the user’s experiences with a system.
What kind of UX research questions could diaries help answer? Depending on the design of your diary study, you may get information on the overall impression of a specific device, usage of features, technological acceptance, emotions associated with task performance, or learnability of an application. Of course you could also get this information with a survey, an interview, or a user test. But only diary methods will give you access to temporal and longitudinal information gathered in a natural context of the interaction.
Advantages of Diary Methods
Studying Temporal Dynamics
As stated in “The User Experience White Paper,” there are several time spans of user experience depending on the moment of usage (see Figure 1). Typical methods used to study UX shed light onto only one kind or moment of the UX. Surveys are often related to Episodic UX: users are asked to assess the interaction after usage. User testing helps collect data on Momentary UX. You may also have an idea of Anticipated UX by conducting focus groups.
Figure 1. Time spans of user experience. (Source: “The User Experience White Paper,” Dagstuhl Seminar on Demarcating User Experience, 2010)
The main advantage of a diary study is that it allows collecting longitudinal information. Let’s imagine you would like to assess a new mobile application. The first part of the diary study may be administered prior to usage by asking users how they imagine the application, what they would expect from it, and if they have any prior experience with similar applications. Then, you might ask them to report their everyday impressions and feelings during usage. Finally, during a debriefing interview or a post-study questionnaire, you may ask the users to reflect on their experience after usage.
Reporting Events and Experiences in Context
Another issue we should be always mindful of is the impact of context on the results. For example, when conducting a user test or an interview, the presence of the researcher may bias the results, leading to a positive evaluation of a system despite poor user performance. This is called a social desirability bias. When we know we are being watched, we will tend to behave in a way we believe is socially acceptable or desirable.
As researchers, we should want to capture life as it is and not how it looks in a controlled setting. The emergence of mobile technology especially highlights this need to study use “in the field” (see Figure 2). That’s when the diary methods come in handy, as they focus on reporting events and experiences in their natural context.
Figure 2. On the left: A participant interacting with a mobile application in a usability lab. On the right: A participant using the same application, but in a natural environment. Do you really think that the user experience is the same in both situations?
My company developed a collaborative environment to address the needs of architects and engineers in the construction sector. The mobile application was tested in a usability lab, and flaws were identified and fixed. But once used on a construction site, there were more problems than expected. Effects of dust, noise, brightness, and handling were underestimated during both the design and evaluation steps. Moreover, collaborative issues in this real on-the-go setting were quite different than those simulated. The subsequent use of a diary study helped improve the system by revealing real user issues and needs in the context of their workplace.
Determining the Antecedents, Correlates, and Consequences of Daily Experiences
As mentioned before, UX is rooted in different time spans, each one being influenced by another. Momentary UX is colored by Anticipated UX in the way that user’s expectations have to be met or exceeded to lead to a positive experience. Then, Momentary UX will be distorted by the filter of cognitive processes when changing into Episodic UX and Cumulative UX. Expectations, mindsets, moods, and social or physical contexts all color user experiences. A diary study captures these influences, shedding light onto how the UX in each time span has formed.
For example, findings from diary studies can reveal that it was the bad mood of the user or brand criticism heard on TV that impacted the user’s assessment of the interface. You may also discover that the game you are evaluating provides enjoyment only when the user interacts with close friends or family, even if the functionality supporting this was neglected by designers. Finally, you may notice that a single flaw you considered as minor discourages the user and leads to system abandonment.
Limitations of Diary Studies
Nothing is perfect! As with most methods, you need to look at the pros and cons of designing a diary study. The main disadvantages of the method are the cost and time associated with:
- Participant recruitment. The quality of the results depends on the participants since the diary is bound to the expressive ability of the writer. This is especially true for a study with a lot of open-ended questions. You also have to achieve a high level of participant commitment to obtain sufficient and reliable diary entries.
- Training or briefing sessions. A diary study often requires detailed training sessions to ensure that participants fully understand what needs to be reported, how, and when.
- Data analysis. Analyzing diary entries is time consuming. This activity takes even more time in the case of a pen-and-paper diary.
Designing a Diary Study
Eliciting Diary Entries
There are three categories of diary protocols based on the way in which entries are elicited:
- Interval-contingent protocol, in which participants have to report their experience at regular predetermined intervals (for example, every two hours or every day).
- Signal-contingent protocol, which uses a signaling device to prompt the participants to make an entry.
- Event-contingent protocol, which requires participants to report each time a specific event occurs.
Protocol selection depends on the purpose of the study and activities under investigation. The interval-contingent option, although widely used, does not help participants remember to make diary entries, which may lead to missing data. Signal-contingent protocols overcome this drawback but are intrusive. An event-contingent protocol might be used especially when you are interested in studying the experience created by a specific event, such as a system error or notification. Rate and timing of self-report have to be set up according to research needs. Do not be too demanding or your diary will become a burden to your participants; a maximum of two to three entries per day should be enough.
Involving Users
One of the major challenges in diary studies is maintaining participant commitment at a high level to ensure diary completion. You have to make sure that participants understand the scope and descriptive depth needed for the diary entries. During the study, you can remind participants of the importance of diary completion. A cash or gift incentive, or the possibility of keeping the assessed device, can be very helpful. However, be careful not to base the incentives on data quantity or you may get tangled up in irrelevant information.
Pen and Paper vs. Electronic Diaries
Because UX studies are often related to the use of technological devices, the choice of an electronic report is often judicious. It simplifies data analysis and gives the opportunity to combine diary entries and log files (for example, date, time, and task performed). E-diaries also provide participants with innovative forms of reporting. Imagine a diary study through voicemail entries, text messages, video, or pictures!
However, you should always try to adjust the method to the target users. If you are conducting a study with elderly people, a pen and paper diary will probably be more appropriate. Your participants should feel comfortable with the process of keeping a diary.
Diary Structure
Diaries may be open format (users record activities and events in their own words) or highly structured (where closed-ended questions are pre-categorized). Do you wish to encourage general reflection or need precise information? A mixed approach is often adopted to collect both qualitative and quantitative data. Do not forget to add a clear set of instructions on how to complete the diary, and stress the importance of recording events as soon as possible after they occur. You may also use non-verbal tools to facilitate completion (see Figure 4), especially for specific target users such as children or people with a disability.
Figure 4. The Self-Assessment Manikin is a non-verbal scale to measure emotions.
Collecting and Analyzing Data
After designing your diary, do not forget to pilot-test it before the study launches. Participants should keep the diary for a maximum of two weeks, as people’s commitment quickly decreases over time. If the diary is electronic, you should monitor its completion for each participant. Do not hesitate to contact participants during the study to ensure that everything is going well.
At the end of the study, conduct follow-up interviews with your participants. That’s your opportunity to have them explain some of their entries. A good understanding of participants’ qualitative answers is required to correctly interpret the data.
The analysis of diary entries depends on how structured the diary is. Quantitative data may be analyzed using any statistical analysis software. Dealing with qualitative data is more challenging but certainly worth the effort because these will provide you with valuable information on subjective experience.
Example of a Before-Usage Diary Study
A diary study of rendezvousing (described in the Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work) examined how people meet up, with the goal of understanding how technology could support them.
Martin Colbert used a diary method because the events of interest were “too rare, private, and geographically dispersed for direct observation, and users too easily forget important details of their behavior to report them accurately long after the event.” Participants had to report their rendezvousing behavior for fourteen days. One diary entry was made for each rendezvous event. Each entry was composed of an open-ended, narrative description in the participant’s own words of what happened and why, and the participant’s responses to a rendezvous questionnaire, which asked for specific details of each event (see Figure 3).
Figure 3. A diary entry participants were supposed to complete following each rendezvous.
The researchers used the results to extract the overall structure of rendezvousing and to build typical rendezvousing scenarios. Eight possible causes of problematic rendezvous were identified, including transportation problems, over-running of previous activities, poor planning, lack of travel information, lack of geographical information, lack of information about other rendezvousers, spontaneous additional tasks, and failure to value success.
The study helped identify what kind of position-aware service was likely to be most useful and how it could be designed to shape rendezvous experiences.
Quantifying UX Over Time
I have used the abridged AttrakDiff tool (www.attrakdiff.de) to evaluate the hedonic and pragmatic quality of a mobile application (see Figure 5). By asking participants to report their feelings and impressions using a short survey, I was able to draw a UX chart representing the evolution of UX over time (see Figure 6).
Figure 5. AttrakDiff survey on a mobile phone.
Figure 6. AttrakDiff scores illustrating how the UX of a mobile application changed over a period of ten days.
The Snippet Technique
The snippet technique developed by the Stanford HCI Group aims at overcoming the limitations of diary studies performed under mobile or active conditions, where there is no time to make thorough entries. Participants capture text, MMS, or voice snippets when on the go and later annotate the snippets on the web (see Figure 7).
Figure 7. Snippets captured during the day serve as memory aides for full diary entries in the evening.[
Retrieved from https://oldmagazine.uxpa.org/dear-diary-using-diaries-to-study-user-experience/
