Figure 1: Still from the movie Star Trek (2009).
Observation techniques are useful for field observation, usability studies, contextual inquiry, and everyday survival in an office environment. The ability to observe what people actually do in context is an incredibly powerful tool for any UX professional. Keen observation enables sharp insight and can result in more appropriate actions. But most people are not great observers. To be really good at making useful observations, you must practice training your attention. Highly effective observers learn to identify and focus on relevant aspects and dismiss the unimportant without bias.
Sci-fi movies are a great way to practice observation skills. They offer a rich version of an environment foreign to us but familiar enough to be able to observe it effectively. They also offer the opportunity, through varied camera angles, to observe action from different perspectives. It is very difficult to learn to observe what is familiar to us, and the fresh technology of science fiction films, no matter how improbable, is a good way to deepen observation skills without bias.
So let’s practice, and let’s make it fun. We’ll use one of my favorite recent sci-fi movies, a re-envisioning of a classic, Star Trek (2009). This movie offers many examples of interactions between humans and a variety of systems (touch screen, manual input, gesture-based, voice-recognition, and more), but for the purpose of this article, we’ll focus on the first scene that takes place on the bridge of the Enterprise. This brief scene involves an initial look at the assembled crew and the activities leading up to the launch and maiden voyage of the brand new starship.
A Brief Description of the Scene
The scene takes place between 40:15 and 42:38 minutes into the film. We’ll start examining the scene with the interaction between Captain Pike and the helmsman, Sulu. Sulu is at his station and is ordered by the Captain to prepare for launch. He interacts with his console and verbally reports readiness to the captain. When given the command to go to “maximum warp,” Sulu initiates the action (pushing a lever forward) but the ship fails to jump to warp speed. In fact, nothing happens. As Sulu struggles to understand the cause of the problem, the first officer, Spock, questions whether he has “disengaged the external initial dampener.” Sulu follows the advice of Spock and the ship is able to launch at maximum warp.
Shortly after takeoff, the Captain asks Ensign Chekov, the ship’s navigator, to “begin ship-wide mission broadcast.” In order to access the ship’s intercom system, Chekov needs to verbally enter an access code. This code, nine-five-victor-victor-two, poses a problem for the heavily Russian-accented ensign. His pronunciation of “victor” as “wictor” causes an error and the voice recognition system won’t allow him to access the intercom. He attempts access again, visibly struggling to pronounce “victor,” and successfully activates the intercom and makes his announcement.
This 2.5 minute scene is a great example because it easily demonstrates six key tips for making useful observations.
Tip 1: Look for Pain Points
Whether they are caused by a system or human error, are manifested as moments of confusion, times of low productivity, or any kind of difficulty, pain points represent obvious moments for more in-depth investigation. In the bridge scene there are two major pain points between the crew and the ship’s systems.
The first pain point is experienced by Sulu, and indicates a problem with system feedback. Why was Sulu able to check readiness of other components but not warned when a function that would prevent launch was still engaged? Once an error occurred, why was there no rapid diagnosis tool available? When systems depend on specialized knowledge of users to fill in gaps in function, there is potential for major usability issues.
The second major pain point in the scene with Chekov and the voice recognition system indicates problems with what I can only loosely term “internationalization.” On an intergalactic ship with crew from all over the universe, it might be reasonable to expect that a voice-commanded system could handle inflections, accents, or other vocal variations that would result from communications by non-native English speaking crew members of different species and cultures.
It is important to note that pain points don’t always indicate system problems. They can also be an indicator of insufficient knowledge or training, process problems, or human error. Regardless of what they indicate, they are important details for closer examination.
Tip 2: Articulate the Unspoken
Non-verbal communication, actions/reactions, and human-human interactions are key nuances that can help you translate what is going on in a situation. The “unspoken” can add context so that it is easier to postulate causation theories and to articulate questions that will help with making insights.
One major non-verbal interaction in this scene occurs when Sulu’s warp attempt fails and we see the other crew members turn and stare at him. As an outside observer, I would hypothesize that either errors are rare and cause for curiosity or that making an error is heinous and warrants shameful staring. Of course in this instance I can’t dig deeper into the stares that Sulu received, but they indicate something going on that I don’t fully understand and that I should pursue.
Tip 3: Attach Numbers to Things
Quantifying aspects of the experience will be important to evaluate the severity and prevalence of a problem or to determine importance and urgency of a task. (How frequently? How many times in a row?) For example:
- Chekov must verbally state the access code twice before he can successfully access the ship’s intercom system
- Four questions are asked of Sulu between the time that the warp failure occurs and when he solves the problem (three from the captain, one from the first officer.)
- Seven tasks are verbally articulated by Sulu after the captain’s first command and before the warp failure: moorings retracted, dock control report ready, thrusters fired, separating from space dock, fleet’s cleared space dock, all ships ready for warp, course laid in.
During a live observation session it is sometimes difficult to count everything, especially frequent things. Tick marks in the margin of your notes will help you keep track of major patterns (ratios such as “attempts-to-success” or “failures-to-completion of task”), and major patterns are most likely where you will find a lot of good insights.
Tip 4: Observe What Does Not Happen
Sometimes important insights can come from indicating when expected or common behaviors are not happening. Noting times when people are not talking, or when users are not interacting with a system, are good starting points. For example, notice that no one in the scene complains about the technology, even though it sometimes seems to cause inefficiency and delay. There are no rolled eyes or knowing looks exchanged. These observations don’t tell us why something is happening. Only further questioning could explain whether this indicates that errors don’t happen often or whether such expressions/reactions are frowned upon. But the lack of behavior does give us a starting point for further probing. I admire Sulu for not lamenting, “Sir, there was no warning message to tell me the dampeners were still engaged!”
Tip 5: Be Descriptive Rather Than Interpretive
You’ll never get from observation to insight if your notes aren’t objective and based on fact. It can be difficult to keep your mind in descriptive mode when observations trigger ideas, memories, or connections. Practicing “sticking to the facts” is a key to success.
For example, a descriptive report of Sulu’s workstation would be: “The console consisted of a screen, a touch interface, and a lever. The pilot sat at the console to operate the ship, his screen viewable to the captain who sat above and behind him.”
An interpretive description could be: “The console was an interesting mix of advanced and archaic instruments: a touch screen and an old-fashioned lever.”
There are many ways to determine when you’re being interpretive, but I find the most common offences can be summed up in three points:
- Your writing is literary. It sounds lyrical, it uses metaphor, or it has too many details, like describing a specific shade of blue.
- Your notes make a lot of references. Observations contain self-references, comparisons, or abstract connections.
- Your writing is explanatory. Your notes seem more instructive than empirical.
A great way to test that your observations are more descriptive than interpretive is to see if what you’ve written can be visualized by someone else. Testing your notes by asking a colleague to make a sketch based on your description is good practice.
If you do want to include interpretive aspects during your observation, indicate when you are being descriptive and when you are being interpretive. I always leave space in my notes for annotations and editorializing. I also have my own shorthand to indicate when I’m being interpretive.
Tip 6: Question Assumptions
As with any research study, assumptions should always be identified and questioned. There are two types of assumptions you need to be aware of: your assumptions and the assumptions of the people and systems you observe.
I assumed the Captain knew everyone when he entered the bridge. The Captain assumed that everyone on the bridge was supposed to be there. But when the Captain engages Chekov he has to ask his name. And when Sulu fails in his warp attempt the Captain asks, “And you are a pilot, right?” (Although the tone of this is rather tongue-in-cheek.) These assumptions could have led to human error (what if Sulu wasn’t qualified to be sitting at that console?), but their relevancy to the task failures needs more evidentiary support.
When Sulu failed to put the ship into warp, the Captain assumed human error. He asks, “Is the parking brake on?” This response to failure is something I keenly identified in my notes (by “keenly” I mean I circled it in red).
Other assumptions that I could test by questioning:
- I assume rank is the same as the military ranks I understand
- I assume the physical setup is ergonomic
- I assume that gestures (for the touch interfaces) are universal
You want to identify and test as many assumptions as possible in order to strengthen the validity of your observations. Unchecked assumptions weaken valuable insights.
Turning 2.5 Minutes of Observation into One Insight
Using these six tips to make observations has led me to an important insight that might help me redesign the Enterprise’s interfaces: errors occur rarely and the system handles them inadequately. In some cases the system does not prevent error, and when error does occur, the system does not allow for graceful recovery. This insight is based on the observations of the assumptions that Sulu’s human error led to the failure to enter warp drive, the lack of complaining, how the crew stared at their failing teammate, and the dogged persistence of those failing in their tasks. Of course, when possible, I would first test this insight as a hypothesis, confirming it through heuristic evaluations, usability tests, or some other supplementary methodology.
But insight isn’t the only foregone conclusion for an observation activity. Sometimes all you need to get out of an observation session is ideas as to where to investigate next. Whether in a usability lab test, in the field, or in the office, great observations can lead to the creation of great interactions. How would you redesign Enterprise’s system to make it more usable?
Retrieved from https://oldmagazine.uxpa.org/to-boldly-go/
Comments are closed.
