The work of a usability practitioner crosses many disciplines: from ergonomics to graphic design, from visual perception to web standards. The situation is no different when designing forms. This is why a background in survey methodology and social research can facilitate a transition into user-centered design.

The field of user interface design is relatively unaware of the well-established question design principles used in the fields of survey methodology and social research. Perhaps this is because usability has arisen largely from the study of ergonomics, human factors, and industrial design, which have considerable focus on the more physical and aesthetic attributes of design. Writing a good question, however, draws on the principles of linguistics, sociology, and psychology—areas that form the foundation of survey methodology and social research.
The problem is, no matter how well laid out the form is, or how streamlined the form filling and processing procedures are, the usability of a form will be significantly compromised if the questions on the form are poorly constructed. This article aims to address this potential gap by providing some models for good question design.

Why Worry About Questions And Answers?

When I tell people that I design forms and have a particular passion for writing good questions, they often respond with incredulity. “Surely writing questions isn’t that hard?” I can understand their perspective; we ask and answer questions hundreds of times in the course of a normal day. “What was your day like?” “Do you want salt and pepper on your sandwich?” “Can you get this finished by the end of the week?” “What time is the party on Saturday?”

We are comfortable with conversational questioning because the grammatical structure of this exchange is well understood by fluent speakers of the language being used. But asking an effective question relies on more than just an understanding of grammar. Consider the following example: “How long have you been using a walking stick?”

Suppose someone first started using a walking stick after she hurt her hip in 2000. She had surgery in 2002 and the surgery gave her two years of relief, during which time she did not use the stick. Unfortunately, the pain returned in 2006, so she started using the walking stick again.

If this question is being asked at the start of 2009, what answer would the person provide? An answer of “eight years” would be correct since it is the elapsed time since she first started using a walking stick. But an answer of “four years”—that is, of actual use—would also be reasonable answer.

If the question were being asked in conversation there would be many cues that would guide the person’s answer. For example, the person may know the individual asking the question and have a sense of why they are asking. Moreover, there’s a good chance that the person would ask for clarification before responding, or include explanations in her answer.

When someone is filling out a form, be it paper or electronic, they don’t have nearly as many cues and conversational techniques to draw on. In this situation, if the respondent doesn’t give up altogether in frustration, they will usually try to guess the designer’s meaning and answer accordingly. The respondent’s intentions are good, but unfortunately their actions—which are a direct response to the question—can result in inconsistency in the data. To make matters worse, looking at the responses for this form won’t raise any alarms, since they will all be valid.
An apparently straightforward question can result in inconsistent data, but that inconsistency will be essentially impossible to detect. This could have serious consequences if you were asking a more complex question about health or taxation.

A Model for Answering Questions

In order to write good questions, you need to know the different ways that a question can fail. In the example above, the problem is one of understanding the question (what does “how long” mean?) and judging the appropriate answer (do they want me to report elapsed or actual time?).

Comprehension and judgment are just two cognitive processes involved in answering a question. There are a number of models that survey methodologists and cognitive psychologists have proposed to describe all the different stages the respondent might go through. In their book “Forms That Work,” Jarrett and Gaffney describe a four stage model, which is, in turn, based on earlier work by Tourangeau, Rips, and Rasinski:

1. Understand the question
2. Find an answer
3. Judge whether the answer fits the question
4. Put the answer on the form

The form designer needs to be conscious of the possibility of error at each step. To see this in action, try answering the following question for yourself:“What did you have for lunch last Tuesday?”

The areas of difficulty that you might encounter include:

1. Understanding the question

Does “last Tuesday” mean the most recent one before today, or the one before that?

2. Finding an answer

Research suggests that to find an answer to this question, most people will: Try and remember what they were doing last Tuesday and use that as a cue for what they would have been eating; or Use a rule like, “I try to take leftovers from the previous night’s dinner to work for lunch each day, so what did I have for dinner the night before?” or “Tuesday is pay day so I always go out and have pasta.” Bad luck if last Tuesday was an exception to the rule!

3. Judging the answer

If last Tuesday was a special celebration at work and you ate something unusual, perhaps you considered whether you should say what you actually ate or what you would normally eat on a Tuesday. And just how sure are you that it was Tuesday, and not Monday or Wednesday, that you had that sushi?

4. Putting the answer on the form

Perhaps you’re all ready to write “rice noodles” when you find that you have to choose from a series of pre-determined response options. There’s a “pasta” option but no “noodles.” What do you do now?

Writing Good Questions

We can use Jarrett and Gaffney’s model to prepare a kind of checklist of things that lead to a well-designed question.

1. Understand the question

• Use the vocabulary that is most suited to the target audience. For example, use “heart” for the general public but “cardiac” for medical professionals.
• Be conscious that even familiar terms may be ambiguous. For example, a 1992 study by Floyd Fowler Jr. found that there was confusion about the following question: “What is the average number of days each week that you have butter?” The confusion came from the word “butter” and whether or not margarine or other potential butter substitutes should be included. Respondents may have also wondered whether “having” butter included both as a condiment (such as on toast) and in cooking.
• Avoid jargon, especially names for products and services that are not well known outside the organization (for example, use “monthly subscription” rather than “SmartBuy package”). While your organization may be trying to promote these brands in the marketplace, forms are not advertising—they are tools for getting something done. As such, the need for the respondent to comprehend the question outweighs the desire to communicate internal terminology. If such terms must be used, provide them as a supplementary part to the question (for example, “monthly subscription (SmartBuy package).”
• Avoid introducing bias through unbalanced and emotively worded questions. For example, “How satisfied or dissatisfied are you with our service,” is preferable to “How satisfied are you with our service?” or “How great was our service to you?”
• Explore only one concept per question. This reduces the burden on short-term memory and eliminates the confusion that arises if the answer to one part of the question is different from the answer to the other part of the question. For example, how does a weekday-only user answer the question, “Do you use the Internet on weekends and weekdays?”

2. Find an answer

There is an enormous amount of information and research on the workings of the human memory, too much to be summarized in a paragraph or two. However, there are key aspects that every question designer should be aware of. Be conscious that:

• A respondent’s ability to find an answer to your question will depend, among other things, on whether it resides in short- or long-term memory.
• There are a number of techniques that can be used to improve memory recall, including keeping the reference period relatively narrow (that is, within six months), anchoring questions to landmarks (such as the day President Obama was inaugurated), and using cues. This last technique draws on the fact that our memories are stored in a connected fashion. Providing a prompt that is associated with the memory trying to be retrieved increases the chances that that memory will be found via the connection. For example, rather than asking someone in North America, “Have you ever been overseas?,” we could ask them whether or not they have visited each continent (Africa, Europe, and so on).
• People will expend the least possible effort in order to get what they need and want. This mechanism, called satisficing, has a particular implication for the design of questions. If the effort involved in answering your question accurately seems, to the respondent, to be disproportionate to the reward they will receive, they will often use a “rule” to give an answer.
  Thus, a market research survey asking about the average amount you spend on your mobile phone each month will probably yield a rule-based answer. For example, “I’m spending more on my current plan than before, but $100 per month sounds too high, so I might say $80 per month.” If, however, the same question is being asked by someone who is trying to find the best mobile phone plan for you, you are likely to be willing to expend more effort in finding an answer and thus do something like consult old phone bills or try to remember the actual amount paid.
• Even if the respondent searches their memory for the answer, that memory may be inaccurate. This is because the more a memory is recalled, the more it changes, is embellished, mixed with imagination, or gets confused with other memories. Form designers need to be skeptical about the ability to collect accurate information about events that happened long ago and/or are seen by the respondent as central to their sense of self or personal narrative.

3. Judge whether the answer fits the question

• Respondents often feel the pressures of a social exchange when filling out a form. For example, they may feel obliged to give an answer to a question, even when the question doesn’t really apply. Similarly, they may refuse to answer questions on sensitive topics if they are placed too early in the form before a relationship has been established.
• Where ambiguity cannot be eliminated from the question, especially because of cultural or linguistic complexities, provide supplementary information to help the respondent judge the suitability of their answer. These tips may describe what edge cases are included or excluded (for example, does the term “work” include activities that are unpaid, voluntary, or for charity?), give characteristic examples (for example, “‘motor vehicles’ includes cars, trucks, and motorcycles”), or define the meaning of terms (for example, “a ‘statutory declaration’ is a written statement declared to be true in the presence of an authorized witness, who is called a ‘signatory’.”)
• When defining the boundaries of a question, be as specific as possible, avoiding terms like “usually,” “regularly,” and “normally.” For example, rather than, “How many eggs do you consume on a typical day?,” a better question would be, “Please think about the last week and all the occasions that you ate eggs. On the days that you ate eggs, on average how many eggs did you eat?”

4. Put the answer on the form

• Ensure the answer field is formatted in a way that closely matches the mental models of the respondent. For example, if your respondents are paid on a monthly basis, ask income questions on a monthly rather than weekly or annual timeframe. Translation from one format to another is an opportunity for error and increases the amount of effort required from the respondent.
  Ensure there is a response option for every respondent, even if it is only “other,” “none of the above,” “don’t know,” or “not applicable.”

Rewriting Problem Questions

Two questions were used in this article to illustrate how difficult it can be to write a good question (“How long have you been using a walking stick?”) and the stages involved in the question answering process (“What did you have for lunch last Tuesday?”).
Applying the principles described in this article, some possible alternatives for asking these questions may be as follows:
For the walking stick question:

• “When did you first use a walking stick?”
• “Do you use a walking stick for only some activities? If so, what activities are those?”
• “In the last four weeks, how often have you used a walking stick?”

The lunch question is probably never going to yield a truly accurate response because it refers to a forgettable and unimportant event. However, if we did need to ask the question, we might choose to ask:

• “What did you eat for lunch on Tuesday 16 December 2008?”
• “Today is Thursday. What did you do for lunch on Tuesday, that is, two days ago?”
• “In the last month have you had the same thing for lunch each Tuesday, or did it vary?”

Which version is the best alternative depends heavily on exactly what data are needed and why.

It should also be apparent that some questions need to be made longer in order to be made more clear. This is one of the reasons that using fewer words, fewer questions, and fewer pages is not always appropriate in form design. The goal should always be greater clarity, whatever that requires.

Topics: Design, User Research, Surveys
Published in: June, 2009 in Forms

Comments are closed.