The First Rule of Usability Testing: Test the Right Users

OK, so the real first rule of usability testing is, “do it.” But we can assume you already know that usability testing is important, and that you need to be doing it in order to make sure your apps and software are creating value for your users—and thus for you—as efficiently as possible.

Given that, the most important consideration when it comes to usability testing is making sure your testers can give you the information you need.

The Best Usability Results Come From Your Users

No one can provide you more information about your app’s user experience than your app’s actual users. Of course, usability testing means a whole lot more than simply surveying users, and it isn’t always feasible or advisable to ask your current customers and clients to fully test your app.

That’s why you need testers who are just like your users to give you real, usable information.

Ask a bunch of software developers how much they like your app aimed at accountants, and you’re going to get information on all the wrong things. Even the most carefully and accurately designed usability test won’t yield the results you’re looking for if you don’t have the right people taking the test.

The more specialized and narrowly-focused your target niche is, the more important it is to find representative usability testers—a banking app aimed at the average consumer can be adequately tested by a wider range of people than a professional-level graphic design program. There’s always a target audience, though, and making sure your usability test group is as close to the target as possible is essential.

Stay tuned for future articles explaining how to figure out exactly who your testers should be and how you can get them to help you out. Until then, we’ll do our best to keep you in the loop!

The Power of Words in UX Research

It’s said that a picture paints a thousand words. But it’s also worth considering that a single word can evoke a powerful image. Consider, for example, what comes to mind when you encounter the word ‘gambling.’ Alternatively, consider the word ‘gaming.’ Likely, different mental images are triggered by each word. Every day, during the course of verbal and written discourse, people have a choice of words to pick from. And it turns out that which words are used can have a significant impact on how the message is received, or the question is interpreted.

Consider, for example, the following pairs of words:

  • Liberal v. progressive
  • Liquor v. spirits
  • Used v. pre-owned

Although the words in each pair are similar to each other, I suspect that each word in the pair brings to mind a slightly different mental image, along with a slightly different emotional response, as well.

Research on the power of words

Researchers were interested in learning about the effect of wording, and how different ways of asking the same question might affect judgments. They devised a study where research participants were shown a video of a car accident and were then asked to estimate the speed of the car that had initiated the collision.

But individuals in each of two groups who had viewed the video were asked a slightly different question:

  • Group 1: At what speed did the first car contact the second car?
  • Group 2: At what speed did the first car smash into the second car?

The difference between these two versions, of course, is that the first example uses the word ‘contact,’ while the second example uses the words ‘smash into.’ Would this difference in wording affect people’s estimates of speed? It turns out that:

  • When the word ‘contact’ was used, people estimated the car to be going 31.8 mph, on average
  • When the words ‘smash into’ were used, people estimated the car to be going 40.5 mph, on average

When asked to recall what had been shown from the video, those who had encountered the words ‘smash into’ also claimed they had seen broken glass, even though in reality, no glass had been broken. The words themselves, then, had a powerful influence both on how people remembered the incident as well as how fast they judged the car to be traveling.

The power of words in UX

But why is all of this important for UX designers and researchers? Well, not surprisingly, the choice of words used when interacting with research participants and with business partners, matters. As we’ve seen, word(s) are powerful drivers of what happens in the mind of the receiver.

As UX researchers, we ask a lot of questions. And how we ask those questions – the actual words we use – can have a significant influence on how participants ‘hear’ and interpret the question and consequently, on how they make judgments and respond. So how do we manage such a situation effectively?

My recommendation is to think carefully about the various ways a question can be phrased, and pay attention to how responses might be influenced. You could also ‘usability test’ the wording prior to your actual study to glean how your questions are being received and interpreted by a potential participant. Using language that is more neutral can also work to your benefit, as stronger language tends to drive mental perceptions and images that are more vivid and ‘extreme.’

Another option is to take a step back and ask a broader question. For example, let’s say you’re trying to learn how research participants feel about a specific aspect of the thing being tested, and you’re particularly interested in any negative emotions they express. The question could be phrased such that any of a variety of terms could be used:

  • Is there anything about this [thing being tested] that makes you feel [aggravated / annoyed / confused / frustrated / irritated]?

Likely, each of these terms brings to mind a distinct mental image and type of emotion. So here, it might be better to take a step back and ask a more open ended question, such as: Tell me about how this [thing] makes you feel. The broader nature of this question leaves room for the participant to express positive and/or negative feelings, rather than being directed to think more narrowly about a specific type of feeling or emotion.

In summary

How we communicate is a vital aspect of life in general, but even moreso for those who facilitate and moderate research studies. Never underestimate the power of words. Although it’s true that a picture does paint a thousand words, a single word can also paint a very vivid picture in the mind of your research participant, thereby driving their behavior and responses, and consequently, your research outcomes.

This is a post by Colleen Roller.  Colleen is forever fascinated with the workings of the human mind, and with the art and science of designing for it. She has written extensively on this topic, authoring columns for UXmatters and UX Magazine. On the personal side, Colleen is a classically trained musician and enjoys performing on alto and soprano recorder. She also designs jewelry, and you can find her necklaces in a popular shop in West Concord, MA.

Usability test questions: does wording matter?

Interaction Designers have a tough job. Designing is hard work! It requires the ability to be simultaneously creative and analytical – not an easy thing. When I changed jobs from being an Interaction Designer to being a UX Researcher, I initially assumed I would escape the rigors of the design role. However, I quickly realized that to be a good researcher, I had to have good research design skills. To get good research outcomes, you need to design effective research studies.

I also realized that, just like in interaction design, there are multiple ways to design in order to achieve a given objective. Each design option has a set of tradeoffs – each has its own strengths and weaknesses. Indeed, I have come to the conclusion that there is no such thing as a perfect design. But even so, a critical requirement of an effective research design is that it addresses and manages potential bias that can leak into a study and potentially skew or pollute research findings.

It’s important for UX researchers to be sensitive to the many ways that bias can affect a study, and to the extent possible, control for it. A key aspect of research design is how we craft the questions asked of research participants. It turns out that how we ask a question has critical implications for how a participant responds, and consequently, on research outcomes.

To see just how important this is, let’s consider a research study where a number of individuals were asked to determine how a judge should rule in a child custody case. After reviewing a list of characteristics about each parent, some individuals were asked which parent should get sole custody, while others were asked which parent should not get sole custody. Would simply asking the question in either a positive or negative way affect people’s response? After all, both questions are just different ways of asking essentially the same thing.

What they found is that the decision to either award or deny custody was indeed influenced by the way the question was asked. When people were asked which parent should get sole custody, they quite naturally focused on and compared the positive characteristics of each parent. When they were asked which parent should not get sole custody, they automatically compared the negative characteristics of each parent.

We can see that the wording of the question actually drove people’s thought process and what they focused on in the decision problem. This is huge! It means that the actual wording of the question has the power to drive attention and focus, and therefore, decision outcomes.

Sometimes as UX researchers, we get into the groove of only asking certain types of questions –
• What did you like about…?
• What would make this better?
• Describe your optimal experience….

Note that all of these questions are framed in a positive way. But what would happen if we asked the question differently?
• What don’t you like about…?
• What would make you stop using….?
• What would happen if the [thing you’re testing] didn’t have this feature/functionality….?

Sometimes the negative version of a question opens the opportunity to get insights from participants that you wouldn’t get otherwise. And asking both the positive and negative version of the question can result in a more holistic understanding of the user experience.


Instead of asking… Ask a broader question…
1. What are the differences between….? 1. How would you compare….?
2. How often do you use….? 2. Tell me about how you use….?
3. How would you rank these….? 3. Which of these is a better approach….?


Another way to glean insights is to consider how broadly or narrowly you frame the question. Often, when you take a step back and ask a ‘broader’ question, you open the door to obtaining answers to questions you hadn’t thought to ask. Asking a broader question also helps control for assumptions that can affect and bias your interpretation of what you observe.

Let’s take a closer look at the questions above. Question #1 makes the assumption that the participant actually notices the differences. But what if s/he doesn’t? Now the participant feels uncomfortable and needs to ‘save face.’ Always remember that the participant will supply an answer to your question. That’s what s/he’s getting compensated for, after all. But, the answer you get may not reflect the reality of the situation.

The narrow wording of question #2 leads the participant to focus only on frequency of use, whereas the revised version allows the participant to answer in a way that allows you to see what’s most important to him. The wording of question #3 leads participants into a mindset of ‘ranking,’ which is a very different task and thought process than answering a broader question about which of the options is a better approach.

It’s worth taking the time to think through various ways of wording your research questions to appreciate how the wording itself may affect participants’ thought process, and consequently how they respond. Truly, research design is both an art and a science – and that’s what makes it so fascinating.

This is a post by Colleen Roller.  Colleen is forever fascinated with the workings of the human mind, and with the art and science of designing for it. She has written extensively on this topic, authoring columns for UXmatters and UX Magazine. On the personal side, Colleen is a classically trained musician and enjoys performing on alto and soprano recorder. She also designs jewelry, and you can find her necklaces in a popular shop in West Concord, MA.

« Previous PageNext Page »