Usability test questions: does wording matter?

Interaction Designers have a tough job. Designing is hard work! It requires the ability to be simultaneously creative and analytical – not an easy thing. When I changed jobs from being an Interaction Designer to being a UX Researcher, I initially assumed I would escape the rigors of the design role. However, I quickly realized that to be a good researcher, I had to have good research design skills. To get good research outcomes, you need to design effective research studies.

I also realized that, just like in interaction design, there are multiple ways to design in order to achieve a given objective. Each design option has a set of tradeoffs – each has its own strengths and weaknesses. Indeed, I have come to the conclusion that there is no such thing as a perfect design. But even so, a critical requirement of an effective research design is that it addresses and manages potential bias that can leak into a study and potentially skew or pollute research findings.

It’s important for UX researchers to be sensitive to the many ways that bias can affect a study, and to the extent possible, control for it. A key aspect of research design is how we craft the questions asked of research participants. It turns out that how we ask a question has critical implications for how a participant responds, and consequently, on research outcomes.

To see just how important this is, let’s consider a research study where a number of individuals were asked to determine how a judge should rule in a child custody case. After reviewing a list of characteristics about each parent, some individuals were asked which parent should get sole custody, while others were asked which parent should not get sole custody. Would simply asking the question in either a positive or negative way affect people’s response? After all, both questions are just different ways of asking essentially the same thing.

What they found is that the decision to either award or deny custody was indeed influenced by the way the question was asked. When people were asked which parent should get sole custody, they quite naturally focused on and compared the positive characteristics of each parent. When they were asked which parent should not get sole custody, they automatically compared the negative characteristics of each parent.

We can see that the wording of the question actually drove people’s thought process and what they focused on in the decision problem. This is huge! It means that the actual wording of the question has the power to drive attention and focus, and therefore, decision outcomes.

Sometimes as UX researchers, we get into the groove of only asking certain types of questions –
• What did you like about…?
• What would make this better?
• Describe your optimal experience….

Note that all of these questions are framed in a positive way. But what would happen if we asked the question differently?
• What don’t you like about…?
• What would make you stop using….?
• What would happen if the [thing you’re testing] didn’t have this feature/functionality….?

Sometimes the negative version of a question opens the opportunity to get insights from participants that you wouldn’t get otherwise. And asking both the positive and negative version of the question can result in a more holistic understanding of the user experience.


Instead of asking… Ask a broader question…
1. What are the differences between….? 1. How would you compare….?
2. How often do you use….? 2. Tell me about how you use….?
3. How would you rank these….? 3. Which of these is a better approach….?


Another way to glean insights is to consider how broadly or narrowly you frame the question. Often, when you take a step back and ask a ‘broader’ question, you open the door to obtaining answers to questions you hadn’t thought to ask. Asking a broader question also helps control for assumptions that can affect and bias your interpretation of what you observe.

Let’s take a closer look at the questions above. Question #1 makes the assumption that the participant actually notices the differences. But what if s/he doesn’t? Now the participant feels uncomfortable and needs to ‘save face.’ Always remember that the participant will supply an answer to your question. That’s what s/he’s getting compensated for, after all. But, the answer you get may not reflect the reality of the situation.

The narrow wording of question #2 leads the participant to focus only on frequency of use, whereas the revised version allows the participant to answer in a way that allows you to see what’s most important to him. The wording of question #3 leads participants into a mindset of ‘ranking,’ which is a very different task and thought process than answering a broader question about which of the options is a better approach.

It’s worth taking the time to think through various ways of wording your research questions to appreciate how the wording itself may affect participants’ thought process, and consequently how they respond. Truly, research design is both an art and a science – and that’s what makes it so fascinating.

This is a post by Colleen Roller.  Colleen is forever fascinated with the workings of the human mind, and with the art and science of designing for it. She has written extensively on this topic, authoring columns for UXmatters and UX Magazine. On the personal side, Colleen is a classically trained musician and enjoys performing on alto and soprano recorder. She also designs jewelry, and you can find her necklaces in a popular shop in West Concord, MA.

6 common challenges in managing UX projects and how to overcome them

As of today, Ive been involved in Web and UX design for 12 years. The last three of them have been shared with my partners at Continuum, leading UX projects for a wide range of clients. Its impossible not to notice certain patterns on how user experience projects wind up, regardless of the scope, the team size or even the deadline and the budget.

These patterns, and the continuous iteration on how to better deal with them, have led us to identify some principles that pave the way to a better experience design. Well talk about them in a future post. But first, lets talk a bit about the most common problems we have faced when doing UX:

Feature creep

This is a well-known problem. Too many stakeholders with similar power positions trying to inject their own vision, insecurity and fear of being the poorest competitor and a lack of prioritization lead to products stacked with functionality that has not met real user needs.

The main assumption behind feature creep is that if its in the product, users will find it. If this were true, certainly we could make more useful products by simply adding more stuff (thats the way Microsoft Office was conceived). A better understanding of our userslimitations and lack of interest in our product makes the team feel the painful need of simplifying and prioritizing.

Google Docs understood this, and they basically offer fewer features but with the big differentiator of sharing and real-time editing. By replacing barely-used functionality with a game-changer offering (plus a free price tag), Google Docs has become the only real threat to Office in two decades.

Deadline creep

Products launching way past their original deadlines (and sometimes even becoming instantly obsolete when launched) is also a documented issue. Besides from feature-creep, the second most common cause we have observed is a constant back-and-forth between the team, where decisions are reversed/undone until the very end and noise is introduced as new stakeholders join late to the table.

The underlying assumption behind deadline creep is “this moment is as good as any other to pitch my idea.” We try to get past our original assumptions as early as possible. From then on, the driver for new changes and pivots should solely be user feedback.

Something that has worked for us is to start the project negotiating with the client deadlines for key decisions and mutually agreeing that these deadlines will be respected, regardless of who shows up later with ideas. Deferring changes until they can be balanced with user feedback is another good technique; get used to saying that is a great idea, we can include it in Iteration Two, after we have launched.

Products that solve no ones needs

Pressure to meet budgets and deadlines already exhausted by theoretical discussions and power plays slowly disconnect the team from the users and their needs. Sometimes the connection was never made in first place, and entire products are built based on overconfident hunches or late-night inspirations that crash painfully and expensively.

We as humans have a tendency to live in denial of things. Sometimes the reality is too harsh, so we exclude ourselves from it to prevent it from spoiling our dreams. This might be ok for our personal realms, but its fatal when developing products that are intended for other people to use. Thats a pain that can be postponed, but never avoided. The earlier this pain is welcomed on the project, the more useful it is, and the less damage it causes.

Sometimes you have to be harsh, even at the cost of losing a client. Our role as UX leaders is to uncover value and create it where it doesnt exist. But you will sooner or later face a lost case: a mix of a really bad idea (or a good idea with terrible timing) and a delusional team or founder. In these cases, saying sorry, we believe theres nothing we can do to make this product usefulwill keep you honest to yourself and to your client. Sometimes this slap in the face earns you extra respect and even encourages the team to reconsider.

Incomplete or defective research

Sometimes we, as UX researchers, trust our gut too much and take shortcuts from meeting an appropriate number of real users and extracting the right insights.

In some cases, an excess of emphasis is put in validating an abstract idea instead of a real solution; in others, too much effort is put into polishing specific controls, losing perspective of the overall value. Our very experience can get in the way if we are not disciplined enough to contrast and challenge all of our assumptions with proper user feedback.

It is important not to work alone as UX practitioners. When theres a team, its harder to keep a bias, because your teammates will remind you of where the project should be focusing. Even you will have a heightened need of objectvity. This is probably because its far easier to see biases in others than in ourselves.

Gap between design and development

Designers and developers often view themselves as separate entities: design feedsdevelopment with designs and specs, and development implements them mechanically, without questioning. But a UX professional cannot unbundle herself from whats being implemented, because thats where the actual experience for the end user is going to take place.

Handing the design to someone else and then forgetting about it causes the experience to break apart; value is not transmitted to the ones in charge to make it a reality. Designers and developers are equally responsible for the user experience. Doing it right takes a lot of communication and a willingness to be mutually influenced towards a unified mindset.

Embrace Agile UX at the development phase. Bring a chair, sit down next to the developer and work in pairs. Free the dev team from having to make user interface decisions; be ready to respond in real-time with graphics, new screens, error handling and copy. This is probably the most overlooked phase in a UX process, and its by far the one which produces the most tangible results for end users. All those little big detailsthat make you fall in love with products exist because someone cared for them long enough to make them appear.

Tendency to Waterfall UX as a way of reducing uncertainty

An approach to UX too close to the academic, the bureaucratic and the theoretical is self-satisfactory: structured, thoroughly executed methodologies are rewarded, regardless of the results they produce or their impact on the end user.

Projects take forever to complete and the team is lost making reports and Gantts look pretty to the board, instead of being in the field, trying, failing and observing. The focus is diverted from effectiveness to mere compliance. Sometimes it’s the client, bound by regulations and standards, that pushes it this way; sometimes are the UX practitioners themselves, trying to validate their work with name-checking and blind following of established practice.

Whatever the cause, everyone the users especiallywill benefit from a flexible approach, rich in tools but always listening for what the project needs, and more reliant on direct feedback than processed and standardized reports.

Keep an obsessive focus on the end experience dont let the methodology blind you. Your end users will never see or appreciate your process, they will only see your results. Nobody says wow, what I love most about this product is that the team got Agile values totally right. Thats what you would say about it. Dont lose sight of your users.

All of these are problems because they ultimately make the user experience suffer. A good UX management maintains the team focused on solving real needs and being effective, from the start to the end. It calls for a holistic involvement; as a UX lead, you cant really disengage any stage of the project, as all of them involve being in touch with users and fine-tuning the output.

Thats why at Continuum we strive to keep our projects as close to Lean UX as they can be. This is not only for the sake of efficiency, but also because its the approach that makes the best use of our common sense and our flexibility as a team. In a future post, well discuss the principles that guide us down that path.

This is a post by Sergio Nouvel, partner at Continuum, a UX, innovation and software boutique. He’s also co-founder of Get on Board, a job board for digital professionals in Latin America.

Usability vs. A/B testing – which one should you use?

These days, I see a lot of people arguing over the type of testing procedures they should follow – Usability testing or A/B testing. Honestly, I find this argument quite vague. In fact, it’s not an argument at all, because, they serve completely different purposes.  We need to clearly understand which one is required when.  Let’s begin by diving into what is what and when you should go for them.

A/B Testing is what you should perform when you have two different designs (A and B), both having certain benefits, but you need to know which one has an edge over the other, or which one has a higher conversion rate.  Let’s put it this way – If you are looking for answers to questions like “Which design results in the most click-throughs by users?” or “Which design layout results in more sales?” or “Which e-mail campaign performs better?”, etc., an A/B Test is what you should go for.  A/B Testing is an effective way of testing how certain design changes (small or big) in the existing product can produce an impact on your returns.  The main advantage of performing such a test is that you can compare between two versions of the same product where the difference in design elements can be as nominal as the color of a particular CTA button or as massive as being completely different from each other.

On the other hand, if you’re looking for answers to questions like “Can users successfully complete the given task?” or “Is the navigation smooth as butter?” or “Do certain elements distract the user from their end goal?” -  Basically anything related to ease of use; Usability Testing is what you should go for.  Unlike A/B Testing, Usability Testing provides insights into user performance metrics rather than establishing the fact which design (A or B) is better.  The coolest thing about usability testing is that you can perform it on designs having any degree of fidelity – ranging from wireframes to high fidelity mockups, or even the actual finished product.  It’s completely up to you.

A/B Testing is quantitative in nature, i.e., it focuses on the “How Many”, whereas Usability Testing is qualitative in nature, i.e., it focuses on the ‘Why’.

While Usability Testing may require you to recruit participants, script your tasks and questions, analyze and make design recommendations based on your findings, A/B Testing requires no such efforts. A/B Testing allows you test your designs with real time traffic.  This sort of a test is performed on a live site, where an equal percentage of people/users are directed towards either designs and the number of click throughs and successful conversions from each design are recorded.  The results are then analyzed to determine which design trumps over the other.

Concluding, I’d like to say that it is very important to understand what type of test is required when.  Not knowing would result in costing you a lot of money, time and effort.  It is often recommended to pair one up with the other.  Why? Well, the benefits have no limits.  You can conduct usability tests to collect inputs (qualitative in nature) from the users and use A/B Tests to get insights into what can your possible design alternatives be or which alternative performs the best.  Conducting a Usability Test also ensures that there is no guess work involved in designing each of the design alternatives, and hence the designs tend to be bulletproof.

Arijit Banerjee is a UI & UX Enthusiast.  Although a power systems engineer by education, he has always found himself inclined toward the world of UX.  He has been associated with several firms and has helped define experiences across a wide range of products.  Apart from that, he’s a terrible singer, a dog lover, and an out and out foodie with decent culinary skills.  You can visit his website or follow him on Twitter.

« Previous PageNext Page »