Which is more effective: lab-based or remote usability testing?

If you have a new website or web app, you probably want to gauge the performance quality of your site. And you probably know that this will involve some sort of usability testing.

But which is the best method for usability testing? A team of researchers studied this very topic. Below is a summary of their study – and their results.

The team compared two methods of usability testing:

1. Traditional lab-based testing method – pick a group of people and have them go through the usability test in a controlled, in-house environment

2. Remote web-based testing method – pick users at random and have them go through the usability test remotely

Their experiments and results were published in a paper titled An Empirical Comparison of Lab and Remote Usability Testing of Web Sites1. Let me give you the highlights.

The experiments

Two user groups tested the usability of a set of two websites: 8 users participated in the experiments in a traditional lab-based environment, and 38 users participated remotely over the web.

Experiment #1

Experiment #1 consisted of 17 tasks to be completed on a website that was meant to be of both informational and transactional value to end-users. It provided information about retirement savings, pension, medical and dental coverage, payroll deduction, direct deposit and financial planning. It also enabled users to set their own payroll deduction.

Experiment #2

Experiment #2 consisted of 13 tasks to be completed on a second website that was meant to be of purely informational value; it provided details about stock quotes, company news, research, and investment strategies.

The tasks in both experiments were designed to judge whether:

• The website was visually appealing
• The menus and links were easy to navigate
• The information was arranged in a logical, easy-to-access manner
• Individual page formatting was good
• Content used appropriate terminologies
• The quality of the web content met the expectations of the target users
• The site was easy to use overall
• The user was able to complete the desired tasks within a reasonable time span
• The user would be interested in returning to the site in the future

The results

The conclusion of the study was based on the following criteria:

• Task completion – percentage of users who successfully completed the given task
• Average time spent to complete each task
• Subjective rating quality

A high correlation was noted in the following conclusions:

• The time spent to complete the tasks and the difficultly experienced in completing tasks were strikingly similar in both user groups – indicating similar behavior irrespective of environments.

• The quality of typed comments and the kind of data extractable from remote users were as rich as those that could be obtained in direct laboratory conditions.

If click streams and screenshots were enabled in remote test conditions, the quality of data obtained will likely be richer. Remote testing conditions are more cost feasible with the ability to include diverse user groups to uncover unique usability issues. Both testing conditions each reveal certain unique usability parameters as well.

In conclusion

The study concludes: The behavior of test users is strikingly similar in lab and remote usability tests. This is reassuring, and indicates that the different environments do not lead to different kinds of behavior.

In other words, take your pick – both methods of usability work and each have their own advantages and disadvantages. In-house testing may ensure that you gather more detailed input, while remote testing is usually less expensive. The most important thing is that you do some form of usability testing.

Footnote 1. An Empirical Comparison of Lab and Remote Usability Testing of Web Sites, by Tom Tullis, Stan Fleischman, Michelle McNulty, Carrie Cianchette, and Marguerite Bergel.

Say hi to the Loop11 team at the Big Design Conference & UPA International

Hey everyone!

We have some exciting news. We’ll soon be on the road in the US and are fired up to meet you!

We will be at the Big Design Conference from May 31 to June 2nd just outside of Dallas, Texas. Stop by and say hello at Booth #12! We would love to chat. The biggest ideas in strategy, UX, design, gaming, mobile, usability and development will be discussed here!

Right afterwards, we will be in Henderson, NV from June 5-7 at the UPA International Conference. Come visit us at Booth #20 and be a part of the most interactive usability & UX conference in the world!

Hope to see y’all there! Don’t be shy.

Toby Biddle and the Loop11 Team

What website usability is – and why you need to user test for it

How many millions of people in the 80s and 90s had trouble figuring out how to program their VCRs?  My older sister was the only one in our house who knew how to do it, and that’s because she was a little nerdy and read the manual.

This is one of the most classic examples of a usability problem.  No one wants to read a manual or call a support line – or even spend more than five minutes trying to figure things out on their own; people want and expect to be able to use products out of the box.  This is the essence of effective usability design, and the same principle applies to website usability.

What is website usability?

Website usability has two aspects:

1. The primary aspect is about meeting your users’ goals and delivering a satisfying user experience.  Is your site clear, concise, and intuitive to them?  Can they quickly and easily find what they’re looking for?  Are the consequences of pressing buttons and clicking links unambiguous to them?

The email management tool Mailchimp.com is one of my favorite examples, because usability is one of its main selling points.  And indeed all of the most common things that you’d want to do with it are laid out clearly on the front page: Create a Campaign, Manage a List, View a Report.  The button for the most common action – creating a campaign – is distinguished by its orange color and large label.  It’s hard to miss.

Ideally your site is so intuitively laid out that the question of usability never enters your users’ minds; it simply works the way they expect it to.  (In fact, users generally only think about usability when they’re frustrated by something that is not usable to them.)

2. The secondary aspect of website usability is more subtle; it’s about fulfilling the goals your company has for the site.  Does your site’s design nudge visitors in the direction you want them to go?  Are the features that are most important to you front and center?

Here’s a quick example: Laura Roeder Studios (lauraroeder.com) offers social media training and tips to small business owners. Building its list of weekly newsletter subscribers is important to them – as shown by the fact that the email newsletter sign-up is given prime real estate on their site:

The value of user testing

Of course we designed our site to be usable, you might be thinking.  Why wouldn’t we? Here’s the thing: you will never know how usable your site truly is until you test it with people outside of your organization.

Is usability subjective?  Could something be intuitive to one person and not another?  Absolutely, and therein lies one of the two main values of user testing: testing across groups reveals quantifiable trends.  If 7 out of 10 people can’t figure out how to navigate to checkout on your website, that tells you something very valuable which you will want to address.

The other main value of user testing is that it’s unambiguous.  Ask a user an open-ended question – Is our website usable to you? – and you will likely get a general reply.  But give someone a set of specific tasks to execute on your website, and look at the results, and you will see unambiguously where the stumbling blocks are, if any.  And that means you can fix them.

User testing enables you to get inside your users’ heads and create a site that will truly be easy and pleasant for them to use.  And that, of course, is win-win for both them and you.

Go ahead and create your first user test to get a quantifiable, unambiguous handle on your site’s usability (it’s both easy and free).

« Previous PageNext Page »