Which is more effective: lab-based or remote usability testing?

If you have a new website or web app, you probably want to gauge the performance quality of your site. And you probably know that this will involve some sort of usability testing.

But which is the best method for usability testing? A team of researchers studied this very topic. Below is a summary of their study – and their results.

The team compared two methods of usability testing:

1. Traditional lab-based testing method – pick a group of people and have them go through the usability test in a controlled, in-house environment

2. Remote web-based testing method – pick users at random and have them go through the usability test remotely

Their experiments and results were published in a paper titled An Empirical Comparison of Lab and Remote Usability Testing of Web Sites1. Let me give you the highlights.

The experiments

Two user groups tested the usability of a set of two websites: 8 users participated in the experiments in a traditional lab-based environment, and 38 users participated remotely over the web.

Experiment #1

Experiment #1 consisted of 17 tasks to be completed on a website that was meant to be of both informational and transactional value to end-users. It provided information about retirement savings, pension, medical and dental coverage, payroll deduction, direct deposit and financial planning. It also enabled users to set their own payroll deduction.

Experiment #2

Experiment #2 consisted of 13 tasks to be completed on a second website that was meant to be of purely informational value; it provided details about stock quotes, company news, research, and investment strategies.

The tasks in both experiments were designed to judge whether:

• The website was visually appealing
• The menus and links were easy to navigate
• The information was arranged in a logical, easy-to-access manner
• Individual page formatting was good
• Content used appropriate terminologies
• The quality of the web content met the expectations of the target users
• The site was easy to use overall
• The user was able to complete the desired tasks within a reasonable time span
• The user would be interested in returning to the site in the future

The results

The conclusion of the study was based on the following criteria:

• Task completion – percentage of users who successfully completed the given task
• Average time spent to complete each task
• Subjective rating quality

A high correlation was noted in the following conclusions:

• The time spent to complete the tasks and the difficultly experienced in completing tasks were strikingly similar in both user groups – indicating similar behavior irrespective of environments.

• The quality of typed comments and the kind of data extractable from remote users were as rich as those that could be obtained in direct laboratory conditions.

If click streams and screenshots were enabled in remote test conditions, the quality of data obtained will likely be richer. Remote testing conditions are more cost feasible with the ability to include diverse user groups to uncover unique usability issues. Both testing conditions each reveal certain unique usability parameters as well.

In conclusion

The study concludes: The behavior of test users is strikingly similar in lab and remote usability tests. This is reassuring, and indicates that the different environments do not lead to different kinds of behavior.

In other words, take your pick – both methods of usability work and each have their own advantages and disadvantages. In-house testing may ensure that you gather more detailed input, while remote testing is usually less expensive. The most important thing is that you do some form of usability testing.

Footnote 1. An Empirical Comparison of Lab and Remote Usability Testing of Web Sites, by Tom Tullis, Stan Fleischman, Michelle McNulty, Carrie Cianchette, and Marguerite Bergel.