Which is more effective: lab-based or remote usability testing?

If you have a new website or web app, you probably want to gauge the performance quality of your site. And you probably know that this will involve some sort of usability testing.

But which is the best method for usability testing? A team of researchers studied this very topic. Below is a summary of their study – and their results.

The team compared two methods of usability testing:

1. Traditional lab-based testing method – pick a group of people and have them go through the usability test in a controlled, in-house environment

2. Remote web-based testing method – pick users at random and have them go through the usability test remotely

Their experiments and results were published in a paper titled An Empirical Comparison of Lab and Remote Usability Testing of Web Sites1. Let me give you the highlights.

The experiments

Two user groups tested the usability of a set of two websites: 8 users participated in the experiments in a traditional lab-based environment, and 38 users participated remotely over the web.

Experiment #1

Experiment #1 consisted of 17 tasks to be completed on a website that was meant to be of both informational and transactional value to end-users. It provided information about retirement savings, pension, medical and dental coverage, payroll deduction, direct deposit and financial planning. It also enabled users to set their own payroll deduction.

Experiment #2

Experiment #2 consisted of 13 tasks to be completed on a second website that was meant to be of purely informational value; it provided details about stock quotes, company news, research, and investment strategies.

The tasks in both experiments were designed to judge whether:

• The website was visually appealing
• The menus and links were easy to navigate
• The information was arranged in a logical, easy-to-access manner
• Individual page formatting was good
• Content used appropriate terminologies
• The quality of the web content met the expectations of the target users
• The site was easy to use overall
• The user was able to complete the desired tasks within a reasonable time span
• The user would be interested in returning to the site in the future

The results

The conclusion of the study was based on the following criteria:

• Task completion – percentage of users who successfully completed the given task
• Average time spent to complete each task
• Subjective rating quality

A high correlation was noted in the following conclusions:

• The time spent to complete the tasks and the difficultly experienced in completing tasks were strikingly similar in both user groups – indicating similar behavior irrespective of environments.

• The quality of typed comments and the kind of data extractable from remote users were as rich as those that could be obtained in direct laboratory conditions.

If click streams and screenshots were enabled in remote test conditions, the quality of data obtained will likely be richer. Remote testing conditions are more cost feasible with the ability to include diverse user groups to uncover unique usability issues. Both testing conditions each reveal certain unique usability parameters as well.

In conclusion

The study concludes: The behavior of test users is strikingly similar in lab and remote usability tests. This is reassuring, and indicates that the different environments do not lead to different kinds of behavior.

In other words, take your pick – both methods of usability work and each have their own advantages and disadvantages. In-house testing may ensure that you gather more detailed input, while remote testing is usually less expensive. The most important thing is that you do some form of usability testing.

Footnote 1. An Empirical Comparison of Lab and Remote Usability Testing of Web Sites, by Tom Tullis, Stan Fleischman, Michelle McNulty, Carrie Cianchette, and Marguerite Bergel.

Say hi to the Loop11 team at the Big Design Conference & UPA International

Hey everyone!

We have some exciting news. We’ll soon be on the road in the US and are fired up to meet you!

We will be at the Big Design Conference from May 31 to June 2nd just outside of Dallas, Texas. Stop by and say hello at Booth #12! We would love to chat. The biggest ideas in strategy, UX, design, gaming, mobile, usability and development will be discussed here!

Right afterwards, we will be in Henderson, NV from June 5-7 at the UPA International Conference. Come visit us at Booth #20 and be a part of the most interactive usability & UX conference in the world!

Hope to see y’all there! Don’t be shy.

Toby Biddle and the Loop11 Team

What website usability is – and why you need to user test for it

How many millions of people in the 80s and 90s had trouble figuring out how to program their VCRs?  My older sister was the only one in our house who knew how to do it, and that’s because she was a little nerdy and read the manual.

This is one of the most classic examples of a usability problem.  No one wants to read a manual or call a support line – or even spend more than five minutes trying to figure things out on their own; people want and expect to be able to use products out of the box.  This is the essence of effective usability design, and the same principle applies to website usability.

What is website usability?

Website usability has two aspects:

1. The primary aspect is about meeting your users’ goals and delivering a satisfying user experience.  Is your site clear, concise, and intuitive to them?  Can they quickly and easily find what they’re looking for?  Are the consequences of pressing buttons and clicking links unambiguous to them?

The email management tool Mailchimp.com is one of my favorite examples, because usability is one of its main selling points.  And indeed all of the most common things that you’d want to do with it are laid out clearly on the front page: Create a Campaign, Manage a List, View a Report.  The button for the most common action – creating a campaign – is distinguished by its orange color and large label.  It’s hard to miss.

Ideally your site is so intuitively laid out that the question of usability never enters your users’ minds; it simply works the way they expect it to.  (In fact, users generally only think about usability when they’re frustrated by something that is not usable to them.)

2. The secondary aspect of website usability is more subtle; it’s about fulfilling the goals your company has for the site.  Does your site’s design nudge visitors in the direction you want them to go?  Are the features that are most important to you front and center?

Here’s a quick example: Laura Roeder Studios (lauraroeder.com) offers social media training and tips to small business owners. Building its list of weekly newsletter subscribers is important to them – as shown by the fact that the email newsletter sign-up is given prime real estate on their site:

The value of user testing

Of course we designed our site to be usable, you might be thinking.  Why wouldn’t we? Here’s the thing: you will never know how usable your site truly is until you test it with people outside of your organization.

Is usability subjective?  Could something be intuitive to one person and not another?  Absolutely, and therein lies one of the two main values of user testing: testing across groups reveals quantifiable trends.  If 7 out of 10 people can’t figure out how to navigate to checkout on your website, that tells you something very valuable which you will want to address.

The other main value of user testing is that it’s unambiguous.  Ask a user an open-ended question – Is our website usable to you? – and you will likely get a general reply.  But give someone a set of specific tasks to execute on your website, and look at the results, and you will see unambiguously where the stumbling blocks are, if any.  And that means you can fix them.

User testing enables you to get inside your users’ heads and create a site that will truly be easy and pleasant for them to use.  And that, of course, is win-win for both them and you.

Go ahead and create your first user test to get a quantifiable, unambiguous handle on your site’s usability (it’s both easy and free).

Microsoft Chooses Loop11 for Usability Software

We are proud to finally announce that Loop11 has beaten a swathe of worldwide rivals to secure a licence deal with Microsoft‘s Office Experience Group.

Under the terms of the licence agreement, Microsoft’s Office Experience Group will utilize Loop11’s to conduct its online usability testing.  The win came after the team’s extensive benchmarking exercise of usability tools in the world marketplace.

We were chosen due to our ability to conduct rapid online usability testing across the range of Microsoft Office applications.  In addition it was the speed of software set up and implementation, ease of use, combined with the ability to meet strict security requirements, as well as the strong analysis capabilities that placed Loop11 in the winning position.

Our CEO, Toby Biddle, commented, “We’re delighted that a giant such as Microsoft has awarded Loop11 the business.  Traditionally, usability testing is conducted in the lab and relies on the more expensive and invasive method of recruiting participants and behavior observation.  Loop11 enables you to conduct fast online usability testing in over 40 languages, making it unbeatable for running worldwide projects from the comfort of your own office.”

Launched in 2009, the software was designed by Loop11’s team of developers to meet the needs of its sister company, usability consulting firm UsabilityOne.  “We were always seeking a cost effective usability tool that generated reliable statistics,” continued Biddle. “Before we developed Loop11, we found lab-based studies to be too small in sample size and although there were other software products in the marketplace, we found them inflexible and prohibitively expensive to be able to run iterative usability studies throughout the development of a website.  The answer was to build our own software based upon our 15 years of usability consulting experience.

“The beauty of Loop11 is its ability to be used by companies not just of Microsoft’s size, but also by small businesses, educational institutions, not-for-profits and government who want to run their own usability studies, continued Biddle. “With the increase in the number of digital applications, it has become more important for businesses to be smart about testing usability.”

If you too want to test like Microsoft grab yourself an annual licence starting at just $1,900 enquire about one NOW!

Clickstream analysis has arrived!

We’re excited to announce the first of a number of new features for Loop11 in 2012 – Clickstream Analysis. The clickstream analysis will replace the ‘Most common navigation path’ by allowing you to analyze task navigation graphically and instantly understand how visitors navigate a task through your website.

The clickstream report provides a graphical representation of participants’ navigation through the website so you can see their journey, as well as the path they took before abandoning or failing a task.

You’ll notice that we made the visualization highly interactive so you can interact with the graph to highlight different pathways, and to see detailed information about specific pages. For example, if you want to dive deeper into your pages, you can hover over the node to see more information at a glance.

Below you can see an example of a task that performed well in usability testing (in this instance with a task completion rate of 92%). A quick look at the analysis shows that 90% of participants went directly to the success page from the homepage. The orange lines are a visual indication of the magnitude of participants who failed the task at different points in their journey through the website. Navigation through the website is clean and uncomplicated, which should be the case when the participant has a clear direction.

By contrast, the clickstream below is for a task that was performed comparatively poorly, where the task completion rate was only 32%. In this example, there is clearly confusion as to where participants should navigate from the home page to complete the task.

This is our first step in tackling clickstream analysis and we look forward to hearing your feedback as you begin to use the report in the coming weeks. We’re excited to bring you the first of a number of new features for the year, so stayed tuned for more!

As always, we welcome your input on how we can make the clickstream analysis more useful for you, so let us know in the comments below.

Server Upgrade

Loop11 will be conducting a scheduled server upgrade on Saturday December 3rd, 2011 beginning at 21:00 PM UTC (Co-ordinated Universal Time), and lasting up to 1 hour.Loop11 Website Maintenance

During this time, the Loop11.com website and systems, will be temporarily inaccessible. Any content hosted on Loop11.com will not be accessible by you or your project participants. During this time a message will be posted on the Loop11 website alerting visitors of the scheduled maintenance. This should only take an hour, and once restored, all your content and data will be back online.

Schedule:

Eastern Daylight Time (North America): Saturday, July 17: 4:00pm

Greenwich Mean Time (Europe): Saturday, July 17: 9:00pm

Eastern Standard Time (Australia): Sunday, July 18: 10:00am

We appreciate your patience and can assure you the service will be restored as quickly as possible. We also wanted to point out some recent improvements:

• Participant instructions page can now be skipped. For anyone running True Intent studies,

the original instructions to participants were not quite suitable. Instructions can now be completely skipped.

• Custom IDs. If you want to track participants so you know who to provide incentives to, you

can customise the end of the URL with a unique ID for each participant. Learn more.

For additional time zones, please reference www.timeanddate.com.

We apologise for any inconvenience caused, if you have any queries relating to this notice please email us at Support@Loop11.com.

Remote, Unmoderated UX Testing Grows by 28%

Jeff Sauro at Measuring Usability, has put together another great blog post comparing preferred methods of usability testing by usability professionals over the past six years. Based on the last three biennial UPA surveys, Jeff determined that there was a 28% increase in people using remote, unmoderated usability testing tools like Loop11. Jeff states that the reason for the increase is that remote testing tools are an inexpensive and effective way of measuring task based, quantitative usability data.

Read the full post here: The Methods UX Professionals Use – MeasuringUsability.com

Also check out: 2011 UPA Survey Results

Scheduled System Maintenance

Loop11 will be conducting scheduled maintenance on Sunday July 17th, 2011 beginning at 22:00 PM UTC (Co-ordinated Universal Time), and lasting up to 4 hours.Loop11 Website Maintenance

During this time, the Loop11.com website will be temporarily inaccessible. Any content hosted on Loop11.com will not be accessible by you or your project participants. During this time a message will be posted on the Loop11 website alerting visitors of the scheduled maintenance.

This is a major service upgrade including database improvements and server updates. We appreciate your patience and can assure you the service will be restored as quickly as possible.

Schedule:

Eastern Daylight Time (North America): Sunday, July 17: 6:00pm

Greenwich Mean Time (Europe): Sunday, July 17: 11:00pm

Eastern Standard Time (Australia): Monday, July 18: 8:00am

For additional time zones, please reference www.timeanddate.com.

We apologise for any inconvenience caused, if you have any queries relating to this notice please email us at Support@Loop11.com.

« Previous PageNext Page »
Want more inspiration?
Join the Fab-UX 5!

Five links to amazing UX articles,sent to you once a week.

No SPAM, just pure UX gold!

No Thanks