Benefits of Online, Unmoderated User Testing

For most usability professionals lab-based research has been the only testing method they’d consider.  However, with the growth of Web 2.0 and Web 3.0 technologies online, unmoderated user testing (also known as remote unmoderated or asynchronous user testing) is gaining traction and being used not only as a compliment to, but in many cases as an alternative to traditional lab-based research.

UnModerated User Testing vs Moderated User TestingIn 1997 when the internet was relatively new to the non-technical public market researchers started experimenting with online surveying to replace the costly and time consuming telephone and mail surveys.  There was much resistance in the market research industry to online surveying, and it was often seen as a “cheap and dirty” way of conducting market research studies.  Despite benefits such as cost, speed and geographical reach, to name a few, many market researchers dismissed online surveying and chose not to offer it as methodology to clients.

Today, there is no doubt online surveying is a common tool in the suite of products offered by market researchers.  Clearly, they just needed time to understand the internet and online surveying to know how to get the most out of the new methodology.

Online user testing is in much the same place today as online surveying was in 1997; resisted by many and even dubbed a “voodoo measurement technique” by a very well respected user interface engineer.  We think he’ll eat his words, just like Thomas Watson, Chairman of IBM did a while ago.

Those who have explored and experimented with online user testing are currently reaping the rewards.  In a recent study by the Usability Professionals Association online user testing experienced an 18% increase in usage by UPA members since 2007, while traditional usability testing experienced a 9% decline over the same period.

Online Unmoderated vs Online Moderated User Testing

There’s online, moderated testing and then there’s online, UNmoderated testing.  The key difference is that with online moderated testing the moderator uses a web-based service that allows users in faraway locations to participate in what is essentially, an online meeting.  The software has the ability to share viewing and control of a web browser that permits the moderator to view the user’s mouse movements and web pages they visit, facilitating discussion similar to that of a traditional lab-based usability test. A good visualisation of the approach can be found here.

In contrast, online unmoderated testing is usually done ‘asynchronously’.  That is, first the researcher designs and initiates a study; the participants perform the tasks in their natural context, at home (or the office) using their own computer; then, once all the participants have completed the tasks, the researcher gathers and analyses the data.  In this approach there is no need for a moderator to be present during the testing. 

Benefits of Online, Unmoderated Testing

There are a lot of great reasons to do some online, unmoderated testing on your website.  Here are a few of the key benefits:

Get quantitative usability metrics
Perhaps the most important benefit of unmoderated testing is that it allows for the collection of quantitative usability metrics; in other words, statistics.  Traditional lab-based testing often consists of between 5 to 8 participants; perhaps double that if you have a bigger research budget.  With such a small sample size it would be wrong to attempt to calculate percentages to assist in reporting, such as the percentage of participants who completed a task successfully or the average time it took to complete a task.  The larger sample sizes that are possible with online testing make the calculation of these and many other usability metrics entirely possible, and these metrics can deliver significantly greater insights into the usability of a website than ever before.

If you run some online testing in conjunction with lab-based testing you’ll get the chance to validate your lab findings to ensure they are a valid and true representation of your websites’ user experience.

Conduct benchmarking studies

Because online testing allows for the collection of quantitative usability metrics it becomes very easy and practical to measure how one website performs against competing websites or other versions of the same website.  This can be particularly valuable if you are redesigning your website and have prepared wireframes of a new design.  Pit them against each other to ensure the new design performs better on those key tasks your site visitors come for.

Test with ‘hard-to-get’ participants

If the target audience of your website is senior business executives, doctors, lawyers or brain surgeons then getting them into your lab for testing can be very difficult and very expensive if you’re offering incentives.  But getting these people to do a 5 to 10 minutes online user test in the comfort of their own office is a much more realistic proposition.

Similarly, if you want to test your website for accessibility by doing testing with people who use assistive technologies (screen readers, electronic pointing devices, alternative keyboards, etc) you’ll never be able to replicate everyone’s unique set-up in your own labs.  Here’s a great opportunity for online testing to step in and take over.

Test with an international audience

Similar to the above point, if your website has an international audience and you want to ensure it can account for any cultural differences that might exist, lab-based testing around the globe is guaranteed to blow your budget…if you had one big enough to begin with.

Online testing easily allows you to recruit participants anywhere in the world.  And if you don’t have a database of your own there are a slew of market research panels that will provide participants that fit your specific criteria.

The ‘Test early, Test often’ principal becomes a reality

With all the different web-based tools around these days that can assist in user research the cost of doing usability testing is always going to be much cheaper than the lab-based alternative.  And once you’ve run a few online studies you’ll be able to run them much faster than anything you can do in a lab.

It follows that if it’s faster and cheaper, you can now conduct more user testing studies more frequently with your currently budget.  For years they’ve said, “Test early, test often”, but how many of us have honestly been able to do that?  Now, it’s become a reality.

Below is an excellent example of a systematic and thorough usability testing process (thanks  There are eight stages of testing in total.  If you use Loop11 for all the stages of task testing, Survey Monkey for the surveys, Optimal Sort for card sorting and Treejack for the IA testing you could do all of this for around $2,500.

Usability Testing Process

For those who want to read more about online user testing there are two books I recommend reading:

[New feature] Now, you can customize the participant interface

We’re excited to announce a new Loop11 feature that’s now available to all users.

When you set up your next usability test, you’ll notice you can now customize the interface participants see. You may upload a logo as well as choose primary and secondary colors for the text and buttons participants will use during the usability study.

For instance, if Amazon were to run a usability test today, rather than the previously standard green background and Loop11 branding, the participant interface can be customized to look like this:

  Amazon 1

 Amazon 2

The customization feature can be configured in Step 1 of the project creation process. When setting up your usability test, you will now be able to “Create a new theme,” as pictured below.


 Amazon 3


Each account may have several different themes, and each theme enables you to add a custom logo and define six unique colors and attributes, as seen below.


Amazon 4


We’re excited to hear your feedback as you experiment with this feature, so please let us know your thoughts in the comments!

Customer experience vs. user experience, and why it matters

The design and user research community is all abuzz with the “customer experience” and its importance as a complement to user experience. What do the two terms mean and can businesses win by taking both areas seriously? That’s what this blog post will explore.

Before we dig deeper, let’s define our terms.

Customer experience vs. user experience

Customer experience encompasses your customer’s, or prospective customer’s, entire experience of your brand. It spans the complete conversion funnel from how they discovered your business, to the moment they landed on the website, their trials and tribulations when using it, their purchase experience, all efforts by your business to re-engage them (e.g. marketing emails, ads and retargeting), and how they experience coming back to purchase a second time. Any help or support team interaction is part of the customer experience, too. And when that customer walks into the brick and mortar location of your business after purchasing online — even that is part of the customer experience.

User experience speaks to a critical segment of the world of interactions described above: it refers to your prospective customer’s relationship with digital interfaces. How that prospective customer, or user, experiences and behaves within your brand’s website, mobile site or software — that’s user experience.

So which one deserves more focus? How should a business ensure both customer experience and user experience are top-notch? And why is this an important distinction?

User experience foreshadows the customer experience 

Simon Sinek gives an interesting TED Talk about how great leaders inspire action, and I’ll save you 18 minutes by summarizing what is arguably the top takeaway: “People don’t buy what you do. They buy why you do it.”

He gives the example of Apple and discusses why they’ve been so successful selling portable music devices when no other computer company has had much luck in that marketplace from a revenue perspective. He posits the iPod sells because Apple believes that technology should be incredibly simple to interact with, and more importantly, we believe it too. We buy iPods because we share that philosophy, and not because we think Apple, a computer company, is the best qualified firm to design and manufacture portable music players.

In the same way, the user experience your business offers online says a lot about:

1. What your business believes.

2. What the customer can expect from your business now and always.

Customers take cues from the web experience you show them — important cues about what it will be like to continue interacting with your company, to buy from your company once or many times, and to get technical or customer support from your company.

In this way, your website’s (or mobile site’s) user experience isn’t just a subcategory of the larger, more relevant beast called “customer experience.” It’s a piece of the customer experience, yes, but your potential customers view your user experience as litmus test that foreshadows what the larger customer experience will be like.

Moreover, your potential customer is highly likely to be a web user first. That means you have a chance to win them over with your user experience and earn the opportunity to give them a larger tour of the “customer experience” mansion. It’s also a chance to lose them in the foyer if the user experience doesn’t stand up to the challenge.

We can all name high-profile companies who have focused on the customer experience and won. Their customers are more likely to be evangelists and more likely to return and purchase again year after year. They also build a tougher skin for the occasional bad service experience when loyalty has been established.

REI, Trader Joe’s, MailChimp — these are businesses that have scored big on the philosophy that an excellent customer experience at every turn and every interaction with the company is of paramount importance.

If you transact a significant chunk of your business on the web, that customer experience begins online and user experience is central. To win your prospects over and deliver stellar user experience, of course, requires testing. You have to ask them, methodically, scientifically, to tell you by the numbers whether your website or mobile site is doing its job and whether they’d recommend it to a friend.

When you’re ready to start testing, sign up for a free trial of Loop11 and your first usability testing project is on us.

[Infographic] Why you must invest in mobile usability testing

Simple guide to benchmarking website usability

If you’re looking to run a benchmarking study to assess website usability, one solid playbook to reference is Jeff Sauro‘s post, published yesterday, on How To Benchmark Website Usability.

Jeff lays out a step-by-step guide describing what you need to consider when setting up your benchmarking study, from designing the study, to executing it and analyzing the results. He walks through critical steps like:

  • Identifying the right users to test
  • Finding the users
  • Defining the tasks
  • Defining test metrics
  • Choosing sample sizes
  • Analyzing the results of your benchmarking study

The post also provides a comparison (based on price and features) of some of the top testing tools, including our usability testing toolset.

Read the full article here on Measuring Usability.

Loop11 is a proud partner in hosting the User Experience Awards

We’re proud to be sponsoring the User Experience Awards this year!  The UX Awards honor exceptional user experience design in digital products and services, as well as their creators.

Submissions run from now until May 15 2013, with the final Awards taking place during NY Internet Week on May 21, 2013. Last year, several startups won awards alongside digital agencies, tech and media companies. There are also cash prizes of up to $1,000 for the Grand Prize. Follow @UXAwards for the latest info.

The event starts at 5:30 and is taking place in Tishman Auditorium at the Parsons School of Design in New York.

If you’re curious about prizes: there will be one Grand Prize, three Gold, three Silver and Bronze. The Grand Prize winner also receives $1000, while Gold winners receive $250. All winners also receive additional prizes from the show’s sponsors.

Submit an entry for the UX Awards

We’d love to see you there!

Search engine findability: can users find your site?

“Findability precedes usability in the alphabet and on the Web. You can’t use what you can’t find.”—Peter Morville, in Ambient Findability

Last month, CEO Toby Biddle recently posted a guest article with our friends at UX Matters on the topic of search engine findability. Testing the usability of your existing website is a given — but we can’t overlook how people find (or perhaps more importantly, fail to find) your site on the web. Studying how people use search engines and keywords to locate sites like yours is what search engine findability is all about.

In brief, the post discusses:

– The importance of search engine findability

– How usability testing can uncover important search engine findability information not visible via your web analytics package

– How to conduct a search engine findability test (learn about a Search Engine Findability Study we ran on our platform to explore which keywords were commonly used when searching for a credit card provider)

Check out the full article here on UX Matters.




Loop11 joins the Unbounce “conversion ecosystem”

We’re excited to be part of the “conversion ecosystem” coined and curated by our partners and friends at You can find Loop11 in the optimization & testing category. 

We welcome and encourage anyone creating a landing page via Unbounce to give Loop11 a test drive; you’ll be surprised what usability insights you might glean from even a simple online usability study. And of course, you can’t beat the savings behind online, unmoderated user testing as compared to the cost of putting prospective users in a room with a computer and guiding them through your study.

A big thank you to the pioneering landing page moguls over at Unbounce.

« Previous PageNext Page »
Want more inspiration?
Join the Fab-UX 5 newsletter!

Five links to amazing UX articles,sent to you once a week.

No SPAM, just pure UX gold!

No Thanks