Seven Things Remote Testing Can Do That Lab Testing Can’t.

The influx of remote usability testing tools has started to change the way usability testing is conducted. There are new techniques and different metrics to consider when conducting user research now.

This post is not aimed at polarizing people or making people pick remote testing over lab based testing, it has been written to differentiate some of the capabilities of the two methods.

So we put a list together of 7 things you can do with remote user testing that you can’t do with lab based testing. Firstly, it’s important to understand that when we talk about remote usability testing tools, we mean full service, unmoderated “quantitative tools”. To understand the difference, you can read Bill Albert’s article on remote testing tools.

Seven Things Remote Testing CAN DO That Lab Testing CAN’T:

  1. 1. Test early, test often…affordably

Remote testing is much more cost effective than lab based testing, which means you can test at every stage of the development lifecycle, from wireframes to a live website. Or conduct benchmarking studies on a weekly or monthly basis to see how changes to your website affect usability. Lab-based testing can cost tens of thousands of dollars, which usually results in lab-based testing not being run very often by most companies. Remote testing tools usually cost just a few hundred dollars, making it easy to test on a much more frequent basis, thus creating the opportunity to run more regular studies.

  1. 2. True Intent Studies

True intent studies focus around a user’s real aim when using a website. What is their goal on your website, how do they go about it, and did they achieve their goal? These are the main objectives. True intent studies can only be run online because you must intercept users as they visit your website so you can observe their natural behaviour.

  1. 3. Get large numbers of “hard to get” users in their own environment

You can test hundreds of people simultaneously, in their natural environment. Whether you need a doctor or French speaker in China, online panels let you tap in to the thousands of unique demographics out there.

  1. 4. Get more in-house people involved

Remote usability testing is a great way of introducing usability testing into a company using limited resources and budget. Typically, most companies only involve a few people in lab-based usability studies. Remote online testing enables other parties to reap the benefits and even get involved. Quantitative metrics will show your IT guys why changes are needed to your website, and the finance guys will have the stats they need to understand why money needs to be spent.

  1. 5. Get statistically significant data

Lab-based usability testing is all about identification of usability issues and qualitative reasoning, not about quantification. Due to the small number of participants in lab testing (usually 5-10) compared to the hundreds or even thousands in remote testing, one simply can’t generate the quantitative reports that high level company decision makers may want to see. However, with remote user testing you can quash any in-house differences of opinion by getting the hard facts about your website’s user experience.

  1. 6. Special technology

    Some interfaces just don’t make any sense to test outside their intended usage environment. Maybe you are running a user test on Facebook. The test involves uploading personal work with personal email contacts, pictures, videos and more. For this, you need the users to have their own email contacts, photos, videos and more. Having them bring their laptop or media to a lab isn’t going to do the study justice. Remote testing would allow them to use their own equipment at their own place…saving time, money and improving the quality of the results.

Or, let’s say you’re testing a recipe website that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment. We also call this the participant’s “technological ecosystem” because it implies that their devices and computing environment have an impact on how they interact.*

*The term “technological ecosystem” and the above example of a recipe website was originally written by Nate Bolt in his article “Pros and Cons of Remote Usability Testing”.

  1. 7. Online accessibility testing

Now, let’s say you’re still testing that recipe website, except this time you need to test it for accessibility. Meaning you need to determine if there are any barriers that might prevent people with unique physical or mental needs from using the step-by-step recipe website in their own homes. It would be virtually impossible to replicate the technology and assistive devices of users with special needs, so the best way to do this is online.

One of the best summaries of lab testing compared to remote testing was given by Nate Bolt in his June, 2010 post “Pros and Cons of Remote Usability Testing”.

As Nate said:

“Both in-person and remote UX research share the same broad purpose: to understand how people interact and behave with the interface you’ve made. There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with timing, security, equipment, or the type of interaction you want to have with participants.”

Happy Testing!

Loop11 Partners with Qualitative Usability Tool

That’s right, now you can get the best of both worlds… quantitative and qualitative. There’s a plethora of information on quantitative versus qualitative research methods. Best practice guides suggest a mix of both to effectively capture a more holistic perspective. A challenge, however, is how to combine qualitative and quantitative testing in a productive way to obtain statistically significant usability data to drive user-centered design. That’s where Loop11‘s partnership with TryMyUI comes in. When Loop11 and TryMyUi are used in tandem, designers, researchers and marketers have a powerful set of metrics to optimize the user experience.

What’s the difference?

The difference between quantitative and qualitative research is often explained using contrasting terminology, like “hard vs. soft”, “numeric vs. non-numeric”, “statistics vs. insights”, “measure vs. explore”, “what vs. why”. The quick contrasts are useful to highlight both the strengths and limitations of each approach if used solely without consideration of the other.

Broadly speaking, quantitative research can provide path and performance analyses by capturing the “what” of user behavior.  Loop11 provides quantitative measures of real user behavior through clickstream analysis. It delivers quantitative metrics such as task completion rates, number of clicks, time on tasks, and detailed path analysis. Data generated from users’ clickstream are automatically presented in real-time reports.

The quantitative measures from Loop11 are especially useful if researchers want to:

–       Identify any usability problems

–       Measure task efficiency and success

–       Compare against competitor usability metrics

–       Run regular usability benchmarking studies

In contrast, a goal of qualitative research is to gain valuable insight into the thought processes behind user’s actions. TryMyUI is remote usability testing tools that capture the “voice of the customer” via a video recording and a written summary documented by the user. As users navigate a website, they “think aloud”, verbalizing their thoughts as they complete the tasks or questions posed by the researcher. For example, a user may express surprise to find the “register” button in a particular location on the site. Quantitative research tracks that the user clicked “register” and what the user clicked on before and after.  The additional insight that the user was “surprised” augments the analysis of the user’s behavior and the researcher observes this first-hand by viewing the narrated video.

What you can learn from the qualitative measures from TryMyUI:

–       How does the user experience match the user’s natural expectations about how things work

–       Why is the user experiencing this with in-context feedback

What’s the linkage?

Loop11 provides the quantitative measures through metrics and reports, and TryMyUI delivers the qualitative insights via narrated videos and written answers to survey questions. With this easy-to-use, on-demand testing tool, any researcher or marketer can simultaneously gain insight into both the “what” and the “why” of user behavior.

Although there is no single usability tool that provides this powerful combination, the combined functionality can be replicated by initiating a project in Loop11 and then augmenting the testing process in TryMyUI.

The process is straightforward:

1:    Open an account on and Both sites offer a free trial for the first project

to help you get started and become familiar with the setup process and results.

2:    At Loop11, create the task scenarios you want the participants to complete.  When you launch

your project copy your unique project URL and head over to TryMyUI.

3:    Paste the Loop11 project URL in “Web Site Address”.  There’s no need to re-enter the task scenarios

as that information is already captured in the Loop11 project URL.

4:    Use the default “Survey Questions” or customize as needed.  Select the demographics.  Select number of testers.

5:    TryMyUI handles the rest and will deliver videos of each test user navigating the website with the “think aloud”

narration.  Users will also provide written answers to any “Survey Questions”.

Happy Testing!


New & Improved

Faster, Smoother, Smarter.

Over the past year, we’ve listened to your feedback and the issues you have raised. So we locked our developers in a room and told them not to come out until the issues you raised were fixed. Our developers didn’t disappoint! We’re now so confident about the improvements to Loop11, we’re offering another free trial to anyone who has already used theirs.

Here are the issues that you raised and we fixed:

Rendering Issues: Some parts of certain websites were not displaying correctly when parsed through Loop11.

Slow Performance: When websites are parsed through Loop11 they run at a speed that’s marginally slower than normal. But on some websites that parsing time was amplified which resulted in slower than average loading times.

Incompatibility: Loop11 was unable to render secure websites running on https.

If you were one of the people who encountered these issues, we apologize for the inconvenience and we appreciate you taking the time to share your feedback with us. Thanks to our super-star engineers those problems have been tackled head on and wrangled to the ground. What does it all mean? Simply put Loop11 is now snag-free!

Our technology update means your website will now work with Loop11 seamlessly. Inaccurate display? Fixed! Slow performance speed? Solved! Incompatibility with secure websites? Say no more… it’s a thing of the past!

We’ve heard from many of you just how invaluable Loop11 has been in improving your websites. Which is why we’re excited to announce the enhanced compatibility of Loop11. So in keeping with our mission to offer usability testing that’s easy-to-use, we’d like to invite you to try Loop11 again with a second free trial. We believe you won’t be disappointed.

To take up the offer of a second free trial email us at to request another one.

If you have any more feedback about Loop11, please let us know. It is your feedback that is driving changes and our continued improvement.

Sincerely yours and happy testing,

Team Loop11

10 Tips For Running Usability Benchmark Tests

With the emergence of tools like Loop11, conducting usability benchmark tests is now more common than ever. Benchmark studies are a great way to see how design or content changes affect website usability. Since benchmark tests are all about tracking usability over time, it’s important to be consistent with the process. With this in mind, Jeff Sauro at has shared his “10 Tips For Benchmark Usability Test”. Here’s a quick rundown:

  1. 1. Recruit for representativeness over randomness
  2. 2. Triangulate using multiple metrics
  3. 3. Estimate Sample Size using the desired margin of error
  4. 4. Counterbalance tasks
  5. 5. Collect both Post-Test and Post-Task Satisfaction
  6. 6. Combine measures into a Single Usability Metric for reporting
  7. 7. Use confidence intervals around all your metrics
  8. 8. Conduct a pilot test
  9. 9. Include some cheater/speeder detection for remote usability tests
  10. 10. When you record task time don’t throw away the failed task-times

To view the full and detailed list, click here to read “10 Tips For Benchmark Usability Test” on the blog.

Happy Testing

High Usability, Low Cost

A recent blog post by Jakob Nielsen, describes how to achieve high UI usability through an in-depth user testing process involving three different methods:

1.      Iterative design

2.      Parallel design

3.      Competitive testing

It is a great and thorough guide to creating and testing designs for high website usability. However, the kind of “test early, test often” approach described here and advocated by usability professionals world-wide is prohibitively expensive – even for large corporate entities. For example, the Nielsen Norman Group currently charges $45,000 for most competitive testing, which is likely to be out reach for most organisations. And that’s before you’ve even started your parallel and iterative design testing.

In this article we wanted to explain how you can run the same kind of testing described by Jakob Nielsen at a fraction of the cost by running online usability testing with Loop11.

Before we get started, it is important to know that there are benefits and drawbacks of both online and lab-based (or moderated) testing. This article is essential reading for anyone about to embark on any form of usability testing.

Iterative Testing with Loop11:

Iterative testing is about testing an initial design, making changes to it and testing it again. The process is repeated several times.

To do this with Loop11, simply create and initial website design, preferably beginning with lo-fidelity wireframes and run your online study with Loop11. Analyse the data and make appropriate changes to the design and test them again with Loop11. This process should be repeated until you have a final design (not wireframes).

Cost: $350 per iteration.

Parallel Design Testing with Loop11:

In parallel design testing, users are asked to test several variations of the same design at the same time. The best areas of each design are then merged into a single design.

Doing this with Loop11 is easy. You’ll need to prepare at least two different designs, run your online study in which you ask participants to complete the same task on multiple designs.
In parallel design testing it is important that each participant only performs tasks on one design otherwise there will be a learning bias.

Cost: $350 per design.

Competitive Testing:

Competitive testing involves comparing your design with your competitors. Doing this with Loop11 is extremely easy and powerful. Simply create a project and ask users to complete the same (or similar) tasks across several websites (yours and your competitors). Since Loop11 requires no downloading or no coding you will be able to test your competitors’ live sites.

Cost: $350.>

So why spend tens of thousands when you can do all of this testing for just a couple of thousand dollars.

Happy Testing!

Top 10 Usability Research Findings of 2010

With the evolution of usability testing and cutting edge tools like Loop11 creating a new wave of research techniques and methodologies, usability professionals are continuously discovering new ways of conducting research, thus finding new insights into usability testing.

So we set out to compile a list of the top ten usability research findings for 2010. Thankfully, Jeff Sauro at saved us a lot of time by already putting together a list of Top 10 Research-Based Usability Findings of 2010. Some of the major findings of 2010 were based around remote and unmoderated testing as well as online surveys and self-reporting. There are a few interesting game-changing surprises on the list, especially at number one. What do you think made it to number one? Check out the top ten list here.

How To Catch A Cheater When Dealing With Large Sample Sizes.

When online user testing, and dealing with large sample sizes , the higher the number of participants, the higher the likelihood of inaccurate data collection. The main culprits for bad data are those participants who are just doing it for the money and not taking the test seriously.

How can we identify these types of cheaters, and what kind of quality control methods can be used to make sure the data is accurate?

Jeff Sauro at Measuring Usability has written a great piece on how to catch cheaters , with some fascinating insights and statistics about cheaters. It also describes which measures can be taken minimize the damage caused by cheaters . The full article can be found here. We highly recommend it.

When two great minds get together!

Wireframe design and testing just got a whole lot easier…

Testing the usability of wireframes has always been a great Loop11 feature. However Loop11 is not a wireframe design tool. On the other hand, Justinmind is a great wireframe design tool, but it didn’t offer comprehensive wireframe usability testing. So we put two and two together and integrated Justinmind with Loop11.

Now you can create your own wireframes with Justinmind and test the usability of them with Loop11. It’s as simple as a click of the mouse!

How does it work?

Justinmind is a wireframe creation and design tool. It also allows you to upload your wireframes into HTML and test them.

This is where Loop11 comes in.

Once you have created your wireframes with the Justinmind wireframe creation tool. Simply use the Justinmind Usernote feature. Their Usernote feature is where you can upload your wireframes into HTML so that they can be tested.

This is where you will find the Loop11 integration… and we all know how simple Loop11 is to use.

So if you’re looking for a simple way to design and test wireframes, look no further.

Happy testing!

« Previous PageNext Page »
Want more inspiration?
Join the Fab-UX 5 newsletter!

Five links to amazing UX articles,sent to you once a week.

No SPAM, just pure UX gold!

No Thanks