Empower Your Team: Data-driven Design Decisions Made Easy

Laboratory-based usability testing has been around for many years. The effectiveness of lab-based testing in uncovering usability problems with websites and applications is widely accepted. However, in recent years the influx of remote usability testing tools has started to change the way usability testing is conducted. One reason for this has been the cost associated with traditional lab-based testing, which can become very expensive. Other reasons have centered on the shortcomings of lab testing, in particular the limited number of participant commonly used.

Here are seven important things that you can do with remote user testing that you can’t do with lab based testing.

1. Test early, test often… affordably

Remote testing is much more cost effective than lab based testing, which means you can test at every stage of the development lifecycle, from wireframes to a live website. Or conduct benchmarking studies on a weekly or monthly basis to see how changes to your website affect usability. Lab-based testing can cost tens of thousands of dollars, which usually results in lab-based testing not being run very often by most companies. Remote testing tools usually cost just a few hundred dollars, making it easy to test on a much more frequent basis, thus creating the opportunity to run more regular studies.

2. True Intent Studies

True intent studies focus around a user’s real aim when using a website. What is their goal on your website, how do they go about it, and did they achieve their goal? These are the main objectives. True intent studies can only be run online because you must intercept users as they visit your website so you can observe their natural behaviour.

3. Get large numbers of “hard to get” users in their own environment

You can test hundreds of people simultaneously, in their natural environment. Whether you need a doctor or French speaker in China, online panels let you tap in to the thousands of unique demographics out there.

4. Get more in-house people involved

Remote usability testing is a great way of introducing usability testing into a company using limited resources and budget. Typically, most companies only involve a few people in lab-based usability studies. Remote online testing enables other parties to reap the benefits and even get involved. Quantitative metrics will show your IT guys why changes are needed to your website, and the finance guys will have the stats they need to understand why money needs to be spent.

5. Get statistically significant data

Lab-based usability testing is all about identification of usability issues and qualitative reasoning, not about quantification. Due to the small number of participants in lab testing (usually 5-10) compared to the hundreds or even thousands in remote testing, one simply can’t generate the quantitative reports that high level company decision makers may want to see. However, with remote user testing you can quash any in-house differences of opinion by getting the hard facts about your website’s user experience.

6. Special technology

Some interfaces just don’t make any sense to test outside their intended usage environment. Maybe you are running a user test on Facebook. The test involves uploading personal work with personal email contacts, pictures, videos and more. For this, you need the users to have their own email contacts, photos, videos and more. Having them bring their laptop or media to a lab isn’t going to do the study justice. Remote testing would allow them to use their own equipment at their own place…saving time, money and improving the quality of the results.

Or, let’s say you’re testing a recipe website that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment. We also call this the participant’s “technological ecosystem” because it implies that their devices and computing environment have an impact on how they interact.

7. Online accessibility testing

Now, let’s say you’re still testing that recipe website, except this time you need to test it for accessibility. Meaning you need to determine if there are any barriers that might prevent people with unique physical or mental needs from using the step-by-step recipe website in their own homes. It would be virtually impossible to replicate the technology and assistive devices of users with special needs, so the best way to do this is online.

Generally, A/B testing is for testing a single page of a website or even a single element on a single page of a website. In contrast, user testing is designed to test entire website and the whole journey that a user takes to complete the goal they had when they first visited the website.

You sure can! Because we do not require you to insert code into your website you can run user tests on any website, including your competitors. It’s the perfect way to run benchmarking studies!

There are many places you can find participants for your user testing. The easiest – and cheapest – place to find them is via social media. If you have a Twitter, Facebook, LinkedIn or other social media account you can simply send your Loop¹¹ project URL out to all your followers and fans asking for their feedback. Similarly, if you have a database of members you can email them directly.

Alternatively, you can create a pop-up invitation in Loop¹¹ to intercept your current website users and ask them to participate in your study.

Finally, if your project requires having participants that meet certain demographic criteria then there are many market research panels that will source participants to meet your exact requirements. Usually these panels charge a small fee for locating your participants. The following companies can provide participants for your projects:

• TestingTime
• Respondent.io
• UserInterviews.com (New accounts get 3 free participants)
• Market Tools
• Pure Profile

Usability testing is about watching users do tasks which should simulate actual usage of your website. An initial usability test should focus on core functionality – areas that will see the most usage. If you are tasked with testing a specific piece of functionality, then naturally your test would need to focus on that area.

For example, if you were testing a rental car website, the core tasks could be: renting a car, renting a car with additional options, finding the address of a rental location and finding the opening hours of a rental location on a Tuesday morning.

Sometimes you have to test more than the core tasks, even if it is the initial usability test. As a balance you can have some core tasks tested first and if there is still time during a test session, test a set of peripheral tasks that include the fringe functionality.

The moment a participant completes an evaluation, the responses can be viewed and reports are generated. Our reports make it easy for you to see how well each task is performed by your participants.

• Understand what percentage of participants have difficulty navigating your website
• See where participants are failing your most important tasks
• Dive into individual responses to view details of a particular participant or to read the comments of open-ended questions.
• View heatmaps showing you where participants click on an individual page
• Generate clickstream data to see how participants really navigate your website.

Conducting monthly benchmarking of your own website is a great way to ensure the usability of your website remains in good standing. Running regular benchmarking studies is an easy as setting up the key tasks you want to observe and copying the entire project each month.

1. Determine what you want to test: What parts of the application and what functions.
2. Why are you testing?
   
   • Do you want a benchmark of usability and performance?
   • Are you trying to find and fix as many usability problems as possible?
   • Are you comparing an application with a competing product?
3. Write a minimum of 5-10 task scenarios that have users attempting realistic tasks in which they interact with the parts of the application you want to test.
4. Recruit a set of users to test. At a minimum aim for between 50 and 100 in total. For goal 2b, test a subset of your users as part of an iterative testing approach.
5. Launch your project.
6. Analyze the results.

Qualitative usability testing has traditionally been based around small sample sizes of 5-20 participants. However, the growth of online testing tools and quantitative usability research is changing the game. Whist many experts agreed that for qualitative, lab-based testing small samples sizes of 5-20 participants is sufficient. Online user testing has created a new wave of analysis such as benchmarking, A/B testing, competitor comparison, validating and much more. These kinds of quantitative analyses require larger numbers of participants to validate the data.

For example: Let’s say your company is testing two different versions of wireframes so management can decide and approve one to implement and allocate resources to. It would not be very good practice to use 10 participants and get a 60%-40% success rate. It would be very hard to validate a study and implement a strategy based on 10 participants. However, if 500-1000 participants were used, that data would be a lot more accurate, and management would have valid data to approve of the findings.

The maximum number of participants you can have in a Loop¹¹ project is 1,000.

If you have updated your Mac computer to macOS 10.15 Catalina or higher, you will need to update your security settings to enable Screen Recording access and Microphone access for the Loop11 browser extension recording.

If you choose to participate in tests with face recording, you'll also need to enable Camera access.

Enabling Microphone and Screen Recording permissions

1. Click the Apple logo in the top-left of your computer screen and select System Preferences from the dropdown menu.

2. Navigate to Security & Privacy in your System Preferences.

3. Click the Privacy tab.

4. Scroll down to Microphone and check the box next to Google Chrome.

5. Complete the same steps for Screen Recording.

Once you allow access to your microphone, you’ll be prompted to share your screen. Select your screen under “Your Entire Screen”, and click “Share”.