In our first case study for 2010, we thought we would have a look at how user friendly 10 of the world’s leading airline websites are. On a recent overseas trip, I was astonished to see how many people continue to take dangerous or banned items, such as scissors and cigarette lighters through the check-in gates at airports. Since security has become radically tougher in recent years we thought we’d explore how easy (or difficult!) it is to find information about the items you’re not supposed to have in your luggage. So…we put the following 10 websites through the loop:
10 Of The World's Leading Airline Websites.
The following task was asked of 1,000 participants (100 per website):
“You are taking an overseas holiday next month. Before you go you want to check whether certain items are considered by the airline to be dangerous or banned. Using the website how can you do this?”
Our participants were sourced from a number of resources, including our Twitter and Facebook accounts, but the vast majority came from Mechanical Turk where we paid the nominal sum of $30 for the bulk of the participants. Thanks to all those who got involved.
Task Completion Rates:
In general, each website had one page dedicated to banned or restricted items, such as these pages from American Airlines and British Airways.
American Airlines Restricted Items Page.
British Airways Banned Items Page
If the participants found the appropriate page they were deemed to have completed the task successfully, otherwise they were considered to have failed it, or they abandoned the task if it all became too hard.
The results indicate that finding information on dangerous and banned items is rather difficult. This perhaps provides some clues as to why so many people on my recent trip were still packing them in their luggage.
Chart showing the task completion rates
The British Airways website was the standout performer with 71% of participants completing the task successfully. Of most concern were Virgin Atlantic and Malaysia Airlines where less than half of participants were able to locate the information. In the case of Malaysia Airlines, just 31% of participants were able to complete the task.
Additionally, a total of 39% of participants abandoned the task on the Malaysia Airlines website even though more than half went directly from the home page to the Baggage Information landing page where they should have easily found the information. It would seem the call to action to “Download now” is not sufficient to indicate the PDF document on the Baggage Information landing page is the place to go for this information.
Malaysia Airlines Webpage.
Average Time to Complete Task:
The average time taken to complete the task on each of the ten websites again shows that the British Airways website was the standout performer, with participants completing the task in an average time of 87 seconds. Malaysia Airlines and Virgin Atlantic once again performed poorly, with the average time for Virgin Atlantic (199 seconds) being more than twice the time taken for those using the British Airways website.
Chart showing the average time to complete the task.
The study also revealed that only Virgin Atlantic and Lufthansa did not have fly-out menus in their main navigation. Fly-out menus, such as those shown on the American Airlines website below; often result in faster navigation since users are able to see at least the second level navigation links without having to make a click.
American Airlines Webpage With Fly-out Menu.
A deeper look at the path analysis for Virgin Atlantic shows that a quarter (24%) of participants went to the correct section of the website, the Passenger Information landing page in the first instance. This is a substantially lower result than British Airways and even Malaysia Airlines where more than half navigated to the correct section first.
Ease of Use Rating:
One of the follow-up questions for participants after completing the task was to rate on a 5-point scale how easy it was to use the website. There was much less variation in these results, which we don’t find surprising. In face-to-face, lab-based user testing we frequently encounter participants who have a terrible time navigating a website but still comment on how easy the website was to use! We always felt this was the moderator effect, but perhaps this extends to unmoderated user testing too!
Chart showing the ease of use rating
Overall Usability Score:
To directly compare the usability of one website to another we decided to follow the ISO definition of usability. ISO 9241-11 defines usability as the “Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.“ This gives us three areas to focus on: effectiveness, efficiency and satisfaction.
Combining the scores for the task completion rate (effectiveness), the average time taken to complete the task (efficiency) and the ease of use rating (satisfaction) we can establish an overall score for each of the ten airline websites, which are shown below.
Chart showing the overal usability score.
Not surprisingly, British Airways was heads and shoulders above the rest while Malaysia Airlines and Virgin Atlantic were well behind. There was little difference between the remaining seven websites. But clearly there’s a lot more work to be done by airline websites to help people avoid packing those banned and dangerous items.
Tested with Loop11
This report was generated entirely from results obtained through running a usability study with Loop11. Loop11 allows you to conduct usability studies on any website without the need to insert code. The real-time reporting provides the following quantitative usability metrics:
- Task completion rate
- Time per task
- Most common success page
- Most common fail page
- Most common first click
- Most common navigation path
- Detailed participant path analysis
- Number of page views to complete tasks
Sign up and you’ll get your first project for free.
What’s Next?
If you’ve got an idea for a case study you’d like to see us do, send the details about it to administration@Loop11.com and we’ll consider it for up-coming case studies.
Tags: Airlines, American Airlines, British Airways, user experience, website usability testing
thanks for the effort you set into sharing it with us. great post
Great article. There’s a lot of good information here, though I did want to let you know something – I am running Mac OS X with the circulating beta of Firefox, and the layout of your blog is kind of bizarre for me. I can read the articles, but the navigation doesn’t function so good.
Glad to see that this site works well on my Blackberry Bold, everything I want to do is functional. Thanks for keeping it up to date with the latest.
I was reading something else about this on another blog. Interesting. Your linear perspective on it is diametrically opposed to what I read earlier. I am still reflecting over the diverse points of view, but I’m inclined heavily toward yours. And irrespective, that’s what is so superb about advanced democracy and the marketplace of thoughts on-line.
Thanks Ziya,
We’ll have a look into this.
Toby
Thanks for publishing about this. There’s a bunch of important tech info on the internet. You’ve got a lot of that info here on your website. I’m impressed – I try to keep a couple blogs pretty current, but it’s a struggle sometimes. You’ve done a solid job with this one. How do you do it?
Great Read! Looking forward to more! Bookmarked your site and will be back again!
I agree with you on this!
Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that: Thanks for lunch!
Thanks for the Information.
thank you.i have fwded your insights to our malaysian airlines. thought they might find this useful.
You should have a look at Ryanair’s website http://www.ryanair.com
Thanks for the benchmark analysis on these sites. I suspect that some of your analysis may have been different if the audience you selected truly came from a diverse user profiles who are actual target audience rather than diverse sources. For e.g. Facebook, Twitter and Mechanical turk have a typical type of audience i.e. slightly on the techie side and not necessarily family, business travellers. This is the reason why in one of your analysis the ‘menu flyouts’ came out positively as ‘faster navigation’. Had it been a typical site user (35-55 year olds m/f? executive secretaries? etc) the analysis would have been different or even perhaps ‘these flyouts get in the way!’.
If you have carefully selected audience based on some criteria then please add it to this fantastic analysis to give the study more relevance.
Hi Vivek,
You are correct that the audience we used to complete the user tests is perhaps not representative of those who would traditionally use these websites. However, we don’t imagine our analysis of task completion rates, average time to complete tasks and ease of use ratings would alter that much even if we used the “correct” audience.
In regards to our analysis of the flyout menus, this analysis was not just based on the results of this study. As a usability consultant with over 10 years of experience, we see this type of thing regularly in lab-based studies. Whether you are a more experienced internet user (a techie, as you describe them) or a less experienced internet user flyout menus frequently result in faster navigation.
Toby
Wicked cheers, very use full information. thank you.