In this presentation, you will learn the basics of how to conduct usability testing. This presentation was delivered as a full-day workshop for DFW-UPA in December 2010.
1. You Watch Customers 2. They Perform Tasks 3. You Note Their Problems 4. Making Recommendations 5. Rinse and repeat (i.e. iterate) Jakob Nielsen’s Definition of Usability Usability is a quality attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process.
Facilitator: This person oversees the entire test process—plan, test, and report. • Participant: Actual or potential customers. Representative users (marketing, designers) should not be used. • Observer: Records events as they occur. Limits interaction with the customer. Does contribute to the report. In today’s workshop, you will play all three roles!
Customers • If your website is difficult to use, customers leave. • If they get lost in your website, customers leave. • If a customer can’t FIND your product, they can’t BUY it. …. Then, they leave! Real feedback directly from customers that use the product, which provides data—not opinions.
Design A Design B We encountered these results from a Usability Test: • Average Time to Book a Limo with Design A = 3:59 • Average Time to Book a Limo with Design B = 8:00 • 12 of 12 customers successfully booked the correct limo with Design A. • 7 of 12 customers successfully booked the correct limo with Design B. • All users were TRAINED and currently used Design B.
Rewarded) For years, companies have been performing usability testing. They just call it a product release. Teams that do usability testing before a release mitigate risk. Plus, they are rewarded. (ex: $300 Million Button)
Money # of users X # of uses per day X # of days per year X Average hourly wage of user X Increase efficiency (in hours) Annual Savings in Productivity • Randy Bias and Deborah Mayhew’s book provides hundreds of formulas to show the return on investment for usability testing. • Here is one formula: Using the previous example, a switch to Design B saves over $2 million USD.
Money Jared Spool The $300,000,000 Fix (per Jared Spool): In the above example, we have a form with two fields, two buttons, and one link. This form appeared on the bottom of a Shopping Cart. What are the problems? Hint: Current and New Customers both had problems.
Money Jared Spool New Customers: New customers did not want to register. They resisted clicking the Register button. New customers wanted to buy what was in their shopping cart. They did not want a relationship. Most wanted to leave. As it turns out, customers are not required to register—it is optional.
Money Jared Spool Existing Customers: Existing customers couldn’t remember their exact email and/or password. They clicked the ―Forgot Password‖ link. Help Desk logs revealed this link was clicked about 167,000 each day! According to Spool, 75% of these people didn’t make a purchase. They did leave!
Money Jared Spool Jared Spool’s $300,000,000 Fix: Make it obvious. The results: The number of customers purchasing went up by 45%. The extra purchases resulted in an extra $15,000,000 the first month. For the first year, the site saw an additional $300,000,000. $ $
Qualitative Usability Testing Provides opportunity to observe users as they perform tasks. Provides a focus on quantitative measures, such how many people are clicking on an ad. Based on expertise and judgment of people in the test (participant, facilitator, observer). Based primarily on click-stream data and log files of real customers Facilitator and observer(s) interprets the observed customer behavior and recommends solutions Data must be interpreted from the click-stream or log files. Data can be done with small sample sizes. Requires large sample sizes.
Quantitative Usability Testing has grew in the last decade. • Many tools exist for quantitative testing tools are out there. • Keynote Systems has been around for a long time now. • You may currently use or know people that use Google Analytics. • Chalkmark is something we will see later today. • Lou Capone will give us a demo of UserZoom (30-minute demo). Web IQ
usability testing for the web has really matured in the last 30 years. • Quantitative usability testing tools has really matured in the last 10 years. • Morae and OvoStudios both offer mature solutions for usability professionals. • IntuitionHQ is very new to the market—I was just made aware of them this week. • We will use Morae today.
Design Walkthrough Formative Usability Test Summative Usability Test Conducted during prototype development to validate initial design decisions, determine stregnths and weaknesses, or to compare various designs. Conducted during the design or development phases. Used to determine if a design objective is being met. Designs change throughout the test. Conducted at the end of development--code is frozen. Used to validate that a product has accomplished its intended goals. Example: Which of the two new designs best meets the customer's expectations? Which design was most preferred by the customers? Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points on an application? Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points in less than 1 minute? Today’s workshop will focus on summative usability testing.
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Step 2: Conducting Your Usability Test • Conduct a pilot test • Facilitate remaining tests Step 3: Analyzing and Reporting Your Usability Test • Compile results • Make recommendations
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Time = Money: Most usability tests are less than 90 minutes. Plus, you are taking your customer’s time. Put first things first—what needs testing!!! (Activities) (People) (Context)
Stephen Anderson Somebody (person) does something (activity) some where or some how (context). This model is Robert Bailey’s Human Performance Model. Let’s use this model to quickly plan out a test!
from Stephen Anderson • Create a group reservation • Plan an event (ex: wedding) • Update profile (ex: Passport) • Review Sailing Route • Make a second payment • Move to a different ship • Upgrade to a new cabin • Buy an excursion deal
Site….. Picture from Stephen Anderson • Captain of the Ship • Group Traveler • Senior Citizen • College Student • Potential customer • Travel Agent • Special Needs Traveler • Worker (on the Ship)
Stephen Anderson • Company website • IVR system • Kiosk (on the ship) • Printout (ex: Itinerary) • Social Media • Email/text messaging • Voice Mail • TV, Radio, DVR • Inside or outside • Public or private • Noisy or quiet • Legal regulations • Social attitudes • Company goals
Have? Ask the project team to list out their overall concerns with their product. Let’s go back to the Royal Caribbean International. Overall Concerns: 1. Ease of learning 2. Ease of use 3. Ease of navigation 4. Usefulness of Help 5. Recovering for errors Specific Concerns: 1. Booking Group Reservations 2. Splitting Group Payments 3. Availability of Group Cabins
the project team. They tend to focus on a single user. Frame It This Way: 1. Who are primary customers? (Define their characteristics.) 2. Is there another group? (How are they like 1st group?) (How are thy different?) 3. Is there another group? (How are they like both groups?) (How are they different?) Travel Agent Event Planner
to Use It? Ask the team. Usually, the team focuses on the delivery device. They will need help with the other contextual factors. Frame It This Way: 1. What device(s) is it on? 2. Is inside or outside? 3. Is it public or private? 4. Is it work or leisure-related? 5. Is it noisy or quiet? 6. Any legal regulations?
Performance Data: • Average Task Completion • Average Time on Task • # of clicks to Help • # of omitted steps • # of errors made • # of hints Subjective Data: • Prefer Design A or B • Design Suggestions • Task Ease Rating • Satisfaction Rating • User Comments • Ranking Features Performance and subjective data are equally valuable. We will come back to data collection in a few minutes.
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks (Activities) (People) (Context)
(circa 1993) Five users will uncover about 85% of the usability issues of a product. Virzi backs up this claim around the same time. Jared Spool (circa 2000) Five users revealed only 35% of the usability issues of a product. This claim is backed up by Schroeder. Laura Faulkner (2003) - Five users uncovered 85% of usability issues on average. - But, the percent ranged from 55-100% (confidence interval). - Ten users uncovered 95% with a lower bound of 82%.
Rules: • Test with 6 customers (per group)—if time permits, use 8 customers. • More customers = more reliable results • More customers = fewer new results • For larger samples, do multiple rounds of smaller tests (rather than 1 large UT) Base Your Number On: • Importance of project • Complexity of the product • Experience of testing team • Expected variety of user reactions
Recruiting agency Incentives: • Gift checks ($100 per session) • Food or gift cards Scheduling: • Slot times • Breaks between participants • Backups, if needed • Floaters, if needed
the Screener Steve Krug has said this: I respectfully disagree. Because… • Your test is a waste of time, if you recruit the wrong people. • Your usability test is dangerous, if you act upon recommendations from a bad recruiting sample. You must screen participants. For recruiting, grade on a curve! Grab a neighbor.
Step1: Planning Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks • People = Usability Participants • Context = Test Scenarios • Activities = Usability Tasks
tasks and scenarios: Scenarios: • Provide the participant with motivation and context • Sets the stage or tells a story • Increases understanding • Makes the situation more realistic • Includes tasks Tasks: • Quick and to the point • Do not need context to increase understanding
You want to book a sailing on Royal Caribbean International for next June with your church group. The group is called ―Saint Francis Summer 2010‖. The group is selling out fast, so you want to book a cabin, which is close to an elevator because your leg hurts from a recent injury. • Open your browser • Click the link labeled ―Royal Caribbean‖ • Tell me the available cabins in the ―Saint Francis Summer 2010‖ group • Tell me a cabin number closest to an elevator • Book the cabin the best suits your needs
have two teams today. --Team Cool --Team Ninja Assassins • Pair up with someone. • Write a scenario with tasks. • We will pass out papers of some sample sites. • You will use your script later today to run some tests.
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Step 2: Conducting Your Usability Test • Conduct a pilot test • Facilitate remaining tests
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Step 2: Conducting Your Usability Test • Conduct a pilot test • Facilitate remaining tests
a complete usability test • Include all test material • Focus on the test scenarios, not findings • At least, one participant should be a pilot • Conduct before test begins (one day prior) • Make changes, if needed • May need to test, again
Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Step 2: Conducting Your Usability Test • Conduct a pilot test • Facilitate remaining tests
Know Your UCD Basics and Design Guidelines 2. Be a Quick Learner 3. Set an Instant Rapport with Participants 4. Have an Excellent Memory 5. Be a Good Listener (Use Active Listening) 6. Don’t Stress Out About Ambiguity 7. Be Very Flexible 8. Have a Long Attention Span 9. Show Empathy (ie Be a People Person) 10. Thing “Big Picture” 11. Be a Good Communicator 12. Be a Good Organizer and Coordinator
• Start with an Easy Task To Build Confidence • Choose the Right Format (remote, on-site UT) • Sit Beside the Person Not Behind the Glass • Use “Think-Out-Loud” Protocol • Give Them Time to Think It Through • Offer Appropriate Encouragement • Lead Participants, Don’t Answer Question (Being an Enabler) • Don’t Act Knowledgeable (Treat Them As the Experts) • Don’t Get Too Involved in Data Collection • Don’t Jump to Conclusions • Don’t Solve Their Problems Immediately (ie act as a Help Desk)
Style Wizard of Oz Style Testing: • Moderator • Tester • Wizard (Person moves screen shots) @suredoc (Keith Anderson) is your guide today. NOTE: Keith is NOT the WWW – Wicked Witch of the West.
Oz….. Here are The Rules: 1. Get into Groups of three. 2. Each person picks a role. 3. Wizard does not talk. 4. Moderator analyzes Tester 5. Tester talks through the task. 6. Complete the Test. 7. Switch Roles. 8. Find two new people. 9. Repeat Steps 1-6. 10. Compile results via K-J Method (well do that later, though).
Lab These are the Test Stations: 1. Two Waiting Participants 2. One Tester in Test Cell 3. Moderator 4. Test Cell Observer 5. Camera Operator 6. TV Operator 7. Remote Viewer Logger 8. Remote Viewer Manager 9. Results Compiler (2 people) 10. Two Observers 11. Rotate between each station We’ll test four different sites today.
Planning Your Usability Test • Define what you are testing • Define which customers you are testing • Define what tasks you are testing • Recruit your customers • Write your usability scenarios and tasks Step 2: Conducting Your Usability Test • Conduct a pilot test • Facilitate remaining tests Step 3: Analyzing and Reporting Your Usability Test • Compile results • Make recommendations
Findings 1. Complete Your Notes 2. Summarize Your Findings 3. Compile and Tabulate Data 4. Analyze Your Findings 5. Prioritize Findings by Severity Level 6. Create Recommendations 7. Develop a Final Document or Presentation 8. Follow-up
session, fill in any gaps. • Basically, complete the short-hand used in the UT. • Mark any areas worth noting. • Use a whiteboard! It is really much easier, in the end.
create a list of the major issues. • Organize by Issue and Participant. • Do Not Start Solution Finding Here! • Below is an Example of How to Summarize: Number Issue Description Customers 1 Unable to locate Group for RCI Sailing from Home Page P1, P2, P3, P4 2 Credit card payment summary incomplete (no SEC) P2, P6 3 Rate Category page does not have sort capability P1, P5, P7 4 Bad error message on Cabin Swap page. P3, P6
simple? • Which tasks were problematic? • Which participants had issues? (Look at it by group.) • Speculate on solutions. • Use Balsamiq to sketch out recommendations. Sketched in Balsamiq in less than 5 minutes.
Showstopper. Unless corrected all users will essentially have a problem. • Severity 2 = High Impact. Significant impact to usability or performance. High customer annoyance and frustration. • Severity 3 = Medium Impact. Some users experience degraded performance or frustration. Help Desk calls possible. • Severity 4 = Low Impact. Probably will not have a significant impact to user performance. Might be cosmetic in nature.
Findings • Remind the team of people-activities-context • Be a problem solver • Present Issues by Severity Level (helps to prioritize) • It great to do a MOCKUP here • Tie recommendation back to USER PERSONAS, too
Consider the audience • Some teams just want an Excel spreadsheet • Provide enough detail (executive summary v. full report) • Create a distribution list for key stakeholders • Purpose (identify issues, prioritize them, make recommendations)
• List all observations on stickies • Post them on a wall • Cluster into groups • Name the group • Vote on each group • If possible, make recommendations
• Write issues on board after each participant (positive and negative) • Place checkmark when the issue is found by another user • Rank issues at the end of the test • If desired, we can think about solutions.
1bigdesign
solmesz1
_limex_
yukiringo
okazakityo
root_recruit
chatwork_hr
abcmisuzu
medley
xxxkinu
transit_kix
degudegu2510
ajstarks
destraynor
addyosmani
sachag
hannesfritz
rmw
dougneiner
rasmusluckow
sugarenia
cromwellryan
zakiwarfel
brad_frost
samlambert
