UXINDIA15- Incorporating Quantitative Research Techniques into your UX Toolkit ( Bill Albert ) by uxindia

The goal of this workshop is to introduce a variety of quantitative UX research methods that are often overlooked during the user-centered design process. Most UX researchers rely heavily on qualitative methods. However, quantitative techniques offer additional insights and provide the necessary data to make the right decision and business decisions. The workshop will review how to collect, analyze, and present the most popular UX metrics. The workshop will also introduce some lesser known, but highly effective UX metrics. The workshop will also touch on quantitative techniques such as open/closed card sorting, surveys, unmoderated usability testing, and first click testing. The workshop will conclude with a discussion on how to integrate these quantitative techniques into your design process. Together, these quantitative techniques will expand your UX toolkit and make you a better-rounded UX researcher.

Transcript

2. INCORPORATING QUANTITATIVE TECHNIQUES INTO YOUR UX TOOLKIT Bill Albert, PhD Executive Director Bentley University User Experience Center 5 October 2015
3. My Background 2
4. Agenda • Why UX quantitative methods matter • Overview of quantitative methods/tools • Discovery • Design/Evaluation • Validation • Data collection tips • Data analysis tips 3
5. Why UX Quantitative Methods Matter 4
6. UX Research Framework 5 Qualitative Quantitative Attitudes Behaviors What is the problem and how big is it? What is the problem, why is it a problem and how to fix it? What are people doing? What are people saying?
7. UX Research Questions 6 UX Research Question Qualitative Quantitative How does our app compare to the competition? No Yes What are the biggest pain points in the design? Yes Yes Has the design improved over time? No Yes Why do our users struggle with the website? Yes No How do our users like the design? Yes Yes What is the experience like for different user groups? Yes Yes Which design color do user like more? No Yes Is the design intuitive? Yes Yes
8. UX Research Questions 7 UX Research Question Attitudes Behavior Do we have the right content? Yes No Do our users understand how to navigate? No Yes Does the terminology make sense? No Yes Do the users like the (visual) look of the application? Yes No Is the workflow intuitive? No Yes Which of the three websites looks the most professional? Yes No What areas of the website are most confusing? No Yes
9. Focus Attention on Things that Matter 8 • Abandonment • Inefficiency • Errors
10. Avoid Problems 9 • Wide-spread customer dissatisfaction • Difficult to learn new system • High costs to remedy fixes
11. Competitive Advantage 10 • Promote a better user experience • Identify and remedy problem areas • Identify new opportunities
12. Track Improvements 11 • Measure against company goals • Determine organizational priorities • Reward success
13. Magnitude of Issues 12 • Identification of an issue is not enough – need to measure the size of the issue • Prioritize design improvements • Identify new design opportunities
14. Convince Management 13 • Move away from opinion • Prioritize design changes • Impact on business goals and customer loyalty
15. Discussion 14 • What is your experience with quantitative methods? • Success stories? • How do you balance qualitative and quantitative methods? • Do your organizations value quantitative methods?
16. Overview of Quantitative UX Methods 15
17. Five Basic Questions 16 • What question do you want to answer? • Where are you in the design process? • What is the state (fidelity) of the product? • What are your budget/time constraints? • What will you do with the information?
18. Common UX Methods 17 Qualitative Quantitative Attitudes Behaviors Usability Lab Testing (formative) Ethnographic Observation Diary Studies Web Analytics A/B Testing Eye Tracking Physiological Click/Mouse Contextual Inquiry/In- Depth Interviews Focus Groups Online Surveys Card Sorting/IA VOC Data Mining Unmoderated Usability
19. User-Centered Design Process 18 Discovery Design Evaluation Design Validate
20. Basic Categories of UX Metrics 19 • Performance Metrics • Self-reported Metrics • Issue-Based Metrics • Physiological Metrics • Comparative and Combined Metrics
21. Discovery Methods 20
22. Discovery: Objectives 21 • Who are the users/customers? • What is their current experience? • What do users want? • What is the competition doing?
23. Surveys 22 • Analyze current user experience – pain points, likes, unmet needs, etc. • Use metrics to determine new functionality and content, drive design priorities
24. Baseline Measures 23 • Collect baseline metrics on current product prior to design • Use metrics to determine design priorities • Metrics used to track improvements Task Completion Rate per Person 8 7 6 5 4 3 2 1 0 <=50% 51%- 60% 61%- 70% 71%- 80% 81%- 90% 91%- 100% Frequenc y Original Redesign
25. Personas 24 • Factor, Discriminate, and Cluster Analysis • Validate personas • Marketing support
26. Case Study: Brand Perception • Client wanted to know how their users would perceive their brand if the color of their products changed • Would the color interfere with their work? • Would they be willing to pay more for certain colors? • Would they think about their brand differently? • How does the product perception change based on region and role? • Bentley UXC research in 2014 using online surveys and images (prototype) • Research conducted in US, Europe, and China markets 25
27. Case Study: Brand Perception 26 Fresh 164 Innovative 140 Attractive 137 Creative 128 Appealing 112 Energetic 103 Friendly 87 Optimistic 84 Cutting edge 81 Warm 80 China Novel 49 Attractive 44 Healthy 44 Innovative 44 Fresh 41 Creative 40 Warm 40 Energetic 36 Optimistic 32 Appealing 31 Cutting edge 31 Europe Fresh 64 Attractive 52 Creative 47 Innovative 47 Friendly 42 Optimistic 36 Inspiring 27 Appealing 26 Energetic 24 Powerful 23 Stimulating 23 US Fresh 59 Appealing 55 Innovative 49 Energetic 43 Attractive 41 Creative 41 Cutting edge 40 Fun 36 Stimulating 34 Friendly 30 Uplifting 29 Overall
28. Case Study: Brand Perception 27 How appropriate is this color in your work environment?
29. Case Study: Brand Perception 28 Overall Role Role in Purchasing Process Bottom 2 Top 2 # Responses Mean US Clinician Equipment User (no purchasing) 10% 38% 39 3.28 Decisive Role 18% 36% 11 3.27 Influential Role 20% 32% 25 3.12 Total Clinical Role 15% 36% 75 3.23 Administrator Equipment User (no purchasing) 20% 40% 5 3.20 Decisive Role 27% 27% 11 2.91 Influential Role 12% 53% 17 3.41 Total Administrator 18% 42% 33 3.21 US Total 16% 38% 108 3.22 Europe Clinician Equipment User (no purchasing) 36% 29% 14 2.86 I have a decisiverole 28% 46% 57 3.18 I have an influential role 44% 21% 34 2.79 Europe Total 34% 35% 105 3.01 China Clinician Equipment User (no purchasing) 33% 33% 3 3.00 Decisive Role 20% 80% 5 3.60 Influential Role 3% 63% 35 3.66 Total Clinical Role 7% 63% 43 3.60 Administrator Decisive Role 0% 100% 16 4.31 Influential Role 2% 84% 45 4.13 Total Administrator 2% 89% 61 4.18 China Total 4% 78% 104 3.94
30. Case Study: Brand Perception • 82% in the US, and more than 90% in Europe and China liked the use of color • 83% US, 81% Europe, 86% China considered visual attractiveness when making a purchase • Blue and green were considered the most appropriate colors, as well as most preferred (for all three regions), while yellow and orange are the least preferred and least appropriate • Participants in China are most willing to pay more for equipment that is visually attractive, including the use of color (78%), followed by US participants (38%) and European participants (35%) 29
31. Discussion 30 • What quantitative techniques do you use in discovery phase? • Success stories? • What are some of the strengths/limitations of each technique?
32. Design/Evaluation Methods 31
33. Design/Evaluation: Objectives 32 • What is the right terminology? • Which design treatment is most effective? • What is the most intuitive information architecture? • What are the pain points in the experience? • Which design treatment is most effective?
34. Design Preferences 33
35. Card Sorting (Open) 34 • Understand how user categorize information – drive information architecture • Cluster analysis to identify categories (items occurring together)
36. Tree Tests 35 • Test the intuitiveness of an information architecture • Closed card sorts test how consistent the categories are selected • Tree tests look at % correct path, directness, and speed
37. Case Study: Information Architecture • Work carried out in 2013 (Bentley UXC) for large retailer • How do users find products in a complex ecommerce website? • How does our client’s site compare to their main competition? • What products are easy/difficult to find on each site? • How does the client site improve over time? • Tree Test using Treejack (www.treejack.com) 36
38. Case Study: Information Architecture 37 Client Site Competitor 1 Site Competitor 2 Site
39. Case Study: Information Architecture 38 Client Site Competitor 1 Site Competitor 2 Site
40. Usability Testing 39 • Performance (success, time, errors) • Self-reported (ease of use, confidence, SUS, NPS) • Physiological (eye tracking)
41. Expectation Metrics 40 • Collect expectations prior to task, collect experience post- task • Map averages by task – use data to drive design priorities
42. Case Study: e-Commerce 41 Overall donations had increased by 50%, and recurring donations increased from 2, up to 19 (a 6,715% increase!) Participant Task 1 Task 2 Task 3 Task 7 Task 8 P1 Missing Missing Missing Missing Missing P2 16 70 250 Missing 9.00 P3 56 10 60 119.00 118.00 P4 59 62 236 111.00 Missing P5 120 20 108 90.00 110.00 P6 Failure 120 322 Missing Missing P7 Failure 86 117 83.00 120.00 P8 Missing Missing Missing Missing Missing Average 63 61 182 101 89 St. Deviation 43 41 102 17 54
43. Discussion 42 • What quantitative techniques do you use as part of your design/evaluation phase? • Success stories? • What are some of the strengths/limitations of the different techniques?
44. Validation Methods 43
45. Validation: Objectives 44 • Does the design meet it’s target goals? • Which design treatment is most effective? • How does a newly launched design compare to the competition?
46. Unmoderated Tools 45 • Collect qualitative and quantitative data • UX metrics such as task success, time, paths, pages • Self-reported metrics post-task and post-session • Open-ended verbatim and video replay
47. Moderated vs. Unmoderated 46 Moderated Unmoderated Greater insight into “why” Less insight into “why” Limited sample size Nearly unlimited sample size Data Collection is time consuming Data Collection is quick Better control of participant engagement Less control of the participant engagement
48. Scorecards 47
49. A/B Testing 48 http://www.abtests.com/test/56002/landing-for-revahealth-com
50. Case Study: Usability Benchmark 49 • Research from 2010, presented at CHEST • Client wanted to benchmark their product (Genuair) against the competition • How does Genuair compare in terms of ease of use and satisfaction? • 48 participants all have COPD http://journal.publications.chestnet.org/article.aspx?articleID=1087181
51. Case Study: Usability Benchmark 50
52. Case Study: Usability Benchmark 51
53. Case Study: Usability Benchmark 52
54. Discussion 53 • What quantitative techniques do you use in validation? phase? • Success stories? • What are some of the strengths/limitations of each technique?
55. Data Collection Tips 54
56. Online Tools 55
57. Choosing the Right Metrics 56 Study Goal Success Time Errors Efficiency Learnability Issus-Based Self-Report Physiological Combined Live Site Card Sorting/IA Completing a transaction X X X X X Comparing products X X X X Frequent use of the same product X X X X X Evaluating navigation or information architecture X X X X Increasing awareness X X X Problem discovery X X Maximizing usability for a critical product X X X Creating an overall positive user experience X X Evaluating impact of subtle changes X Comparing alternative designs X X X X X
58. Finding Participants • Recruiters • Panel companies • Internal (company) lists • Friends/families • Websites • My recommendation: Pay for the recruit! 57
59. Sample Size • Depends on: • Goals of the study • How much error you are willing to except • How much variation there is in the population • My general rule of thumb • Target 100 participants per distinct user type (novices vs. experts) • Margin of error around +/- 5% • Above 500 is generally not useful 58
60. Survey Recommendations 59 • Keep it short! • Screen participants carefully • Pilot the study – make sure it is usable
61. Tips in Self-Reported Metrics 60 • Use speed traps and consistency checks • Collect post-task and post-session data • Use Likert scales more than semantic differential
62. Tips in Measuring Task Success 61 • Task must have a clear end-state • Task answers cannot be guessed • Choose distractor answers carefully • Confirm there are no other acceptable answers
63. Tips in Measuring Completion Times 62 • Use reminders and warnings if answering too quickly or no activity after a few minutes • Do not use think-aloud protocol (use RTA) • Allow users to “give up” if they can’t complete the task • Decide when you will stop the timer (moderated only)
64. Tips in Eye Tracking 63 • Decide if you want metrics and/or visualizations • The artifact you use will impact the analysis • Control for exposure time
65. Tips in Measuring Engagement 64 • Define engagement – interest? Usefulness? Likelihood to use in the future? • Use combination of behavior and self-reported, and ideally physiological metrics • Look at comparative products/designs
66. Data Analysis Tips 65
67. Task Success Analysis 66 • Binary Success (success/failure) is most common • Look by task and compare user groups • Look at confidence intervals! • Aggregate across tasks for overall performance
68. Task Success Analysis 67 • Determine thresholds to see the big picture • Analyze success by experience • Analyze based on how the task was accomplished
69. Task Completion Time Analysis 68 • Only analyze successful tasks • Visualize time data with scatterplot • Identify and remove outliers (maximum) • Identify and remove minimum times • Report as median, not mean
70. Task Completion Time Analysis 69 • Analyze time data per task and across all tasks • Compare across designs and user groups • Statistical tests to determine significance Average Time-on-Task 160 140 120 100 80 60 40 20 0 Task 1 Task 2 Task 3 Task 4 Task 5 Second s
71. Task Completion Time Data Analysis 70 • Measure the percentage of participants who meet specific time criteria (same or different criteria for each task) • Measure percentage of participants who completed all tasks within a specific time 50% 20% 25% 40% 45% 20% 0% 40% 60% % of Participants Under One Minute 100% 80% Task 1 Task 2 Task 3 Task 4 Task 5 % Participants
72. Self-Reported – Post-Task Analysis 71 • Use variety of metrics – ease of use, confidence, usefulness, expectations • Probe special aspects such as navigation, content, etc. • Calculate % Top 2 Box and % Bottom 2 Box • Analyze verbatim comments based on positive/negative sentiment
73. Self-Reported – Post-Session Analysis 72 • Use post-session survey such as SUS (System Usability Scale) • Consider rating statements to cover the entire experience Frequency Distribution of SUS Scores for 129 Conditions from 50 Studies 50 45 40 35 30 25 20 15 10 5 0 <=40 41-50 81-90 91-100 51-60 61-70 71-80 Average SUS Scores Frequency Percentiles: 10th 47.4 25th 56.7 50th 68.9 75th 76.7 90th 81.2 Mean 66.4 www.measuringusability.com
74. Issues-Based Analysis 73 • Metrics for issues should be classified by severity and type • Use metrics to compare designs, iterations, groups – drive design resources
75. Card Sorting (Closed) Data 74 • Compare IA’s with different # categories - examine agreement across users • Validate IA with Tree Tests – look at success speed, and directness
76. Comparative Analysis 75 • Compare to expert performance (efficiency) • Examine changes over time (learnability) 140 120 100 80 60 40 20 0 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task Time (sec) Avg User Time Avg Expert Time
77. Eye Tracking Analysis 76 • Common metrics include: Dwell time, # fixations, Revisits, and Time to first fixation, sequence, and hit ratio • Control for exposure time
78. Combining Data 77 • Combine metrics together to calculate a “UX Score” • Z-scores or “Percentages” method Overall Usability Index 4.00 3.00 2.00 1.00 0.00 -1.000 -2.00 -3.00 -4.00 -5.00 1 0 20 30 4 0 5 0 6 0 70 80 9 0 Age (in years) Performance Z-Score Study 1 Linear (Study 1) Study 2 Linear (Study 2)
79. Thank You! Bill Albert, PhD Executive Director walbert@bentley.edu @UXMetrics Bentley Univ. User Experience Center www.bentley.edu/uxc @BentleyUXC LinkedIn Group – Bentley UXC