Validating Assumptions with 12 UX Research Methods

This presentation reviews three big questions UX research can help answer: What do people need? What do people want? Can people use it? These questions can be answered with 12 different methods: observation, interview, diary study, experience sampling, photo diary, collaging, A/B testing, Wizard of Oz, fake doors, online usability test, card sorting, and noticeability test.

1. I’M HERE TO PREVENT YOU FROM PERFECTLY EXECUTING THE WRONG PLAN.   @tsharon  
2. HI, I’M TOMER
3. VALIDATING ASSUMPTIONS WITH 12 UX RESEARCH METHODS TOMER SHARON, SR. UX RESEARCHER, GOOGLE @tsharon
4. RESOURCES •  ARTICLES BY JEREMY JOHNSON, KYLE SOUCY, & STEVE PORTIGAL •  MY EXPERIENCE
5. PERFECTLY EXECUTING THE WRONG PLAN.  
6. QUESTIONS I ASK MY TEAMS
7. WHAT DO YOU WANT TO KNOW?
8. WHY?
9. WHAT KNOWLEDGE GAPS ARE YOU HOPING TO FILL?
10. WHAT DECISIONS ARE YOU TRYING TO MAKE BASED ON RESEARCH RESULTS?
11. WHAT ARE YOUR ASSUMPTIONS ABOUT USERS, THAT IF PROVEN FALSE, WILL CAUSE YOU TO FAIL?
12. FROM ASSUMPTIONS TO HYPOTHESES
13. WE BELIEVE THAT [DOING THIS] FOR [THESE PEOPLE] WILL ACHIEVE [THIS OUTCOME]. WE’LL KNOW THIS IS TRUE WHEN WE SEE [THIS MARKET FEEDBACK].
14. READ MORE SERIOUSLY, THESE ARE BOOK EVERY DESIGNER MUST READ ASAP
16. USER EXPERIENCE RESEARCH
17. PROVIDING INSIGHTS INTO PRODUCT USERS, THEIR PERSPECTIVES, AND THEIR ABILITIES TO THE RIGHT PEOPLE AT THE RIGHT TIME.
18. 3 QUESTIONS UX RESEARCH CAN ANSWER
19. WHAT DO PEOPLE NEED?
20. WHAT DO PEOPLE WANT?
21. CAN PEOPLE USE IT?
22. UX RESEARCH IS ABOUT DOING THE RIGHT THING, NOT JUST DOING THINGS RIGHT
23. NEEDS > WANTS > USABILITY DOING THE RIGHT THING DOING THINGS RIGHT
24. FIRST RULE OF UX RESEARCH
25. DON’T LISTEN TO USERS
26. OBSERVE BEHAVIOR
27. I DON’T LISTEN TO USERS BECAUSE OF THE PSYCHOLOGY OF ATTITUDE & BEHAVIOR
28. HIGH, NEGATIVE RELATIONSHIP BETWEEN ATTITUDE AND BEHAVIOR
29. STUDENTS WERE ASKED ABOUT THEIR ATTITUDE TOWARD CHEATING. A FEW WEEKS LATER, THEY WERE ASKED TO GRADE THEIR OWN TESTS. THERE WAS CLOSE TO ZERO CORRELATION BETWEEN ATTITUDE TOWARD CHEATING AND ACTUAL CHEATING. [COREY, 1937]
31. WHAT DOES IT MEAN TO YOU?
32. NEVER ASK WHAT THEY NEED DO YOU THINK THIS IS A GOOD THING TO HAVE?
33. BECAUSE HUMANS SUCK AT PREDICTING THE FUTURE YET THEY ARE HAPPY TO GIVE YOU THEIR OPINION
34. INSTEAD OBSERVE WHAT THEY DO SHOW ME THE MOST FREQUENT TASK YOU TYPICALLY DO.
35. NEVER ASK WHAT IS THEIR FEEDBACK WHAT DO YOU THINK ABOUT THIS COOL “SHARE” BUTTON?
36. BECAUSE THEY’LL TELL YOU WHAT YOU WANT TO HEAR AND THEY HAVE NO IDEA WHAT THEY ARE TALKING ABOUT
37. INSTEAD WATCH THEM USE IT CAN YOU SHOW ME HOW YOU DO THAT?
38. WHAT DO PEOPLE NEED?
39. THESE METHODS WILL HELP YOU PERFECTLY EXECUTE THE RIGHT PLAN
40. OBSERVATION
41. FIELD STUDY CONTEXTUAL INQUIRY FLY-ON-THE-WALL SHADOWING ETHNOGRAPHY GUIDED TOUR
42. SHOW ME HOW YOU DO THAT
47. OBSERVATION A SMALL RESEARCH TEAM OBSERVES USERS IN THE CONTEXT OF USAGE, IN AN UNOBTRUSIVE MANNER , TO GATHER INSIGHTS THAT CANNOT OTHERWISE BE OBTAINED. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PREPARE A FIELD GUIDE •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION OF OBSERVATION •  OBTAIN PERMISSION FOR STUDY/RECORDING •  PREPARE FIELD KITS •  ASSIGN OBSERVATION ROLES •  OBSERVE •  TAKE LOTS OF NOTES AND PHOTOS ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS CONS EASY TO MISINTERPRET, TIME CONSUMING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS VIDEO RECORDING – GOPRO, FLIP, SMARTPHONE PHOTOS – SMARTPHONE APPS – EthOS, RECORDIUM (iOS) READ MORE  
48. INTERVIEW
49. CONVERSE LIKE A TALK SHOW HOST, THINK LIKE A WRITER, UNDERSTAND SUBTEXT LIKE A PSYCHIATRIST, HAVE AN EAR LIKE A MUSICIAN. - LAWRENCE GROBEL
52. INTERVIEW GATHERING INFORMATION THROUGH DIRECT DIALOGUE PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PREPARE QUESTIONS •  CHOOSE & RECRUIT PARTICIPANTS •  SET A SCHEDULE AND LOCATION •  START AND FINISH WITH EASY QUESTIONS •  LISTEN MORE THAN YOU TALK •  TAKE LOTS OF NOTES ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS, BUILDS CREDIBILITY WITH STAKEHOLDERS CONS RATIONALIZATION, IT’S HARD TO ASK THE RIGHT QUESTIONS   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS RAINBOW SPREADSHEET PEN AND PAPER READ MORE  
53. WHAT’S WORST THAN 5 ENGINEERS IN A ROOM DRINKING LATTE?  
59. DIARY STUDY
62. DIARY STUDY A GROUP OF PEOPLE KEEPS A DIARY AS THEY USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON DIARY ENTRIES •  ENCOURAGE PARTICIPANTS ANALYSIS READ DIARIES, COMPOSE HYPOTHESES, CODE AND ANALYZE TO DETERMINE USAGE PATTERNS AND COMMON ISSUES. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG FORM & SPREADSHEET VOICE MESSAGES EMAIL PRINT DIARY READ MORE  
63. EXPERIENCE SAMPLING
66. EXPERIENCE SAMPLING “REALITY MINING”, CAPTURING EXPERIENCES AT THE MOMENT OF THEIR OCCURANCE WITHIN THE CONTEXT OF EVERYDAY LIFE. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PHRASE A QUESTION •  DETERMINE SAMPLING RATE •  CHOOSE A TOOL •  DECIDE HOW TO COLLECT DATA •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON RESPONSES •  ENCOURAGE PARTICIPANTS •  BEGIN ANALYSIS EARLY ANALYSIS CLEAN THE DATA FIRST, DECIDE ON CATEGORIES, CLASSIFY, ADJUST CATEGORIES, ANALYZE AS A TEAM FIRST, THEN INDIVIDUALLY, EYEBALL THE DATA, GENERATE FREQUENCY CHARTS, IDENTIFY THEMES, PREPARE CREATIVE BRIEFS PROS DISCOVER HOW PRODUCTS INTEGRATE INTO PEOPLE’S ROUTINES IN UNANTICIPATED WAYS, CHEAP CONS RECRUITING AND MANAGING PARTICIPANTS ARE CHALLENGING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS HUNDREDS+ TOOLS FORM & SPREADSHEET EMAIL APP – PACO (ANDROID) SMS READ MORE TED TALKS: BOB EVANS & MIHALY CSIKSZENTMIHALYI  
67. PHOTO DIARY
69. PHOTO DIARY A GROUP OF PEOPLE TAKE PICTURES AS THEY USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON PHOTOS •  ENCOURAGE PARTICIPANTS ANALYSIS FLIP THROUGH PHOTOS, COMPOSE HYPOTHESES, CLASSIFY AND CATEGORIZE TO DETERMINE PATTERNS AND COMMON ISSUES. AFFINITY DIAGRAM WITH STAKEHOLDERS. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG PRIVATE SOCIAL MEDIA PAGE EMAIL READ MORE  
70. COLLAGING
75. COLLAGING A GROUP OF PEOPLE TAKE PICTURES AS THEY USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  CHOOSE & RECRUIT PARTICIPANTS •  PREPARE COLLAGE BOARD AND MATERIALS •  MAGAZINES •  SHARPIES •  SCISSORS •  ASK PARTICIPANTS TO WORK IN GROUPS OF 3 •  LEAVE THE ROOM •  AFTER 20 MINUTES COME BACK AND PROBE •  TAKE PICTURES OF COLLAGES ANALYSIS WITH PARTICIPANTS FIRST, THEN CODE AND ANALYZE WITH STAKEHOLDERS IN A WORKSHOP SETUP. PROS GETTING ANSWERS TO QUESTIONS YOU DON’T KNOW TO ASK CONS STAKEHOLDER BUY-IN IS TOUGH   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 9 (3X3) TOOLS REMOTE COLLAGING WITH DESKTOP-SHARING TOOLS READ MORE “COLLAGING”  
76. WHAT DO PEOPLE WANT?
77. THESE METHODS WILL HELP YOU EXECUTE YOUR USERS’ *PLAN
78. A/B TESTING
80. A/B TESTING CREATING MULTIPLE VERSIONS OF A SCREEN OR FEATURE AND SHOWING EACH VERSION TO A DIFFERENT SET OF USERS IN PRODUCTION TO FIND OUT WHICH VERSION PRODUCES BETTER METRICS. PLAY-BY-PLAY •  IDENTIFY A DESIGN DECISION TO EVALUATE •  CREATE CONDITION DESIGNS •  IDENTIFY METRICS TO TRACK (E.G., ENGAGEMENT, REVENUE, CONVERSION TO PAYING, REGISTRATION, RETENTION) •  DIRECT 50% OF TRAFFIC AT EACH CONDITION (IF 2 CONDITIONS) •  TRACK METRICS ANALYSIS CALCULATE DESCRIPTIVE STATISTICS PER EVALUATED METRICS AND COMPARE. PROS STATISTICALLY SIGNIFICANT DATA, REAL USAGE DATA, VALIDATES DESIGN DECISIONS, GREAT FEEDBACK LOOP CONS CAN’T EVALUATE TONS OF DESIGN DECISIONS, CAN’T TELL YOU WHY   TIMING WHEN IT’S TIME TO MAKE A DESIGN DECISION. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS GOOGLE ANALYTICS – CONTENT EXPERIMENTS OPTIMIZELY READ MORE  
81. WIZARD OF OZ
84. WIZARD OF OZ INSTEAD OF DOING ALL THE DESIGN AND ENGINEERING WORK TO BUILD THE PRODUCT FOR EVERYBODY ALL AT ONCE, DO THE WORK MANUALLY WITH A FEW PARTICIPANTS. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PART IN YOUR PRODUCT •  CHARACTERIZE THE “MACHINE” •  DEVELOP PROTOCOL (IF, THEN) •  PRACTICE! •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION FOR THE STUDY •  ASK PARTICIPANTS TO THINK OUT LOUD •  TAKE LOTS OF NOTES ANALYSIS OPPORTUNITIES FOR IMPROVEMENT, INSIGHTS ABOUT VALUE. PROS REDUCES WASTED ENGINEERING WORK CONS CAN’T ALWAYS FULLY IMMITATE A REAL PRODUCT   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS 4-12 TOOLS INTERCOM PAPER MOCKS “MACHINE” READ MORE  
85. FAKE DOORS
87. FAKE DOORS INSTEAD OF DOING ALL THE DESIGN AND ENGINEERING WORK TO BUILD A PRODUCT OR FEATURE, FAKE IT, AND DETERMINE WHAT % OF PEOPLE WOULD BE INTERESTED IN IT. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PRODUCT/FEATURE •  PRETEND IT IS THERE •  INSTRUMENT SENSORS •  TRACK SIGNALS ANALYSIS CALCULATE % OF PEOPLE WHO SHOWED INTEREST AND EVALUATE IN CONJUNCTION WITH INTERVIEW DATA. PROS REDUCES WASTED DESIGN & ENGINEERING WORK CONS DOESN’T ALWAYS WORK AS A STANDALONE STUDY   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS - READ MORE VIMEO.COM/24744647  
88. CAN PEOPLE USE IT?
89. THESE METHODS WILL HELP YOU PERFECTLY EXECUTE *YOUR PLAN
90. ONLINE USABILITY TEST
91. http://youtu.be/ZvpHsp_NJEo
92. ONLINE USABILITY TEST AN ONLINE RESEARCH TECHNIQUE DURING WHICH USERS ARE ASKED TO COMPLETE TASKS WHILE UX METRICS ARE COLLECTED BEHIND THE SCENES. PLAY-BY-PLAY •  IDENTIFY A DESIGN TO STUDY •  IDENTIFY A COMPARISON DESIGN (COMPETITOR, EARLY VERSION) •  PREPARE A SCRIPT, INCLUDING TASKS •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE PARTICIPATION ANALYSIS COMPARE METRICS, ANALYZE METRICS IN CONJUNCTION WITH QUALITATIVE RESEARCH. PROS PURE COMPARISON, EXTREMELY EFFECTIVE WITH STAKEHOLDERS CONS ONLY TELLS YOU WHAT HAPPENED, NOT WHY   TIMING START AS EARLY AS POSSIBLE, TEST AS OFTEN AS POSSIBLE. # OF PARTICIPANTS ANY NUMBER ABOVE 4 PER DESIGN WORKS. TOOLS LOOP 11 USERZOOM WEB EFFECTIVE READ MORE  
93. CARD SORTING
94. http://youtu.be/-89cj71-Vfg
96. CARD SORTING A TACTICAL RESEARCH TECHNIQUE FOR LEARNING HOW TO STRUCTURE INFORMATION IN A PRODUCT. PARTICIPANTS ARE ASKED TO PUT A SET OF ITEMS INTO LOGICAL GROUPS. PLAY-BY-PLAY •  SELECT CONTENT (CURRENT OR FUTURE) •  PREPARE CARDS (<100, + EMPTY, #) •  SELECT A TOOL •  DETERMINE IF OPEN OR CLOSED SORT •  DETERMINE IF INDIVIDUAL OR GROUP WORK •  DETERMINE IF IN-PERSON OR REMOTE •  BRIEF PARTICIPANTS •  OBSERVE AND TAKE NOTES •  KEEP THE MOMENTUM GOING •  DON’T LEAD THE WITNESS •  PROMPT QUIET PARTICIPANTS ANALYSIS CLUSTER ANALYSIS (PRODUCE DENDROGRAM), EYEBALL GROUPINGS, UNIFY GROUP NAMES, ANALYZE ORAL FEEDBACK & COMMENTS. PROS HIERARCHY FROM USERS’ PERSPECTIVES, UNCOVERS USER TERMINOLOGY CONS RELATIVELY MANY PARTICIPANTS   TIMING AS EARLY AS POSSIBLE, BEFORE A DESIGN OR REDESIGN. # OF PARTICIPANTS 20-30 TOOLS XSORT (MAC ONLY) REMOTE: USERZOOM, OPTIMAL USABILITY, WEBSORT READ MORE  
97. NOTICEABILITY TEST
100. NOTICEABILITY TEST A RESEARCH TECHNIQUE FOR LEARNING WHETHER PEOPLE NOTICE KEY ELEMENTS IN YOUR DESIGN PLAY-BY-PLAY AFTER USERS COMPLETE A TASK ASK THEM TO RE-ASSEMBLE THE KEY SCREEN, SHUT UP, & WATCH WHAT HAPPENS. ANALYSIS DID THEY PUT KEY ELEMENTS IN PLACE? DID THEY LEAVE OUT WHAT DOESN’T BELONG? DID THEY DRAW ELEMENTS THAT WEREN’T THERE? PROS CHEAP, CRYSTAL CLEAR RESULTS – DID THEY NOTICE SOMETHING CONS PERCEIVED AS NOT SCIENTIFIC ENOUGH   TIMING AS A PART OF AN ITERATIVE DESIGN PROCESS. # OF PARTICIPANTS 3-12 TOOLS PRINTOUTS OF SCREENS READ MORE SOON  
101. RECAP
102. PERFECTLY EXECUTING THE WRONG PLAN.  
103. PROVIDING INSIGHTS INTO PRODUCT USERS, THEIR PERSPECTIVES, AND THEIR ABILITIES TO THE RIGHT PEOPLE AT THE RIGHT TIME.
104. WHAT DO PEOPLE NEED?
105. WHAT DO PEOPLE WANT?
106. CAN PEOPLE USE IT?
107. DON’T LISTEN TO USERS
108. OBSERVE BEHAVIOR
109. NEEDS OBSERVATION INTERVIEW DIARY STUDY EXPERIENCE SAMPLING PHOTO DIARY COLLAGING WANTS A/B TESTING WIZARD OF OZ FAKE DOORS USABILITY ONLINE USABILITY TEST CARD SORTING NOTICEABILITY TEST
110. LAST POINT
111. “DON’T BELIEVE EVERYTHING YOU SEE ON A POWERPOINT SLIDE JUST BECAUSE THERE’S A PICTURE WITH A QUOTE NEXT TO IT.” - ABRAHAM LINCOLN
112. THANK YOU @tsharon Facebook.com/itsourresearch http://itsourresear.ch