Validating Assumptions with 12 UX Research Methods

Validating Assumptions with 12 UX Research Methods

This presentation reviews three big questions UX research can help answer: What do people need? What do people want? Can people use it? These questions can be answered with 12 different methods: observation, interview, diary study, experience sampling, photo diary, collaging, A/B testing, Wizard of Oz, fake doors, online usability test, card sorting, and noticeability test.

3c0b492a599715b69b5ed1363a9c2c4f?s=128

Tomer Sharon

June 04, 2013
Tweet

Transcript

  1. I’M HERE TO PREVENT YOU FROM PERFECTLY EXECUTING THE WRONG

    PLAN.   @tsharon  
  2. HI, I’M TOMER

  3. VALIDATING ASSUMPTIONS WITH 12 UX RESEARCH METHODS TOMER SHARON, SR.

    UX RESEARCHER, GOOGLE @tsharon
  4. RESOURCES •  ARTICLES BY JEREMY JOHNSON, KYLE SOUCY, & STEVE

    PORTIGAL •  MY EXPERIENCE
  5. PERFECTLY EXECUTING THE WRONG PLAN.  

  6. QUESTIONS I ASK MY TEAMS

  7. WHAT DO YOU WANT TO KNOW?

  8. WHY?

  9. WHAT KNOWLEDGE GAPS ARE YOU HOPING TO FILL?

  10. WHAT DECISIONS ARE YOU TRYING TO MAKE BASED ON RESEARCH

    RESULTS?
  11. WHAT ARE YOUR ASSUMPTIONS ABOUT USERS, THAT IF PROVEN FALSE,

    WILL CAUSE YOU TO FAIL?
  12. FROM ASSUMPTIONS TO HYPOTHESES

  13. WE BELIEVE THAT [DOING THIS] FOR [THESE PEOPLE] WILL ACHIEVE

    [THIS OUTCOME]. WE’LL KNOW THIS IS TRUE WHEN WE SEE [THIS MARKET FEEDBACK].
  14. READ MORE SERIOUSLY, THESE ARE BOOK EVERY DESIGNER MUST READ

    ASAP
  15. None
  16. USER EXPERIENCE RESEARCH

  17. PROVIDING INSIGHTS INTO PRODUCT USERS, THEIR PERSPECTIVES, AND THEIR ABILITIES

    TO THE RIGHT PEOPLE AT THE RIGHT TIME.
  18. 3 QUESTIONS UX RESEARCH CAN ANSWER

  19. WHAT DO PEOPLE NEED?

  20. WHAT DO PEOPLE WANT?

  21. CAN PEOPLE USE IT?

  22. UX RESEARCH IS ABOUT DOING THE RIGHT THING, NOT JUST

    DOING THINGS RIGHT
  23. NEEDS > WANTS > USABILITY DOING THE RIGHT THING DOING

    THINGS RIGHT
  24. FIRST RULE OF UX RESEARCH

  25. DON’T LISTEN TO USERS

  26. OBSERVE BEHAVIOR

  27. I DON’T LISTEN TO USERS BECAUSE OF THE PSYCHOLOGY OF

    ATTITUDE & BEHAVIOR
  28. HIGH, NEGATIVE RELATIONSHIP BETWEEN ATTITUDE AND BEHAVIOR

  29. STUDENTS WERE ASKED ABOUT THEIR ATTITUDE TOWARD CHEATING. A FEW

    WEEKS LATER, THEY WERE ASKED TO GRADE THEIR OWN TESTS. THERE WAS CLOSE TO ZERO CORRELATION BETWEEN ATTITUDE TOWARD CHEATING AND ACTUAL CHEATING. [COREY, 1937]
  30. None
  31. WHAT DOES IT MEAN TO YOU?

  32. NEVER ASK WHAT THEY NEED DO YOU THINK THIS IS

    A GOOD THING TO HAVE?
  33. BECAUSE HUMANS SUCK AT PREDICTING THE FUTURE YET THEY ARE

    HAPPY TO GIVE YOU THEIR OPINION
  34. INSTEAD OBSERVE WHAT THEY DO SHOW ME THE MOST FREQUENT

    TASK YOU TYPICALLY DO.
  35. NEVER ASK WHAT IS THEIR FEEDBACK WHAT DO YOU THINK

    ABOUT THIS COOL “SHARE” BUTTON?
  36. BECAUSE THEY’LL TELL YOU WHAT YOU WANT TO HEAR AND

    THEY HAVE NO IDEA WHAT THEY ARE TALKING ABOUT
  37. INSTEAD WATCH THEM USE IT CAN YOU SHOW ME HOW

    YOU DO THAT?
  38. WHAT DO PEOPLE NEED?

  39. THESE METHODS WILL HELP YOU PERFECTLY EXECUTE THE RIGHT PLAN

  40. OBSERVATION

  41. FIELD STUDY CONTEXTUAL INQUIRY FLY-ON-THE-WALL SHADOWING ETHNOGRAPHY GUIDED TOUR

  42. SHOW ME HOW YOU DO THAT

  43. None
  44. None
  45. None
  46. None
  47. OBSERVATION A SMALL RESEARCH TEAM OBSERVES USERS IN THE CONTEXT

    OF USAGE, IN AN UNOBTRUSIVE MANNER , TO GATHER INSIGHTS THAT CANNOT OTHERWISE BE OBTAINED. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PREPARE A FIELD GUIDE •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION OF OBSERVATION •  OBTAIN PERMISSION FOR STUDY/RECORDING •  PREPARE FIELD KITS •  ASSIGN OBSERVATION ROLES •  OBSERVE •  TAKE LOTS OF NOTES AND PHOTOS ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS CONS EASY TO MISINTERPRET, TIME CONSUMING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS VIDEO RECORDING – GOPRO, FLIP, SMARTPHONE PHOTOS – SMARTPHONE APPS – EthOS, RECORDIUM (iOS) READ MORE  
  48. INTERVIEW

  49. CONVERSE LIKE A TALK SHOW HOST, THINK LIKE A WRITER,

    UNDERSTAND SUBTEXT LIKE A PSYCHIATRIST, HAVE AN EAR LIKE A MUSICIAN. - LAWRENCE GROBEL
  50. None
  51. None
  52. INTERVIEW GATHERING INFORMATION THROUGH DIRECT DIALOGUE PLAY-BY-PLAY •  IDENTIFY A

    DOMAIN TO STUDY •  PREPARE QUESTIONS •  CHOOSE & RECRUIT PARTICIPANTS •  SET A SCHEDULE AND LOCATION •  START AND FINISH WITH EASY QUESTIONS •  LISTEN MORE THAN YOU TALK •  TAKE LOTS OF NOTES ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS, BUILDS CREDIBILITY WITH STAKEHOLDERS CONS RATIONALIZATION, IT’S HARD TO ASK THE RIGHT QUESTIONS   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS RAINBOW SPREADSHEET PEN AND PAPER READ MORE  
  53. WHAT’S WORST THAN 5 ENGINEERS IN A ROOM DRINKING LATTE?

     
  54. None
  55. None
  56. None
  57. None
  58. None
  59. DIARY STUDY

  60. None
  61. None
  62. DIARY STUDY A GROUP OF PEOPLE KEEPS A DIARY AS

    THEY USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON DIARY ENTRIES •  ENCOURAGE PARTICIPANTS ANALYSIS READ DIARIES, COMPOSE HYPOTHESES, CODE AND ANALYZE TO DETERMINE USAGE PATTERNS AND COMMON ISSUES. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG FORM & SPREADSHEET VOICE MESSAGES EMAIL PRINT DIARY READ MORE  
  63. EXPERIENCE SAMPLING

  64. None
  65. None
  66. EXPERIENCE SAMPLING “REALITY MINING”, CAPTURING EXPERIENCES AT THE MOMENT OF

    THEIR OCCURANCE WITHIN THE CONTEXT OF EVERYDAY LIFE. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PHRASE A QUESTION •  DETERMINE SAMPLING RATE •  CHOOSE A TOOL •  DECIDE HOW TO COLLECT DATA •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON RESPONSES •  ENCOURAGE PARTICIPANTS •  BEGIN ANALYSIS EARLY ANALYSIS CLEAN THE DATA FIRST, DECIDE ON CATEGORIES, CLASSIFY, ADJUST CATEGORIES, ANALYZE AS A TEAM FIRST, THEN INDIVIDUALLY, EYEBALL THE DATA, GENERATE FREQUENCY CHARTS, IDENTIFY THEMES, PREPARE CREATIVE BRIEFS PROS DISCOVER HOW PRODUCTS INTEGRATE INTO PEOPLE’S ROUTINES IN UNANTICIPATED WAYS, CHEAP CONS RECRUITING AND MANAGING PARTICIPANTS ARE CHALLENGING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS HUNDREDS+ TOOLS FORM & SPREADSHEET EMAIL APP – PACO (ANDROID) SMS READ MORE TED TALKS: BOB EVANS & MIHALY CSIKSZENTMIHALYI  
  67. PHOTO DIARY

  68. None
  69. PHOTO DIARY A GROUP OF PEOPLE TAKE PICTURES AS THEY

    USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON PHOTOS •  ENCOURAGE PARTICIPANTS ANALYSIS FLIP THROUGH PHOTOS, COMPOSE HYPOTHESES, CLASSIFY AND CATEGORIZE TO DETERMINE PATTERNS AND COMMON ISSUES. AFFINITY DIAGRAM WITH STAKEHOLDERS. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG PRIVATE SOCIAL MEDIA PAGE EMAIL READ MORE  
  70. COLLAGING

  71. None
  72. None
  73. None
  74. None
  75. COLLAGING A GROUP OF PEOPLE TAKE PICTURES AS THEY USE

    A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  CHOOSE & RECRUIT PARTICIPANTS •  PREPARE COLLAGE BOARD AND MATERIALS •  MAGAZINES •  SHARPIES •  SCISSORS •  ASK PARTICIPANTS TO WORK IN GROUPS OF 3 •  LEAVE THE ROOM •  AFTER 20 MINUTES COME BACK AND PROBE •  TAKE PICTURES OF COLLAGES ANALYSIS WITH PARTICIPANTS FIRST, THEN CODE AND ANALYZE WITH STAKEHOLDERS IN A WORKSHOP SETUP. PROS GETTING ANSWERS TO QUESTIONS YOU DON’T KNOW TO ASK CONS STAKEHOLDER BUY-IN IS TOUGH   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 9 (3X3) TOOLS REMOTE COLLAGING WITH DESKTOP-SHARING TOOLS READ MORE “COLLAGING”  
  76. WHAT DO PEOPLE WANT?

  77. THESE METHODS WILL HELP YOU EXECUTE YOUR USERS’ *PLAN

  78. A/B TESTING

  79. None
  80. A/B TESTING CREATING MULTIPLE VERSIONS OF A SCREEN OR FEATURE

    AND SHOWING EACH VERSION TO A DIFFERENT SET OF USERS IN PRODUCTION TO FIND OUT WHICH VERSION PRODUCES BETTER METRICS. PLAY-BY-PLAY •  IDENTIFY A DESIGN DECISION TO EVALUATE •  CREATE CONDITION DESIGNS •  IDENTIFY METRICS TO TRACK (E.G., ENGAGEMENT, REVENUE, CONVERSION TO PAYING, REGISTRATION, RETENTION) •  DIRECT 50% OF TRAFFIC AT EACH CONDITION (IF 2 CONDITIONS) •  TRACK METRICS ANALYSIS CALCULATE DESCRIPTIVE STATISTICS PER EVALUATED METRICS AND COMPARE. PROS STATISTICALLY SIGNIFICANT DATA, REAL USAGE DATA, VALIDATES DESIGN DECISIONS, GREAT FEEDBACK LOOP CONS CAN’T EVALUATE TONS OF DESIGN DECISIONS, CAN’T TELL YOU WHY   TIMING WHEN IT’S TIME TO MAKE A DESIGN DECISION. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS GOOGLE ANALYTICS – CONTENT EXPERIMENTS OPTIMIZELY READ MORE  
  81. WIZARD OF OZ

  82. None
  83. None
  84. WIZARD OF OZ INSTEAD OF DOING ALL THE DESIGN AND

    ENGINEERING WORK TO BUILD THE PRODUCT FOR EVERYBODY ALL AT ONCE, DO THE WORK MANUALLY WITH A FEW PARTICIPANTS. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PART IN YOUR PRODUCT •  CHARACTERIZE THE “MACHINE” •  DEVELOP PROTOCOL (IF, THEN) •  PRACTICE! •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION FOR THE STUDY •  ASK PARTICIPANTS TO THINK OUT LOUD •  TAKE LOTS OF NOTES ANALYSIS OPPORTUNITIES FOR IMPROVEMENT, INSIGHTS ABOUT VALUE. PROS REDUCES WASTED ENGINEERING WORK CONS CAN’T ALWAYS FULLY IMMITATE A REAL PRODUCT   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS 4-12 TOOLS INTERCOM PAPER MOCKS “MACHINE” READ MORE  
  85. FAKE DOORS

  86. None
  87. FAKE DOORS INSTEAD OF DOING ALL THE DESIGN AND ENGINEERING

    WORK TO BUILD A PRODUCT OR FEATURE, FAKE IT, AND DETERMINE WHAT % OF PEOPLE WOULD BE INTERESTED IN IT. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PRODUCT/FEATURE •  PRETEND IT IS THERE •  INSTRUMENT SENSORS •  TRACK SIGNALS ANALYSIS CALCULATE % OF PEOPLE WHO SHOWED INTEREST AND EVALUATE IN CONJUNCTION WITH INTERVIEW DATA. PROS REDUCES WASTED DESIGN & ENGINEERING WORK CONS DOESN’T ALWAYS WORK AS A STANDALONE STUDY   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS - READ MORE VIMEO.COM/24744647  
  88. CAN PEOPLE USE IT?

  89. THESE METHODS WILL HELP YOU PERFECTLY EXECUTE *YOUR PLAN

  90. ONLINE USABILITY TEST

  91. http://youtu.be/ZvpHsp_NJEo

  92. ONLINE USABILITY TEST AN ONLINE RESEARCH TECHNIQUE DURING WHICH USERS

    ARE ASKED TO COMPLETE TASKS WHILE UX METRICS ARE COLLECTED BEHIND THE SCENES. PLAY-BY-PLAY •  IDENTIFY A DESIGN TO STUDY •  IDENTIFY A COMPARISON DESIGN (COMPETITOR, EARLY VERSION) •  PREPARE A SCRIPT, INCLUDING TASKS •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE PARTICIPATION ANALYSIS COMPARE METRICS, ANALYZE METRICS IN CONJUNCTION WITH QUALITATIVE RESEARCH. PROS PURE COMPARISON, EXTREMELY EFFECTIVE WITH STAKEHOLDERS CONS ONLY TELLS YOU WHAT HAPPENED, NOT WHY   TIMING START AS EARLY AS POSSIBLE, TEST AS OFTEN AS POSSIBLE. # OF PARTICIPANTS ANY NUMBER ABOVE 4 PER DESIGN WORKS. TOOLS LOOP 11 USERZOOM WEB EFFECTIVE READ MORE  
  93. CARD SORTING

  94. http://youtu.be/-89cj71-Vfg

  95. None
  96. CARD SORTING A TACTICAL RESEARCH TECHNIQUE FOR LEARNING HOW TO

    STRUCTURE INFORMATION IN A PRODUCT. PARTICIPANTS ARE ASKED TO PUT A SET OF ITEMS INTO LOGICAL GROUPS. PLAY-BY-PLAY •  SELECT CONTENT (CURRENT OR FUTURE) •  PREPARE CARDS (<100, + EMPTY, #) •  SELECT A TOOL •  DETERMINE IF OPEN OR CLOSED SORT •  DETERMINE IF INDIVIDUAL OR GROUP WORK •  DETERMINE IF IN-PERSON OR REMOTE •  BRIEF PARTICIPANTS •  OBSERVE AND TAKE NOTES •  KEEP THE MOMENTUM GOING •  DON’T LEAD THE WITNESS •  PROMPT QUIET PARTICIPANTS ANALYSIS CLUSTER ANALYSIS (PRODUCE DENDROGRAM), EYEBALL GROUPINGS, UNIFY GROUP NAMES, ANALYZE ORAL FEEDBACK & COMMENTS. PROS HIERARCHY FROM USERS’ PERSPECTIVES, UNCOVERS USER TERMINOLOGY CONS RELATIVELY MANY PARTICIPANTS   TIMING AS EARLY AS POSSIBLE, BEFORE A DESIGN OR REDESIGN. # OF PARTICIPANTS 20-30 TOOLS XSORT (MAC ONLY) REMOTE: USERZOOM, OPTIMAL USABILITY, WEBSORT READ MORE  
  97. NOTICEABILITY TEST

  98. None
  99. None
  100. NOTICEABILITY TEST A RESEARCH TECHNIQUE FOR LEARNING WHETHER PEOPLE NOTICE

    KEY ELEMENTS IN YOUR DESIGN PLAY-BY-PLAY AFTER USERS COMPLETE A TASK ASK THEM TO RE-ASSEMBLE THE KEY SCREEN, SHUT UP, & WATCH WHAT HAPPENS. ANALYSIS DID THEY PUT KEY ELEMENTS IN PLACE? DID THEY LEAVE OUT WHAT DOESN’T BELONG? DID THEY DRAW ELEMENTS THAT WEREN’T THERE? PROS CHEAP, CRYSTAL CLEAR RESULTS – DID THEY NOTICE SOMETHING CONS PERCEIVED AS NOT SCIENTIFIC ENOUGH   TIMING AS A PART OF AN ITERATIVE DESIGN PROCESS. # OF PARTICIPANTS 3-12 TOOLS PRINTOUTS OF SCREENS READ MORE SOON  
  101. RECAP

  102. PERFECTLY EXECUTING THE WRONG PLAN.  

  103. PROVIDING INSIGHTS INTO PRODUCT USERS, THEIR PERSPECTIVES, AND THEIR ABILITIES

    TO THE RIGHT PEOPLE AT THE RIGHT TIME.
  104. WHAT DO PEOPLE NEED?

  105. WHAT DO PEOPLE WANT?

  106. CAN PEOPLE USE IT?

  107. DON’T LISTEN TO USERS

  108. OBSERVE BEHAVIOR

  109. NEEDS OBSERVATION INTERVIEW DIARY STUDY EXPERIENCE SAMPLING PHOTO DIARY COLLAGING

    WANTS A/B TESTING WIZARD OF OZ FAKE DOORS USABILITY ONLINE USABILITY TEST CARD SORTING NOTICEABILITY TEST
  110. LAST POINT

  111. “DON’T BELIEVE EVERYTHING YOU SEE ON A POWERPOINT SLIDE JUST

    BECAUSE THERE’S A PICTURE WITH A QUOTE NEXT TO IT.” - ABRAHAM LINCOLN
  112. THANK YOU @tsharon Facebook.com/itsourresearch http://itsourresear.ch