Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Validating Assumptions with 12 UX Research Methods

Validating Assumptions with 12 UX Research Methods

This presentation reviews three big questions UX research can help answer: What do people need? What do people want? Can people use it? These questions can be answered with 12 different methods: observation, interview, diary study, experience sampling, photo diary, collaging, A/B testing, Wizard of Oz, fake doors, online usability test, card sorting, and noticeability test.

Tomer Sharon

June 04, 2013
Tweet

More Decks by Tomer Sharon

Other Decks in Business

Transcript

  1. WE BELIEVE THAT [DOING THIS] FOR [THESE PEOPLE] WILL ACHIEVE

    [THIS OUTCOME]. WE’LL KNOW THIS IS TRUE WHEN WE SEE [THIS MARKET FEEDBACK].
  2. STUDENTS WERE ASKED ABOUT THEIR ATTITUDE TOWARD CHEATING. A FEW

    WEEKS LATER, THEY WERE ASKED TO GRADE THEIR OWN TESTS. THERE WAS CLOSE TO ZERO CORRELATION BETWEEN ATTITUDE TOWARD CHEATING AND ACTUAL CHEATING. [COREY, 1937]
  3. NEVER ASK WHAT IS THEIR FEEDBACK WHAT DO YOU THINK

    ABOUT THIS COOL “SHARE” BUTTON?
  4. BECAUSE THEY’LL TELL YOU WHAT YOU WANT TO HEAR AND

    THEY HAVE NO IDEA WHAT THEY ARE TALKING ABOUT
  5. OBSERVATION A SMALL RESEARCH TEAM OBSERVES USERS IN THE CONTEXT

    OF USAGE, IN AN UNOBTRUSIVE MANNER , TO GATHER INSIGHTS THAT CANNOT OTHERWISE BE OBTAINED. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PREPARE A FIELD GUIDE •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION OF OBSERVATION •  OBTAIN PERMISSION FOR STUDY/RECORDING •  PREPARE FIELD KITS •  ASSIGN OBSERVATION ROLES •  OBSERVE •  TAKE LOTS OF NOTES AND PHOTOS ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS CONS EASY TO MISINTERPRET, TIME CONSUMING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS VIDEO RECORDING – GOPRO, FLIP, SMARTPHONE PHOTOS – SMARTPHONE APPS – EthOS, RECORDIUM (iOS) READ MORE  
  6. CONVERSE LIKE A TALK SHOW HOST, THINK LIKE A WRITER,

    UNDERSTAND SUBTEXT LIKE A PSYCHIATRIST, HAVE AN EAR LIKE A MUSICIAN. - LAWRENCE GROBEL
  7. INTERVIEW GATHERING INFORMATION THROUGH DIRECT DIALOGUE PLAY-BY-PLAY •  IDENTIFY A

    DOMAIN TO STUDY •  PREPARE QUESTIONS •  CHOOSE & RECRUIT PARTICIPANTS •  SET A SCHEDULE AND LOCATION •  START AND FINISH WITH EASY QUESTIONS •  LISTEN MORE THAN YOU TALK •  TAKE LOTS OF NOTES ANALYSIS USE CASE STORIES AFFINITY DIAGRAM KJ TECHNIQUE PROS DEEPENS EMPATHY, CHALLENGES ASSUMPTIONS, BUILDS CREDIBILITY WITH STAKEHOLDERS CONS RATIONALIZATION, IT’S HARD TO ASK THE RIGHT QUESTIONS   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-16 TOOLS RAINBOW SPREADSHEET PEN AND PAPER READ MORE  
  8. DIARY STUDY A GROUP OF PEOPLE KEEPS A DIARY AS

    THEY USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON DIARY ENTRIES •  ENCOURAGE PARTICIPANTS ANALYSIS READ DIARIES, COMPOSE HYPOTHESES, CODE AND ANALYZE TO DETERMINE USAGE PATTERNS AND COMMON ISSUES. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG FORM & SPREADSHEET VOICE MESSAGES EMAIL PRINT DIARY READ MORE  
  9. EXPERIENCE SAMPLING “REALITY MINING”, CAPTURING EXPERIENCES AT THE MOMENT OF

    THEIR OCCURANCE WITHIN THE CONTEXT OF EVERYDAY LIFE. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  PHRASE A QUESTION •  DETERMINE SAMPLING RATE •  CHOOSE A TOOL •  DECIDE HOW TO COLLECT DATA •  CHOOSE & RECRUIT PARTICIPANTS (EVALUATE WRITING SKILLS) •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON RESPONSES •  ENCOURAGE PARTICIPANTS •  BEGIN ANALYSIS EARLY ANALYSIS CLEAN THE DATA FIRST, DECIDE ON CATEGORIES, CLASSIFY, ADJUST CATEGORIES, ANALYZE AS A TEAM FIRST, THEN INDIVIDUALLY, EYEBALL THE DATA, GENERATE FREQUENCY CHARTS, IDENTIFY THEMES, PREPARE CREATIVE BRIEFS PROS DISCOVER HOW PRODUCTS INTEGRATE INTO PEOPLE’S ROUTINES IN UNANTICIPATED WAYS, CHEAP CONS RECRUITING AND MANAGING PARTICIPANTS ARE CHALLENGING   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS HUNDREDS+ TOOLS FORM & SPREADSHEET EMAIL APP – PACO (ANDROID) SMS READ MORE TED TALKS: BOB EVANS & MIHALY CSIKSZENTMIHALYI  
  10. PHOTO DIARY A GROUP OF PEOPLE TAKE PICTURES AS THEY

    USE A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  DETERMING IF STUDY IS STRUCTURED OR UNSTRUCTURED •  DETERMINE THE DURATION OF THE STUDY •  DETERMINE SAMPLING RATE •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE RESPONDENTS AND RESPONSES •  PROVIDE FEEDBACK ON PHOTOS •  ENCOURAGE PARTICIPANTS ANALYSIS FLIP THROUGH PHOTOS, COMPOSE HYPOTHESES, CLASSIFY AND CATEGORIZE TO DETERMINE PATTERNS AND COMMON ISSUES. AFFINITY DIAGRAM WITH STAKEHOLDERS. PROS GEOGRAPHICALLY DISTRIBUTED, LONGITUDINAL, CHEAP CONS HIGHLY DEPENDS ON PARTICIPANTS’ COOPERATION LEVEL   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 8-20 TOOLS PRIVATE BLOG PRIVATE SOCIAL MEDIA PAGE EMAIL READ MORE  
  11. COLLAGING A GROUP OF PEOPLE TAKE PICTURES AS THEY USE

    A PRODUCT, OR GO THROUGH CERTAIN ROUTINES OR TASKS. PLAY-BY-PLAY •  IDENTIFY A DOMAIN TO STUDY •  CHOOSE & RECRUIT PARTICIPANTS •  PREPARE COLLAGE BOARD AND MATERIALS •  MAGAZINES •  SHARPIES •  SCISSORS •  ASK PARTICIPANTS TO WORK IN GROUPS OF 3 •  LEAVE THE ROOM •  AFTER 20 MINUTES COME BACK AND PROBE •  TAKE PICTURES OF COLLAGES ANALYSIS WITH PARTICIPANTS FIRST, THEN CODE AND ANALYZE WITH STAKEHOLDERS IN A WORKSHOP SETUP. PROS GETTING ANSWERS TO QUESTIONS YOU DON’T KNOW TO ASK CONS STAKEHOLDER BUY-IN IS TOUGH   TIMING AS EARLY AS POSSIBLE OR ON AN ONGOING BASIS. # OF PARTICIPANTS 9 (3X3) TOOLS REMOTE COLLAGING WITH DESKTOP-SHARING TOOLS READ MORE “COLLAGING”  
  12. A/B TESTING CREATING MULTIPLE VERSIONS OF A SCREEN OR FEATURE

    AND SHOWING EACH VERSION TO A DIFFERENT SET OF USERS IN PRODUCTION TO FIND OUT WHICH VERSION PRODUCES BETTER METRICS. PLAY-BY-PLAY •  IDENTIFY A DESIGN DECISION TO EVALUATE •  CREATE CONDITION DESIGNS •  IDENTIFY METRICS TO TRACK (E.G., ENGAGEMENT, REVENUE, CONVERSION TO PAYING, REGISTRATION, RETENTION) •  DIRECT 50% OF TRAFFIC AT EACH CONDITION (IF 2 CONDITIONS) •  TRACK METRICS ANALYSIS CALCULATE DESCRIPTIVE STATISTICS PER EVALUATED METRICS AND COMPARE. PROS STATISTICALLY SIGNIFICANT DATA, REAL USAGE DATA, VALIDATES DESIGN DECISIONS, GREAT FEEDBACK LOOP CONS CAN’T EVALUATE TONS OF DESIGN DECISIONS, CAN’T TELL YOU WHY   TIMING WHEN IT’S TIME TO MAKE A DESIGN DECISION. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS GOOGLE ANALYTICS – CONTENT EXPERIMENTS OPTIMIZELY READ MORE  
  13. WIZARD OF OZ INSTEAD OF DOING ALL THE DESIGN AND

    ENGINEERING WORK TO BUILD THE PRODUCT FOR EVERYBODY ALL AT ONCE, DO THE WORK MANUALLY WITH A FEW PARTICIPANTS. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PART IN YOUR PRODUCT •  CHARACTERIZE THE “MACHINE” •  DEVELOP PROTOCOL (IF, THEN) •  PRACTICE! •  CHOOSE & RECRUIT PARTICIPANTS •  IDENTIFY THE BEST LOCATION FOR THE STUDY •  ASK PARTICIPANTS TO THINK OUT LOUD •  TAKE LOTS OF NOTES ANALYSIS OPPORTUNITIES FOR IMPROVEMENT, INSIGHTS ABOUT VALUE. PROS REDUCES WASTED ENGINEERING WORK CONS CAN’T ALWAYS FULLY IMMITATE A REAL PRODUCT   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS 4-12 TOOLS INTERCOM PAPER MOCKS “MACHINE” READ MORE  
  14. FAKE DOORS INSTEAD OF DOING ALL THE DESIGN AND ENGINEERING

    WORK TO BUILD A PRODUCT OR FEATURE, FAKE IT, AND DETERMINE WHAT % OF PEOPLE WOULD BE INTERESTED IN IT. PLAY-BY-PLAY •  IDENTIFY THE HARD-TO-DEVELOP PRODUCT/FEATURE •  PRETEND IT IS THERE •  INSTRUMENT SENSORS •  TRACK SIGNALS ANALYSIS CALCULATE % OF PEOPLE WHO SHOWED INTEREST AND EVALUATE IN CONJUNCTION WITH INTERVIEW DATA. PROS REDUCES WASTED DESIGN & ENGINEERING WORK CONS DOESN’T ALWAYS WORK AS A STANDALONE STUDY   TIMING WHEN IT’S A PAIN IN THE A$$ TO DEVELOP SOMETHING. # OF PARTICIPANTS ANY NUMBER MORE THAN 2 WORKS. TOOLS - READ MORE VIMEO.COM/24744647  
  15. ONLINE USABILITY TEST AN ONLINE RESEARCH TECHNIQUE DURING WHICH USERS

    ARE ASKED TO COMPLETE TASKS WHILE UX METRICS ARE COLLECTED BEHIND THE SCENES. PLAY-BY-PLAY •  IDENTIFY A DESIGN TO STUDY •  IDENTIFY A COMPARISON DESIGN (COMPETITOR, EARLY VERSION) •  PREPARE A SCRIPT, INCLUDING TASKS •  CHOOSE & RECRUIT PARTICIPANTS •  MANAGE PARTICIPATION ANALYSIS COMPARE METRICS, ANALYZE METRICS IN CONJUNCTION WITH QUALITATIVE RESEARCH. PROS PURE COMPARISON, EXTREMELY EFFECTIVE WITH STAKEHOLDERS CONS ONLY TELLS YOU WHAT HAPPENED, NOT WHY   TIMING START AS EARLY AS POSSIBLE, TEST AS OFTEN AS POSSIBLE. # OF PARTICIPANTS ANY NUMBER ABOVE 4 PER DESIGN WORKS. TOOLS LOOP 11 USERZOOM WEB EFFECTIVE READ MORE  
  16. CARD SORTING A TACTICAL RESEARCH TECHNIQUE FOR LEARNING HOW TO

    STRUCTURE INFORMATION IN A PRODUCT. PARTICIPANTS ARE ASKED TO PUT A SET OF ITEMS INTO LOGICAL GROUPS. PLAY-BY-PLAY •  SELECT CONTENT (CURRENT OR FUTURE) •  PREPARE CARDS (<100, + EMPTY, #) •  SELECT A TOOL •  DETERMINE IF OPEN OR CLOSED SORT •  DETERMINE IF INDIVIDUAL OR GROUP WORK •  DETERMINE IF IN-PERSON OR REMOTE •  BRIEF PARTICIPANTS •  OBSERVE AND TAKE NOTES •  KEEP THE MOMENTUM GOING •  DON’T LEAD THE WITNESS •  PROMPT QUIET PARTICIPANTS ANALYSIS CLUSTER ANALYSIS (PRODUCE DENDROGRAM), EYEBALL GROUPINGS, UNIFY GROUP NAMES, ANALYZE ORAL FEEDBACK & COMMENTS. PROS HIERARCHY FROM USERS’ PERSPECTIVES, UNCOVERS USER TERMINOLOGY CONS RELATIVELY MANY PARTICIPANTS   TIMING AS EARLY AS POSSIBLE, BEFORE A DESIGN OR REDESIGN. # OF PARTICIPANTS 20-30 TOOLS XSORT (MAC ONLY) REMOTE: USERZOOM, OPTIMAL USABILITY, WEBSORT READ MORE  
  17. NOTICEABILITY TEST A RESEARCH TECHNIQUE FOR LEARNING WHETHER PEOPLE NOTICE

    KEY ELEMENTS IN YOUR DESIGN PLAY-BY-PLAY AFTER USERS COMPLETE A TASK ASK THEM TO RE-ASSEMBLE THE KEY SCREEN, SHUT UP, & WATCH WHAT HAPPENS. ANALYSIS DID THEY PUT KEY ELEMENTS IN PLACE? DID THEY LEAVE OUT WHAT DOESN’T BELONG? DID THEY DRAW ELEMENTS THAT WEREN’T THERE? PROS CHEAP, CRYSTAL CLEAR RESULTS – DID THEY NOTICE SOMETHING CONS PERCEIVED AS NOT SCIENTIFIC ENOUGH   TIMING AS A PART OF AN ITERATIVE DESIGN PROCESS. # OF PARTICIPANTS 3-12 TOOLS PRINTOUTS OF SCREENS READ MORE SOON  
  18. NEEDS OBSERVATION INTERVIEW DIARY STUDY EXPERIENCE SAMPLING PHOTO DIARY COLLAGING

    WANTS A/B TESTING WIZARD OF OZ FAKE DOORS USABILITY ONLINE USABILITY TEST CARD SORTING NOTICEABILITY TEST
  19. “DON’T BELIEVE EVERYTHING YOU SEE ON A POWERPOINT SLIDE JUST

    BECAUSE THERE’S A PICTURE WITH A QUOTE NEXT TO IT.” - ABRAHAM LINCOLN