Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A more complete story: Three adventures in design research

A more complete story: Three adventures in design research

Survey design, controlled experiment design, and a think aloud sneak attack

Chrissie Brodigan

February 15, 2016
Tweet

More Decks by Chrissie Brodigan

Other Decks in Research

Transcript

  1. Chis Bog, Cols Soy o Cmit Ar (CA)

    Measuring 

    Hard to Measure Things:
    Uncover a more complete story
    Chrissie Brodigan
    @tenaciouscb
    1

    View Slide

  2. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    2
    About CB
    Hi, I’m Chrissie Brodigan
    ✴ Live in Sausalito
    ✴ Trained as a historian
    ✴ Focus on gender & labor
    ✴ Competitive figure skater
    ✴ Built GitHub’s early UXR
    practice (2013 - 2016)
    Twitter: @tenaciouscb

    View Slide

  3. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    3
    Career narrative
    Writi
    Research
    (8 years of
    training)
    Design Ethnography
    Writing

    View Slide

  4. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    4
    Airborne Heroines
    Job requirements
    ✴ Age 21 – 27
    ✴ Unmarried
    ✴ Weight – not over 135 lbs
    ✴ Registered nurse
    ✴ No eyeglasses

    View Slide

  5. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    5
    Larry Levine
    You’ve written a clear, but
    incomplete narrative. 


    Go talk to these women and
    listen to their stories.

    View Slide

  6. Chis Bog, Cols Soy o Cmit Ar (CA) 6
    I wa h no r.

    View Slide

  7. Chis Bog, Cols Soy o Cmit Ar (CA) 7
    I ha ffice h y.

    View Slide

  8. Chis Bog, Cols Soy o Cmit Ar (CA) 8
    I tal o 17 core,
    mo w I ke d’t
    e or ut.

    View Slide

  9. Chis Bog, Cols Soy o Cmit Ar (CA) 9
    A more complete story
    ✴ Experienced both highly
    marginalizing &
    empowering 

    work conditions.
    ✴ Skilled, professional, &
    organized workers in their
    own labor union.
    ✴ Were part of a process
    that changed
    constitutional law.

    View Slide

  10. Chis Bog, Cols Soy o Cmit Ar (CA)

    10
    My point?
    There is nothing like connecting with
    people. 


    Listening to stories can flip what you
    think you know and what the data tells
    you on it’s head.

    View Slide

  11. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    11
    Story flipping & 

    what we’ll cover:
    1. Survey Project(s)
    2. Controlled Experiment
    3. Think Aloud Protocol

    View Slide

  12. Chis Bog, Cols Soy o Cmit Ar (CA)

    12
    1
    Surveys

    View Slide

  13. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    Surveys help us to measure
    hard-to-measure things
    ✴ Emotions
    ✴ Intentions
    ✴ Motivations + Goals
    ✴ Workflow workarounds
    ✴ Prior knowledge
    ✴ Perception
    13

    View Slide

  14. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    START
    FINISH
    2
    3
    Finding the story will usually
    require more than one research
    attempt.
    14

    View Slide

  15. Chis Bog, Cols Soy o Cmit Ar (CA)

    15
    It's like driving a car at night. You
    never see further than your
    headlights, but you can make the
    whole trip that way. –E.L. Doctorow

    View Slide

  16. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    1. Tools & Workflows
    survey
    START
    FINISH
    2
    3
    2. New Users
    Tools & Workflows
    survey
    3. Inactive Users
    “365” survey
    The path to a more complete user
    story may be completely surprising!
    16

    View Slide

  17. Chis Bog, Cols Soy o Cmit Ar (CA)
    Q. How familiar are you with 

    git for version control?
    17
    ✴76% of people
    arriving from
    the U.S. were 

    brand new to
    git.
    ✴3-point scale.

    View Slide

  18. Chis Bog, Cols Soy o Cmit Ar (CA) 18
    We changed our approach
    Our team realized we were asking new
    users to tell us about skills they didn’t
    have.

    We changed strategy to ask about skills
    outside of git and GitHub that new users
    did have. (Kind of like a census.)

    View Slide

  19. Chis Bog, Cols Soy o Cmit Ar (CA) 19
    We designed a new instrument
    1. Tools in a developer toolkit
    2. Channels for tool discovery
    3. Biggest personal challenge
    4. Ways to solve that challenge
    5. Demographics

    View Slide

  20. Chis Bog, Cols Soy o Cmit Ar (CA) 20
    You have a story

    View Slide

  21. Chis Bog, Cols Soy o Cmit Ar (CA) 21
    We respect your data

    View Slide

  22. Chis Bog, Cols Soy o Cmit Ar (CA) 22
    Our design provided us with a
    cross-sectional view of data
    Analyze a snapshot
    of data vs. studying
    multiple data points
    (time-series data). 

    17 escalator accidents in 2014.

    View Slide

  23. Chis Bog, Cols Soy o Cmit Ar (CA) 23
    We looked at who responded
    We always begin
    analysis by identifying
    the “Who.”
    Here, we realized that
    we had a blazing blind
    spot –new users.

    View Slide

  24. Chis Bog, Cols Soy o Cmit Ar (CA) 24
    So, we changed our
    approach again …
    It was important to meet new
    users where they were at vs.
    where we were at.

    View Slide

  25. Chis Bog, Cols Soy o Cmit Ar (CA) 25
    We divided the 

    35-question survey into
    several smaller surveys,
    and rolled them out in
    waves over email.
    We sent shorter surveys in waves

    View Slide

  26. Chis Bog, Cols Soy o Cmit Ar (CA) 26
    The “hows” and “whys” were still a mystery. 


    To better understand outcomes, we needed to
    study tools, workflows, & skills-development
    over time.
    Establishing a baseline

    View Slide

  27. Chis Bog, Cols Soy o Cmit Ar (CA) 27
    Gather & analyze a single cohort’s data with two
    types of studies:

    ✴ Prospective – identify outcomes as they happen in
    real time.

    ✴ Retrospective – look back at variables over time and
    identify how they contributed to known outcomes.
    We “accidentally on-purpose”
    designed a longitudinal study

    View Slide

  28. Chis Bog, Cols Soy o Cmit Ar (CA) 28
    A film shot
    intermittently from
    2002 - 2013

    View Slide

  29. Chis Bog, Cols Soy o Cmit Ar (CA) 29

    The Harvard Grant Study

    Followed 268 men for 75 years as they
    both died & aged on into their 90s.

    View Slide

  30. Chis Bog, Cols Soy o Cmit Ar (CA) 30
    Understanding
    new users
    We took a cohort of 90,000
    new accounts created in
    September 2015 & divided
    them into two groups.

    View Slide

  31. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    31
    1. Explorers
    2. Creators

    View Slide

  32. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    32
    Evolution vs.
    Replacement
    Are new users different because people change ? 

    “evolution”


    Or, is the product attracting a new type of user?

    “replacement”

    View Slide

  33. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    33
    Obvious vs.
    Interesting
    1. First, when reading graphs identify
    the strongest pattern.

    2. Next, cover up what’s obvious & look
    for what’s interesting.

    View Slide

  34. Chis Bog, Cols Soy o Cmit Ar (CA) 34
    Q. What’s in your toolkit?
    Obvious:
    Tenured
    accounts are
    more likely to use
    a text editor
    than an IDE.

    View Slide

  35. Chis Bog, Cols Soy o Cmit Ar (CA) 35
    Q. Experience with tools?
    Obvious
    Interesting!
    Newcomers are
    as likely to say
    they use neither
    an IDE or a Text
    Editor, as to say
    they use one.

    View Slide

  36. Chis Bog, Cols Soy o Cmit Ar (CA) 36
    Q. Where do you go for referral?
    Obvious: 


    Both newcomers
    & tenured users
    act similarly
    with regards to
    tool discovery,
    primarily using
    Google.

    View Slide

  37. Chis Bog, Cols Soy o Cmit Ar (CA) 37
    Q. Primary text editor?
    Interesting!


    New accounts are
    more likely to be
    using Notepad++.
    29% of the sample .

    View Slide

  38. Chis Bog, Cols Soy o Cmit Ar (CA) 38
    Obvious
    Obvious
    Interesting!
    Atom’s use is
    much smaller
    among new users
    than we predicted.
    Q. Primary text editor?

    View Slide

  39. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    39
    52.8K Following

    View Slide

  40. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    40
    Github’s Free Text Editor

    View Slide

  41. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    41
    Google + % of people who
    don’t use a text editor =
    growth opportunity for Atom
    Combine observations

    View Slide

  42. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    42
    Atom is missing from 

    key search terms

    View Slide

  43. Chis Bog, Cols Soy o Cmit Ar (CA) 43
    How do you study people
    who aren’t engaged in
    your product?

    View Slide

  44. Chis Bog, Cols Soy o Cmit Ar (CA) 44
    Design an exit survey:
    1. What were you looking for …?
    2. Why did you stop using . . . . . ?
    3. What product are you using?
    4. What’s one thing we could
    have done better?

    View Slide

  45. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    45
    Q. Which VCS are you using?
    Strong pattern in the
    yellows & greens,
    which represent
    “Nothing” & “SVN.”
    As programming
    experience increases
    people are much more
    likely to be using
    another VCS vs.
    GitHub.

    View Slide

  46. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    46
    Q. What’s one thing we could
    have done better?
    Free private repos
    are NOT
    universally the
    most valuable
    GitHub good.


    Only among the 

    most experienced
    programmers are
    FPR a plurality of
    requests.

    View Slide

  47. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    47
    We’re talking about 

    free private repositories, so
    let’s discuss how to measure
    pricing your product.
    $

    View Slide

  48. Chis Bog, Cols Soy o Cmit Ar (CA) 48
    2
    Controlled
    Experiment

    View Slide

  49. Chis Bog, Cols Soy o Cmit Ar (CA) 49
    Q. How much would you pay?
    $
    $
    Photo credit: William Warby (Flickr)

    View Slide

  50. Chis Bog, Cols Soy o Cmit Ar (CA) 50
    Q. How much would you pay?
    $
    $
    Photo credit: William Warby (Flickr)

    View Slide

  51. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    51
    Instead ask about
    value –product goods
    ✴ Mug
    ✴ T-shirt
    ✴ Hoodie
    ✴ Feature(s)
    ✴ Experiences

    View Slide

  52. Chis Bog, Cols Soy o Cmit Ar (CA) 52
    The Golden Ticket
    ✴ Classic controlled
    experiment, but with a
    nice twist.
    ✴ 39,800 eligible
    candidates between the
    treatment & control.
    ✴ Coupons for free private
    repositories (FPR) to
    individuals with 1+ year
    of tenure.

    View Slide

  53. Chis Bog, Cols Soy o Cmit Ar (CA) 53
    Experiment design:

    39,800 humans
    Treatment
    (19,949)
    3 arms of
    6,600 each
    Exit Survey
    (2,039) 


    Shared
    Feedback
    Control: 19,851
    Screener

    (4,418)
    Redeemed
    their (FPR)
    coupon

    View Slide

  54. Chis Bog, Cols Soy o Cmit Ar (CA) 54
    3
    1 5 N
    Three Arms + Control:
    1, 3, or 5 

    private
    repositories
    No free
    repositories

    View Slide

  55. Chis Bog, Cols Soy o Cmit Ar (CA) 55
    Golden Ticket email
    ✴ Sent a total of 39,800 emails
    ✴ “Free private repositories for @name”
    ✴ “Free for life”
    ✴ Misunderstandings about the offer
    ✴ Good email deliverability, but . . .
    ✴ Overall low redemption rate

    View Slide

  56. Chis Bog, Cols Soy o Cmit Ar (CA) 56
    Golden Ticket screener
    ✴The original product
    draw.
    ✴Experience with
    competitor products.
    ✴Technical & social
    challenges.

    View Slide

  57. Chis Bog, Cols Soy o Cmit Ar (CA) 57
    Roll experiments out slowly. 

    Measure twice, cut once.

    View Slide

  58. Chis Bog, Cols Soy o Cmit Ar (CA) 58

    View Slide

  59. Chis Bog, Cols Soy o Cmit Ar (CA) 59
    Twitter

    Leaks

    View Slide

  60. Chis Bog, Cols Soy o Cmit Ar (CA) 60
    Unfair treatment

    View Slide

  61. Chis Bog, Cols Soy o Cmit Ar (CA) 61
    Too good to be true?

    View Slide

  62. Chis Bog, Cols Soy o Cmit Ar (CA) 62
    Measuring different
    types of data
    Human behaviors with activity data:
    1. Coupon redemption 

    2. Repository creation

    Perception of value with survey data:

    3. Attitudinal data

    View Slide

  63. Chis Bog, Cols Soy o Cmit Ar (CA) 63
    The exit survey provides us
    with insight into why people
    did or did NOT engage in one or
    both of the first two activities
    (redemption & creation).
    Understanding attitudes helps
    inform what levers we can
    design and pull with
    experiences to effect change in
    behaviors.
    Attitudinal Data

    View Slide

  64. Chis Bog, Cols Soy o Cmit Ar (CA) 64
    Q. Which would you value
    the most?
    Good # %
    Private repositories 663 36%
    GitHub T-shirt 324 17%
    Merged Pull Request 311 17%
    Git Training 265 14%
    GitHub Training 189 10%
    “Other” 103 6%
    64% indicated they
    would get more value
    out of something
    else.

    24% wanted practical
    training in Git or
    GitHub.

    34% reported that
    publicly consumable
    goods ( t-shirt) would
    be more valuable.

    View Slide

  65. Chis Bog, Cols Soy o Cmit Ar (CA) 65
    Quantitative data is
    empirical evidence.
    $$ E[Y_{i} | T_{i} = 1] - E[Y_{i} | T_{i} = 0] $$

    View Slide

  66. Chis Bog, Cols Soy o Cmit Ar (CA) 66
    Qualitative data is
    data with soul.

    View Slide

  67. Chis Bog, Cols Soy o Cmit Ar (CA) 67
    Open text responses
    No amount of machine learning can
    surface the quality of insights that
    reading open text responses does.

    View Slide

  68. “Git Udan”
    68

    View Slide

  69. “Fre e f m
    te w unte
    re r”
    69

    View Slide

  70. “at t fie re
    rite, or
    pit es up
    fi pe”
    70

    View Slide

  71. Chis Bog, Cols Soy o Cmit Ar (CA) 71
    Private appears to
    be understood as
    private only to me
    vs. 

    working with
    other people
    privately.
    Unlimited
    Collaborators

    View Slide

  72. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    72
    3
    Think Aloud

    View Slide

  73. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    73
    Faster Horses Syndrome
    Listen to the how,
    why, when, and
    where, behind
    customer requests
    ….

    View Slide

  74. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    74
    The Collaboration Study
    ✴ Customers told us they needed a
    complex feature: branch permissions.
    ✴ Competitor products offered branch
    permissions.
    ✴ Designing for a large audience, means
    we need to be thoughtful and
    deliberate.
    Solve for human motivations & goals behind feature requests.

    View Slide

  75. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    75
    Fork v. Branch:
    Choosing a 

    collaboration model

    View Slide

  76. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    76
    ?
    Think Aloud Protocol

    View Slide

  77. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    77
    Feature prioritization
    ✴ Branch Permissions
    ✴ Automatically syncing forks
    ✴ Sign-off
    ✴ Only merge with passing tests
    ✴ Undo button
    ✴ Disable force push
    ✴ Private forks
    ✴ Prevent merging from the command line

    View Slide

  78. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    78
    Feature prioritization
    ✴ Branch Permissions
    ✴ Automatically syncing forks
    ✴ Sign-off
    ✴ Only merge with passing tests
    ✴ Undo button
    ✴ Disable force push
    ✴ Private forks
    ✴ Prevent merging from the command line

    View Slide

  79. “Wha?! The’s a 

    un to? Whe?”

    View Slide

  80. “Tel bo im n a 

    un to w ha
    hed u.”

    View Slide

  81. Chis Bog, Cols Soy o Cmit Ar (CA)
    C
    81
    ✴ Include items that don’t exist, but sound like
    they might.
    ✴ Listen to people define what they think the
    “feature” is.
    ✴ Ask how, where, when, & why they would use
    the feature.
    Sneak Attack

    View Slide

  82. Chis Bog, Cols Soy o Cmit Ar (CA)

    82
    It's like driving a car at night. You
    never see further than your
    headlights, but you can make the
    whole trip that way. –E.L. Doctorow

    View Slide

  83. Chis Bog, Cols Soy o Cmit Ar (CA)

    83
    Finding the story
    ✴ Products have new users, tenured users, and
    inactive users, understanding each experience
    provides a more complete view.
    ✴ Researching hard-to-reach places– reading open
    text responses & listening to humans share their
    motivations and goals is how you find the story.
    ✴ Research is your flashlight.

    View Slide

  84. Chis Bog, Cols Soy o Cmit Ar (CA) 84
    Wit rec al
    av uc. –@so

    View Slide

  85. Chis Bog, Cols Soy o Cmit Ar (CA)

    85
    Wrapping Up
    1. What’s obvious vs. interesting in your data?

    2. How can you gather and use attitudinal data to
    study perception of value?

    3. Where does a sneak attack make sense?

    4. How will you uncover a more complete story?

    View Slide

  86. Chis Bog, Cols Soy o Cmit Ar (CA)

    86
    Thank you

    Questions?
    @tenaciouscb

    View Slide