Measuring Hard-to-Measure Things

Measuring Hard-to-Measure Things

It's all about humans.

F091db85bc34be872c938ba9b0901e60?s=128

Chrissie Brodigan

June 16, 2015
Tweet

Transcript

  1. the best way to build and ship software Hard-to-Measure Things

    @chrissiebrodigan, GitHub User Research
  2. the best way to build and ship software A little

    about me …. 2 ! •Live in Sausalito •Trained historian •UX designer •GitHub’s first researcher
  3. the best way to build and ship software

  4. the best way to build and ship software

  5. the best way to build and ship software GitHub 3

    years ago … Justin & I research things together now!!
  6. the best way to build and ship software What would

    interest people about user research at Monitorama?
  7. the best way to build and ship software

  8. “ the best way to build and ship software 8

    ! WHAT ARE THE WAYS HIDDEN VARIABLES SCREW US? – @D2FN
  9. the best way to build and ship software All products

    have blind spots
  10. the best way to build and ship software

  11. the best way to build and ship software Research is

    GitHub’s flashlight
  12. the best way to build and ship software Why? User

    research What is happening Monitoring tools
  13. the best way to build and ship software WTF are

    those humans doing?
  14. the best way to build and ship software How do

    you uncover why something is happening?
  15. “ the best way to build and ship software 15

    ! GAP FROM 
 METRIC/GRAPH TO INSIGHT CAN BE HUGE – @RANDOMMOOD
  16. the best way to build and ship software

  17. “ the best way to build and ship software 17

    ! RESEARCH: WITHOUT IT ALL YOU HAVE IS LUCK. - @SBOAK
  18. the best way to build and ship software Human instrument

    for socially meaningful questions
  19. the best way to build and ship software Get out

    from behind the numbers & in front of people
  20. the best way to build and ship software

  21. the best way to build and ship software When you

    talk with people look for … 21 ! 1. Goals 2. Motivations 3. Workarounds 
 Humans do really interesting things with software when they’re confused & frustrated
  22. the best way to build and ship software Example: Someone

    is running a large job
  23. the best way to build and ship software Solution: Limit

    or block them?
  24. the best way to build and ship software Modest Proposal

    … 24 ! 1. Talk with them about their goals 2. Look for their motivations 3. Design a better solution *https://en.wikipedia.org/wiki/A_Modest_Proposal
  25. the best way to build and ship software Develop trust

    in research Listen to the right people
  26. the best way to build and ship software Let me

    show you …
  27. the best way to build and ship software Who uses

    GitHub.com?
  28. the best way to build and ship software Who uses

    GitHub Enterprise?
  29. the best way to build and ship software Who administers

    GitHub Enterprise?
  30. the best way to build and ship software ♥♥ ♥

    HUG
  31. the best way to build and ship software Product that’s

    out of our hands and on your hardware
  32. the best way to build and ship software Abundance of

    data GitHub.com
  33. the best way to build and ship software Absence of

    data GitHub Enterprise
  34. the best way to build and ship software How do

    you study a BLACK BOX?
  35. the best way to build and ship software Story #1

    System Administrator Happiness
  36. the best way to build and ship software Designed a

    bi-annual survey
  37. the best way to build and ship software Instrument measures

    … 37 ! 1. Demographics 2. Product Experience 3. Net Promoter Score
  38. “ the best way to build and ship software 38

    ! FASTER, SMALLER PACKAGES AND MORE RELIABLE UPDATE PROCEDURE (I WET MY PANTS EACH TIME I UPGRADE.) Where do we need to improve the most?
  39. the best way to build and ship software Instrument is

    designed to be … 39 ! • Repeatable • Longitudinal (company-level) • Actionable
  40. the best way to build and ship software Takeaway: Increase

    product focus on the administrator experience
  41. the best way to build and ship software Story #2.

    Slow rollout of a large feature
  42. the best way to build and ship software Git Large

    File Storage (Git LFS) on GitHub
  43. the best way to build and ship software 3. Early

    access program
  44. the best way to build and ship software Things that

    kept our team up at night …
  45. the best way to build and ship software Pricing &

    Billing Fair & easy?
  46. the best way to build and ship software Infrastructure Responsible

    & stable?
  47. the best way to build and ship software

  48. the best way to build and ship software Support Generate

    volume?
  49. the best way to build and ship software 3-part research

    plan
  50. the best way to build and ship software Research plan

    … 50 ! 1. Survey 2. Pre-release 3. Early access program: • Entry screener • Controlled experiment • Exit survey
  51. the best way to build and ship software 1. Survey

    Sample size: 1,800
  52. the best way to build and ship software 2. Pre-release

    Sample size: 20
  53. the best way to build and ship software 3. Early

    access program Sample size: 13, 000+
  54. the best way to build and ship software EAP runs

    as a controlled experiment
  55. the best way to build and ship software

  56. the best way to build and ship software

  57. the best way to build and ship software

  58. the best way to build and ship software Six weeks

  59. the best way to build and ship software Sent two

    exit surveys
  60. the best way to build and ship software

  61. the best way to build and ship software If we

    could grant you one wish to improve GitHub, what would you wish for?
  62. the best way to build and ship software All 3

    data sources changed our knowledge of the world
  63. the best way to build and ship software But, we

    forgot about something important
  64. the best way to build and ship software We’ve been

    telling people for years DON’T use large files with Git
  65. the best way to build and ship software Reified with

    file size limits on GitHub.com
  66. the best way to build and ship software Asked people

    to leap into a new belief system, adopt a tool, & change behavior
  67. the best way to build and ship software Create a

    path to change behaviors
  68. the best way to build and ship software Framework for

    behaviors … 68 ! 1. Current – Used to doing 2. New – Open to try to achieve goals 3. Aspirational – Future big changes. Current New Aspirational Thank you Diane Loviglio for teaching me!
  69. the best way to build and ship software Git LFS

    1. 2. 3. 4. Current behaviors Use Dropbox Reality Don’t use Git 
 with large files New behaviors Use Git LFS Aspirational
 behaviors Growth
  70. the best way to build and ship software Going from

    current to aspirational behavior is kind of crazy
  71. the best way to build and ship software Takeaway: Adoption

    takes time & trust especially across team workflows
  72. the best way to build and ship software Next EAP:

    Org/user permissions
  73. the best way to build and ship software

  74. the best way to build and ship software

  75. the best way to build and ship software

  76. the best way to build and ship software Will belief

    systems change with new permissions?
  77. the best way to build and ship software Story #3.

    The undo button
  78. the best way to build and ship software

  79. the best way to build and ship software Branching v.

    forking
  80. the best way to build and ship software Activity: Feature

    prioritization
  81. the best way to build and ship software Most important

    to least important: 81 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
  82. the best way to build and ship software Most important

    to least important: 82 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
  83. the best way to build and ship software “What! There’s

    an undo button?”
  84. the best way to build and ship software “What do

    you mean by undo button?”
  85. the best way to build and ship software “You tell

    me, where would you use an undo button?”
  86. the best way to build and ship software “How would

    an undo button help you?”
  87. the best way to build and ship software “Tell me

    about a story when an undo button have helped you?”
  88. “ the best way to build and ship software 88

    ! WHAT ARE THE WAYS HIDDEN VARIABLES HELP US CONNECT WITH HUMANS? -@TENACIOUSCB
  89. the best way to build and ship software Hard-to-measure things

    are fun to measure
  90. the best way to build and ship software Guarantee: If

    you think in terms of human behaviors & talk with people You will become better at discovering & solving problems
  91. the best way to build and ship software You are

    my people, I want to hang out, help out
  92. the best way to build and ship software • Pull

    Requests • Issues • Admin-UI • Permissions • New User Experiences Are you interested? cb@github.com
  93. the best way to build and ship software ♥ Thank

    You ♥ cb@github.com @tenacicouscb