Measuring Hard-to-Measure Things

Measuring Hard-to-Measure Things

It's all about humans.

F091db85bc34be872c938ba9b0901e60?s=128

Chrissie Brodigan

June 16, 2015
Tweet

Transcript

  1. 1.

    the best way to build and ship software Hard-to-Measure Things

    @chrissiebrodigan, GitHub User Research
  2. 2.

    the best way to build and ship software A little

    about me …. 2 ! •Live in Sausalito •Trained historian •UX designer •GitHub’s first researcher
  3. 5.

    the best way to build and ship software GitHub 3

    years ago … Justin & I research things together now!!
  4. 6.

    the best way to build and ship software What would

    interest people about user research at Monitorama?
  5. 8.

    “ the best way to build and ship software 8

    ! WHAT ARE THE WAYS HIDDEN VARIABLES SCREW US? – @D2FN
  6. 12.

    the best way to build and ship software Why? User

    research What is happening Monitoring tools
  7. 14.

    the best way to build and ship software How do

    you uncover why something is happening?
  8. 15.

    “ the best way to build and ship software 15

    ! GAP FROM 
 METRIC/GRAPH TO INSIGHT CAN BE HUGE – @RANDOMMOOD
  9. 17.

    “ the best way to build and ship software 17

    ! RESEARCH: WITHOUT IT ALL YOU HAVE IS LUCK. - @SBOAK
  10. 18.
  11. 19.

    the best way to build and ship software Get out

    from behind the numbers & in front of people
  12. 21.

    the best way to build and ship software When you

    talk with people look for … 21 ! 1. Goals 2. Motivations 3. Workarounds 
 Humans do really interesting things with software when they’re confused & frustrated
  13. 24.

    the best way to build and ship software Modest Proposal

    … 24 ! 1. Talk with them about their goals 2. Look for their motivations 3. Design a better solution *https://en.wikipedia.org/wiki/A_Modest_Proposal
  14. 25.

    the best way to build and ship software Develop trust

    in research Listen to the right people
  15. 31.

    the best way to build and ship software Product that’s

    out of our hands and on your hardware
  16. 35.

    the best way to build and ship software Story #1

    System Administrator Happiness
  17. 37.

    the best way to build and ship software Instrument measures

    … 37 ! 1. Demographics 2. Product Experience 3. Net Promoter Score
  18. 38.

    “ the best way to build and ship software 38

    ! FASTER, SMALLER PACKAGES AND MORE RELIABLE UPDATE PROCEDURE (I WET MY PANTS EACH TIME I UPGRADE.) Where do we need to improve the most?
  19. 39.

    the best way to build and ship software Instrument is

    designed to be … 39 ! • Repeatable • Longitudinal (company-level) • Actionable
  20. 40.

    the best way to build and ship software Takeaway: Increase

    product focus on the administrator experience
  21. 41.

    the best way to build and ship software Story #2.

    Slow rollout of a large feature
  22. 42.

    the best way to build and ship software Git Large

    File Storage (Git LFS) on GitHub
  23. 44.
  24. 50.

    the best way to build and ship software Research plan

    … 50 ! 1. Survey 2. Pre-release 3. Early access program: • Entry screener • Controlled experiment • Exit survey
  25. 53.

    the best way to build and ship software 3. Early

    access program Sample size: 13, 000+
  26. 54.
  27. 61.

    the best way to build and ship software If we

    could grant you one wish to improve GitHub, what would you wish for?
  28. 62.

    the best way to build and ship software All 3

    data sources changed our knowledge of the world
  29. 63.

    the best way to build and ship software But, we

    forgot about something important
  30. 64.

    the best way to build and ship software We’ve been

    telling people for years DON’T use large files with Git
  31. 65.
  32. 66.

    the best way to build and ship software Asked people

    to leap into a new belief system, adopt a tool, & change behavior
  33. 68.

    the best way to build and ship software Framework for

    behaviors … 68 ! 1. Current – Used to doing 2. New – Open to try to achieve goals 3. Aspirational – Future big changes. Current New Aspirational Thank you Diane Loviglio for teaching me!
  34. 69.

    the best way to build and ship software Git LFS

    1. 2. 3. 4. Current behaviors Use Dropbox Reality Don’t use Git 
 with large files New behaviors Use Git LFS Aspirational
 behaviors Growth
  35. 70.

    the best way to build and ship software Going from

    current to aspirational behavior is kind of crazy
  36. 71.

    the best way to build and ship software Takeaway: Adoption

    takes time & trust especially across team workflows
  37. 76.

    the best way to build and ship software Will belief

    systems change with new permissions?
  38. 81.

    the best way to build and ship software Most important

    to least important: 81 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
  39. 82.

    the best way to build and ship software Most important

    to least important: 82 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
  40. 84.
  41. 85.

    the best way to build and ship software “You tell

    me, where would you use an undo button?”
  42. 87.

    the best way to build and ship software “Tell me

    about a story when an undo button have helped you?”
  43. 88.

    “ the best way to build and ship software 88

    ! WHAT ARE THE WAYS HIDDEN VARIABLES HELP US CONNECT WITH HUMANS? -@TENACIOUSCB
  44. 90.

    the best way to build and ship software Guarantee: If

    you think in terms of human behaviors & talk with people You will become better at discovering & solving problems
  45. 91.

    the best way to build and ship software You are

    my people, I want to hang out, help out
  46. 92.

    the best way to build and ship software • Pull

    Requests • Issues • Admin-UI • Permissions • New User Experiences Are you interested? cb@github.com
  47. 93.

    the best way to build and ship software ♥ Thank

    You ♥ cb@github.com @tenacicouscb