Measuring Hard-to-Measure Things

It's all about humans.

Transcript

1. the best way to build and ship software Hard-to-Measure Things @chrissiebrodigan, GitHub User Research
2. the best way to build and ship software A little about me …. 2 ! •Live in Sausalito •Trained historian •UX designer •GitHub’s first researcher
3. the best way to build and ship software
4. the best way to build and ship software
5. the best way to build and ship software GitHub 3 years ago … Justin & I research things together now!!
6. the best way to build and ship software What would interest people about user research at Monitorama?
7. the best way to build and ship software
8. “ the best way to build and ship software 8 ! WHAT ARE THE WAYS HIDDEN VARIABLES SCREW US? – @D2FN
9. the best way to build and ship software All products have blind spots
10. the best way to build and ship software
11. the best way to build and ship software Research is GitHub’s flashlight
12. the best way to build and ship software Why? User research What is happening Monitoring tools
13. the best way to build and ship software WTF are those humans doing?
14. the best way to build and ship software How do you uncover why something is happening?
15. “ the best way to build and ship software 15 ! GAP FROM 
 METRIC/GRAPH TO INSIGHT CAN BE HUGE – @RANDOMMOOD
16. the best way to build and ship software
17. “ the best way to build and ship software 17 ! RESEARCH: WITHOUT IT ALL YOU HAVE IS LUCK. - @SBOAK
18. the best way to build and ship software Human instrument for socially meaningful questions
19. the best way to build and ship software Get out from behind the numbers & in front of people
20. the best way to build and ship software
21. the best way to build and ship software When you talk with people look for … 21 ! 1. Goals 2. Motivations 3. Workarounds 
 Humans do really interesting things with software when they’re confused & frustrated
22. the best way to build and ship software Example: Someone is running a large job
23. the best way to build and ship software Solution: Limit or block them?
24. the best way to build and ship software Modest Proposal … 24 ! 1. Talk with them about their goals 2. Look for their motivations 3. Design a better solution *https://en.wikipedia.org/wiki/A_Modest_Proposal
25. the best way to build and ship software Develop trust in research Listen to the right people
26. the best way to build and ship software Let me show you …
27. the best way to build and ship software Who uses GitHub.com?
28. the best way to build and ship software Who uses GitHub Enterprise?
29. the best way to build and ship software Who administers GitHub Enterprise?
30. the best way to build and ship software ♥♥ ♥ HUG
31. the best way to build and ship software Product that’s out of our hands and on your hardware
32. the best way to build and ship software Abundance of data GitHub.com
33. the best way to build and ship software Absence of data GitHub Enterprise
34. the best way to build and ship software How do you study a BLACK BOX?
35. the best way to build and ship software Story #1 System Administrator Happiness
36. the best way to build and ship software Designed a bi-annual survey
37. the best way to build and ship software Instrument measures … 37 ! 1. Demographics 2. Product Experience 3. Net Promoter Score
38. “ the best way to build and ship software 38 ! FASTER, SMALLER PACKAGES AND MORE RELIABLE UPDATE PROCEDURE (I WET MY PANTS EACH TIME I UPGRADE.) Where do we need to improve the most?
39. the best way to build and ship software Instrument is designed to be … 39 ! • Repeatable • Longitudinal (company-level) • Actionable
40. the best way to build and ship software Takeaway: Increase product focus on the administrator experience
41. the best way to build and ship software Story #2. Slow rollout of a large feature
42. the best way to build and ship software Git Large File Storage (Git LFS) on GitHub
43. the best way to build and ship software 3. Early access program
44. the best way to build and ship software Things that kept our team up at night …
45. the best way to build and ship software Pricing & Billing Fair & easy?
46. the best way to build and ship software Infrastructure Responsible & stable?
47. the best way to build and ship software
48. the best way to build and ship software Support Generate volume?
49. the best way to build and ship software 3-part research plan
50. the best way to build and ship software Research plan … 50 ! 1. Survey 2. Pre-release 3. Early access program: • Entry screener • Controlled experiment • Exit survey
51. the best way to build and ship software 1. Survey Sample size: 1,800
52. the best way to build and ship software 2. Pre-release Sample size: 20
53. the best way to build and ship software 3. Early access program Sample size: 13, 000+
54. the best way to build and ship software EAP runs as a controlled experiment
55. the best way to build and ship software
56. the best way to build and ship software
57. the best way to build and ship software
58. the best way to build and ship software Six weeks
59. the best way to build and ship software Sent two exit surveys
60. the best way to build and ship software
61. the best way to build and ship software If we could grant you one wish to improve GitHub, what would you wish for?
62. the best way to build and ship software All 3 data sources changed our knowledge of the world
63. the best way to build and ship software But, we forgot about something important
64. the best way to build and ship software We’ve been telling people for years DON’T use large files with Git
65. the best way to build and ship software Reified with file size limits on GitHub.com
66. the best way to build and ship software Asked people to leap into a new belief system, adopt a tool, & change behavior
67. the best way to build and ship software Create a path to change behaviors
68. the best way to build and ship software Framework for behaviors … 68 ! 1. Current – Used to doing 2. New – Open to try to achieve goals 3. Aspirational – Future big changes. Current New Aspirational Thank you Diane Loviglio for teaching me!
69. the best way to build and ship software Git LFS 1. 2. 3. 4. Current behaviors Use Dropbox Reality Don’t use Git 
 with large files New behaviors Use Git LFS Aspirational
 behaviors Growth
70. the best way to build and ship software Going from current to aspirational behavior is kind of crazy
71. the best way to build and ship software Takeaway: Adoption takes time & trust especially across team workflows
72. the best way to build and ship software Next EAP: Org/user permissions
73. the best way to build and ship software
74. the best way to build and ship software
75. the best way to build and ship software
76. the best way to build and ship software Will belief systems change with new permissions?
77. the best way to build and ship software Story #3. The undo button
78. the best way to build and ship software
79. the best way to build and ship software Branching v. forking
80. the best way to build and ship software Activity: Feature prioritization
81. the best way to build and ship software Most important to least important: 81 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
82. the best way to build and ship software Most important to least important: 82 ! •Branch Permissions •Automatically syncing forks •Sign-off •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line
83. the best way to build and ship software “What! There’s an undo button?”
84. the best way to build and ship software “What do you mean by undo button?”
85. the best way to build and ship software “You tell me, where would you use an undo button?”
86. the best way to build and ship software “How would an undo button help you?”
87. the best way to build and ship software “Tell me about a story when an undo button have helped you?”
88. “ the best way to build and ship software 88 ! WHAT ARE THE WAYS HIDDEN VARIABLES HELP US CONNECT WITH HUMANS? -@TENACIOUSCB
89. the best way to build and ship software Hard-to-measure things are fun to measure
90. the best way to build and ship software Guarantee: If you think in terms of human behaviors & talk with people You will become better at discovering & solving problems
91. the best way to build and ship software You are my people, I want to hang out, help out
92. the best way to build and ship software • Pull Requests • Issues • Admin-UI • Permissions • New User Experiences Are you interested? cb@github.com
93. the best way to build and ship software ♥ Thank You ♥ cb@github.com @tenacicouscb