Measuring Hard-to-Measure Things

Slide 1

Slide 1 text

the best way to build and ship software Hard-to-Measure Things @chrissiebrodigan, GitHub User Research

Slide 2

Slide 2 text

the best way to build and ship software A little about me …. 2 ! •Live in Sausalito •Trained historian •UX designer •GitHub’s ﬁrst researcher

Slide 3

Slide 3 text

the best way to build and ship software

Slide 4

Slide 4 text

the best way to build and ship software

Slide 5

Slide 5 text

the best way to build and ship software GitHub 3 years ago … Justin & I research things together now!!

Slide 6

Slide 6 text

the best way to build and ship software What would interest people about user research at Monitorama?

Slide 7

Slide 7 text

the best way to build and ship software

Slide 8

Slide 8 text

“ the best way to build and ship software 8 ! WHAT ARE THE WAYS HIDDEN VARIABLES SCREW US? – @D2FN

Slide 9

Slide 9 text

the best way to build and ship software All products have blind spots

Slide 10

Slide 10 text

the best way to build and ship software

Slide 11

Slide 11 text

the best way to build and ship software Research is GitHub’s ﬂashlight

Slide 12

Slide 12 text

the best way to build and ship software Why? User research What is happening Monitoring tools

Slide 13

Slide 13 text

the best way to build and ship software WTF are those humans doing?

Slide 14

Slide 14 text

the best way to build and ship software How do you uncover why something is happening?

Slide 15

Slide 15 text

“ the best way to build and ship software 15 ! GAP FROM   METRIC/GRAPH TO INSIGHT CAN BE HUGE – @RANDOMMOOD

Slide 16

Slide 16 text

the best way to build and ship software

Slide 17

Slide 17 text

“ the best way to build and ship software 17 ! RESEARCH: WITHOUT IT ALL YOU HAVE IS LUCK. - @SBOAK

Slide 18

Slide 18 text

the best way to build and ship software Human instrument for socially meaningful questions

Slide 19

Slide 19 text

the best way to build and ship software Get out from behind the numbers & in front of people

Slide 20

Slide 20 text

the best way to build and ship software

Slide 21

Slide 21 text

the best way to build and ship software When you talk with people look for … 21 ! 1. Goals 2. Motivations 3. Workarounds   Humans do really interesting things with software when they’re confused & frustrated

Slide 22

Slide 22 text

the best way to build and ship software Example: Someone is running a large job

Slide 23

Slide 23 text

the best way to build and ship software Solution: Limit or block them?

Slide 24

Slide 24 text

the best way to build and ship software Modest Proposal … 24 ! 1. Talk with them about their goals 2. Look for their motivations 3. Design a better solution *https://en.wikipedia.org/wiki/A_Modest_Proposal

Slide 25

Slide 25 text

the best way to build and ship software Develop trust in research Listen to the right people

Slide 26

Slide 26 text

the best way to build and ship software Let me show you …

Slide 27

Slide 27 text

the best way to build and ship software Who uses GitHub.com?

Slide 28

Slide 28 text

the best way to build and ship software Who uses GitHub Enterprise?

Slide 29

Slide 29 text

the best way to build and ship software Who administers GitHub Enterprise?

Slide 30

Slide 30 text

the best way to build and ship software ♥♥ ♥ HUG

Slide 31

Slide 31 text

the best way to build and ship software Product that’s out of our hands and on your hardware

Slide 32

Slide 32 text

the best way to build and ship software Abundance of data GitHub.com

Slide 33

Slide 33 text

the best way to build and ship software Absence of data GitHub Enterprise

Slide 34

Slide 34 text

the best way to build and ship software How do you study a BLACK BOX?

Slide 35

Slide 35 text

the best way to build and ship software Story #1 System Administrator Happiness

Slide 36

Slide 36 text

the best way to build and ship software Designed a bi-annual survey

Slide 37

Slide 37 text

the best way to build and ship software Instrument measures … 37 ! 1. Demographics 2. Product Experience 3. Net Promoter Score

Slide 38

Slide 38 text

“ the best way to build and ship software 38 ! FASTER, SMALLER PACKAGES AND MORE RELIABLE UPDATE PROCEDURE (I WET MY PANTS EACH TIME I UPGRADE.) Where do we need to improve the most?

Slide 39

Slide 39 text

the best way to build and ship software Instrument is designed to be … 39 ! • Repeatable • Longitudinal (company-level) • Actionable

Slide 40

Slide 40 text

the best way to build and ship software Takeaway: Increase product focus on the administrator experience

Slide 41

Slide 41 text

the best way to build and ship software Story #2. Slow rollout of a large feature

Slide 42

Slide 42 text

the best way to build and ship software Git Large File Storage (Git LFS) on GitHub

Slide 43

Slide 43 text

the best way to build and ship software 3. Early access program

Slide 44

Slide 44 text

the best way to build and ship software Things that kept our team up at night …

Slide 45

Slide 45 text

the best way to build and ship software Pricing & Billing Fair & easy?

Slide 46

Slide 46 text

the best way to build and ship software Infrastructure Responsible & stable?

Slide 47

Slide 47 text

the best way to build and ship software

Slide 48

Slide 48 text

the best way to build and ship software Support Generate volume?

Slide 49

Slide 49 text

the best way to build and ship software 3-part research plan

Slide 50

Slide 50 text

the best way to build and ship software Research plan … 50 ! 1. Survey 2. Pre-release 3. Early access program: • Entry screener • Controlled experiment • Exit survey

Slide 51

Slide 51 text

the best way to build and ship software 1. Survey Sample size: 1,800

Slide 52

Slide 52 text

the best way to build and ship software 2. Pre-release Sample size: 20

Slide 53

Slide 53 text

the best way to build and ship software 3. Early access program Sample size: 13, 000+

Slide 54

Slide 54 text

the best way to build and ship software EAP runs as a controlled experiment

Slide 55

Slide 55 text

the best way to build and ship software

Slide 56

Slide 56 text

the best way to build and ship software

Slide 57

Slide 57 text

the best way to build and ship software

Slide 58

Slide 58 text

the best way to build and ship software Six weeks

Slide 59

Slide 59 text

the best way to build and ship software Sent two exit surveys

Slide 60

Slide 60 text

the best way to build and ship software

Slide 61

Slide 61 text

the best way to build and ship software If we could grant you one wish to improve GitHub, what would you wish for?

Slide 62

Slide 62 text

the best way to build and ship software All 3 data sources changed our knowledge of the world

Slide 63

Slide 63 text

the best way to build and ship software But, we forgot about something important

Slide 64

Slide 64 text

the best way to build and ship software We’ve been telling people for years DON’T use large ﬁles with Git

Slide 65

Slide 65 text

the best way to build and ship software Reiﬁed with ﬁle size limits on GitHub.com

Slide 66

Slide 66 text

the best way to build and ship software Asked people to leap into a new belief system, adopt a tool, & change behavior

Slide 67

Slide 67 text

the best way to build and ship software Create a path to change behaviors

Slide 68

Slide 68 text

the best way to build and ship software Framework for behaviors … 68 ! 1. Current – Used to doing 2. New – Open to try to achieve goals 3. Aspirational – Future big changes. Current New Aspirational Thank you Diane Loviglio for teaching me!

Slide 69

Slide 69 text

the best way to build and ship software Git LFS 1. 2. 3. 4. Current behaviors Use Dropbox Reality Don’t use Git   with large files New behaviors Use Git LFS Aspirational  behaviors Growth

Slide 70

Slide 70 text

the best way to build and ship software Going from current to aspirational behavior is kind of crazy

Slide 71

Slide 71 text

the best way to build and ship software Takeaway: Adoption takes time & trust especially across team workﬂows

Slide 72

Slide 72 text

the best way to build and ship software Next EAP: Org/user permissions

Slide 73

Slide 73 text

the best way to build and ship software

Slide 74

Slide 74 text

the best way to build and ship software

Slide 75

Slide 75 text

the best way to build and ship software

Slide 76

Slide 76 text

the best way to build and ship software Will belief systems change with new permissions?

Slide 77

Slide 77 text

the best way to build and ship software Story #3. The undo button

Slide 78

Slide 78 text

the best way to build and ship software

Slide 79

Slide 79 text

the best way to build and ship software Branching v. forking

Slide 80

Slide 80 text

the best way to build and ship software Activity: Feature prioritization

Slide 81

Slide 81 text

the best way to build and ship software Most important to least important: 81 ! •Branch Permissions •Automatically syncing forks •Sign-oﬀ •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line

Slide 82

Slide 82 text

the best way to build and ship software Most important to least important: 82 ! •Branch Permissions •Automatically syncing forks •Sign-oﬀ •Only merge with passing tests •Undo button •Disable force push •Private forks •Prevent merging from the command line

Slide 83

Slide 83 text

the best way to build and ship software “What! There’s an undo button?”

Slide 84

Slide 84 text

the best way to build and ship software “What do you mean by undo button?”

Slide 85

Slide 85 text

the best way to build and ship software “You tell me, where would you use an undo button?”

Slide 86

Slide 86 text

the best way to build and ship software “How would an undo button help you?”

Slide 87

Slide 87 text

the best way to build and ship software “Tell me about a story when an undo button have helped you?”

Slide 88

Slide 88 text

“ the best way to build and ship software 88 ! WHAT ARE THE WAYS HIDDEN VARIABLES HELP US CONNECT WITH HUMANS? -@TENACIOUSCB

Slide 89

Slide 89 text

the best way to build and ship software Hard-to-measure things are fun to measure

Slide 90

Slide 90 text

the best way to build and ship software Guarantee: If you think in terms of human behaviors & talk with people You will become better at discovering & solving problems

Slide 91

Slide 91 text

the best way to build and ship software You are my people, I want to hang out, help out

Slide 92

Slide 92 text

the best way to build and ship software • Pull Requests • Issues • Admin-UI • Permissions • New User Experiences Are you interested? cb@github.com

Slide 93

Slide 93 text

the best way to build and ship software ♥ Thank You ♥ cb@github.com @tenacicouscb