the best way to build and ship software
Hard-to-Measure Things
@chrissiebrodigan, GitHub User Research
Slide 2
Slide 2 text
the best way to build and ship software
A little about me ….
2
!
•Live in Sausalito
•Trained historian
•UX designer
•GitHub’s first researcher
Slide 3
Slide 3 text
the best way to build and ship software
Slide 4
Slide 4 text
the best way to build and ship software
Slide 5
Slide 5 text
the best way to build and ship software
GitHub 3 years ago …
Justin & I research things together now!!
Slide 6
Slide 6 text
the best way to build and ship software
What would interest people about
user research at Monitorama?
Slide 7
Slide 7 text
the best way to build and ship software
Slide 8
Slide 8 text
“
the best way to build and ship software 8
!
WHAT ARE THE WAYS
HIDDEN VARIABLES
SCREW US?
– @D2FN
Slide 9
Slide 9 text
the best way to build and ship software
All products have
blind spots
Slide 10
Slide 10 text
the best way to build and ship software
Slide 11
Slide 11 text
the best way to build and ship software
Research is
GitHub’s flashlight
Slide 12
Slide 12 text
the best way to build and ship software
Why?
User
research
What is
happening
Monitoring
tools
Slide 13
Slide 13 text
the best way to build and ship software
WTF are those
humans doing?
Slide 14
Slide 14 text
the best way to build and ship software
How do you uncover why
something is happening?
Slide 15
Slide 15 text
“
the best way to build and ship software 15
!
GAP FROM
METRIC/GRAPH TO
INSIGHT CAN BE HUGE
– @RANDOMMOOD
Slide 16
Slide 16 text
the best way to build and ship software
Slide 17
Slide 17 text
“
the best way to build and ship software 17
!
RESEARCH:
WITHOUT IT ALL YOU
HAVE IS LUCK.
- @SBOAK
Slide 18
Slide 18 text
the best way to build and ship software
Human instrument for
socially meaningful questions
Slide 19
Slide 19 text
the best way to build and ship software
Get out from behind the
numbers & in front of people
Slide 20
Slide 20 text
the best way to build and ship software
Slide 21
Slide 21 text
the best way to build and ship software
When you talk with people look for …
21
!
1. Goals
2. Motivations
3. Workarounds
Humans do really interesting things with software
when they’re confused & frustrated
Slide 22
Slide 22 text
the best way to build and ship software
Example:
Someone is running a large job
Slide 23
Slide 23 text
the best way to build and ship software
Solution:
Limit or block them?
Slide 24
Slide 24 text
the best way to build and ship software
Modest Proposal …
24
!
1. Talk with them about their goals
2. Look for their motivations
3. Design a better solution
*https://en.wikipedia.org/wiki/A_Modest_Proposal
Slide 25
Slide 25 text
the best way to build and ship software
Develop trust in research
Listen to the right people
Slide 26
Slide 26 text
the best way to build and ship software
Let me show you …
Slide 27
Slide 27 text
the best way to build and ship software
Who uses GitHub.com?
Slide 28
Slide 28 text
the best way to build and ship software
Who uses
GitHub Enterprise?
Slide 29
Slide 29 text
the best way to build and ship software
Who administers
GitHub Enterprise?
Slide 30
Slide 30 text
the best way to build and ship software
♥♥
♥ HUG
Slide 31
Slide 31 text
the best way to build and ship software
Product that’s out of our
hands and on your hardware
Slide 32
Slide 32 text
the best way to build and ship software
Abundance of data
GitHub.com
Slide 33
Slide 33 text
the best way to build and ship software
Absence of data
GitHub Enterprise
Slide 34
Slide 34 text
the best way to build and ship software
How do you study a
BLACK BOX?
Slide 35
Slide 35 text
the best way to build and ship software
Story #1
System Administrator
Happiness
Slide 36
Slide 36 text
the best way to build and ship software
Designed a
bi-annual survey
Slide 37
Slide 37 text
the best way to build and ship software
Instrument measures …
37
!
1. Demographics
2. Product Experience
3. Net Promoter Score
Slide 38
Slide 38 text
“
the best way to build and ship software 38
!
FASTER, SMALLER PACKAGES
AND MORE RELIABLE UPDATE
PROCEDURE (I WET MY PANTS
EACH TIME I UPGRADE.)
Where do we need to improve the most?
Slide 39
Slide 39 text
the best way to build and ship software
Instrument is designed to be …
39
!
• Repeatable
• Longitudinal (company-level)
• Actionable
Slide 40
Slide 40 text
the best way to build and ship software
Takeaway:
Increase product focus on the
administrator experience
Slide 41
Slide 41 text
the best way to build and ship software
Story #2.
Slow rollout of a
large feature
Slide 42
Slide 42 text
the best way to build and ship software
Git Large File Storage
(Git LFS)
on GitHub
Slide 43
Slide 43 text
the best way to build and ship software
3. Early access program
Slide 44
Slide 44 text
the best way to build and ship software
Things that kept our
team up at night …
Slide 45
Slide 45 text
the best way to build and ship software
Pricing & Billing
Fair & easy?
Slide 46
Slide 46 text
the best way to build and ship software
Infrastructure
Responsible & stable?
Slide 47
Slide 47 text
the best way to build and ship software
Slide 48
Slide 48 text
the best way to build and ship software
Support
Generate volume?
Slide 49
Slide 49 text
the best way to build and ship software
3-part research plan
Slide 50
Slide 50 text
the best way to build and ship software
Research plan …
50
!
1. Survey
2. Pre-release
3. Early access program:
• Entry screener
• Controlled experiment
• Exit survey
Slide 51
Slide 51 text
the best way to build and ship software
1. Survey
Sample size: 1,800
Slide 52
Slide 52 text
the best way to build and ship software
2. Pre-release
Sample size: 20
Slide 53
Slide 53 text
the best way to build and ship software
3. Early access program
Sample size: 13, 000+
Slide 54
Slide 54 text
the best way to build and ship software
EAP runs as a
controlled experiment
Slide 55
Slide 55 text
the best way to build and ship software
Slide 56
Slide 56 text
the best way to build and ship software
Slide 57
Slide 57 text
the best way to build and ship software
Slide 58
Slide 58 text
the best way to build and ship software
Six weeks
Slide 59
Slide 59 text
the best way to build and ship software
Sent two exit surveys
Slide 60
Slide 60 text
the best way to build and ship software
Slide 61
Slide 61 text
the best way to build and ship software
If we could grant you
one wish to improve GitHub,
what would you wish for?
Slide 62
Slide 62 text
the best way to build and ship software
All 3 data sources changed
our knowledge of the world
Slide 63
Slide 63 text
the best way to build and ship software
But, we forgot about
something important
Slide 64
Slide 64 text
the best way to build and ship software
We’ve been telling
people for years DON’T
use large files with Git
Slide 65
Slide 65 text
the best way to build and ship software
Reified with
file size limits on
GitHub.com
Slide 66
Slide 66 text
the best way to build and ship software
Asked people to leap into
a new belief system, adopt
a tool, & change behavior
Slide 67
Slide 67 text
the best way to build and ship software
Create a path to
change behaviors
Slide 68
Slide 68 text
the best way to build and ship software
Framework for behaviors …
68
!
1. Current – Used to doing
2. New – Open to try to achieve goals
3. Aspirational – Future big changes.
Current
New
Aspirational
Thank you Diane Loviglio for teaching me!
Slide 69
Slide 69 text
the best way to build and ship software
Git LFS
1.
2.
3.
4.
Current
behaviors
Use Dropbox
Reality
Don’t use Git
with large
files
New
behaviors
Use Git LFS
Aspirational
behaviors
Growth
Slide 70
Slide 70 text
the best way to build and ship software
Going from current to
aspirational behavior is
kind of crazy
Slide 71
Slide 71 text
the best way to build and ship software
Takeaway:
Adoption takes time & trust
especially across team workflows
Slide 72
Slide 72 text
the best way to build and ship software
Next EAP:
Org/user permissions
Slide 73
Slide 73 text
the best way to build and ship software
Slide 74
Slide 74 text
the best way to build and ship software
Slide 75
Slide 75 text
the best way to build and ship software
Slide 76
Slide 76 text
the best way to build and ship software
Will belief systems
change with new
permissions?
Slide 77
Slide 77 text
the best way to build and ship software
Story #3.
The undo button
Slide 78
Slide 78 text
the best way to build and ship software
Slide 79
Slide 79 text
the best way to build and ship software
Branching v. forking
Slide 80
Slide 80 text
the best way to build and ship software
Activity:
Feature prioritization
Slide 81
Slide 81 text
the best way to build and ship software
Most important to least important:
81
!
•Branch Permissions
•Automatically syncing forks
•Sign-off
•Only merge with passing tests
•Undo button
•Disable force push
•Private forks
•Prevent merging from the command line
Slide 82
Slide 82 text
the best way to build and ship software
Most important to least important:
82
!
•Branch Permissions
•Automatically syncing forks
•Sign-off
•Only merge with passing tests
•Undo button
•Disable force push
•Private forks
•Prevent merging from the command line
Slide 83
Slide 83 text
the best way to build and ship software
“What! There’s an
undo button?”
Slide 84
Slide 84 text
the best way to build and ship software
“What do you mean by
undo button?”
Slide 85
Slide 85 text
the best way to build and ship software
“You tell me, where would
you use an undo button?”
Slide 86
Slide 86 text
the best way to build and ship software
“How would an undo
button help you?”
Slide 87
Slide 87 text
the best way to build and ship software
“Tell me about a story when an
undo button have helped you?”
Slide 88
Slide 88 text
“
the best way to build and ship software 88
!
WHAT ARE THE WAYS
HIDDEN VARIABLES HELP
US CONNECT WITH
HUMANS?
-@TENACIOUSCB
Slide 89
Slide 89 text
the best way to build and ship software
Hard-to-measure things
are fun to measure
Slide 90
Slide 90 text
the best way to build and ship software
Guarantee:
If you think in terms of human behaviors
& talk with people
You will become better at
discovering & solving problems
Slide 91
Slide 91 text
the best way to build and ship software
You are my people,
I want to hang out, help out
Slide 92
Slide 92 text
the best way to build and ship software
• Pull Requests
• Issues
• Admin-UI
• Permissions
• New User Experiences
Are you interested?
[email protected]
Slide 93
Slide 93 text
the best way to build and ship software
♥ Thank You ♥
[email protected] @tenacicouscb