Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
Measuring
Hard to Measure Things:
Uncover a more complete story
Chrissie Brodigan
@tenaciouscb
1
Slide 2
Slide 2 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
2
About CB
Hi, I’m Chrissie Brodigan
✴ Live in Sausalito
✴ Trained as a historian
✴ Focus on gender & labor
✴ Competitive figure skater
✴ Built GitHub’s early UXR
practice (2013 - 2016)
Twitter: @tenaciouscb
Slide 3
Slide 3 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
3
Career narrative
Writi
Research
(8 years of
training)
Design Ethnography
Writing
Slide 4
Slide 4 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
4
Airborne Heroines
Job requirements
✴ Age 21 – 27
✴ Unmarried
✴ Weight – not over 135 lbs
✴ Registered nurse
✴ No eyeglasses
Slide 5
Slide 5 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
5
Larry Levine
You’ve written a clear, but
incomplete narrative.
Go talk to these women and
listen to their stories.
Slide 6
Slide 6 text
Chis Bog, Cols Soy o Cmit Ar (CA) 6
I wa h no r.
Slide 7
Slide 7 text
Chis Bog, Cols Soy o Cmit Ar (CA) 7
I ha ffice h y.
Slide 8
Slide 8 text
Chis Bog, Cols Soy o Cmit Ar (CA) 8
I tal o 17 core,
mo w I ke d’t
e or ut.
Slide 9
Slide 9 text
Chis Bog, Cols Soy o Cmit Ar (CA) 9
A more complete story
✴ Experienced both highly
marginalizing &
empowering
work conditions.
✴ Skilled, professional, &
organized workers in their
own labor union.
✴ Were part of a process
that changed
constitutional law.
Slide 10
Slide 10 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
10
My point?
There is nothing like connecting with
people.
Listening to stories can flip what you
think you know and what the data tells
you on it’s head.
Slide 11
Slide 11 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
11
Story flipping &
what we’ll cover:
1. Survey Project(s)
2. Controlled Experiment
3. Think Aloud Protocol
Slide 12
Slide 12 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
12
1
Surveys
Slide 13
Slide 13 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
Surveys help us to measure
hard-to-measure things
✴ Emotions
✴ Intentions
✴ Motivations + Goals
✴ Workflow workarounds
✴ Prior knowledge
✴ Perception
13
Slide 14
Slide 14 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
START
FINISH
2
3
Finding the story will usually
require more than one research
attempt.
14
Slide 15
Slide 15 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
15
It's like driving a car at night. You
never see further than your
headlights, but you can make the
whole trip that way. –E.L. Doctorow
Slide 16
Slide 16 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
1. Tools & Workflows
survey
START
FINISH
2
3
2. New Users
Tools & Workflows
survey
3. Inactive Users
“365” survey
The path to a more complete user
story may be completely surprising!
16
Slide 17
Slide 17 text
Chis Bog, Cols Soy o Cmit Ar (CA)
Q. How familiar are you with
git for version control?
17
✴76% of people
arriving from
the U.S. were
brand new to
git.
✴3-point scale.
Slide 18
Slide 18 text
Chis Bog, Cols Soy o Cmit Ar (CA) 18
We changed our approach
Our team realized we were asking new
users to tell us about skills they didn’t
have.
We changed strategy to ask about skills
outside of git and GitHub that new users
did have. (Kind of like a census.)
Slide 19
Slide 19 text
Chis Bog, Cols Soy o Cmit Ar (CA) 19
We designed a new instrument
1. Tools in a developer toolkit
2. Channels for tool discovery
3. Biggest personal challenge
4. Ways to solve that challenge
5. Demographics
Slide 20
Slide 20 text
Chis Bog, Cols Soy o Cmit Ar (CA) 20
You have a story
Slide 21
Slide 21 text
Chis Bog, Cols Soy o Cmit Ar (CA) 21
We respect your data
Slide 22
Slide 22 text
Chis Bog, Cols Soy o Cmit Ar (CA) 22
Our design provided us with a
cross-sectional view of data
Analyze a snapshot
of data vs. studying
multiple data points
(time-series data).
17 escalator accidents in 2014.
Slide 23
Slide 23 text
Chis Bog, Cols Soy o Cmit Ar (CA) 23
We looked at who responded
We always begin
analysis by identifying
the “Who.”
Here, we realized that
we had a blazing blind
spot –new users.
Slide 24
Slide 24 text
Chis Bog, Cols Soy o Cmit Ar (CA) 24
So, we changed our
approach again …
It was important to meet new
users where they were at vs.
where we were at.
Slide 25
Slide 25 text
Chis Bog, Cols Soy o Cmit Ar (CA) 25
We divided the
35-question survey into
several smaller surveys,
and rolled them out in
waves over email.
We sent shorter surveys in waves
Slide 26
Slide 26 text
Chis Bog, Cols Soy o Cmit Ar (CA) 26
The “hows” and “whys” were still a mystery.
To better understand outcomes, we needed to
study tools, workflows, & skills-development
over time.
Establishing a baseline
Slide 27
Slide 27 text
Chis Bog, Cols Soy o Cmit Ar (CA) 27
Gather & analyze a single cohort’s data with two
types of studies:
✴ Prospective – identify outcomes as they happen in
real time.
✴ Retrospective – look back at variables over time and
identify how they contributed to known outcomes.
We “accidentally on-purpose”
designed a longitudinal study
Slide 28
Slide 28 text
Chis Bog, Cols Soy o Cmit Ar (CA) 28
A film shot
intermittently from
2002 - 2013
Slide 29
Slide 29 text
Chis Bog, Cols Soy o Cmit Ar (CA) 29
The Harvard Grant Study
Followed 268 men for 75 years as they
both died & aged on into their 90s.
Slide 30
Slide 30 text
Chis Bog, Cols Soy o Cmit Ar (CA) 30
Understanding
new users
We took a cohort of 90,000
new accounts created in
September 2015 & divided
them into two groups.
Slide 31
Slide 31 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
31
1. Explorers
2. Creators
Slide 32
Slide 32 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
32
Evolution vs.
Replacement
Are new users different because people change ?
“evolution”
Or, is the product attracting a new type of user?
“replacement”
Slide 33
Slide 33 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
33
Obvious vs.
Interesting
1. First, when reading graphs identify
the strongest pattern.
2. Next, cover up what’s obvious & look
for what’s interesting.
Slide 34
Slide 34 text
Chis Bog, Cols Soy o Cmit Ar (CA) 34
Q. What’s in your toolkit?
Obvious:
Tenured
accounts are
more likely to use
a text editor
than an IDE.
Slide 35
Slide 35 text
Chis Bog, Cols Soy o Cmit Ar (CA) 35
Q. Experience with tools?
Obvious
Interesting!
Newcomers are
as likely to say
they use neither
an IDE or a Text
Editor, as to say
they use one.
Slide 36
Slide 36 text
Chis Bog, Cols Soy o Cmit Ar (CA) 36
Q. Where do you go for referral?
Obvious:
Both newcomers
& tenured users
act similarly
with regards to
tool discovery,
primarily using
Google.
Slide 37
Slide 37 text
Chis Bog, Cols Soy o Cmit Ar (CA) 37
Q. Primary text editor?
Interesting!
New accounts are
more likely to be
using Notepad++.
29% of the sample .
Slide 38
Slide 38 text
Chis Bog, Cols Soy o Cmit Ar (CA) 38
Obvious
Obvious
Interesting!
Atom’s use is
much smaller
among new users
than we predicted.
Q. Primary text editor?
Slide 39
Slide 39 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
39
52.8K Following
Slide 40
Slide 40 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
40
Github’s Free Text Editor
Slide 41
Slide 41 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
41
Google + % of people who
don’t use a text editor =
growth opportunity for Atom
Combine observations
Slide 42
Slide 42 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
42
Atom is missing from
key search terms
Slide 43
Slide 43 text
Chis Bog, Cols Soy o Cmit Ar (CA) 43
How do you study people
who aren’t engaged in
your product?
Slide 44
Slide 44 text
Chis Bog, Cols Soy o Cmit Ar (CA) 44
Design an exit survey:
1. What were you looking for …?
2. Why did you stop using . . . . . ?
3. What product are you using?
4. What’s one thing we could
have done better?
Slide 45
Slide 45 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
45
Q. Which VCS are you using?
Strong pattern in the
yellows & greens,
which represent
“Nothing” & “SVN.”
As programming
experience increases
people are much more
likely to be using
another VCS vs.
GitHub.
Slide 46
Slide 46 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
46
Q. What’s one thing we could
have done better?
Free private repos
are NOT
universally the
most valuable
GitHub good.
Only among the
most experienced
programmers are
FPR a plurality of
requests.
Slide 47
Slide 47 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
47
We’re talking about
free private repositories, so
let’s discuss how to measure
pricing your product.
$
Slide 48
Slide 48 text
Chis Bog, Cols Soy o Cmit Ar (CA) 48
2
Controlled
Experiment
Slide 49
Slide 49 text
Chis Bog, Cols Soy o Cmit Ar (CA) 49
Q. How much would you pay?
$
$
Photo credit: William Warby (Flickr)
Slide 50
Slide 50 text
Chis Bog, Cols Soy o Cmit Ar (CA) 50
Q. How much would you pay?
$
$
Photo credit: William Warby (Flickr)
Slide 51
Slide 51 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
51
Instead ask about
value –product goods
✴ Mug
✴ T-shirt
✴ Hoodie
✴ Feature(s)
✴ Experiences
Slide 52
Slide 52 text
Chis Bog, Cols Soy o Cmit Ar (CA) 52
The Golden Ticket
✴ Classic controlled
experiment, but with a
nice twist.
✴ 39,800 eligible
candidates between the
treatment & control.
✴ Coupons for free private
repositories (FPR) to
individuals with 1+ year
of tenure.
Slide 53
Slide 53 text
Chis Bog, Cols Soy o Cmit Ar (CA) 53
Experiment design:
39,800 humans
Treatment
(19,949)
3 arms of
6,600 each
Exit Survey
(2,039)
Shared
Feedback
Control: 19,851
Screener
(4,418)
Redeemed
their (FPR)
coupon
Slide 54
Slide 54 text
Chis Bog, Cols Soy o Cmit Ar (CA) 54
3
1 5 N
Three Arms + Control:
1, 3, or 5
private
repositories
No free
repositories
Slide 55
Slide 55 text
Chis Bog, Cols Soy o Cmit Ar (CA) 55
Golden Ticket email
✴ Sent a total of 39,800 emails
✴ “Free private repositories for @name”
✴ “Free for life”
✴ Misunderstandings about the offer
✴ Good email deliverability, but . . .
✴ Overall low redemption rate
Slide 56
Slide 56 text
Chis Bog, Cols Soy o Cmit Ar (CA) 56
Golden Ticket screener
✴The original product
draw.
✴Experience with
competitor products.
✴Technical & social
challenges.
Slide 57
Slide 57 text
Chis Bog, Cols Soy o Cmit Ar (CA) 57
Roll experiments out slowly.
Measure twice, cut once.
Slide 58
Slide 58 text
Chis Bog, Cols Soy o Cmit Ar (CA) 58
Slide 59
Slide 59 text
Chis Bog, Cols Soy o Cmit Ar (CA) 59
Twitter
Leaks
Slide 60
Slide 60 text
Chis Bog, Cols Soy o Cmit Ar (CA) 60
Unfair treatment
Slide 61
Slide 61 text
Chis Bog, Cols Soy o Cmit Ar (CA) 61
Too good to be true?
Slide 62
Slide 62 text
Chis Bog, Cols Soy o Cmit Ar (CA) 62
Measuring different
types of data
Human behaviors with activity data:
1. Coupon redemption
2. Repository creation
Perception of value with survey data:
3. Attitudinal data
Slide 63
Slide 63 text
Chis Bog, Cols Soy o Cmit Ar (CA) 63
The exit survey provides us
with insight into why people
did or did NOT engage in one or
both of the first two activities
(redemption & creation).
Understanding attitudes helps
inform what levers we can
design and pull with
experiences to effect change in
behaviors.
Attitudinal Data
Slide 64
Slide 64 text
Chis Bog, Cols Soy o Cmit Ar (CA) 64
Q. Which would you value
the most?
Good # %
Private repositories 663 36%
GitHub T-shirt 324 17%
Merged Pull Request 311 17%
Git Training 265 14%
GitHub Training 189 10%
“Other” 103 6%
64% indicated they
would get more value
out of something
else.
24% wanted practical
training in Git or
GitHub.
34% reported that
publicly consumable
goods ( t-shirt) would
be more valuable.
Slide 65
Slide 65 text
Chis Bog, Cols Soy o Cmit Ar (CA) 65
Quantitative data is
empirical evidence.
$$ E[Y_{i} | T_{i} = 1] - E[Y_{i} | T_{i} = 0] $$
Slide 66
Slide 66 text
Chis Bog, Cols Soy o Cmit Ar (CA) 66
Qualitative data is
data with soul.
Slide 67
Slide 67 text
Chis Bog, Cols Soy o Cmit Ar (CA) 67
Open text responses
No amount of machine learning can
surface the quality of insights that
reading open text responses does.
Slide 68
Slide 68 text
“Git Udan”
68
Slide 69
Slide 69 text
“Fre e f m
te w unte
re r”
69
Slide 70
Slide 70 text
“at t fie re
rite, or
pit es up
fi pe”
70
Slide 71
Slide 71 text
Chis Bog, Cols Soy o Cmit Ar (CA) 71
Private appears to
be understood as
private only to me
vs.
working with
other people
privately.
Unlimited
Collaborators
Slide 72
Slide 72 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
72
3
Think Aloud
Slide 73
Slide 73 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
73
Faster Horses Syndrome
Listen to the how,
why, when, and
where, behind
customer requests
….
Slide 74
Slide 74 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
74
The Collaboration Study
✴ Customers told us they needed a
complex feature: branch permissions.
✴ Competitor products offered branch
permissions.
✴ Designing for a large audience, means
we need to be thoughtful and
deliberate.
Solve for human motivations & goals behind feature requests.
Slide 75
Slide 75 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
75
Fork v. Branch:
Choosing a
collaboration model
Slide 76
Slide 76 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
76
?
Think Aloud Protocol
Slide 77
Slide 77 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
77
Feature prioritization
✴ Branch Permissions
✴ Automatically syncing forks
✴ Sign-off
✴ Only merge with passing tests
✴ Undo button
✴ Disable force push
✴ Private forks
✴ Prevent merging from the command line
Slide 78
Slide 78 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
78
Feature prioritization
✴ Branch Permissions
✴ Automatically syncing forks
✴ Sign-off
✴ Only merge with passing tests
✴ Undo button
✴ Disable force push
✴ Private forks
✴ Prevent merging from the command line
Slide 79
Slide 79 text
“Wha?! The’s a
un to? Whe?”
Slide 80
Slide 80 text
“Tel bo im n a
un to w ha
hed u.”
Slide 81
Slide 81 text
Chis Bog, Cols Soy o Cmit Ar (CA)
C
81
✴ Include items that don’t exist, but sound like
they might.
✴ Listen to people define what they think the
“feature” is.
✴ Ask how, where, when, & why they would use
the feature.
Sneak Attack
Slide 82
Slide 82 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
82
It's like driving a car at night. You
never see further than your
headlights, but you can make the
whole trip that way. –E.L. Doctorow
Slide 83
Slide 83 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
83
Finding the story
✴ Products have new users, tenured users, and
inactive users, understanding each experience
provides a more complete view.
✴ Researching hard-to-reach places– reading open
text responses & listening to humans share their
motivations and goals is how you find the story.
✴ Research is your flashlight.
Slide 84
Slide 84 text
Chis Bog, Cols Soy o Cmit Ar (CA) 84
Wit rec al
av uc. –@so
Slide 85
Slide 85 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
85
Wrapping Up
1. What’s obvious vs. interesting in your data?
2. How can you gather and use attitudinal data to
study perception of value?
3. Where does a sneak attack make sense?
4. How will you uncover a more complete story?
Slide 86
Slide 86 text
Chis Bog, Cols Soy o Cmit Ar (CA)
⚡
86
Thank you
Questions?
@tenaciouscb