Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)

Slide 1

Slide 1 text

Measuring   Hard To Measure Things 1 Chrissie Brodigan  January 21, 2016, @tenaciouscb GitHub

Slide 2

Slide 2 text

I’m Probably Nervous This is also a dream come true! 2 GitHub

Slide 3

Slide 3 text

Hi, It’s Nice To Meet You ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive ﬁgure skater ✴ Synchronized swimming   (keep it weird mom) ✴ GitHub’s ﬁrst UXR 3 GitHub

Slide 4

Slide 4 text

Technical background Writing Research Design Ethnography Writing GitHub

Slide 5

Slide 5 text

“Single, Unmarried”5 GitHub ✴Age 21 – 27 ✴Unmarried ✴Weight – not over 135 lbs ✴Registered nurse ✴No eyeglasses

Slide 6

Slide 6 text

Larry 6 GitHub ✴You’ve written a clear, but incomplete story. ✴You need to go talk to these women. ✴You need to listen to their stories.

Slide 7

Slide 7 text

“I wa h, no r.” 7 GitHub

Slide 8

Slide 8 text

“ I ha ﬃce h y.” 8 GitHub

Slide 9

Slide 9 text

“ I tal o 17 core, mo w ne l o un.” 9 GitHub

Slide 10

Slide 10 text

Listening to people changed everything • Experienced both highly marginalizing & empowering   work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub

Slide 11

Slide 11 text

There’s nothing like connecting with people. Listening to stories can ﬂip what you think you know, what the data says, on its head. GitHub

Slide 12

Slide 12 text

12 I believe that research studies begin with possibility & conclude with discovery.    There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub

Slide 13

Slide 13 text

13 I believe that research studies begin with possibility & conclude with discovery.    There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub

Slide 14

Slide 14 text

14 I believe that research studies begin with possibility & conclude with discovery.    There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub

Slide 15

Slide 15 text

In this talk, we’ll cover: ✴ GitHub product & background ✴ 3 research techniques & stories ✴ Q&A / Discussion 15 “Tools & Workﬂows survey – Cross-sectional study “The Golden Ticket” – Controlled pricing experiment “Collaboration” study – Exploratory “think aloud” GitHub Three Stories

Slide 16

Slide 16 text

GitHub  ✴ Hosted git version control  ✴ Business: - Free plans for open source - Paid private plans - On-premise Enterprise  ✴ Powered-by Pull Requests  (Code review workﬂows) 16 GitHub

Slide 17

Slide 17 text

Organize code in repositories 17 Photo Credit: Don DeBold, Flicker GitHub

Slide 18

Slide 18 text

18 Hosted mostly in the cloud GitHub

Slide 19

Slide 19 text

GitHub 4 years ago … 19 @kneath persisted & hired me in 2013   Justin is a research superfan! GitHub

Slide 20

Slide 20 text

“Without research … ⚡ ⚡ ⚡ GitHub … all you have is luck.”–@sboag

Slide 21

Slide 21 text

Let’s get into the research. GitHub

Slide 22

Slide 22 text

Hard to Measure ✴ Emotions ✴ Intentions ✴ Motivations + Goals ✴ Workﬂow workarounds ✴ Prior knowledge ✴ Perception 22 GitHub

Slide 23

Slide 23 text

Researchers are  human instruments ✴ Researchers guide customers through interviews, encouraging them to share experiences that depict the why to the what of data.  ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub

Slide 24

Slide 24 text

Story #1. Tools & Workﬂows Survey ✴ Cross-sectional study ✴ Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.

Slide 25

Slide 25 text

Upon sign-up we learned: Q. How familiar are you with Git for version control? 76% of people arriving from the U.S. were   brand new to git. 3-point scale. GitHub

Slide 26

Slide 26 text

We realized that we were asking about skills people didn’t have. " We shifted strategy to ask people about what they do know. GitHub

Slide 27

Slide 27 text

Tools & Workﬂows Instrument Design  (35 questions) 1. Tools in your developer toolkit 2. Channels used for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics (human age, etc.) 27 GitHub

Slide 28

Slide 28 text

GitHub

Slide 29

Slide 29 text

Optional Demographics GitHub

Slide 30

Slide 30 text

Cross-sectional study View & interpret a single photo vs. time-series data, which looks at many moments to understand change over time. 17 escalator accidents in 2014. 30 GitHub

Slide 31

Slide 31 text

Who? We always begin analysis by identifying the “Who.” And, we realized that we had a blind spot   –new users.  31 GitHub

Slide 32

Slide 32 text

Studying   new users # # # 32 1. The beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to ﬁnd them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workﬂows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub

Slide 33

Slide 33 text

Evolve our approach to meet new users where they are vs. where we are. GitHub

Slide 34

Slide 34 text

We divided up the 35-question survey into several smaller surveys, which we rolled out in waves. We used the opportunity to design a 12-month longitudinal study. 34 GitHub

Slide 35

Slide 35 text

  Longitudinal study   (our ﬂashlight into the hows & whys)   Observe a single cohort over time, gathering data about points of interest at repeated intervals.   We analyze the data with both prospective and retrospective studies. GitHub

Slide 36

Slide 36 text

Boyhood  A ﬁlm shot intermittently from 2002 - 2013 36 GitHub

Slide 37

Slide 37 text

The Harvard Grant Study   Followed 268 men for 75 years as they both died & aged on into their 90s. 37 GitHub

Slide 38

Slide 38 text

New Account Creators Study Researchers and their methodology will   naturally age alongside their subjects. 38 GitHub

Slide 39

Slide 39 text

We took a cohort of 90,000 new accounts created in September 2015 & divided them into two groups. GitHub

Slide 40

Slide 40 text

New Account Creators Study (NAC) Explorers Creators GitHub

Slide 41

Slide 41 text

We’re 5 months into the NAC, so we’ll look at a cross-sectional view of the current data. Think: 17 escalator accidents last year GitHub

Slide 42

Slide 42 text

Are new users different because people change over time? (evolution)    Or, is GitHub attracting a new type of user?  (replacement) GitHub

Slide 43

Slide 43 text

43 1. First, when reading graphs identify the strongest pattern.  2. Next, cover up what’s obvious & look for what’s interesting. Obvious vs. Interesting GitHub

Slide 44

Slide 44 text

Q. What’s in your toolkit? Obvious: Tenured accounts are more likely to use a text editor than an IDE. GitHub

Slide 45

Slide 45 text

Experience with tools Obvious Interesting! Newcomers are as likely to say they use neither an IDE or a Text Editor, as to say they use one. GitHub

Slide 46

Slide 46 text

Q. Primary text editor? New accounts are more likely to be using Notepad++. 29% of the sample GitHub

Slide 47

Slide 47 text

Q. Primary text editor? Interesting! Atom’s use is much smaller among new users than we thought it would be. Obvious Obvious GitHub

Slide 48

Slide 48 text

52.8K Following 48 GitHub

Slide 49

Slide 49 text

Q. Where do you go for advice? One area where both newcomers & tenured users act similarly – tool discovery. GitHub

Slide 50

Slide 50 text

When we put all 3 insights together & looked more closely at the world,   we noticed a big blind spot. GitHub

Slide 51

Slide 51 text

Google + % of people who don’t use a text editor = . . .audience opportunity GitHub

Slide 52

Slide 52 text

GitHub Atom doesn’t show up until the 4th page

Slide 53

Slide 53 text

We’ve been talking a engaged users,   let’s talk about inactive users. GitHub

Slide 54

Slide 54 text

How do you study inactive users? GitHub

Slide 55

Slide 55 text

Github “365” Survey 55 GitHub Thesis: People are burning brightly somewhere, just not at GitHub.

Slide 56

Slide 56 text

Will you share why @name? Tips: ✴Include a question from the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub

Slide 57

Slide 57 text

Q. Which VCS are you using? Insight: Strong pattern in the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub

Slide 58

Slide 58 text

Q. If we could have done one thing … Insight:    Free private repos are NOT universally the most valuable GitHub good.    Only among the   most experienced programmers are FPR a plurality of requests. GitHub

Slide 59

Slide 59 text

With an exit survey ask … 59 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What’s one thing we could have done better? GitHub

Slide 60

Slide 60 text

We’re talking about free private repositories, so let’s discuss how to measure something like pricing your product. GitHub

Slide 61

Slide 61 text

61 Q. How much would you pay for GitHub? $ $ $ GitHub ? Photo credit: William Warby (Flickr)

Slide 62

Slide 62 text

62 Q. How much would you pay for GitHub? ? ? ? GitHub ? Photo credit: William Warby (Flickr)

Slide 63

Slide 63 text

Ask about value –GitHub goods ✴ Mug ✴ T-shirt ✴ Hoodie ✴ Feature(s) ✴ Experiences 63 GitHub

Slide 64

Slide 64 text

Story #2. The Golden Ticket ✴Classic controlled experiment, but with a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub

Slide 65

Slide 65 text

Experiment Design  39,800 Humans 65 Treatment (19,949) 3 arms of 6,600 Exit Survey (2,039)     Shared Feedback Control: 19,851 Screener  (4,418) Redeemed their (FPR) coupon GitHub

Slide 66

Slide 66 text

Data & Measurement 1. Coupon redemption 2. Repository creation 3. Perception of value GitHub

Slide 67

Slide 67 text

67 … from the exit survey tells us why people did or didn’t engage in one or both of the ﬁrst two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub

Slide 68

Slide 68 text

Golden Ticket Email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub

Slide 69

Slide 69 text

Golden Ticket Email ✴ Your original draw to GitHub ✴ Experience with other VCS ✴ If you used a competitor product ✴ Technical & social challenges GitHub

Slide 70

Slide 70 text

Roll your experiment out slowly.   Measure twice, cut once. GitHub

Slide 71

Slide 71 text

GitHub

Slide 72

Slide 72 text

Twitter  Leaks 72 GitHub

Slide 73

Slide 73 text

Unfair   Treatment 73 GitHub

Slide 74

Slide 74 text

Too Good To Be True? 74 GitHub

Slide 75

Slide 75 text

We were interested in what people valued most:   free private repositories or some other good? 75 GitHub

Slide 76

Slide 76 text

Q. Which would you value the most? GitHub

Slide 77

Slide 77 text

Placing a value on GitHub Goods 77 Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else.  24% wanted practical training in Git or GitHub.   34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub

Slide 78

Slide 78 text

Open Text Responses 78 No amount of machine learning or text analysis   can surface the insights reading open text does. GitHub

Slide 79

Slide 79 text

“GitHub underpants” GitHub

Slide 80

Slide 80 text

Q. If we could grant you one wish to make GitHub even better, what would you wish for? Tip: “Wishes” help surface emerging trends GitHub

Slide 81

Slide 81 text

“Fre e f m te w unte re r” 81 GitHub ⋆

Slide 82

Slide 82 text

“at t ﬁe re rite, or pit es a p ﬁe pe” 82 GitHub ⋆

Slide 83

Slide 83 text

Unlimited Collaborators 83 Private appears to be understood as private only to me vs.   working with other people privately. GitHub

Slide 84

Slide 84 text

Private for only you. GitHub

Slide 85

Slide 85 text

Invite your friends to your private code base. GitHub

Slide 86

Slide 86 text

Faster Horses Speaking of listening to customers and anyone who spends their time listening to customer requests …. GitHub

Slide 87

Slide 87 text

Story #3. The Collaboration Study ✴ Customers told us they needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub

Slide 88

Slide 88 text

Fork v. Branch: Choosing a collaboration model GitHub

Slide 89

Slide 89 text

Feature Prioritization 89 ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub

Slide 90

Slide 90 text

Feature Prioritization 90 ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub

Slide 91

Slide 91 text

“Wha?! The’s a   un to? Whe?” 91 GitHub

Slide 92

Slide 92 text

“Tel bo im n a   un to w ha le y.” 92 GitHub

Slide 93

Slide 93 text

93 Include items in your list that maybe don’t exist, but sound like they might. Listen to people deﬁne what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack

Slide 94

Slide 94 text

Wrapping Up 1. What’s obvious vs. interesting in your data?  2. How can you use attitudinal data to study perception of value?  3. Where does a sneak attack make sense? GitHub

Slide 95

Slide 95 text

GitHub   Design & Research Teams are Growing! cb@github.com GitHub

Slide 96

Slide 96 text

Medium (medium.com/@tenaciouscb) 96 ✴ Product Pricing, Attitudinal Data, & GitHub Underpants ✴ What’s Obvious vs. What’s Interesting ✴ GitHub Transformers: Tools & Workﬂows ✴ New Year, New User Journeys ✴ 365 Project: Listening to inactive users ✴ Measuring Hard-to-Measure Things GitHub

Slide 97

Slide 97 text

Speakerdeck (speakerdeck.com/chrissiebrodigan) 97 GitHub

Slide 98

Slide 98 text

Thank You 98 @tenaciouscb | cb@github.com — GitHub