Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)

This talk covers some of the hard-to-measure things we're studying at GitHub. Stories include: cross-sectional survey project, pricing experiment with attitudinal data, and a Think Aloud with a research sneak attack.

Transcript

1. Measuring 
 Hard To Measure Things 1 Chrissie Brodigan
 January 21, 2016, @tenaciouscb GitHub
2. I’m Probably Nervous This is also a dream come true! 2 GitHub
3. Hi, It’s Nice To Meet You ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Synchronized swimming 
 (keep it weird mom) ✴ GitHub’s first UXR 3 GitHub
4. Technical background Writing Research Design Ethnography Writing GitHub
5. “Single, Unmarried”5 GitHub ✴Age 21 – 27 ✴Unmarried ✴Weight – not over 135 lbs ✴Registered nurse ✴No eyeglasses
6. Larry 6 GitHub ✴You’ve written a clear, but incomplete story. ✴You need to go talk to these women. ✴You need to listen to their stories.
7. “I wa h, no r.” 7 GitHub
8. “ I ha ffice h y.” 8 GitHub
9. “ I tal o 17 core, mo w ne l o un.” 9 GitHub
10. Listening to people changed everything • Experienced both highly marginalizing & empowering 
 work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub
11. There’s nothing like connecting with people. Listening to stories can flip what you think you know, what the data says, on its head. GitHub
12. 12 I believe that research studies begin with possibility & conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
13. 13 I believe that research studies begin with possibility & conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
14. 14 I believe that research studies begin with possibility & conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
15. In this talk, we’ll cover: ✴ GitHub product & background ✴ 3 research techniques & stories ✴ Q&A / Discussion 15 “Tools & Workflows survey – Cross-sectional study “The Golden Ticket” – Controlled pricing experiment “Collaboration” study – Exploratory “think aloud” GitHub Three Stories
16. GitHub
 ✴ Hosted git version control
 ✴ Business: - Free plans for open source - Paid private plans - On-premise Enterprise
 ✴ Powered-by Pull Requests
 (Code review workflows) 16 GitHub
17. Organize code in repositories 17 Photo Credit: Don DeBold, Flicker GitHub
18. 18 Hosted mostly in the cloud GitHub
19. GitHub 4 years ago … 19 @kneath persisted & hired me in 2013 
 Justin is a research superfan! GitHub
20. “Without research … ⚡ ⚡ ⚡ GitHub … all you have is luck.”–@sboag
21. Let’s get into the research. GitHub
22. Hard to Measure ✴ Emotions ✴ Intentions ✴ Motivations + Goals ✴ Workflow workarounds ✴ Prior knowledge ✴ Perception 22 GitHub
23. Researchers are
 human instruments ✴ Researchers guide customers through interviews, encouraging them to share experiences that depict the why to the what of data.
 ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub
24. Story #1. Tools & Workflows Survey ✴ Cross-sectional study ✴ Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.
25. Upon sign-up we learned: Q. How familiar are you with Git for version control? 76% of people arriving from the U.S. were 
 brand new to git. 3-point scale. GitHub
26. We realized that we were asking about skills people didn’t have. " We shifted strategy to ask people about what they do know. GitHub
27. Tools & Workflows Instrument Design
 (35 questions) 1. Tools in your developer toolkit 2. Channels used for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics (human age, etc.) 27 GitHub
28. GitHub
29. Optional Demographics GitHub
30. Cross-sectional study View & interpret a single photo vs. time-series data, which looks at many moments to understand change over time. 17 escalator accidents in 2014. 30 GitHub
31. Who? We always begin analysis by identifying the “Who.” And, we realized that we had a blind spot 
 –new users.
 31 GitHub
32. Studying 
 new users # # # 32 1. The beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub
33. Evolve our approach to meet new users where they are vs. where we are. GitHub
34. We divided up the 35-question survey into several smaller surveys, which we rolled out in waves. We used the opportunity to design a 12-month longitudinal study. 34 GitHub
35. 
 Longitudinal study 
 (our flashlight into the hows & whys) 
 Observe a single cohort over time, gathering data about points of interest at repeated intervals. 
 We analyze the data with both prospective and retrospective studies. GitHub
36. Boyhood
 A film shot intermittently from 2002 - 2013 36 GitHub
37. The Harvard Grant Study 
 Followed 268 men for 75 years as they both died & aged on into their 90s. 37 GitHub
38. New Account Creators Study Researchers and their methodology will 
 naturally age alongside their subjects. 38 GitHub
39. We took a cohort of 90,000 new accounts created in September 2015 & divided them into two groups. GitHub
40. New Account Creators Study (NAC) Explorers Creators GitHub
41. We’re 5 months into the NAC, so we’ll look at a cross-sectional view of the current data. Think: 17 escalator accidents last year GitHub
42. Are new users different because people change over time? (evolution)
 
 Or, is GitHub attracting a new type of user?
 (replacement) GitHub
43. 43 1. First, when reading graphs identify the strongest pattern.
 2. Next, cover up what’s obvious & look for what’s interesting. Obvious vs. Interesting GitHub
44. Q. What’s in your toolkit? Obvious: Tenured accounts are more likely to use a text editor than an IDE. GitHub
45. Experience with tools Obvious Interesting! Newcomers are as likely to say they use neither an IDE or a Text Editor, as to say they use one. GitHub
46. Q. Primary text editor? New accounts are more likely to be using Notepad++. 29% of the sample GitHub
47. Q. Primary text editor? Interesting! Atom’s use is much smaller among new users than we thought it would be. Obvious Obvious GitHub
48. 52.8K Following 48 GitHub
49. Q. Where do you go for advice? One area where both newcomers & tenured users act similarly – tool discovery. GitHub
50. When we put all 3 insights together & looked more closely at the world, 
 we noticed a big blind spot. GitHub
51. Google + % of people who don’t use a text editor = . . .audience opportunity GitHub
52. GitHub Atom doesn’t show up until the 4th page
53. We’ve been talking a engaged users, 
 let’s talk about inactive users. GitHub
54. How do you study inactive users? GitHub
55. Github “365” Survey 55 GitHub Thesis: People are burning brightly somewhere, just not at GitHub.
56. Will you share why @name? Tips: ✴Include a question from the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub
57. Q. Which VCS are you using? Insight: Strong pattern in the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub
58. Q. If we could have done one thing … Insight:
 
 Free private repos are NOT universally the most valuable GitHub good.
 
 Only among the 
 most experienced programmers are FPR a plurality of requests. GitHub
59. With an exit survey ask … 59 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What’s one thing we could have done better? GitHub
60. We’re talking about free private repositories, so let’s discuss how to measure something like pricing your product. GitHub
61. 61 Q. How much would you pay for GitHub? $ $ $ GitHub ? Photo credit: William Warby (Flickr)
62. 62 Q. How much would you pay for GitHub? ? ? ? GitHub ? Photo credit: William Warby (Flickr)
63. Ask about value –GitHub goods ✴ Mug ✴ T-shirt ✴ Hoodie ✴ Feature(s) ✴ Experiences 63 GitHub
64. Story #2. The Golden Ticket ✴Classic controlled experiment, but with a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub
65. Experiment Design
 39,800 Humans 65 Treatment (19,949) 3 arms of 6,600 Exit Survey (2,039) 
 
 Shared Feedback Control: 19,851 Screener
 (4,418) Redeemed their (FPR) coupon GitHub
66. Data & Measurement 1. Coupon redemption 2. Repository creation 3. Perception of value GitHub
67. 67 … from the exit survey tells us why people did or didn’t engage in one or both of the first two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub
68. Golden Ticket Email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub
69. Golden Ticket Email ✴ Your original draw to GitHub ✴ Experience with other VCS ✴ If you used a competitor product ✴ Technical & social challenges GitHub
70. Roll your experiment out slowly. 
 Measure twice, cut once. GitHub
71. GitHub
72. Twitter
 Leaks 72 GitHub
73. Unfair 
 Treatment 73 GitHub
74. Too Good To Be True? 74 GitHub
75. We were interested in what people valued most: 
 free private repositories or some other good? 75 GitHub
76. Q. Which would you value the most? GitHub
77. Placing a value on GitHub Goods 77 Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else.
 24% wanted practical training in Git or GitHub. 
 34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub
78. Open Text Responses 78 No amount of machine learning or text analysis 
 can surface the insights reading open text does. GitHub
79. “GitHub underpants” GitHub
80. Q. If we could grant you one wish to make GitHub even better, what would you wish for? Tip: “Wishes” help surface emerging trends GitHub
81. “Fre e f m te w unte re r” 81 GitHub ⋆
82. “at t fie re rite, or pit es a p fie pe” 82 GitHub ⋆
83. Unlimited Collaborators 83 Private appears to be understood as private only to me vs. 
 working with other people privately. GitHub
84. Private for only you. GitHub
85. Invite your friends to your private code base. GitHub
86. Faster Horses Speaking of listening to customers and anyone who spends their time listening to customer requests …. GitHub
87. Story #3. The Collaboration Study ✴ Customers told us they needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub
88. Fork v. Branch: Choosing a collaboration model GitHub
89. Feature Prioritization 89 ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
90. Feature Prioritization 90 ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
91. “Wha?! The’s a 
 un to? Whe?” 91 GitHub
92. “Tel bo im n a 
 un to w ha le y.” 92 GitHub
93. 93 Include items in your list that maybe don’t exist, but sound like they might. Listen to people define what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack
94. Wrapping Up 1. What’s obvious vs. interesting in your data?
 2. How can you use attitudinal data to study perception of value?
 3. Where does a sneak attack make sense? GitHub
95. GitHub 
 Design & Research Teams are Growing! cb@github.com GitHub
96. Medium (medium.com/@tenaciouscb) 96 ✴ Product Pricing, Attitudinal Data, & GitHub Underpants ✴ What’s Obvious vs. What’s Interesting ✴ GitHub Transformers: Tools & Workflows ✴ New Year, New User Journeys ✴ 365 Project: Listening to inactive users ✴ Measuring Hard-to-Measure Things GitHub
97. Speakerdeck (speakerdeck.com/chrissiebrodigan) 97 GitHub
98. Thank You 98 @tenaciouscb | cb@github.com — GitHub