Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)

Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)

This talk covers some of the hard-to-measure things we're studying at GitHub. Stories include: cross-sectional survey project, pricing experiment with attitudinal data, and a Think Aloud with a research sneak attack.

F091db85bc34be872c938ba9b0901e60?s=128

Chrissie Brodigan

January 16, 2016
Tweet

Transcript

  1. Measuring 
 Hard To Measure Things 1 Chrissie Brodigan
 January

    21, 2016, @tenaciouscb GitHub
  2. I’m Probably Nervous This is also a dream come true!

    2 GitHub
  3. Hi, It’s Nice To Meet You ✴ Live in Sausalito

    ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Synchronized swimming 
 (keep it weird mom) ✴ GitHub’s first UXR 3 GitHub
  4. Technical background Writing Research Design Ethnography Writing GitHub

  5. “Single, Unmarried”5 GitHub ✴Age 21 – 27 ✴Unmarried ✴Weight –

    not over 135 lbs ✴Registered nurse ✴No eyeglasses
  6. Larry 6 GitHub ✴You’ve written a clear, but incomplete story.

    ✴You need to go talk to these women. ✴You need to listen to their stories.
  7. “I wa h, no r.” 7 GitHub

  8. “ I ha ffice h y.” 8 GitHub

  9. “ I tal o 17 core, mo w ne l

    o un.” 9 GitHub
  10. Listening to people changed everything • Experienced both highly marginalizing

    & empowering 
 work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub
  11. There’s nothing like connecting with people. Listening to stories can

    flip what you think you know, what the data says, on its head. GitHub
  12. 12 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  13. 13 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  14. 14 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  15. In this talk, we’ll cover: ✴ GitHub product & background

    ✴ 3 research techniques & stories ✴ Q&A / Discussion 15 “Tools & Workflows survey – Cross-sectional study “The Golden Ticket” – Controlled pricing experiment “Collaboration” study – Exploratory “think aloud” GitHub Three Stories
  16. GitHub
 ✴ Hosted git version control
 ✴ Business: - Free

    plans for open source - Paid private plans - On-premise Enterprise
 ✴ Powered-by Pull Requests
 (Code review workflows) 16 GitHub
  17. Organize code in repositories 17 Photo Credit: Don DeBold, Flicker

    GitHub
  18. 18 Hosted mostly in the cloud GitHub

  19. GitHub 4 years ago … 19 @kneath persisted & hired

    me in 2013 
 Justin is a research superfan! GitHub
  20. “Without research … ⚡ ⚡ ⚡ GitHub … all you

    have is luck.”–@sboag
  21. Let’s get into the research. GitHub

  22. Hard to Measure ✴ Emotions ✴ Intentions ✴ Motivations +

    Goals ✴ Workflow workarounds ✴ Prior knowledge ✴ Perception 22 GitHub
  23. Researchers are
 human instruments ✴ Researchers guide customers through interviews,

    encouraging them to share experiences that depict the why to the what of data.
 ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub
  24. Story #1. Tools & Workflows Survey ✴ Cross-sectional study ✴

    Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.
  25. Upon sign-up we learned: Q. How familiar are you with

    Git for version control? 76% of people arriving from the U.S. were 
 brand new to git. 3-point scale. GitHub
  26. We realized that we were asking about skills people didn’t

    have. " We shifted strategy to ask people about what they do know. GitHub
  27. Tools & Workflows Instrument Design
 (35 questions) 1. Tools in

    your developer toolkit 2. Channels used for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics (human age, etc.) 27 GitHub
  28. GitHub

  29. Optional Demographics GitHub

  30. Cross-sectional study View & interpret a single photo vs. time-series

    data, which looks at many moments to understand change over time. 17 escalator accidents in 2014. 30 GitHub
  31. Who? We always begin analysis by identifying the “Who.” And,

    we realized that we had a blind spot 
 –new users.
 31 GitHub
  32. Studying 
 new users # # # 32 1. The

    beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub
  33. Evolve our approach to meet new users where they are

    vs. where we are. GitHub
  34. We divided up the 35-question survey into several smaller surveys,

    which we rolled out in waves. We used the opportunity to design a 12-month longitudinal study. 34 GitHub
  35. 
 Longitudinal study 
 (our flashlight into the hows &

    whys) 
 Observe a single cohort over time, gathering data about points of interest at repeated intervals. 
 We analyze the data with both prospective and retrospective studies. GitHub
  36. Boyhood
 A film shot intermittently from 2002 - 2013 36

    GitHub
  37. The Harvard Grant Study 
 Followed 268 men for 75

    years as they both died & aged on into their 90s. 37 GitHub
  38. New Account Creators Study Researchers and their methodology will 


    naturally age alongside their subjects. 38 GitHub
  39. We took a cohort of 90,000 new accounts created in

    September 2015 & divided them into two groups. GitHub
  40. New Account Creators Study (NAC) Explorers Creators GitHub

  41. We’re 5 months into the NAC, so we’ll look at

    a cross-sectional view of the current data. Think: 17 escalator accidents last year GitHub
  42. Are new users different because people change over time? (evolution)


    
 Or, is GitHub attracting a new type of user?
 (replacement) GitHub
  43. 43 1. First, when reading graphs identify the strongest pattern.


    2. Next, cover up what’s obvious & look for what’s interesting. Obvious vs. Interesting GitHub
  44. Q. What’s in your toolkit? Obvious: Tenured accounts are more

    likely to use a text editor than an IDE. GitHub
  45. Experience with tools Obvious Interesting! Newcomers are as likely to

    say they use neither an IDE or a Text Editor, as to say they use one. GitHub
  46. Q. Primary text editor? New accounts are more likely to

    be using Notepad++. 29% of the sample GitHub
  47. Q. Primary text editor? Interesting! Atom’s use is much smaller

    among new users than we thought it would be. Obvious Obvious GitHub
  48. 52.8K Following 48 GitHub

  49. Q. Where do you go for advice? One area where

    both newcomers & tenured users act similarly – tool discovery. GitHub
  50. When we put all 3 insights together & looked more

    closely at the world, 
 we noticed a big blind spot. GitHub
  51. Google + % of people who don’t use a text

    editor = . . .audience opportunity GitHub
  52. GitHub Atom doesn’t show up until the 4th page

  53. We’ve been talking a engaged users, 
 let’s talk about

    inactive users. GitHub
  54. How do you study inactive users? GitHub

  55. Github “365” Survey 55 GitHub Thesis: People are burning brightly

    somewhere, just not at GitHub.
  56. Will you share why @name? Tips: ✴Include a question from

    the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub
  57. Q. Which VCS are you using? Insight: Strong pattern in

    the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub
  58. Q. If we could have done one thing … Insight:


    
 Free private repos are NOT universally the most valuable GitHub good.
 
 Only among the 
 most experienced programmers are FPR a plurality of requests. GitHub
  59. With an exit survey ask … 59 1. What were

    you looking for …? 2. Why did you stop using . . . . . ? 3. What’s one thing we could have done better? GitHub
  60. We’re talking about free private repositories, so let’s discuss how

    to measure something like pricing your product. GitHub
  61. 61 Q. How much would you pay for GitHub? $

    $ $ GitHub ? Photo credit: William Warby (Flickr)
  62. 62 Q. How much would you pay for GitHub? ?

    ? ? GitHub ? Photo credit: William Warby (Flickr)
  63. Ask about value –GitHub goods ✴ Mug ✴ T-shirt ✴

    Hoodie ✴ Feature(s) ✴ Experiences 63 GitHub
  64. Story #2. The Golden Ticket ✴Classic controlled experiment, but with

    a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub
  65. Experiment Design
 39,800 Humans 65 Treatment (19,949) 3 arms of

    6,600 Exit Survey (2,039) 
 
 Shared Feedback Control: 19,851 Screener
 (4,418) Redeemed their (FPR) coupon GitHub
  66. Data & Measurement 1. Coupon redemption 2. Repository creation 3.

    Perception of value GitHub
  67. 67 … from the exit survey tells us why people

    did or didn’t engage in one or both of the first two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub
  68. Golden Ticket Email ✴ Sent a total of 39,800 emails

    ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub
  69. Golden Ticket Email ✴ Your original draw to GitHub ✴

    Experience with other VCS ✴ If you used a competitor product ✴ Technical & social challenges GitHub
  70. Roll your experiment out slowly. 
 Measure twice, cut once.

    GitHub
  71. GitHub

  72. Twitter
 Leaks 72 GitHub

  73. Unfair 
 Treatment 73 GitHub

  74. Too Good To Be True? 74 GitHub

  75. We were interested in what people valued most: 
 free

    private repositories or some other good? 75 GitHub
  76. Q. Which would you value the most? GitHub

  77. Placing a value on GitHub Goods 77 Good # %

    Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else.
 24% wanted practical training in Git or GitHub. 
 34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub
  78. Open Text Responses 78 No amount of machine learning or

    text analysis 
 can surface the insights reading open text does. GitHub
  79. “GitHub underpants” GitHub

  80. Q. If we could grant you one wish to make

    GitHub even better, what would you wish for? Tip: “Wishes” help surface emerging trends GitHub
  81. “Fre e f m te w unte re r” 81

    GitHub ⋆
  82. “at t fie re rite, or pit es a p

    fie pe” 82 GitHub ⋆
  83. Unlimited Collaborators 83 Private appears to be understood as private

    only to me vs. 
 working with other people privately. GitHub
  84. Private for only you. GitHub

  85. Invite your friends to your private code base. GitHub

  86. Faster Horses Speaking of listening to customers and anyone who

    spends their time listening to customer requests …. GitHub
  87. Story #3. The Collaboration Study ✴ Customers told us they

    needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub
  88. Fork v. Branch: Choosing a collaboration model GitHub

  89. Feature Prioritization 89 ✴ Branch Permissions ✴ Automatically syncing forks

    ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
  90. Feature Prioritization 90 ✴ Branch Permissions ✴ Automatically syncing forks

    ✴ Sign-off ✴ Only merge with passing tests ✴Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
  91. “Wha?! The’s a 
 un to? Whe?” 91 GitHub

  92. “Tel bo im n a 
 un to w ha

    le y.” 92 GitHub
  93. 93 Include items in your list that maybe don’t exist,

    but sound like they might. Listen to people define what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack
  94. Wrapping Up 1. What’s obvious vs. interesting in your data?


    2. How can you use attitudinal data to study perception of value?
 3. Where does a sneak attack make sense? GitHub
  95. GitHub 
 Design & Research Teams are Growing! cb@github.com GitHub

  96. Medium (medium.com/@tenaciouscb) 96 ✴ Product Pricing, Attitudinal Data, & GitHub

    Underpants ✴ What’s Obvious vs. What’s Interesting ✴ GitHub Transformers: Tools & Workflows ✴ New Year, New User Journeys ✴ 365 Project: Listening to inactive users ✴ Measuring Hard-to-Measure Things GitHub
  97. Speakerdeck (speakerdeck.com/chrissiebrodigan) 97 GitHub

  98. Thank You 98 @tenaciouscb | cb@github.com — GitHub