Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)
This talk covers some of the hard-to-measure things we're studying at GitHub. Stories include: cross-sectional survey project, pricing experiment with attitudinal data, and a Think Aloud with a research sneak attack.
Hi, It’s Nice To Meet You ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Synchronized swimming (keep it weird mom) ✴ GitHub’s first UXR 3 GitHub
Listening to people changed everything • Experienced both highly marginalizing & empowering work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub
Researchers are human instruments ✴ Researchers guide customers through interviews, encouraging them to share experiences that depict the why to the what of data. ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub
Story #1. Tools & Workflows Survey ✴ Cross-sectional study ✴ Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.
Upon sign-up we learned: Q. How familiar are you with Git for version control? 76% of people arriving from the U.S. were brand new to git. 3-point scale. GitHub
Cross-sectional study View & interpret a single photo vs. time-series data, which looks at many moments to understand change over time. 17 escalator accidents in 2014. 30 GitHub
Studying new users # # # 32 1. The beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub
We divided up the 35-question survey into several smaller surveys, which we rolled out in waves. We used the opportunity to design a 12-month longitudinal study. 34 GitHub
Longitudinal study (our flashlight into the hows & whys)
Observe a single cohort over time, gathering data about points of interest at repeated intervals. We analyze the data with both prospective and retrospective studies. GitHub
43 1. First, when reading graphs identify the strongest pattern. 2. Next, cover up what’s obvious & look for what’s interesting. Obvious vs. Interesting GitHub
Will you share why @name? Tips: ✴Include a question from the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub
Q. Which VCS are you using? Insight: Strong pattern in the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub
With an exit survey ask … 59 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What’s one thing we could have done better? GitHub
Story #2. The Golden Ticket ✴Classic controlled experiment, but with a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub
67 … from the exit survey tells us why people did or didn’t engage in one or both of the first two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub
Golden Ticket Email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub
Golden Ticket Email ✴ Your original draw to GitHub ✴ Experience with other VCS ✴ If you used a competitor product ✴ Technical & social challenges GitHub
Placing a value on GitHub Goods 77 Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else. 24% wanted practical training in Git or GitHub.
34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub
Story #3. The Collaboration Study ✴ Customers told us they needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub
93 Include items in your list that maybe don’t exist, but sound like they might. Listen to people define what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack
Wrapping Up 1. What’s obvious vs. interesting in your data? 2. How can you use attitudinal data to study perception of value? 3. Where does a sneak attack make sense? GitHub
chrissiebrodigan
stktu
tatsuropianooo
.png){kind=avatar}
researchconf
hide2kano
xtalent
shimacos
kikuzo
smartfukushilab1
kuehara
kei18
marcelosomers
samanthasiow
kastner
holman
shpigford
eileencodes
addyosmani
brianwarren
cassininazir
geoffreycrofte
trishagee