Chis Bog, Cols Soy o Cmit Ar (CA) C 2 About CB Hi, I’m Chrissie Brodigan ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Built GitHub’s early UXR practice (2013 - 2016) Twitter: @tenaciouscb
Chis Bog, Cols Soy o Cmit Ar (CA) C 4 Airborne Heroines Job requirements ✴ Age 21 – 27 ✴ Unmarried ✴ Weight – not over 135 lbs ✴ Registered nurse ✴ No eyeglasses
Chis Bog, Cols Soy o Cmit Ar (CA) 9 A more complete story ✴ Experienced both highly marginalizing & empowering work conditions. ✴ Skilled, professional, & organized workers in their own labor union. ✴ Were part of a process that changed constitutional law.
Chis Bog, Cols Soy o Cmit Ar (CA) C Surveys help us to measure hard-to-measure things ✴ Emotions ✴ Intentions ✴ Motivations + Goals ✴ Workflow workarounds ✴ Prior knowledge ✴ Perception 13
Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 15 It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way. –E.L. Doctorow
Chis Bog, Cols Soy o Cmit Ar (CA) C 1. Tools & Workflows survey START FINISH 2 3 2. New Users Tools & Workflows survey 3. Inactive Users “365” survey The path to a more complete user story may be completely surprising! 16
Chis Bog, Cols Soy o Cmit Ar (CA) Q. How familiar are you with git for version control? 17 ✴76% of people arriving from the U.S. were brand new to git. ✴3-point scale.
Chis Bog, Cols Soy o Cmit Ar (CA) 19 We designed a new instrument 1. Tools in a developer toolkit 2. Channels for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics
Chis Bog, Cols Soy o Cmit Ar (CA) 22 Our design provided us with a cross-sectional view of data Analyze a snapshot of data vs. studying multiple data points (time-series data). 17 escalator accidents in 2014.
Chis Bog, Cols Soy o Cmit Ar (CA) 23 We looked at who responded We always begin analysis by identifying the “Who.” Here, we realized that we had a blazing blind spot –new users.
Chis Bog, Cols Soy o Cmit Ar (CA) 25 We divided the 35-question survey into several smaller surveys, and rolled them out in waves over email. We sent shorter surveys in waves
Chis Bog, Cols Soy o Cmit Ar (CA) 27 Gather & analyze a single cohort’s data with two types of studies: ✴ Prospective – identify outcomes as they happen in real time. ✴ Retrospective – look back at variables over time and identify how they contributed to known outcomes. We “accidentally on-purpose” designed a longitudinal study
Chis Bog, Cols Soy o Cmit Ar (CA) 30 Understanding new users We took a cohort of 90,000 new accounts created in September 2015 & divided them into two groups.
Chis Bog, Cols Soy o Cmit Ar (CA) C 33 Obvious vs. Interesting 1. First, when reading graphs identify the strongest pattern. 2. Next, cover up what’s obvious & look for what’s interesting.
Chis Bog, Cols Soy o Cmit Ar (CA) 35 Q. Experience with tools? Obvious Interesting! Newcomers are as likely to say they use neither an IDE or a Text Editor, as to say they use one.
Chis Bog, Cols Soy o Cmit Ar (CA) 38 Obvious Obvious Interesting! Atom’s use is much smaller among new users than we predicted. Q. Primary text editor?
Chis Bog, Cols Soy o Cmit Ar (CA) 44 Design an exit survey: 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What product are you using? 4. What’s one thing we could have done better?
Chis Bog, Cols Soy o Cmit Ar (CA) C 45 Q. Which VCS are you using? Strong pattern in the yellows & greens, which represent “Nothing” & “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub.
Chis Bog, Cols Soy o Cmit Ar (CA) C 46 Q. What’s one thing we could have done better? Free private repos are NOT universally the most valuable GitHub good.
Only among the most experienced programmers are FPR a plurality of requests.
Chis Bog, Cols Soy o Cmit Ar (CA) 52 The Golden Ticket ✴ Classic controlled experiment, but with a nice twist. ✴ 39,800 eligible candidates between the treatment & control. ✴ Coupons for free private repositories (FPR) to individuals with 1+ year of tenure.
Chis Bog, Cols Soy o Cmit Ar (CA) 55 Golden Ticket email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate
Chis Bog, Cols Soy o Cmit Ar (CA) 56 Golden Ticket screener ✴The original product draw. ✴Experience with competitor products. ✴Technical & social challenges.
Chis Bog, Cols Soy o Cmit Ar (CA) 63 The exit survey provides us with insight into why people did or did NOT engage in one or both of the first two activities (redemption & creation). Understanding attitudes helps inform what levers we can design and pull with experiences to effect change in behaviors. Attitudinal Data
Chis Bog, Cols Soy o Cmit Ar (CA) 64 Q. Which would you value the most? Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else. 24% wanted practical training in Git or GitHub.
34% reported that publicly consumable goods ( t-shirt) would be more valuable.
Chis Bog, Cols Soy o Cmit Ar (CA) 67 Open text responses No amount of machine learning can surface the quality of insights that reading open text responses does.
Chis Bog, Cols Soy o Cmit Ar (CA) 71 Private appears to be understood as private only to me vs. working with other people privately. Unlimited Collaborators
Chis Bog, Cols Soy o Cmit Ar (CA) C 74 The Collaboration Study ✴ Customers told us they needed a complex feature: branch permissions. ✴ Competitor products offered branch permissions. ✴ Designing for a large audience, means we need to be thoughtful and deliberate. Solve for human motivations & goals behind feature requests.
Chis Bog, Cols Soy o Cmit Ar (CA) C 77 Feature prioritization ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line
Chis Bog, Cols Soy o Cmit Ar (CA) C 78 Feature prioritization ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line
Chis Bog, Cols Soy o Cmit Ar (CA) C 81 ✴ Include items that don’t exist, but sound like they might. ✴ Listen to people define what they think the “feature” is. ✴ Ask how, where, when, & why they would use the feature. Sneak Attack
Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 82 It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way. –E.L. Doctorow
Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 83 Finding the story ✴ Products have new users, tenured users, and inactive users, understanding each experience provides a more complete view. ✴ Researching hard-to-reach places– reading open text responses & listening to humans share their motivations and goals is how you find the story. ✴ Research is your flashlight.
Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 85 Wrapping Up 1. What’s obvious vs. interesting in your data? 2. How can you gather and use attitudinal data to study perception of value? 3. Where does a sneak attack make sense? 4. How will you uncover a more complete story?
chrissiebrodigan
masakat0
ya9do
forest1040
schulta
shimacos
bakanaouji
kuehara
koharite
rina_glutton
yoshipon
.png){kind=avatar}
researchconf
upura
michaelherold
eileencodes
samanthasiow
carmenintech
moore
kevinliebholz
jensimmons
thoeni
tanoku
chrisshort
jennybc
iamctodd