A more complete story: Three adventures in design research

Survey design, controlled experiment design, and a think aloud sneak attack

Transcript

1. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ Measuring 
 Hard to Measure Things: Uncover a more complete story Chrissie Brodigan @tenaciouscb 1
2. Chis Bog, Cols Soy o Cmit Ar (CA) C 2 About CB Hi, I’m Chrissie Brodigan ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Built GitHub’s early UXR practice (2013 - 2016) Twitter: @tenaciouscb
3. Chis Bog, Cols Soy o Cmit Ar (CA) C 3 Career narrative Writi Research (8 years of training) Design Ethnography Writing
4. Chis Bog, Cols Soy o Cmit Ar (CA) C 4 Airborne Heroines Job requirements ✴ Age 21 – 27 ✴ Unmarried ✴ Weight – not over 135 lbs ✴ Registered nurse ✴ No eyeglasses
5. Chis Bog, Cols Soy o Cmit Ar (CA) C 5 Larry Levine You’ve written a clear, but incomplete narrative. 
 
 Go talk to these women and listen to their stories.
6. Chis Bog, Cols Soy o Cmit Ar (CA) 6 I wa h no r.
7. Chis Bog, Cols Soy o Cmit Ar (CA) 7 I ha ffice h y.
8. Chis Bog, Cols Soy o Cmit Ar (CA) 8 I tal o 17 core, mo w I ke d’t e or ut.
9. Chis Bog, Cols Soy o Cmit Ar (CA) 9 A more complete story ✴ Experienced both highly marginalizing & empowering 
 work conditions. ✴ Skilled, professional, & organized workers in their own labor union. ✴ Were part of a process that changed constitutional law.
10. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 10 My point? There is nothing like connecting with people. 
 
 Listening to stories can flip what you think you know and what the data tells you on it’s head.
11. Chis Bog, Cols Soy o Cmit Ar (CA) C 11 Story flipping & 
 what we’ll cover: 1. Survey Project(s) 2. Controlled Experiment 3. Think Aloud Protocol
12. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 12 1 Surveys
13. Chis Bog, Cols Soy o Cmit Ar (CA) C Surveys help us to measure hard-to-measure things ✴ Emotions ✴ Intentions ✴ Motivations + Goals ✴ Workflow workarounds ✴ Prior knowledge ✴ Perception 13
14. Chis Bog, Cols Soy o Cmit Ar (CA) C START FINISH 2 3 Finding the story will usually require more than one research attempt. 14
15. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 15 It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way. –E.L. Doctorow
16. Chis Bog, Cols Soy o Cmit Ar (CA) C 1. Tools & Workflows survey START FINISH 2 3 2. New Users Tools & Workflows survey 3. Inactive Users “365” survey The path to a more complete user story may be completely surprising! 16
17. Chis Bog, Cols Soy o Cmit Ar (CA) Q. How familiar are you with 
 git for version control? 17 ✴76% of people arriving from the U.S. were 
 brand new to git. ✴3-point scale.
18. Chis Bog, Cols Soy o Cmit Ar (CA) 18 We changed our approach Our team realized we were asking new users to tell us about skills they didn’t have. 
 We changed strategy to ask about skills outside of git and GitHub that new users did have. (Kind of like a census.)
19. Chis Bog, Cols Soy o Cmit Ar (CA) 19 We designed a new instrument 1. Tools in a developer toolkit 2. Channels for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics
20. Chis Bog, Cols Soy o Cmit Ar (CA) 20 You have a story
21. Chis Bog, Cols Soy o Cmit Ar (CA) 21 We respect your data
22. Chis Bog, Cols Soy o Cmit Ar (CA) 22 Our design provided us with a cross-sectional view of data Analyze a snapshot of data vs. studying multiple data points (time-series data). 
 17 escalator accidents in 2014.
23. Chis Bog, Cols Soy o Cmit Ar (CA) 23 We looked at who responded We always begin analysis by identifying the “Who.” Here, we realized that we had a blazing blind spot –new users.
24. Chis Bog, Cols Soy o Cmit Ar (CA) 24 So, we changed our approach again … It was important to meet new users where they were at vs. where we were at.
25. Chis Bog, Cols Soy o Cmit Ar (CA) 25 We divided the 
 35-question survey into several smaller surveys, and rolled them out in waves over email. We sent shorter surveys in waves
26. Chis Bog, Cols Soy o Cmit Ar (CA) 26 The “hows” and “whys” were still a mystery. 
 
 To better understand outcomes, we needed to study tools, workflows, & skills-development over time. Establishing a baseline
27. Chis Bog, Cols Soy o Cmit Ar (CA) 27 Gather & analyze a single cohort’s data with two types of studies:
 ✴ Prospective – identify outcomes as they happen in real time.
 ✴ Retrospective – look back at variables over time and identify how they contributed to known outcomes. We “accidentally on-purpose” designed a longitudinal study
28. Chis Bog, Cols Soy o Cmit Ar (CA) 28 A film shot intermittently from 2002 - 2013
29. Chis Bog, Cols Soy o Cmit Ar (CA) 29 
 The Harvard Grant Study 
 Followed 268 men for 75 years as they both died & aged on into their 90s.
30. Chis Bog, Cols Soy o Cmit Ar (CA) 30 Understanding new users We took a cohort of 90,000 new accounts created in September 2015 & divided them into two groups.
31. Chis Bog, Cols Soy o Cmit Ar (CA) C 31 1. Explorers 2. Creators
32. Chis Bog, Cols Soy o Cmit Ar (CA) C 32 Evolution vs. Replacement Are new users different because people change ? 
 “evolution”
 
 Or, is the product attracting a new type of user?
 “replacement”
33. Chis Bog, Cols Soy o Cmit Ar (CA) C 33 Obvious vs. Interesting 1. First, when reading graphs identify the strongest pattern.
 2. Next, cover up what’s obvious & look for what’s interesting.
34. Chis Bog, Cols Soy o Cmit Ar (CA) 34 Q. What’s in your toolkit? Obvious: Tenured accounts are more likely to use a text editor than an IDE.
35. Chis Bog, Cols Soy o Cmit Ar (CA) 35 Q. Experience with tools? Obvious Interesting! Newcomers are as likely to say they use neither an IDE or a Text Editor, as to say they use one.
36. Chis Bog, Cols Soy o Cmit Ar (CA) 36 Q. Where do you go for referral? Obvious: 
 
 Both newcomers & tenured users act similarly with regards to tool discovery, primarily using Google.
37. Chis Bog, Cols Soy o Cmit Ar (CA) 37 Q. Primary text editor? Interesting!
 
 New accounts are more likely to be using Notepad++. 29% of the sample .
38. Chis Bog, Cols Soy o Cmit Ar (CA) 38 Obvious Obvious Interesting! Atom’s use is much smaller among new users than we predicted. Q. Primary text editor?
39. Chis Bog, Cols Soy o Cmit Ar (CA) C 39 52.8K Following
40. Chis Bog, Cols Soy o Cmit Ar (CA) C 40 Github’s Free Text Editor
41. Chis Bog, Cols Soy o Cmit Ar (CA) C 41 Google + % of people who don’t use a text editor = growth opportunity for Atom Combine observations
42. Chis Bog, Cols Soy o Cmit Ar (CA) C 42 Atom is missing from 
 key search terms
43. Chis Bog, Cols Soy o Cmit Ar (CA) 43 How do you study people who aren’t engaged in your product?
44. Chis Bog, Cols Soy o Cmit Ar (CA) 44 Design an exit survey: 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What product are you using? 4. What’s one thing we could have done better?
45. Chis Bog, Cols Soy o Cmit Ar (CA) C 45 Q. Which VCS are you using? Strong pattern in the yellows & greens, which represent “Nothing” & “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub.
46. Chis Bog, Cols Soy o Cmit Ar (CA) C 46 Q. What’s one thing we could have done better? Free private repos are NOT universally the most valuable GitHub good.
 
 Only among the 
 most experienced programmers are FPR a plurality of requests.
47. Chis Bog, Cols Soy o Cmit Ar (CA) C 47 We’re talking about 
 free private repositories, so let’s discuss how to measure pricing your product. $
48. Chis Bog, Cols Soy o Cmit Ar (CA) 48 2 Controlled Experiment
49. Chis Bog, Cols Soy o Cmit Ar (CA) 49 Q. How much would you pay? $ $ Photo credit: William Warby (Flickr)
50. Chis Bog, Cols Soy o Cmit Ar (CA) 50 Q. How much would you pay? $ $ Photo credit: William Warby (Flickr)
51. Chis Bog, Cols Soy o Cmit Ar (CA) C 51 Instead ask about value –product goods ✴ Mug ✴ T-shirt ✴ Hoodie ✴ Feature(s) ✴ Experiences
52. Chis Bog, Cols Soy o Cmit Ar (CA) 52 The Golden Ticket ✴ Classic controlled experiment, but with a nice twist. ✴ 39,800 eligible candidates between the treatment & control. ✴ Coupons for free private repositories (FPR) to individuals with 1+ year of tenure.
53. Chis Bog, Cols Soy o Cmit Ar (CA) 53 Experiment design:
 39,800 humans Treatment (19,949) 3 arms of 6,600 each Exit Survey (2,039) 
 
 Shared Feedback Control: 19,851 Screener
 (4,418) Redeemed their (FPR) coupon
54. Chis Bog, Cols Soy o Cmit Ar (CA) 54 3 1 5 N Three Arms + Control: 1, 3, or 5 
 private repositories No free repositories
55. Chis Bog, Cols Soy o Cmit Ar (CA) 55 Golden Ticket email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate
56. Chis Bog, Cols Soy o Cmit Ar (CA) 56 Golden Ticket screener ✴The original product draw. ✴Experience with competitor products. ✴Technical & social challenges.
57. Chis Bog, Cols Soy o Cmit Ar (CA) 57 Roll experiments out slowly. 
 Measure twice, cut once.
58. Chis Bog, Cols Soy o Cmit Ar (CA) 58
59. Chis Bog, Cols Soy o Cmit Ar (CA) 59 Twitter
 Leaks
60. Chis Bog, Cols Soy o Cmit Ar (CA) 60 Unfair treatment
61. Chis Bog, Cols Soy o Cmit Ar (CA) 61 Too good to be true?
62. Chis Bog, Cols Soy o Cmit Ar (CA) 62 Measuring different types of data Human behaviors with activity data: 1. Coupon redemption 
 2. Repository creation 
 Perception of value with survey data: 
 3. Attitudinal data
63. Chis Bog, Cols Soy o Cmit Ar (CA) 63 The exit survey provides us with insight into why people did or did NOT engage in one or both of the first two activities (redemption & creation). Understanding attitudes helps inform what levers we can design and pull with experiences to effect change in behaviors. Attitudinal Data
64. Chis Bog, Cols Soy o Cmit Ar (CA) 64 Q. Which would you value the most? Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else.
 24% wanted practical training in Git or GitHub. 
 34% reported that publicly consumable goods ( t-shirt) would be more valuable.
65. Chis Bog, Cols Soy o Cmit Ar (CA) 65 Quantitative data is empirical evidence. $$ E[Y_{i} | T_{i} = 1] - E[Y_{i} | T_{i} = 0] $$
66. Chis Bog, Cols Soy o Cmit Ar (CA) 66 Qualitative data is data with soul.
67. Chis Bog, Cols Soy o Cmit Ar (CA) 67 Open text responses No amount of machine learning can surface the quality of insights that reading open text responses does.
68. “Git Udan” 68
69. “Fre e f m te w unte re r” 69
70. “at t fie re rite, or pit es up fi pe” 70
71. Chis Bog, Cols Soy o Cmit Ar (CA) 71 Private appears to be understood as private only to me vs. 
 working with other people privately. Unlimited Collaborators
72. Chis Bog, Cols Soy o Cmit Ar (CA) C 72 3 Think Aloud
73. Chis Bog, Cols Soy o Cmit Ar (CA) C 73 Faster Horses Syndrome Listen to the how, why, when, and where, behind customer requests ….
74. Chis Bog, Cols Soy o Cmit Ar (CA) C 74 The Collaboration Study ✴ Customers told us they needed a complex feature: branch permissions. ✴ Competitor products offered branch permissions. ✴ Designing for a large audience, means we need to be thoughtful and deliberate. Solve for human motivations & goals behind feature requests.
75. Chis Bog, Cols Soy o Cmit Ar (CA) C 75 Fork v. Branch: Choosing a 
 collaboration model
76. Chis Bog, Cols Soy o Cmit Ar (CA) C 76 ? Think Aloud Protocol
77. Chis Bog, Cols Soy o Cmit Ar (CA) C 77 Feature prioritization ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line
78. Chis Bog, Cols Soy o Cmit Ar (CA) C 78 Feature prioritization ✴ Branch Permissions ✴ Automatically syncing forks ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line
79. “Wha?! The’s a 
 un to? Whe?”
80. “Tel bo im n a 
 un to w ha hed u.”
81. Chis Bog, Cols Soy o Cmit Ar (CA) C 81 ✴ Include items that don’t exist, but sound like they might. ✴ Listen to people define what they think the “feature” is. ✴ Ask how, where, when, & why they would use the feature. Sneak Attack
82. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 82 It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way. –E.L. Doctorow
83. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 83 Finding the story ✴ Products have new users, tenured users, and inactive users, understanding each experience provides a more complete view. ✴ Researching hard-to-reach places– reading open text responses & listening to humans share their motivations and goals is how you find the story. ✴ Research is your flashlight.
84. Chis Bog, Cols Soy o Cmit Ar (CA) 84 Wit rec al av uc. –@so
85. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 85 Wrapping Up 1. What’s obvious vs. interesting in your data?
 2. How can you gather and use attitudinal data to study perception of value?
 3. Where does a sneak attack make sense?
 4. How will you uncover a more complete story?
86. Chis Bog, Cols Soy o Cmit Ar (CA) ⚡ 86 Thank you 
 Questions? @tenaciouscb