From data to decisions

2013 talk about how Khan Academy mucks through messy data to make decisions

Transcript

1. From meaningful data to meaningful decisions
2. “Khan Academy, I’ve heard of that...”
3. “...isn’t that a bunch of youtube videos?”
5. A history in 60 seconds 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 BZZZzzzzzzzzz! Please yell, “SHUT UP, BEN.”
9. Meanwhile, has 1,600,000,000 views
10. From data to decisions
11. Epic results
12. Epic results “How we boosted conversions by 158%.”
13. Epic results “We increased revenue 15% in 5 mins.”
14. Epic results “+40% signups by changing one color!”
15. Epic results “I lost 15 lbs staring at some graphs, and so can you!”
16. Epic results Messy reality
17. Epic results Messy reality “I’m confused.”
18. Epic results Messy reality “Conversions were way up at first, but...”
19. Messy reality Epic results “Visits are up, engagement is down.”
20. Epic results Messy reality “Our data’s all messed up.”
21. Epic results Messy reality Messy reality Epic results Messy reality Messy reality Messy reality Messy reality Messy reality Messy reality
22. [EXPERIMENT_A] [EXPERIMENT_B] [EXPERIMENT_C]
23. “The Struggling Model” [EXPERIMENT_A]
24. When is a student struggling?
26. Old struggling New struggling - vs -
27. Old struggling New struggling - vs - 30 attempts, still no mastery
28. Old struggling New struggling - vs - 30 attempts, still no mastery Emphasize last few problems
29. What happened?
30. Nothing happened?
31. Messy reality
32. Let’s try that again.
33. Let’s try that again. for students without coaches
34. Let’s try that again. for students without coaches Nothing happened
35. Let’s try that again. for students with coaches
36. Let’s try that again. for students with coaches +8.3% skills mastered
37. Cohorts. [LESSON_LEARNED]
38. Humans + data.
39. “Change the homepage” [EXPERIMENT_B]
40. Old homepage New homepage - vs -
41. Old homepage
42. New homepage
43. What happened?
44. What happened? Engagement went UP Learning went UP Return visits went DOWN
45. Go go gadget brain.
46. “We’ve lost that moment of ‘oh shit’ awe.”
47. “Users don’t know what we offer.”
48. Messy reality
49. Bug.
50. Bug? Hunt uncertainty. [LESSON_LEARNED]
51. Not alone. Puzzling, messy outcomes?
52. “the difference between theory and practice is greater in practice than in theory”
53. “the difference between theory and practice is greater in practice than in theory” “the reality is that most ideas fail to move the metrics they were designed to improve”
54. “the difference between theory and practice is greater in practice than in theory” “the reality is that most ideas fail to move the metrics they were designed to improve” “weeks to months to properly analyze and get to the often surprising root cause.”
57. “The ?????????? Test” [EXPERIMENT_C]
58. - vs -
59. “The ?????????? Test” [EXPERIMENT_C] A/A
60. A versus A
61. A versus A Skills mastered? Same.
62. A versus A Skills mastered? Same. Retention? Same.
63. A versus A Signups? Same. Skills mastered? Same. Retention? Same.
64. A versus A Signups? Same. Skills mastered? Same. Visits? Same. Retention? Same.
65. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Retention? Same.
66. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Retention? Same. Videos finished? Same.
67. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Problems done? Same. Retention? Same. Videos finished? Same.
68. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Problems done? Same. Problems answered correctly? Same. Retention? Same. Videos finished? Same.
69. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Problems answered correctly? Same. Retention? Same. Videos finished? Same.
70. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same.
71. Fewer users asking and answering questions?
72. While we’re all debating whether to ship version A or version A...
85. Messy reality
87. Multiple comparisons problem
88. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same.
89. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same. Users who started registering but didn’t finish?
90. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same. Problems answered incorrectly? Users who started registering but didn’t finish?
91. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same. Problems answered incorrectly? Users who started registering but didn’t finish? Updates to profiles?
92. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same. Problems answered incorrectly? Users who started registering but didn’t finish? Updates to profiles? Number of email subscriptions?
93. A versus A Signups? Same. Skills mastered? Same. Videos started? Same. Visits? Same. Profiles made? Same. Problems done? Same. Discussion participants? Stat sig. Problems answered correctly? Same. Retention? Same. Videos finished? Same. Problems answered incorrectly? Users who started registering but didn’t finish? Updates to profiles? Users voting on discussion posts? Number of email subscriptions?
94. A versus A One Stat sig. 60+ experiments
95. Hire smart people to help you science your data. [LESSON_LEARNED]
96. K, data’s good. Now what?
97. You may find it easier to make decisions with data than we do.
98. Customer lifetime value
99. Customer lifetime value can still be tricky.
100. Customer lifetime value Remember ? can still be tricky.
101. “Bing had a bug in an experiment, which resulted in very poor search results...”
102. “Bing had a bug in an experiment, which resulted in very poor search results...” “revenue per user went up over 30%!”
103. Principles
104. “Early Proficiencies” [BONUS_EXPERIMENT]
105. When has Rene mastered scientific notation?
107. Old “strict” proficiencies New “early” proficiencies - vs -
108. Old “strict” proficiencies New “early” proficiencies - vs - Very strict proof of proficiency
109. Old “strict” proficiencies New “early” proficiencies - vs - Very strict proof of proficiency Use more data, be less strict
110. What happened?
111. What happened? “Proficient” skills went UP Retention went UP Return visits went DOWN
112. What happened? “Proficient” skills went UP Retention went UP Return visits went DOWN
113. What happened? “Proficient” skills went UP Retention went UP Return visits went DOWN
114. Not a bug this time.
115. Principles make decisions. Not a bug this time.
116. Principles make decisions. Data just informs. Not a bug this time.
117. I am so jealous of customer lifetime value.
118. I am so jealous of customer lifetime value. I am so jealous of customer lifetime value. student learning.
119. We’re not there yet. But here’s a teaser.
120. Growth mindset
121. “The more you learn today, the smarter you’ll be tomorrow.”
122. “The more you learn today, the smarter you’ll be tomorrow.” 5% more problems attempted 5% more skills mastered
123. “The more you learn today, the smarter you’ll be tomorrow.” 5% more problems attempted 5% more skills mastered 40,000 data points in days
124. Let’s dream for a second.
133. What will happen if we can test all effects on student lifetime learning?
134. What will happen if we can quickly find the best educational content?
135. What will happen if all of this content is free, for anybody, on any device?
136. Sci-fi sh*t.
137. Epic challenge.
138. Epic challenge. khanacademy.org/careers
139. @kamens bjk5.com