A/B Testing Got You Elected Mister President

Transcript

1. A/B Testing Got you elected Mister President
2. @samphippen @samphippen 
3. Should I make this change?
4. Users A group: 50% B group: 50% Site change Old site
5. Measure some metric
6. Do maths on the two groups
7. ???
8. Profit
9. Lemme show you my favourite A/B test
16. Also some videos
18. +$60 million
20. Protips
21. Same user always sees same version
22. Caching
23. Roughly same performance
24. Also for feature flagging
25. A super lightning fast guide on how to do it and what it looks like
26. gem 'split'
27. require 'split/dashboard' run Rack::URLMap.new \ "/" => YourApp::Application, "/split" => Split::Dashboard.new
28. <% ab_test("experiment_name", "a", "b") do |c| %> <a href="/win" class="btn <%= c %>"> Get points? </a> <% end %>
29. What it looks like
33. https://github.com/ andrew/split
34. How to interpret the results
35. Stats time
36. Confidence Value
37. P =0.95 is used in medical trials
38. Common mistake: Assumption of normality
40. This will probably work for you
41. How to design the experiment
42. Step 1: clearly state your hypothesis
43. Example: I will get more donations if our button is jimmy wale’s face
44. Formally: Null Hypothesis: there will be no increase in donations if we use jimmy wales face
45. Formally: positive Hypothesis: there will be an increase in donations if we use jimmy wales face
46. Step 2: Pick a statistical test
47. Example: difference of proportions (the standard A/b test)
48. http://stattrek.com/ hypothesis-test/ difference-in- proportions.aspx
49. Step 3: Decide an experiment length (number of days)
50. Example: we get 200 hits a day, let’s test for 15 days for 3000 hits
51. Alternatively: A fixed sample size Stop after 10000 users
52. Step 4: Split
53. Half the users get jimmy wales face half the users get whatever the button was before
54. Step 5: inspect results and analyse
55. Let’s talk about analysis
56. Let’s work two examples (one null, one positive)
57. With jimmy Without Jimmy Users in test 100 100 Users that clicked 27 18
58. Confidence = 93.6% Too low at 95% to conclude that this is better
59. common mistake: Sample size
60. With jimmy Without Jimmy Users in test 1000 1000 Users that clicked 270 180
61. 99.9% confidence High enough for us to declare this better
62. Confounding factors ARE bad
63. this is hard stuff I hope you understood :) ask me questions @samphippen