how_to_ab_test_with_confidence_railsconf.pdf

Transcript

1. How to A/B Test with con fi dence @fglc2 Photo by Ivan Aleksic on Unsplash
3. The Plan • Intro: What's an A/B Test? • Test setup errors • Errors during the test • Test analysis errors • Best practices Photo by Javier Allegue Barros on Unsplash
4. What is an A/B test?
5. Buy Now Order Or
6. 🧛🙋🙋🙋🧕🧑✈👨🌾👩💼💁🧑🎨 🧑🎤👩💼🙋👷🙋👩🏭🕵🙋🧑🚀🧝 👨🎓💁👨🏭💂👩🌾🧛🧑✈💁🧝💁 🙋🕵👩🏭👨🚀🙋🧕👨🦱👰👨🎓🕵 👩🔧🧑🚒👩🚀🧝👨🎓🥷🧑🏭🧕🧑✈🧟
7. 💁👨🏭🙋🙋🧕🧕🧝 👩🏭👨🚀🧛👩💼💁👰👨🎓 🕵🧟💁🧑🎨🧑🎤🧕👨🎓 🙋💂👨🌾👩🏭 🕵👩🚀🧝👨🎓👨🦱🧑✈👩🔧 🕵🥷🧑🏭🧑✈👩🌾👩💼👷 🙋🙋🧑🚒🙋🧑🚀🧑✈💁 🧝🧛🙋🙋 Buy Now Order 49 orders 56 orders
8. Is the difference real?
9. • Layouts / designs / fl ows • Algorithms (eg recommendation engines) • Anything where you can measure a di ff erence Not just buttons!
10. Jargon
11. Signi fi cance • Is the observed di ff erence is just noise? • p value of 0.05 = 5% chance it’s a fl uke • The statistical test depends on the type of metric • No guarantees on the magnitude of the di ff erence
12. Test power Photo by Michael Longmire on Unsplash Test power
13. Test power • How small a change do I want to detect? • 10% to 20% is much easier to measure than 0.1% to 0.2%
14. Sample size • Check this is feasible! • Ideally you don’t look / change anything until sample size reached • Be wary of very short experiments
15. Bayesian A/B testing
16. Bayesian A/B testing
17. Bayesian A/B testing • Allows you to model your existing knowledge & uncertainties • Can be better at with low base rates • The underlying maths are a bit more complicated
18. Test setup errors
19. Group Randomisation Photo by Macau Photo Agency on Unsplash
20. class User < ActiveRecord::Base def ab_group if id % 2 == 0 'experiment' else 'control' end end end
21. class User < ActiveRecord::Base def ab_group(experiment) hash = Digest::SHA1.hexdigest( “#{experiment}-#{id}" ).to_i(16) if hash % 2 == 0 'experiment' else 'control' end end end
22. Non random split • Newer users in other group • Older users in one group • New users were less loyal!
23. Starting too early
24. Home Page 50,000 Users Home Page 50,000 Users
25. 30,000 Users 30,000 Users Home Page 50,000 Users Home Page 50,000 Users
26. 15,000 Users 15,000 Users 30,000 Users 30,000 Users Home Page 50,000 Users Home Page 50,000 Users
27. Checkout Page A Checkout Page B 5,000 Users 5,000 Users 15,000 Users 15,000 Users 30,000 Users 30,000 Users Home Page 50,000 Users Home Page 50,000 Users
28. 2600 conversions 2500 conversions Checkout Page A Checkout Page B 5,000 Users 5,000 Users 15,000 Users 15,000 Users 30,000 Users 30,000 Users Home Page 50,000 Users Home Page 50,000 Users
29. 2600 conversions 2500 conversions Home Page 100,000 Users 60,000 Users 30,000 Users Checkout Page A Checkout Page B 5,000 Users 5,000 Users
30. Not agreeing setup • Scope of the test (what pages, users, countries ...) • What is the goal? How do we measure it? • Agree *one* metric
31. Errors during the test Photo by Sarah Kilian on Unsplash
32. A test measures the impact of all differences
33. Ecommerce Service Recommendation Service
34. Ecommerce Service Recommendation Service 10x more crashes
35. Repeated signi fi cance testing • Invalidates signi fi cance calculation • Di ffi cult to resist! • Stick to your Sample Size • This is fi ne with Bayesian A/B testing
36. Test analysis errors Photo by Isaac Smith on Unsplash
37. Do the maths • Use the appropriate statistical test • Signi fi cance on one metric does not imply signi fi cance on another
38. Outliers Photo by Ministerie van Buitenlandse Zaken
39. Photo by Ministerie van Buitenlandse Zaken
40. Photo by Ministerie van Buitenlandse Zaken
41. Understanding the domain
42. -4 -3 -2 -1 0 week 1 week 2 week 3
43. -4 -2 0 2 4 6 8 week 1 week 2 week 3 week 4 week 5 week 6 week 7
44. Results splitting
45. 💰
46. 💰
47. We aren't neutral
48. If the result is 'right' 🎉
49. If the result is 'wrong' • Start looking at result splits • Start digging for potential errors • Hey what about this other metric • Well documented test can help
50. Best practices Photo by SpaceX on Unsplash
51. Don't reinvent the wheel • Split, Vanity gems do a good job • Consider platforms (Optimizely, Google Optimize) • But understand your tool, drawbacks
52. Resist the urge to check/tinker • Repeated signi fi cance testing • Changing the test while it is running (restart the test if necessary)
53. A/A tests • Do the full process but with no di ff erence between the variants • Allows you to practise
54. Be wary of overtesting • Let's test everything! • Can be paralysing/time consuming • Not a substitute for vision / talking to your users
55. Document your test • Metric (inc. outliers etc.) • Success criteria • Scope • Sample size / test power • Signi fi cance calculation/process • Meaningful variant names
56. Thank you! @fglc2
57. Further Reading • https://www.evanmiller.org/how-not-to-run-an-ab-test.html • https://making.lyst.com/bayesian-calculator/ • https://www.chrisstucchio.com/blog/2014/ bayesian_ab_decision_rule.html @fglc2