Know your impact: An introduction to AB testing

As developers we care a lot about metrics and benchmarks related to our code and its speed. You know how to improve the test coverage or how to make this one request faster. But how do you improve your conversion rate, how can you validate if a feature was a success?

In this talk I will tell about my own journey from low motivation to AB-Testing and how small measurements can keep you happy. I will give you an introduction about the statistical background of AB Testing and some best practices you can easily apply.

Transcript

1. What motivates you?
2. What motivates you? • Autonomy: when, what? • Mastery: not too easy, not too hard • Purpose: what is the outcome?
3. What motivates you? • Autonomy: when, what? • Mastery: not too easy, not too hard • Purpose: what is the outcome?
4. Know your impact: An introduction to AB Testing Jalyna Schröder @jalynas | CTO @ wundertax.de
5. What is an AB Test?
7. What about significance?
8. Standard Normal Curve 80%, z = 1.64 99%, z = 2.58
9. Example Version A vs. B • n = 1000 • ^p = 0.5 and 0.55
10. A
11. https://abtestguide.com/calc/
12. Why to AB Test?
13. Market conditions are constantly changing
14. Unbiased Results
15. Learning all the time
16. Less of Highest-Paid-Person’s-Opinion
17. Kill your darlings
18. Clear Product
19. Make money
20. How to AB Test?
21. How to implement?
22. https://github.com/lucaong/ablab
23. experiment :homepage_background do goal 'sign up' group :students, description: 'photo of real students' group :workplace, description: 'photo of workplace with laptop' end
24. def homepage_background case experiment(:homepage_background).group when :students image_url('student_background.jpg') when :workplace image_url('workplace_background.jpg') else # Control Group... nil end end
25. def index experiment(:homepage_background).track_view! end
26. def create if User.create(user_params) experiment(:homepage_background).track_success! redirect_to some_path end end
28. https://github.com/splitrb/split https://github.com/assaf/vanity
29. What to test?
31. What to test? • Homepage (Sign Up Conversion) • Product Pages (Purchase Conversion) • Checkout Pages (Purchase Conversion) • Emails, Email subjects (Opening Rate) • For new features (e.g. new payment provider)
32. What to test? • Homepage (Sign Up Conversion) • Product Pages (Purchase Conversion) • Checkout Pages (Purchase Conversion) • Emails, Email subjects (Opening Rate) • For new features (e.g. new payment provider) • Simply everywhere where you can
33. How to setup an AB Test? 1. Use data to identify problems/opportunities
 (user interviews, heatmaps, recordings, analytic data) 2. Create assumptions about which solution could fix the problem (and pre-validate!) 3. Test big changes first
34. Learnings • Analyze your funnel • Results are not what you expect • Do not test small changes and use vwo.com/ab-split-test-duration • Document all your test results • Avoid Parallel Testing • If you can, use server-side testing • Test stuff that you know will decrease CR • “Let’s just test it”
35. What you can not test • Very low traffic pages (e.g. B2B, Backoffices) • Tiny changes
36. Limitations
38. Do user testing
40. Thank you @jalynas
41. Example Version A vs. B • n = 1000 • ^p = 0.55 and 0.5 • A: With 95% confidence 0.519 ≤ p1 ≤ 0.581 • B: With 95% confidence 0.469 ≤ p2 ≤ 0.531
42. Example Version A • n = 1000 • ^p = 0.55 • SE(^P) = • ^p ± 1.96 * SE(^P) = 0.55 ± 0.031 => 3% Margin • With 95% confidence 0.519 ≤ p ≤ 0.581 = 0.0157
43. Example Version B • n = 1000 • ^p = 0.5 • SE(^P) = • ^p ± 1.96 * SE(^P) = 0.5 ± 0.031 • With 95% confidence 0.469 ≤ p ≤ 0.531 = 0.0158 Version A: 0.519 ≤ p ≤ 0.581
44. Standard Error
 in binomial distribution
45. Confidence Interval Samples p } 5%