A/B Testing - What? Why? How?

We are all working towards one single goal. To create a best user experience. But most of the time we are wrong about what we think is good for the user. We usually have no idea if a feature is useful to the user until it is used by the customers. My talk is about letting the users help you understand what they need and if a particular feature is really useful for the users by using the technique of A/B testing. Attend my talk to find out how to run A/B tests in your company at scale and what it takes to build the optimal tests

Transcript

1. A/B testing - What? Why? How? @narendra_shetty
2. Narendra Shetty Sr. Software Engineer @ Twilio London
4. What is A/B testing?
5. Running experiments to optimize customer experience What is A/B testing?
6. Sign up Sign up A B
7. Sign up Sign up A B
8. Why A/B testing?
9. We’re, most of the time, wrong about what we think is ‘good’ Why A/B testing?
10. We want our feature to be profitable. Why A/B testing?
11. We want customer sentiment to drive product development Why A/B testing?
12. We want to make product decision based on fact Why A/B testing?
13. To Learn Why A/B testing?
14. Why not A/B testing?
15. You don’t have sufficient traffic Why not A/B testing?
16. Your success metrics are ill defined. Why not A/B testing?
17. You Like Being An Expert Why not A/B testing?
18. Process of A/B testing
19. Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Hypothesis !== idea
20. Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Data you can measure to prove your hypothesis is right or wrong
21. Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Design & Implement
22. Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Build Measure Learn Build Measure Learn Build Measure Learn
23. 90% tests fail
24. 90% tests fail … and sometimes you don’t know why tests fail.
26. Base: 150px Variant: 200px
27. Base: 150px Variant: 200px 300px
28. Base: 150px Variant: 200px 300px 500px
29. 150px 500px A B
30. Neutral
31. Base: 150px Variant: 500px Result: Increase in page load
32. 150px 300px A B
33. 150px 300px A B Wins!!
34. Web vs Native A/B Testing Process
35. Web A/B Testing Process
36. Ideas Web
37. Ideas Build Web
38. Ideas Build Deploy Web
39. Ideas Build Deploy Users 1 day Web
40. Native A/B Testing Process
41. Ideas Native
42. Ideas Build Native
43. Ideas Build Submit 1-2 weeks Native
44. Ideas Build Submit Review 1-2 weeks 2-3 days Native
45. Ideas Build Submit Review Stable 1-2 weeks 2-3 days 2-3 days Native
46. Ideas Build Submit Review Stable Users ~3 weeks 1-2 weeks 2-3 days 2-3 days Native
47. Big shot AB testing Common pitfalls
48. Fringe AB testing Common pitfalls
49. Assumed Reproducibility Common pitfalls
50. https://booking.design/would-you-like-fries-with-that-4edf46849380
51. A https://booking.design/would-you-like-fries-with-that-4edf46849380 B
52. A B Neutral https://booking.design/would-you-like-fries-with-that-4edf46849380
53. https://twitter.com/lukew/status/469862551304818688
54. A/B testing when done right
55. Test everything A/B testing when done right
56. Test atomically A/B testing when done right
57. Build tools you can trust A/B testing when done right
58. Question when you don’t understand A/B testing when done right
59. Build a culture of data driven development A/B testing when done right
60. @narendra_shetty @narendrashetty @narendrashetty Thank you!