The perils and pitfalls of A/B testing

We work in an environment where metrics are immediate, and it’s alluring to believe our impact on them is easily split, measured, and determined. We’re also promised, by an abundance of new analytics products, that testing is easy, fast, and that we’ll be making better decisions with less effort.
The reality is that A/B testing is really difficult. I’ll share real-world experiences to show how A/B testing can distract, slow, and cause energy to be focussed on solving the wrong problems. We’ll also walk through some scenarios where the chances of testing success is likely to be increased.

Transcript

1. A B
2. $
4. Buy Buy
6. Gains come with losses.
10. Gains come with losses.
11. You’re 
 rushing.
12. You’re neglecting QA.
13. You’re drowning.
15. You’re bogged down in data.
19. You’re ditching good ideas.
20. You’re not focussed.
21. It’s taking
 too long.
22. Source: https://blog.kissmetrics.com/when-not-to-test/
23. Source: https://blog.kissmetrics.com/when-not-to-test/
25. Timing is everything.
27. It’s not you.
28. You’re doing 2x the work.
29. Data analytics ain’t free.
30. You believe the pseudoscience.
31. You really 
 don’t know.
32. Don’t know 
 your customer.
33. It’s not all
 about winning.
34. Some wins 
 are quiet.
35. You can’t test everything.
36. Ice ice baby.
37. Impact Confidence Ease
38. Impact Confidence Ease 8 5 3
39. 16/30
40. Impact Confidence Ease 3 7 8
41. 18/30
42. Impact Confidence Ease 1 9 10
43. 20/30
44. One on. One off.
45. If the stakes are high.
46. Value time. Have a theory. Volume is key. Design well.
47. Do test.
48. Choose your problems.
49. Be brave.
50. @njmcgee