Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
A/B testing - What? Why? How? @narendra_shetty
Slide 2
Slide 2 text
Narendra Shetty Sr. Software Engineer @ Twilio London
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
What is A/B testing?
Slide 5
Slide 5 text
Running experiments to optimize customer experience What is A/B testing?
Slide 6
Slide 6 text
Sign up Sign up A B
Slide 7
Slide 7 text
Sign up Sign up A B
Slide 8
Slide 8 text
Why A/B testing?
Slide 9
Slide 9 text
We’re, most of the time, wrong about what we think is ‘good’ Why A/B testing?
Slide 10
Slide 10 text
We want our feature to be profitable. Why A/B testing?
Slide 11
Slide 11 text
We want customer sentiment to drive product development Why A/B testing?
Slide 12
Slide 12 text
We want to make product decision based on fact Why A/B testing?
Slide 13
Slide 13 text
To Learn Why A/B testing?
Slide 14
Slide 14 text
Why not A/B testing?
Slide 15
Slide 15 text
You don’t have sufficient traffic Why not A/B testing?
Slide 16
Slide 16 text
Your success metrics are ill defined. Why not A/B testing?
Slide 17
Slide 17 text
You Like Being An Expert Why not A/B testing?
Slide 18
Slide 18 text
Process of A/B testing
Slide 19
Slide 19 text
Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Hypothesis !== idea
Slide 20
Slide 20 text
Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Data you can measure to prove your hypothesis is right or wrong
Slide 21
Slide 21 text
Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Design & Implement
Slide 22
Slide 22 text
Process of A/B testing Start with an hypothesis Define a metric Build Analyze & Learn Build Measure Learn Build Measure Learn Build Measure Learn
Slide 23
Slide 23 text
90% tests fail
Slide 24
Slide 24 text
90% tests fail … and sometimes you don’t know why tests fail.
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
Base: 150px Variant: 200px
Slide 27
Slide 27 text
Base: 150px Variant: 200px 300px
Slide 28
Slide 28 text
Base: 150px Variant: 200px 300px 500px
Slide 29
Slide 29 text
150px 500px A B
Slide 30
Slide 30 text
Neutral
Slide 31
Slide 31 text
Base: 150px Variant: 500px Result: Increase in page load
Slide 32
Slide 32 text
150px 300px A B
Slide 33
Slide 33 text
150px 300px A B Wins!!
Slide 34
Slide 34 text
Web vs Native A/B Testing Process
Slide 35
Slide 35 text
Web A/B Testing Process
Slide 36
Slide 36 text
Ideas Web
Slide 37
Slide 37 text
Ideas Build Web
Slide 38
Slide 38 text
Ideas Build Deploy Web
Slide 39
Slide 39 text
Ideas Build Deploy Users 1 day Web
Slide 40
Slide 40 text
Native A/B Testing Process
Slide 41
Slide 41 text
Ideas Native
Slide 42
Slide 42 text
Ideas Build Native
Slide 43
Slide 43 text
Ideas Build Submit 1-2 weeks Native
Slide 44
Slide 44 text
Ideas Build Submit Review 1-2 weeks 2-3 days Native
Slide 45
Slide 45 text
Ideas Build Submit Review Stable 1-2 weeks 2-3 days 2-3 days Native
Slide 46
Slide 46 text
Ideas Build Submit Review Stable Users ~3 weeks 1-2 weeks 2-3 days 2-3 days Native
Slide 47
Slide 47 text
Big shot AB testing Common pitfalls
Slide 48
Slide 48 text
Fringe AB testing Common pitfalls
Slide 49
Slide 49 text
Assumed Reproducibility Common pitfalls
Slide 50
Slide 50 text
https://booking.design/would-you-like-fries-with-that-4edf46849380
Slide 51
Slide 51 text
A https://booking.design/would-you-like-fries-with-that-4edf46849380 B
Slide 52
Slide 52 text
A B Neutral https://booking.design/would-you-like-fries-with-that-4edf46849380
Slide 53
Slide 53 text
https://twitter.com/lukew/status/469862551304818688
Slide 54
Slide 54 text
A/B testing when done right
Slide 55
Slide 55 text
Test everything A/B testing when done right
Slide 56
Slide 56 text
Test atomically A/B testing when done right
Slide 57
Slide 57 text
Build tools you can trust A/B testing when done right
Slide 58
Slide 58 text
Question when you don’t understand A/B testing when done right
Slide 59
Slide 59 text
Build a culture of data driven development A/B testing when done right
Slide 60
Slide 60 text
@narendra_shetty @narendrashetty @narendrashetty Thank you!