Tweet
Tweet

Slide 1

Slide 1 text

http://chrisjpowers.com @chrisjpowers The Truth About A/B Experimentation

Slide 2

Slide 2 text

Chris Powers 10 years of development 5 Years of Experimentation

Slide 3

Slide 3 text

http://chrisjpowers.com @chrisjpowers

Slide 4

Slide 4 text

http://chrisjpowers.com @chrisjpowers

Slide 5

Slide 5 text

http://chrisjpowers.com @chrisjpowers Why Use Experiments?

Slide 6

Slide 6 text

http://chrisjpowers.com @chrisjpowers

Slide 7

Slide 7 text

http://chrisjpowers.com @chrisjpowers

Slide 8

Slide 8 text

http://chrisjpowers.com @chrisjpowers Scientific Method Create a Hypothesis Run an Experiment Collect Data Analyze

Slide 9

Slide 9 text

http://chrisjpowers.com @chrisjpowers What is A/B Experimentation?

Slide 10

Slide 10 text

http://chrisjpowers.com @chrisjpowers Split Testing (A/B)

Slide 11

Slide 11 text

http://chrisjpowers.com @chrisjpowers Multivariant Testing VARIATION 2 VARIATION 3

Slide 12

Slide 12 text

http://chrisjpowers.com @chrisjpowers Multivariate Testing Control White Text Black Text Control Control Variant 3 Variant 6 Blue BG Variant 1 Variant 4 Variant 7 Green BG Variant 2 Variant 5 Variant 8

Slide 13

Slide 13 text

http://chrisjpowers.com @chrisjpowers Great Expectations

Slide 14

Slide 14 text

http://chrisjpowers.com @chrisjpowers Big Lifts 14% 62%

Slide 15

Slide 15 text

http://chrisjpowers.com @chrisjpowers Enormous Lifts 228%

Slide 16

Slide 16 text

http://chrisjpowers.com @chrisjpowers Incredible Lifts (Literally) 450%

Slide 17

Slide 17 text

http://chrisjpowers.com @chrisjpowers

Slide 18

Slide 18 text

http://chrisjpowers.com @chrisjpowers Realistic Goals 0.5% 5% => 5.5% 5% => 5.025%

Slide 19

Slide 19 text

http://chrisjpowers.com @chrisjpowers $1B * 0.00025 = $250,000

Slide 20

Slide 20 text

http://chrisjpowers.com @chrisjpowers Example Experiments

Slide 21

Slide 21 text

http://chrisjpowers.com @chrisjpowers

Slide 22

Slide 22 text

http://chrisjpowers.com @chrisjpowers Inline Deal Options 8 Million Samples 46 days Lift: 0.81% (± 0.69%)

Slide 23

Slide 23 text

http://chrisjpowers.com @chrisjpowers

Slide 24

Slide 24 text

http://chrisjpowers.com @chrisjpowers Free Shipping Text 12.5 Million Samples 63 days Lift: 0.78% (± 0.67%)

Slide 25

Slide 25 text

http://chrisjpowers.com @chrisjpowers

Slide 26

Slide 26 text

http://chrisjpowers.com @chrisjpowers Top Goods Categories 80 Million Samples 8 days Lift: 0.07% (± 0.47%)

Slide 27

Slide 27 text

http://chrisjpowers.com @chrisjpowers

Slide 28

Slide 28 text

http://chrisjpowers.com @chrisjpowers Brands Search 14 Million Samples 55 days Lift: -0.79% (± 0.41%)

Slide 29

Slide 29 text

http://chrisjpowers.com @chrisjpowers Flat 55% Negative 30% Positive 15%

Slide 30

Slide 30 text

http://chrisjpowers.com @chrisjpowers Significance and Power

Slide 31

Slide 31 text

http://chrisjpowers.com @chrisjpowers

Slide 32

Slide 32 text

http://chrisjpowers.com @chrisjpowers Statistical Significance T-Score p < 0.05 95% Confidence Controls False Positive Results

Slide 33

Slide 33 text

http://chrisjpowers.com @chrisjpowers 5% False Positive Rate

Slide 34

Slide 34 text

http://chrisjpowers.com @chrisjpowers Statistical Significance “If my p-value is less than 0.05, then I have a 95% chance that my reading is correct.”

Slide 35

Slide 35 text

http://chrisjpowers.com @chrisjpowers

Slide 36

Slide 36 text

http://chrisjpowers.com @chrisjpowers Statistical Significance “If my p-value is less than 0.05, then I have a 95% chance that my reading is correct.” Incorrect

Slide 37

Slide 37 text

http://chrisjpowers.com @chrisjpowers Statistical Significance “If my treatment has no actual effect (null hypothesis) then there’s a 5% chance I will measure a p-value less than 0.05.” correct

Slide 38

Slide 38 text

http://chrisjpowers.com @chrisjpowers Statistical Power 80% Confidence Controls False Negative Results Sets required sample size

Slide 39

Slide 39 text

http://chrisjpowers.com @chrisjpowers 20% False NEGATIVE Rate

Slide 40

Slide 40 text

http://chrisjpowers.com @chrisjpowers

Slide 41

Slide 41 text

http://chrisjpowers.com @chrisjpowers How Many Samples?

Slide 42

Slide 42 text

http://chrisjpowers.com @chrisjpowers Sample Size Equation Coefficient: 5% False Positive, 20% False Negative

Slide 43

Slide 43 text

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size scales linearly with variance.

Slide 44

Slide 44 text

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size gets smaller (squared) with larger values.

Slide 45

Slide 45 text

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size gets larger (squared) as you try to measure smaller lifts.

Slide 46

Slide 46 text

http://chrisjpowers.com @chrisjpowers Sensitivity to Effect Size 20%: 6K Samples per Treatment 10%: 25K Samples per Treatment 5%: 100K Samples per Treatment 1%: 2.5M Samples per Treatment 0.5%: 10M Samples per Treatment

Slide 47

Slide 47 text

http://chrisjpowers.com @chrisjpowers Sensitivity to Effect Size 0 2500000 5000000 7500000 10000000 0.5% 1% 5% 10% 20%

Slide 48

Slide 48 text

http://chrisjpowers.com @chrisjpowers

Slide 49

Slide 49 text

http://chrisjpowers.com @chrisjpowers

Slide 50

Slide 50 text

http://chrisjpowers.com @chrisjpowers You May Need These Lifts

Slide 51

Slide 51 text

http://chrisjpowers.com @chrisjpowers Living with Small Samples

Slide 52

Slide 52 text

http://chrisjpowers.com @chrisjpowers

Slide 53

Slide 53 text

http://chrisjpowers.com @chrisjpowers

Slide 54

Slide 54 text

http://chrisjpowers.com @chrisjpowers

Slide 55

Slide 55 text

http://chrisjpowers.com @chrisjpowers

Slide 56

Slide 56 text

http://chrisjpowers.com @chrisjpowers

Slide 57

Slide 57 text

http://chrisjpowers.com @chrisjpowers Run experiments at 50/50. Only amateurs run experiments at 1%. - Ron Kohavi

Slide 58

Slide 58 text

http://chrisjpowers.com @chrisjpowers The Peeking Problem

Slide 59

Slide 59 text

http://chrisjpowers.com @chrisjpowers

Slide 60

Slide 60 text

http://chrisjpowers.com @chrisjpowers Peeking Problem Peeking regularly and stopping experiments when significance is achieved raises the 5% false positive rate to… 30%

Slide 61

Slide 61 text

http://chrisjpowers.com @chrisjpowers 30% False Positive Rate

Slide 62

Slide 62 text

http://chrisjpowers.com @chrisjpowers Flat 55% Negative 30% Positive 15%

Slide 63

Slide 63 text

http://chrisjpowers.com @chrisjpowers False 17% Flat 39% Negative 30% Positive 15%

Slide 64

Slide 64 text

http://chrisjpowers.com @chrisjpowers Directionality

Slide 65

Slide 65 text

http://chrisjpowers.com @chrisjpowers

Slide 66

Slide 66 text

http://chrisjpowers.com @chrisjpowers Mitigating Risk

Slide 67

Slide 67 text

http://chrisjpowers.com @chrisjpowers

Slide 68

Slide 68 text

http://chrisjpowers.com @chrisjpowers Losing Money -100 -75 -50 -25 0 Start Sample Size

Slide 69

Slide 69 text

http://chrisjpowers.com @chrisjpowers

Slide 70

Slide 70 text

http://chrisjpowers.com @chrisjpowers Using a Smoke Test Smoke Test: 5% Sensitivity, 100K Users Main Test: 0.5% Sensitivity, 10M Users

Slide 71

Slide 71 text

http://chrisjpowers.com @chrisjpowers Group Sequential Analysis How can we peek while still controlling our false positive rate?

Slide 72

Slide 72 text

http://chrisjpowers.com @chrisjpowers

Slide 73

Slide 73 text

http://chrisjpowers.com @chrisjpowers

Slide 74

Slide 74 text

http://chrisjpowers.com @chrisjpowers

Slide 75

Slide 75 text

http://chrisjpowers.com @chrisjpowers

Slide 76

Slide 76 text

http://chrisjpowers.com @chrisjpowers Organizational Hurdles

Slide 77

Slide 77 text

http://chrisjpowers.com @chrisjpowers HIghest Paid Person in the Office

Slide 78

Slide 78 text

http://chrisjpowers.com @chrisjpowers

Slide 79

Slide 79 text

http://chrisjpowers.com @chrisjpowers

Slide 80

Slide 80 text

http://chrisjpowers.com @chrisjpowers

Slide 81

Slide 81 text

http://chrisjpowers.com @chrisjpowers One Last Thought

Slide 82

Slide 82 text

http://chrisjpowers.com @chrisjpowers 40 Experiments

Slide 83

Slide 83 text

http://chrisjpowers.com @chrisjpowers 2 ^ 40 1,099,511,627,776

Slide 84

Slide 84 text

http://chrisjpowers.com @chrisjpowers 99.5% of Experiences

Slide 85

Slide 85 text

http://chrisjpowers.com @chrisjpowers Thank You

Slide 86

Slide 86 text

Questions?