Tweet
Tweet

Slide 1

Slide 1 text

Jessica Stringham github.com/jessstringham/talks @JessStringham Experiment Assignment on the Web

Slide 2

Slide 2 text

In A/B Testing: Should we show a visitor A or B?

Slide 3

Slide 3 text

In A/B Testing: Should we show a visitor A or B? random.choice?

Slide 4

Slide 4 text

Yelp’s Mission Connecting people with great local businesses.

Slide 5

Slide 5 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 6

Slide 6 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 7

Slide 7 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 8

Slide 8 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 9

Slide 9 text

Rule: Follow experiment design

Slide 10

Slide 10 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

100% visitors 0% visitors 40% visitors love it

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Does the color make a difference?

Slide 17

Slide 17 text

A/B Test 50% red 50% blue

Slide 18

Slide 18 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 19

Slide 19 text

Implement variants

Slide 20

Slide 20 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 21

Slide 21 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 22

Slide 22 text

Does the color make a difference?

Slide 23

Slide 23 text

Rule: Follow experiment design ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 24

Slide 24 text

Experiments on the web ● Design ● Implementation ○ Assignment ● Data collection and analysis

Slide 25

Slide 25 text

A/B Test 50% red 50% blue

Slide 26

Slide 26 text

Randomly assign

Slide 27

Slide 27 text

Randomly assign *with enough traffic

Slide 28

Slide 28 text

Randomly assign

Slide 29

Slide 29 text

Rule: Avoid introducing bias

Slide 30

Slide 30 text

Goal of A/B Test: Does the color make a difference?

Slide 31

Slide 31 text

Are we comparing just the effects of color?

Slide 32

Slide 32 text

Are we comparing just the effects of color? (Spoilers: probably not without random assignment)

Slide 33

Slide 33 text

What if we don’t randomly assign?

Slide 34

Slide 34 text

Week 1 Week 2 Assign based on time (not random)

Slide 35

Slide 35 text

Unknown variables Week 1 Week 2

Slide 36

Slide 36 text

Week 1 Week 2

Slide 37

Slide 37 text

20% visitors love it

Slide 38

Slide 38 text

Week 1 Week 2 Are we comparing just the effects of color?

Slide 39

Slide 39 text

Week 1 Week 2 No, because of unknown variables

Slide 40

Slide 40 text

Bias Week 1 Week 2

Slide 41

Slide 41 text

How can we avoid bias here? Week 1 Week 2

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Random assignment = red = blue

Slide 44

Slide 44 text

Unknown variable

Slide 45

Slide 45 text

Are we comparing just the effects of color?

Slide 46

Slide 46 text

Are we comparing just the effects of color? Mostly...

Slide 47

Slide 47 text

Why random? Avoid bias from unknown variables Week 1 Week 2

Slide 48

Slide 48 text

Can introduce bias in other ways ???

Slide 49

Slide 49 text

Rule: Avoid introducing bias

Slide 50

Slide 50 text

Assignment Rules Follow experiment design Avoid introducing bias

Slide 51

Slide 51 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 52

Slide 52 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 53

Slide 53 text

Random assignment from scratch ● random.choice ● Experimental units ● Deterministic assignment

Slide 54

Slide 54 text

Random assignment from scratch ● random.choice ● Experimental units ● Deterministic assignment

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

@app.route('/reviews/') def reviews(...): ... color = 'red' ...

Slide 57

Slide 57 text

@app.route('/reviews/') def reviews(...): ... - color = 'red' + color = choose_color_assignment() ...

Slide 58

Slide 58 text

>>> choose_color_assignment() red >>> choose_color_assignment() blue

Slide 59

Slide 59 text

def choose_color_assignment(): ...

Slide 60

Slide 60 text

def choose_color_assignment(): return random.choice(['red', 'blue'])

Slide 61

Slide 61 text

/reviews/1

Slide 62

Slide 62 text

/reviews/2

Slide 63

Slide 63 text

/reviews/3

Slide 64

Slide 64 text

/reviews/4

Slide 65

Slide 65 text

/reviews/4 /reviews/3 /reviews/2 /reviews/1

Slide 66

Slide 66 text

Random assignment from scratch ● random.choice ● Experimental units ● Deterministic assignment

Slide 67

Slide 67 text

experimental unit

Slide 68

Slide 68 text

@app.route('/reviews/') def reviews(...): ... color = choose_color_assignment() ...

Slide 69

Slide 69 text

= review page request

Slide 70

Slide 70 text

A visitor will be assigned to multiple colors!

Slide 71

Slide 71 text

/reviews/4 /reviews/3 /reviews/2 /reviews/1 That doesn’t work for this experiment

Slide 72

Slide 72 text

= person?

Slide 73

Slide 73 text

@app.route('/reviews/') def reviews(...): ... color = choose_color_assignment() ...

Slide 74

Slide 74 text

= user_id ≈ person

Slide 75

Slide 75 text

Assignment Rules Follow experiment design Avoid introducing bias

Slide 76

Slide 76 text

Assignment Rules Follow experiment design Avoid introducing bias

Slide 77

Slide 77 text

Random assignment from scratch ● random.choice ● Experimental units ● Deterministic assignment

Slide 78

Slide 78 text

Random assignment from scratch ● random.choice ● Experimental units ● Deterministic assignment

Slide 79

Slide 79 text

def choose_color_assignment()

Slide 80

Slide 80 text

def choose_color_assignment(user_id)

Slide 81

Slide 81 text

>>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=1) red

Slide 82

Slide 82 text

>>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=4) blue

Slide 83

Slide 83 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 84

Slide 84 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 85

Slide 85 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 86

Slide 86 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i] hashlib.md5

Slide 87

Slide 87 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 88

Slide 88 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 89

Slide 89 text

# Same user_id -> same value >>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=1) red

Slide 90

Slide 90 text

# Different ids get different values. >>> choose_color_assignment(user_id=100) red >>> choose_color_assignment(user_id=101) blue >>> choose_color_assignment(user_id=102) red

Slide 91

Slide 91 text

Random assignment from scratch

Slide 92

Slide 92 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 93

Slide 93 text

Experiment assignment on the web ● Experiments on the web ● Random assignment from scratch ● Assignment in practice

Slide 94

Slide 94 text

Assignment in practice ● Independence with salts ● Assignment groups ● Additional logic

Slide 95

Slide 95 text

Assignment in practice ● Independence with salts ● Assignment groups ● Additional logic

Slide 96

Slide 96 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 97

Slide 97 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]

Slide 98

Slide 98 text

def choose_size_assignment(user_id): key = "{}|size".format(user_id) assignment_i = hash_func(key) % 2 return ['big', 'small'][assignment_i]

Slide 99

Slide 99 text

def choose_size_assignment(user_id): key = "{}|size".format(user_id) assignment_i = hash_func(key) % 2 return ['big', 'small'][assignment_i]

Slide 100

Slide 100 text

color red blue

Slide 101

Slide 101 text

Assigned to “big” color red blue

Slide 102

Slide 102 text

Shuffling is a tool that can help avoid bias But watch out - experiment interactions

Slide 103

Slide 103 text

Assignment in practice ● Independence with salts ● Assignment groups ● Additional logic

Slide 104

Slide 104 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 ...

Slide 105

Slide 105 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 ...

Slide 106

Slide 106 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 ...

Slide 107

Slide 107 text

Divide users* into 20 groups *or experimental units

Slide 108

Slide 108 text

def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 if assignment_i < 10: return 'red' else: return 'blue'

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

No content

Slide 111

Slide 111 text

No content

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

No content

Slide 114

Slide 114 text

Assignment groups

Slide 115

Slide 115 text

Assignment in practice ● Independence with salts ● Assignment groups ● Additional logic

Slide 116

Slide 116 text

Assignment in practice ● Independence with salts ● Assignment groups ● Additional logic

Slide 117

Slide 117 text

if request.user.state == 'OR': color = 'red' else: color = random_color_assignment(request)

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

No content

Slide 120

Slide 120 text

No content

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

random_color_assignment

Slide 123

Slide 123 text

random_color_assignment

Slide 124

Slide 124 text

random_color_assignment

Slide 125

Slide 125 text

Actually shown...

Slide 126

Slide 126 text

Actually shown...

Slide 127

Slide 127 text

Bias!

Slide 128

Slide 128 text

Actually shown...

Slide 129

Slide 129 text

No content

Slide 130

Slide 130 text

No content

Slide 131

Slide 131 text

Assignment in practice ● Independence with salts ● Abstraction ● Additional logic This got complicated! A/A tests are a good idea.

Slide 132

Slide 132 text

Summary

Slide 133

Slide 133 text

Consider experimental units /reviews/4 /reviews/3 /reviews/2 /reviews/1

Slide 134

Slide 134 text

Assignment from scratch def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 if assignment_i < 10: return 'red' else: return 'blue'

Slide 135

Slide 135 text

No content

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

@YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp

Slide 138

Slide 138 text

Questions?

Slide 139

Slide 139 text

No content