Jessica Stringham - Experiment Assignment on the Web

A popular way of improving websites is to run experiments on it. We split users into groups, show two or more variations of the site, measure how well each one does, and then show the best version to everyone. In this talk, I'll walk through a toy Python program that does the first step: splits users into groups.
A few interesting problems arise: grouping users, whitelists, and scaling. I'll share different ways to address them. I'll also give examples of things that can go terribly wrong when designing experiment assignment code.

https://us.pycon.org/2017/schedule/presentation/389/

Transcript

1. Jessica Stringham github.com/jessstringham/talks @JessStringham Experiment Assignment on the Web
2. In A/B Testing: Should we show a visitor A or B?
3. In A/B Testing: Should we show a visitor A or B? random.choice?
4. Yelp’s Mission Connecting people with great local businesses.
5. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
6. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
7. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
8. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
9. Rule: Follow experiment design
10. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
13. 100% visitors 0% visitors 40% visitors love it
16. Does the color make a difference?
17. A/B Test 50% red 50% blue
18. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
19. Implement variants
20. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
21. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
22. Does the color make a difference?
23. Rule: Follow experiment design • Design • Implementation ◦ Assignment • Data collection and analysis
24. Experiments on the web • Design • Implementation ◦ Assignment • Data collection and analysis
25. A/B Test 50% red 50% blue
26. Randomly assign
27. Randomly assign *with enough traffic
28. Randomly assign
29. Rule: Avoid introducing bias
30. Goal of A/B Test: Does the color make a difference?
31. Are we comparing just the effects of color?
32. Are we comparing just the effects of color? (Spoilers: probably not without random assignment)
33. What if we don’t randomly assign?
34. Week 1 Week 2 Assign based on time (not random)
35. Unknown variables Week 1 Week 2
36. Week 1 Week 2
37. 20% visitors love it
38. Week 1 Week 2 Are we comparing just the effects of color?
39. Week 1 Week 2 No, because of unknown variables
40. Bias Week 1 Week 2
41. How can we avoid bias here? Week 1 Week 2
43. Random assignment = red = blue
44. Unknown variable
45. Are we comparing just the effects of color?
46. Are we comparing just the effects of color? Mostly...
47. Why random? Avoid bias from unknown variables Week 1 Week 2
48. Can introduce bias in other ways ???
49. Rule: Avoid introducing bias
50. Assignment Rules Follow experiment design Avoid introducing bias
51. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
52. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
53. Random assignment from scratch • random.choice • Experimental units • Deterministic assignment
54. Random assignment from scratch • random.choice • Experimental units • Deterministic assignment
56. @app.route('/reviews/<page_id>') def reviews(...): ... color = 'red' ...
57. @app.route('/reviews/<page_id>') def reviews(...): ... - color = 'red' + color = choose_color_assignment() ...
58. >>> choose_color_assignment() red >>> choose_color_assignment() blue
59. def choose_color_assignment(): ...
60. def choose_color_assignment(): return random.choice(['red', 'blue'])
61. /reviews/1
62. /reviews/2
63. /reviews/3
64. /reviews/4
65. /reviews/4 /reviews/3 /reviews/2 /reviews/1
66. Random assignment from scratch • random.choice • Experimental units • Deterministic assignment
67. experimental unit
68. @app.route('/reviews/<page_id>') def reviews(...): ... color = choose_color_assignment() ...
69. = review page request
70. A visitor will be assigned to multiple colors!
71. /reviews/4 /reviews/3 /reviews/2 /reviews/1 That doesn’t work for this experiment
72. = person?
73. @app.route('/reviews/<page_id>') def reviews(...): ... color = choose_color_assignment() ...
74. = user_id ≈ person
75. Assignment Rules Follow experiment design Avoid introducing bias
76. Assignment Rules Follow experiment design Avoid introducing bias
77. Random assignment from scratch • random.choice • Experimental units • Deterministic assignment
78. Random assignment from scratch • random.choice • Experimental units • Deterministic assignment
79. def choose_color_assignment()
80. def choose_color_assignment(user_id)
81. >>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=1) red
82. >>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=4) blue
83. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
84. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
85. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
86. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i] hashlib.md5
87. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
88. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
89. # Same user_id -> same value >>> choose_color_assignment(user_id=1) red >>> choose_color_assignment(user_id=1) red
90. # Different ids get different values. >>> choose_color_assignment(user_id=100) red >>> choose_color_assignment(user_id=101) blue >>> choose_color_assignment(user_id=102) red
91. Random assignment from scratch
92. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
93. Experiment assignment on the web • Experiments on the web • Random assignment from scratch • Assignment in practice
94. Assignment in practice • Independence with salts • Assignment groups • Additional logic
95. Assignment in practice • Independence with salts • Assignment groups • Additional logic
96. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
97. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 return ['red', 'blue'][assignment_i]
98. def choose_size_assignment(user_id): key = "{}|size".format(user_id) assignment_i = hash_func(key) % 2 return ['big', 'small'][assignment_i]
99. def choose_size_assignment(user_id): key = "{}|size".format(user_id) assignment_i = hash_func(key) % 2 return ['big', 'small'][assignment_i]
100. color red blue
101. Assigned to “big” color red blue
102. Shuffling is a tool that can help avoid bias But watch out - experiment interactions
103. Assignment in practice • Independence with salts • Assignment groups • Additional logic
104. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 ...
105. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 2 ...
106. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 ...
107. Divide users* into 20 groups *or experimental units
108. def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 if assignment_i < 10: return 'red' else: return 'blue'
114. Assignment groups
115. Assignment in practice • Independence with salts • Assignment groups • Additional logic
116. Assignment in practice • Independence with salts • Assignment groups • Additional logic
117. if request.user.state == 'OR': color = 'red' else: color = random_color_assignment(request)
122. random_color_assignment
123. random_color_assignment
124. random_color_assignment
125. Actually shown...
126. Actually shown...
127. Bias!
128. Actually shown...
131. Assignment in practice • Independence with salts • Abstraction • Additional logic This got complicated! A/A tests are a good idea.
132. Summary
133. Consider experimental units /reviews/4 /reviews/3 /reviews/2 /reviews/1
134. Assignment from scratch def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i = hash_func(key) % 20 if assignment_i < 10: return 'red' else: return 'blue'
137. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp
138. Questions?