A/B Testing on Android - Yelp Engineering

Making Measurable Improvements to your App with Experiments

RECORDING: https://youtu.be/fuHaos3XFWo

Experimentation is a great way to prove the value of your changes in your app; you split your user base into groups, each receiving a different treatment, and based on their usage you decide which treatment is better. Sounds easy? Well… In practice, things get tricky. I want to share with you how we run experiments at Yelp and how we manage to do that effectively. In this talk you will learn about:

- Different types of experiments and ways to assign treatments.
- Structuring your app to support experimentation.
- Reporting results and pitfalls that can ruin your analysis.

By the end of this you'll be itching to start the next experiment on your app!

Transcript

1. A/B Testing Muhammad Siddeek @MSiddeek
2. Connecting people with
 great local businesses. Yelp’s Mission
3. A/B Testing Statistical procedure to show that some change to your app
 affects your users’ experience “
4. A/B Testing
5. A/B Testing
6. A/B Testing
7. A/B Testing
8. A/B Testing Status Quo (A) Treatment (B)
9. A/B Testing Status Quo (A) Treatment (B)
10. A/B Testing
11. A/B Testing A
12. A/B Testing A B
13. A/B Testing A B
14. A/B Testing A B
15. A/B Testing A B
16. A/B Testing A B
17. A/B Testing A B
18. A/B Testing A B
19. Steps
20. Steps I. DESIGN
21. Steps I. DESIGN II. ANALYTICS
22. Steps I. DESIGN II. ANALYTICS III. IMPLEMENTATION
23. I. Design
24. “If pizza ordering is made simpler, our app will make more money. ✘
25. “IF we change variable X, THEN we’ll get result Y, BECAUSE we have reasoning Z.
26. “IF we make pineapples a default topping on our “Pizza Order” page, THEN daily orders will increase, BECAUSE pineapple is the most frequently selected topping. ✔
27. IF We Change Variable X
28. IF We Change Variable X USER FUNNEL
29. IF We Change Variable X USER FUNNEL MESSAGING
30. IF We Change Variable X USER FUNNEL MESSAGING NAVIGATION
31. IF We Change Variable X USER FUNNEL MESSAGING NAVIGATION CONTENT GATING
32. IF We Change Variable X USER FUNNEL MESSAGING NAVIGATION CONTENT GATING CTAs
33. IF We Change Variable X USER FUNNEL MESSAGING NAVIGATION CONTENT GATING CTAs VISUAL CONTENT
34. THEN We Get Result Y
35. THEN We Get Result Y CLICK THROUGH RATE
36. THEN We Get Result Y CLICK THROUGH RATE SESSION LENGTH
37. THEN We Get Result Y CLICK THROUGH RATE SESSION LENGTH REVENUE
38. THEN We Get Result Y CLICK THROUGH RATE SESSION LENGTH REVENUE Low-Level High-Level
39. THEN We Get Result Y CLICK THROUGH RATE SESSION LENGTH REVENUE Low-Level High-Level Quicker Feedback Harder To Analyze
40. THEN We Get Result Y SENSITIVE ROBUST
41. BECAUSE We Have Reason Z
42. How Long to Run?
43. How Long to Run?
44. -15 +15 Day 0
45. -15 +15 Day 1
46. -15 +15 Day 2
47. -15 +15 Day 3
48. -15 +15 Day 4
49. -15 +15 Day 5
50. -15 +15 Day 6
51. -15 +15 Day 6 wins with 100% confidence
52. -15 +15 Day 6 wins with 100% confidence
53. #days = #events to establish confidence #events per day
54. #days = #events to establish confidence #events per day http://powerandsamplesize.com/ https://abtestguide.com/
55. #days = #events to establish confidence #events per day check your existing metrics
56. #days = #events to establish confidence #events per day round up to multiple of 7
57. • Population size • Session length • Events/session • Number of signed in users • … Test Over? Sanity Check!
58. Then, Evaluate!
59. A B Example 1
60. A B Example 1 Tie
61. A B Example 2
62. A B Example 2 A wins
63. A B Example 2
64. A B Example 2 Tie
65. A B Example 3
66. A B Example 3 Tie?
67. II. Analytics
68. Events
69. Tap Events
70. Events Tap PageView Notification ApiRequest
71. Session 0 Events < 20mins Tap PageView Notification ApiRequest
72. Sessions Session 0
73. Sessions > 20mins Session 0 Session 1 Session 2
74. User I Sessions > 20mins Session 0 Session 1 Session 2
75. User I Users
76. Users User I User II User III
77. User I User II User III Dimensions DE DE 7.1 8.0 Country IT 7.1 App
 Version
78. User I User II User III Custom Dimensions DE DE 7.1 8.0 Country App
 Version IT 7.1 A A Experiment Bucket B
79. III. Implementation
80. Bucket Assignment
81. Bucket[] buckets = {Bucket.A, Bucket.B};
82. Bucket[] buckets = {Bucket.A, Bucket.B};
83. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { }
84. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { if (user.getCountry() == “IT”) { return Bucket.B; } else { return Bucket.A; } }
85. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { if (user.getCountry() == “IT”) { return Bucket.B; } else { return Bucket.A; } } ✘
86. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { }
87. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { return buckets[new Random().nextInt() % buckets.length]; }
88. getBucket();
89. getBucket(); Bucket.B
90. getBucket(); Bucket.B getBucket(); Bucket.A
91. getBucket(); Bucket.B getBucket(); Bucket.A getBucket(); Bucket.B
92. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket() { return buckets[new Random().nextInt() % buckets.length]; }
93. Bucket[] buckets = {Bucket.A, Bucket.B}; int rand = new Random().nextInt(); Bucket getBucket() { return buckets[rand % buckets.length]; }
94. getBucket(); Bucket.B
95. getBucket(); Bucket.B getBucket(); Bucket.B
96. getBucket(); Bucket.B getBucket(); Bucket.B System.exit(0); getBucket(); Bucket.A
97. Bucket[] buckets = {Bucket.A, Bucket.B}; int rand = new Random().nextInt(); Bucket getBucket() { return buckets[rand % buckets.length]; }
98. Bucket[] buckets = {Bucket.A, Bucket.B}; int rand = new Random().nextInt(); Bucket getBucket() { if (getCachedBucket() == null) { cacheBucket(buckets[rand % buckets.length]); } return getCachedBucket(); }
99. Bucket[] buckets = {Bucket.A, Bucket.B}; int rand = new Random().nextInt(); Bucket getBucket() { if (getCachedBucket() == null) { cacheBucket(buckets[rand % buckets.length]); } return getCachedBucket(); } Local Random Experiment ✔
100. getBucket(); Bucket.B getBucket(); Bucket.B
101. getBucket(); Bucket.B getBucket(); Bucket.B app.clearCache(); getBucket(); Bucket.A
102. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId) { return buckets[userId % buckets.length]; }
103. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId) { return buckets[userId % buckets.length]; } Unit of Diversion
104. int userId = 26; getBucket(userId); Bucket.A
105. int userId = 26; getBucket(userId); Bucket.A System.exit(0); getBucket(userId); Bucket.A
106. int userId = 26; getBucket(userId); Bucket.A System.exit(0); getBucket(userId); Bucket.A System.exit(0); getBucket(userId); Bucket.A
107. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId) { return buckets[userId % buckets.length]; }
108. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[userId % buckets.length]; }
109. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[userId % buckets.length]; }
110. 0 1 btnClr rateApp
111. 0 1 btnClr rateApp payText autoVid fakeSlp
112. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[userId % buckets.length]; }
113. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; }
114. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; } {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} * 2 / 10
115. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; } {0, 2, 4, 6, 8, 10, 12, 14, 16, 18} / 10
116. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; } {0, 0, 0, 0, 0, 1, 1, 1, 1, 1}
117. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; } {0, 0, 0, 0, 0, 1, 1, 1, 1, 1}
118. 0 1 btnClr rateApp payText autoVid fakeSlp
119. 0 1 2 3 4 5 6 7 8 9 btnClr rateApp payText autoVid fakeSlp
120. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { return buckets[(userId % 10) * buckets.length / 10]; }
121. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { int bucketId = userId + expId.hashCode(); return buckets[(bucketId % 10) * buckets.length / 10]; }
122. 0 1 2 3 4 5 6 7 8 9 btnClr rateApp payText autoVid fakeSlp
123. 0 1 2 3 4 5 6 7 8 9 btnClr rateApp payText autoVid fakeSlp
124. Bucket[] buckets = {Bucket.A, Bucket.B}; Bucket getBucket(int userId, String expId) { int bucketId = userId ⊕ expId.hashCode(); return buckets[(bucketId % 10) * buckets.length / 10]; } Local Persistent Experiment ✔
125. 0 1 2 3 4 5 6 7 8 9 exp1 50% 50%
126. 0 1 2 3 4 5 6 7 8 9 exp1 50% 50% exp2 80% 20%
127. 0 1 2 3 4 5 6 7 8 9 exp1 50% 50% exp2 80% 20% exp3 40% 20% 20% 20%
128. 0 1 2 3 4 5 6 7 8 9 exp1 50% 50% exp2 80% 20% exp3 40% 20% 20% 20% flag1 DE US (any)
129. Future<Bucket> getBucket(String expId) { return experimentService.getBucket(expId); } Remote Experiment
130. Future<Bucket> getBucket(String expId) { return experimentService.getBucket(expId); } Remote Experiment !!!
131. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment
132. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment Simple Client is Simple Client is Complex
133. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment Persistent Persistent Not Persistent Simple Client is Simple Client is Complex
134. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment Persistent Persistent Not Persistent Simple Client is Simple Client is Complex Easy to Re-configure Not Configurable Not Configurable
135. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment Persistent Persistent Not Persistent Simple Client is Simple Client is Complex Easy to Re-configure Not Configurable Not Configurable Asynchronous Synchronous Synchronous
136. Local Persistent
 Experiment Local Random
 Experiment Remote
 Experiment Persistent Persistent Not Persistent Simple Client is Simple Client is Complex Easy to Re-configure Not Configurable Not Configurable Asynchronous Synchronous Synchronous
137. Bucket Reporting
138. NewUser AppStart Welcome
139. NewUser screenBldr = new HitBuilders .ScreenViewBuilder(); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart Welcome Welcome
140. NewUser SignUp Join Point of Diversion screenBldr = new HitBuilders .ScreenViewBuilder(); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart Welcome Welcome
141. NewUser screenBldr = new HitBuilders .ScreenViewBuilder() .setCustomDimension( cdIndex, cdValue ); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart SignUp Join Welcome Welcome
142. NewUser screenBldr = new HitBuilders .ScreenViewBuilder() .setCustomDimension( cdIndex, cdValue ); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart SignUp Join Welcome Welcome
143. NewUser screenBldr = new HitBuilders .ScreenViewBuilder() .setCustomDimension( cdIndex, cdValue ); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart SignUp Join Welcome Welcome 0-199
144. NewUser screenBldr = new HitBuilders .ScreenViewBuilder() .setCustomDimension( cdIndex, cdValue ); tracker.setScreenName(“Welcome”); tracker.send(screenBldr.build()); AppStart SignUp Join Welcome Welcome 0-199 any string
145. NewUser int cdIndex = 13; String cdValue = “Join”; AppStart SignUp Join Welcome Welcome
146. NewUser int cdIndex = CD.exp_wlcmBtn; String cdValue = “Join”; AppStart SignUp Join Welcome Welcome
147. NewUser int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “SignUp”; break; default: cdValue = “Join”; break; } AppStart SignUp Join Welcome Welcome
148. NewUser int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “wlcmBtnSignUp”; break; default: cdValue = “wlcmBtnJoin”; break; } AppStart wlcmBtnSignUp wlcmBtnJoin Welcome Welcome
149. AppStart RateApp Welcome Welcome NewsFeed
150. int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “wlcmRateOn”; break; default: cdValue = “wlcmRateOff”; break; } AppStart RateApp Welcome Welcome NewsFeed
151. wlcmRateOff wlcmRateOn int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “wlcmRateOn”; break; default: cdValue = “wlcmRateOff”; break; } AppStart Welcome Welcome RateApp NewsFeed
152. wlcmRateOff wlcmRateOn int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “wlcmRateOn”; break; default: cdValue = “wlcmRateOff”; break; } AppStart Welcome Welcome RateApp NewsFeed
153. wlcmRateOff wlcmRateOn int cdIndex = CD.exp_wlcmBtn; String cdValue; switch (getBucket()) { case Bucket.B: cdValue = “wlcmRateOn”; break; default: cdValue = “wlcmRateOff”; break; } DeepLink Welcome Welcome RateApp NewsFeed
154. wlcmRateOff wlcmRateOn int cdIndex = CD.exp_wlcmBtn; String cdValue = “<no-bucket>”; switch (getBucket()) { case Bucket.B: cdValue = “wlcmRateOn”; break; case Bucket.A: cdValue = “wlcmRateOff”; break; } DeepLink Welcome Welcome RateApp NewsFeed
155. Tips - Display and Report Fresh State
156. Tip - Debug Utilities
157. $ adb shell setprop log.tag.GAv4 DEBUG $ adb logcat -s GAv4 Tip - Test Custom Dimensions E2E
158. $ adb shell setprop log.tag.GAv4 DEBUG $ adb logcat -s GAv4 09-10 15:46:17.975 12296-12326/com.msiddeek.app D/GAv4:
 Hit delivery requested: ht=1505051177805, _s=2, _v=ma10.2.40, a=474619346, aid=com.msiddeek.nomnom, an=Nom Nom, av=1.0, cd=Welcome, cd1=bucket_a, sr=360x592, t=screenview, tid=12, ul=en-us, v=1, cid=89296d03-6e60-434e-8cf7-842d496ff220, Tip - Test Custom Dimensions E2E
159. $ adb shell setprop log.tag.GAv4 DEBUG $ adb logcat -s GAv4 09-10 15:46:17.975 12296-12326/com.msiddeek.app D/GAv4:
 Hit delivery requested: ht=1505051177805, _s=2, _v=ma10.2.40, a=474619346, aid=com.msiddeek.nomnom, an=Nom Nom, av=1.0, cd=Welcome, cd1=bucket_a, sr=360x592, t=screenview, tid=12, ul=en-us, v=1, cid=89296d03-6e60-434e-8cf7-842d496ff220, Tip - Test Custom Dimensions E2E
160. Tip - Unit-test
161. Tip - GA Developer View
162. Tip - Docs! Docs! Docs!
163. Tip - External QA Setup
164. Tip - Monitor Early
165. Tip - Don’t Disturb Running Experiments
166. Closing Thoughts
167. Closing Thoughts
168. Closing Thoughts
169. Closing Thoughts
170. Closing Thoughts
171. Closing Thoughts
172. www.yelp.com/careers We're Hiring!
173. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp