The Realities of A/B Testing

A/B testing isn’t just about shipping and learning. There’s also some hard work involved. What if you have several A/B tests running on top of one another? Should you duplicate lots of code or introduce lots of branching? What does statistical relevance even mean?
In this talk, I’ll dig into the realities of A/B testing. I’ll discuss how to deal with A/B testing in practice, how to continuously split test an application with a non-trivial number of variants and not go mad, and how to keep your quality standards high.

Presented on App Builders 2019, Lugano

Transcript

1. The Realities of A/B Testing Aydar Mukhametzyanov iOS Developer at Freeletics
2. Agenda •👶 Introduction to A/B Testing •🌱 A/B Test Lifecycle • ☝ Learnings •🎁 Bonus
3. Who here knows what A/B Testing is?
4. Who here has run an A/B Test?
5. 👶 Introduction to A/B Testing
6. Idea! What are Feature Flags 153
7. if isDiamondFeatureEnabled { openDiamondViewController() } else { openDuckViewController() } What are Feature Flags
8. ? isDiamondFeatureEnabled What are Feature Flags
9. ? What are Feature Flags
10. 50% — true 50% — false isEnabled What is the A/B Test 50% 50% ? isEnabled
11. Analysis Decision 50% 50% ? What is the A/B Test
12. BUY! BUY! A B A B BUY! BUY! A B FREE! C What is the A/B Test
13. 🌱 A/B Test Lifecycle
14. 🌱 A/B Test Lifecycle Strategy Implementation Result
15. Before starting an A/B Test usertesting.com (Product + Design)
16. A/B Test Strategy •Hypothesis •Success criteria de fi nition •Target for experiment •Tra ff i c split (Product + Data Analyst)
17. A/B Test Strategy Tra ff i c split All users 100 %
18. A/B Test Strategy Tra ff i c split A 50 % B 50 % One more A/B Test?
19. A/B Test Strategy Tra ff i c split Not a f f
20. A/B Test Strategy Tra ff i c split Not a f f
21. A/B Test Strategy Quickly High quality Low price
22. A/B Test Strategy Short A/B Test High quality data Less a ff ected users
23. A/B Test Strategy (Product + Data Analyst) •Hypothesis •Success criteria de fi nition •Target for experiment •Tra ffi c split
24. 🌱 A/B Test Lifecycle Strategy Implementation Result
25. A/B Test Implementation (Engineers) •Remote feature fl ag •Tracking events •Cases A and B
26. A/B Test Implementation Remote con fi g
27. A/B Test Implementation Remote con fi g
28. A/B Test Implementation Remote con fi g
29. A/B Test Implementation Remote con fi g
30. A/B Test Implementation Remote con fi g
31. remoteConfig = RemoteConfig.remoteConfig() remoteConfig.setDefaults(fromPlist: "RemoteConfigDefaults") //... remoteConfig.fetch { [weak self] status, error in if status == .success { print("Config fetched!") self?.remoteConfig.activateFetched() } else { print("Config not fetched") } } A/B Test Implementation Remote con fi g
32. remoteConfig = RemoteConfig.remoteConfig() remoteConfig.setDefaults(fromPlist: "RemoteConfigDefaults") //... remoteConfig.fetch { [weak self] status, error in if status == .success { print("Config fetched!") self?.remoteConfig.activateFetched() } else { print("Config not fetched") } } A/B Test Implementation Remote con fi g
33. A/B Test Implementation •Remote feature fl ag •Tracking events •Cases A and B
34. A/B Test Implementation Tracking events
35. A/B Test Implementation Tracking events Analytics.logEvent("share_image", parameters: [ "name": name as NSObject, "full_text": text as NSObject ])
36. A/B Test Implementation •Remote feature fl ag •Tracking events •Cases A and B
37. A/B Test Implementation Cases A and B Just do it ☝ Learnings if isDiamondFeatureEnabled { openDiamondViewController() } else { openDuckViewController() }
38. Clarify future scenarios A/B Test Implementation Learnings
39. Clarify future scenarios • How long should an AB test be alive? • If the test is successful, then … • If the test failed, then … A/B Test Implementation Learnings
40. Clarify future scenarios Software architecture A/B Test Implementation Learnings
41. v1 v2 A/B Test Implementation Learnings struct FixedStructure { let workout: Workout } struct DynamicStructure { let workouts: [Workout] }
42. v1 v2 A/B Test Implementation Learnings struct FixedStructure { let workout: Workout } struct DynamicStructure { let workouts: [Workout] } struct UniversalStructure { // ... }
43. v1 v2 A/B Test Implementation Learnings Nothing bad in copy-pasting (sometimes) struct FixedStructure { let workout: Workout } struct DynamicStructure { let workouts: [Workout] }
44. Take shortcuts A/B Test Implementation Learnings
45. Take shortcuts A/B Test Implementation Learnings
46. Take shortcuts A/B Test Implementation Learnings 💪 Training fl ow ? 🚀 Onboarding fl ow +
47. t 100% DELIVERY INTEGRATION A/B TEST DEVELOPMENT Take shortcuts A/B Test Implementation Learnings
48. t 100% DELIVERY DEV PROT. A/B TEST INTEGRATION DEVELOPMENT INTEGRATION Take shortcuts A/B Test Implementation Learnings
49. Prototype • Targeted to one locale and speci fi c group • Looks like designed • Tracks user behavior Take shortcuts A/B Test Implementation Learnings
50. Prototype by images Take shortcuts A/B Test Implementation Learnings
51. Prototype by images A/B Test Implementation Learnings Start Image Non trivial UI element Button ✅ Easy to implement ✅ Easy to implement ❌ Needs a prototype
52. imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25, topCapHeight: 0) Prototype by images A/B Test Implementation Learnings
53. Prototype by images A/B Test Implementation Learnings
54. Prototype by images A/B Test Implementation Learnings
55. Prototype by images A/B Test Implementation Learnings
56. Cleanup ASAP A/B Test Implementation Learnings
57. Cleanup ASAP A/B Test Implementation Learnings
58. Test A/B Test Implementation Learnings
59. A/B Test Implementation Learnings Secret screen for “on the fl y” con fi guration Test Helps with manual testing
60. Test A/B Test Implementation Learnings UI Tests UI Tests for automatic testing
61. Test A/B Test Implementation Learnings UI Tests App ProcessInfo.processInfo.environment UI Tests for automatic testing
62. Feature fl ag may change A/B Test Implementation Learnings
63. Feature fl ag may change A/B Test Implementation Learnings
64. Feature fl ag may change A/B Test Implementation Learnings
65. Feature fl ag may change A/B Test Implementation Learnings isEnabled == true isEnabled == false
66. Feature fl ag may change A/B Test Implementation Learnings isEnabled == true id Model UI Feature fl ag
67. isEnabled == false Feature fl ag may change A/B Test Implementation Learnings id Model UI Feature fl ag didSelect Out of bounds
68. •Clarify future scenarios •Take shortcuts •Cleanup ASAP •Test •Feature fl ag may change A/B Test Implementation Learnings
69. 🌱 A/B Test Lifecycle Strategy Implementation Result
70. A/B Test Results (Data Analyst) •Fetch tracked data •Process data •Report results
71. A/B Test Results
72. A/B Test Results awesome_script.py
73. A/B Test Results chartio.com
74. A/B Test Results Conversion 32%
75. A/B Test Results Conversion Probability 32%
76. A/B Test Results Conversion Probability 32% 20%
77. A/B Test Results Conversion Probability 32% 20%
78. A/B Test Results 32% 34% B A
79. A/B Test Results 32% 34% B A
80. A/B Test Results 32% 34% B A
81. 👍 Positive trend A/B Test Results 👎 Negative trend 😐 Flat - case A results - case B results
82. A/B Test Results •What performs better? •How much better?
83. 🎁 Bonus
84. Tests at Freeletics A B
85. Tests at Freeletics A B A. -10% B. +12% C. +18% Conversion change:
86. Tests at Freeletics A B A. -10% B. +12% C. +18% Conversion change:
87. Tests at Freeletics A B
88. Tests at Freeletics A B A. -13% B. +30% C. +42% Conversion change:
89. Tests at Freeletics A B A. -13% B. +30% C. +42% Conversion change:
90. Summing up • Strategy • Implementation • Results 🌱 A/B Test life cycle • UserTesting • Firebase Remote Con fi g • Firebase Analytics • BigQuery • Chartio 🛠 Tools ☝ Learnings • Clarify future scenarios • Take shortcuts • Cleanup ASAP • Test • Feature fl ag may change
91. aydarin Thank you! 🎉 aydar.mukh@gmail.com