A look into AB testing

Introduction to AB testing and some personal learnings. Presented on the Munich iOS Developers Meetup.

Transcript

1. A look into AB testing Aydar Mukhametzyanov, Freeletics
2. Agenda •What is the “Feature Flag” •Feature Flags at Freeletics •Learnings
3. What is the “Feature Flag”
4. Idea! What is the “Feature Flag”
5. What is the “Feature Flag”
6. What is the “Feature Flag”
7. What is the “Feature Flag”
8. What is the “Feature Flag” ? 50% — true 50% — false isEnabled
9. What is the “Feature Flag” Analysis Decision
10. Feature Flags at Freeletics
11. Feature Flags at Freeletics
12. Feature Flags at Freeletics
13. Feature Flags at Freeletics
14. Feature Flags at Freeletics
15. Feature Flags at Freeletics
16. Feature Flags at Freeletics remoteConfig = RemoteConfig.remoteConfig() remoteConfig.setDefaults(fromPlist: "RemoteConfigDefaults") … remoteConfig.fetch { (status, error) -> Void in if status == .success { print("Config fetched!") self.remoteConfig.activateFetched() } else { print("Config not fetched") } }
17. Feature Flags at Freeletics 14
18. Learnings
19. Learnings Clarify future scenarios
20. Learnings Clarify future scenarios • How long should an AB test be alive? • If the test is successful, then … • If the test failed, then …
21. Clarify future scenarios Learnings Code quality
22. Learnings Learn fast
23. Learnings Learn fast ?
24. Learnings t 100% DELIVERY INTEGR. AB TEST DEV Learn fast
25. Learnings t 100% DELIVERY DEV PROT. AB TEST INTEGR. DEV INTEGR. Learn fast
26. Learnings Prototype • Targeted to one locale and specific group • Looks like designed • Tracks user behavior Learn fast
27. Learnings Prototype by images Learn fast
28. Learnings Prototype by images imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25, topCapHeight: 0) Learn fast
29. Learnings Prototype by images imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25, topCapHeight: 0) Learn fast
30. Learnings Prototype by images imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25, topCapHeight: 0) Learn fast
31. Learnings Prototype by images Learn fast
32. Learnings Cleanup ASAP
33. Learnings Cleanup ASAP
34. Learnings Test
35. Learnings Test
36. Learnings Tips for coding
37. Learnings Tips for coding Nothing bad in copy-pasting
38. Learnings Tips for coding Nothing bad in copy-pasting v1 v2
39. Be ready for feature flag changes Learnings Tips for coding
40. Learnings Tips for coding Be ready for feature flag changes TableView Model FeatureFlag Storage numberOfRows() isEnabled() false 100 cellForRow(99) isEnabled() true User scrolls down Choose extended model Choose compact model Out of bounds Feature Flag update
41. Learnings • Clarify future scenarios • Learn fast • Build prototypes • Cleanup ASAP • Test • Debug panel • Tips for coding • Nothing bad in copy-pasting • Be ready for feature flag changes
42. Thank you! aydar.mukh aydar.mukh@gmail.com aydarin Questions