How (not) to A/B Test

Deck from my talk given at the SLUG meetup! Video here: https://youtu.be/NVQuBow1B10

Transcript

1. How (not) to A/B Test Lessons learned in experimentation at Lyft Kyle Bashour • @kylebshr • 27 March 2019
2. What is an A/B test? @kylebshr
3. Hypothesis @kylebshr
4. Hypothesis @kylebshr
5. Experiment Setup FeatureFlag.yellowSignUpButton.on { button.setStyle(background: .yellow, text: .black) } .off { button.setStyle(background: .blue, text: .white) } @kylebshr
6. Experiment Setup // App launch: download assignment FeatureFlag.yellowSignUpButton.on { button.setStyle(background: .yellow, text: .black) } .off { button.setStyle(background: .blue, text: .white) } @kylebshr
7. Experiment Setup // App launch: download assignment // Only expose once feature flag is accessed FeatureFlag.yellowSignUpButton.on { button.setStyle(background: .yellow, text: .black) } .off { button.setStyle(background: .blue, text: .white) } @kylebshr
8. Monitor Results @kylebshr
9. Ship It! @kylebshr
10. A/B Testing Products @kylebshr
11. Lyft.app @kylebshr
12. Project X @kylebshr
13. Implementation @kylebshr
14. Implementation @kylebshr
15. Implementation @kylebshr
16. Implementation @kylebshr
17. Implementation @kylebshr
18. Sidenote @UIApplicationMain class MyAppDelegate: UIResponder, UIApplicationDelegate {...} @kylebshr
19. Sidenote // main.swift import UIKit UIApplicationMain( CommandLine.argc, CommandLine.unsafeArgv, nil, // Or a UIApplication subclass NSStringFromClass(MyAppDelegate.self) ) @kylebshr
20. Implementation @kylebshr
21. Results @kylebshr
22. What’s broken? And why? @kylebshr
23. It’s hard to maintain two apps @kylebshr
24. @kylebshr
25. Demand Graph Redesign @kylebshr
26. Implementation protocol GraphDisplaying { func display(_ data: [Int]) func bezierCurve(from points: [Int]) ... } class BarGraphView: UIView, GraphDisplaying {...} @kylebshr
27. Implementation typealias GraphView = UIView & GraphDisplaying final class DemandGraphView: UIView { private let graphView: GraphView override init(frame: CGRect) { self.graphView = BarGraphView() super.init(frame: frame) } func display(_ model: DemandViewModel) { self.graphView.display(model.demand) } } @kylebshr
28. Implementation class InteractiveBarGraphView: UIView, GraphDisplaying {...} @kylebshr
29. Implementation final class DemandGraphView: UIView { private let graphView: GraphView override init(frame: CGRect) { if FeatureFlag.demandGraphV2.getValue() { self.graphView = InteractiveBarGraphView() } else { self.graphView = BarGraphView() } super.init(frame: frame) } func display(_ model: DemandViewModel) { self.graphView.display(model.demand) } } @kylebshr
30. Results @kylebshr
31. What does a good A/B test look like? @kylebshr
32. Driver Home Redesign @kylebshr
33. Driver Home Redesign @kylebshr
34. Implementation // DriverHomeViewController.swift private let newsFeedViewController = NewsfeedFeedViewController() override func viewDidLoad() { self.setUpNewsFeedPanel() } @kylebshr
35. Implementation private lazy var newsFeedViewController = NewsfeedFeedViewController() private lazy var incentivesViewController = IncentivesViewController() override func viewDidLoad() { FeatureFlag.homeTabIncentives.on { self.setUpIncentivesPanel() } .off { self.setUpNewsFeedPanel() } } @kylebshr
36. Results @kylebshr
37. Swift Recap • Learned what @UIApplicationMain does • Protocols can help with clean A/B structure • In other cases, lazy instantiation can help @kylebshr
38. A/B Testing Recap • Limit scope • Solid hypothesis • Build for shipping @kylebshr
39. Thanks for listening! Questions? @kylebshr
40. @kylebshr