2016 - Or Weizman - A/B Testing: Harder than just a color change

Description
Is your Product Manager asking you to test out different text or button colors? Not sure where to start? This talk will contain methodology and two case studies from Yelp’s Transaction Platform on how to properly run an experiment and get the best result. Learn about how to run a simple button color experiment, avoid pitfalls, test, and analyze the results with confidence. Statistical confidence!
​
Abstract
A/B testing is a common practice for websites...but where do you begin? This data-driven approach allows you to launch experiments and features with confidence. So how do you prepare, launch, and analyze an A/B experiment? How do you know for how long to keep it running? What about which metrics to track?

This talk will present a procedure developed to run an A/B experiment, from planning the task and understanding the key metrics to analyzing the results. We will cover both simple and more complex case study, which help us understand the challenges involved in running experiments.

This talk will cover a topic that will enable developers to make more data-driven decisions but has not been covered at Pycon. By providing case studies as motivation and a procedure to implement A/B testing this talk will excite the audience. Yelp runs multiple experiments on different aspects and the Transaction Platform team has gotten unique experience of needing to create experiments with limited traffic which will be discussed in the talk.

Bio
Or Weizman is an engineer for Yelp's Transaction Platform team, which enables users to transact with Yelp's extensive set of businesses through many third party providers.

https://youtu.be/7SA3a_AXA1g
​

Transcript

1. A/B Testing Harder than just a color change Or Weizman weizman@yelp.com
2. Yelp’s Mission Connecting people with great local businesses.
3. Yelp Stats As of Q2 2016 92M 32 72% 108M
6. 15%
7. A/B Experiment? • What? • When? • Why? ? DATA DECISIONS
8. How? • Generate Hypothesis • Gather Necessary Data • Implementation and Testing • Roll out Experiment • Analyze Results
9. Case Study 1: Greener Pastures
10. Generate Hypothesis Analyze Roll Out Implement & Test Gather Data Hypothesis
12. Gather Necessary Data Analyze Roll Out Implement & Test Gather Data Hypothesis
13. What kind of data? • Click Through Rate • Weekly Traffic • What else?
14. • Key Metric • Secondary Metrics User lands on Biz page User clicks on Start Order User adds items User completes order User lands on Biz page User clicks on Start Order
15. Math Time! • Minimum Detectable Effect of 5% • Baseline conversion • Statistical power (1 - β) of 95% • Significance level α of 5% 307,967 users per variation
16. Implementation and Testing Analyze Roll Out Implement & Test Gather Data Hypothesis
17. 50% variation 50% status quo
18. Kwargs are your friends not food self.button_color_cohort = kwargs.get('button_color_cohort', 'status_quo') if self.button_color_cohort == 'status_quo': button_color = 'orange' else: button_color = 'green'
19. Unit tests aren’t enough! • Run through flows • Visual Inspection • Integration Tests
20. Roll out Experiment Analyze Roll Out Implement & Test Gather Data Hypothesis
21. Steady • If possible roll out in this order ◦ Internally for testing ◦ To a small percentage traffic ◦ To predetermined traffic percentage ▪ 50/50 in our case
22. Analyze Results Analyze Roll Out Implement & Test Gather Data Hypothesis
23. Results • 5.26% relative increase • Great Success!
24. Good News Everyone!
25. Case Study 2: An abundance of experiments
26. Generate Hypothesis Analyze Roll Out Implement & Test Gather Data Hypothesis
33. Gather Necessary Data Analyze Roll Out Implement & Test Gather Data Hypothesis
34. More than 1 Experimental Variation • Correct for multiple • α = 5%/(# of cohorts - 1)
35. Implementation and Testing Analyze Roll Out Implement & Test Gather Data Hypothesis
36. Let's Build It!
37. Let's Build It!
38. Let's Build It! Left Layout Right Layout Orange Green Red Blue Dark Orange
39. Let's Build It! Alternatively run sequentially but this is slow....
40. Sharing the Pool Experiment 1 off on
41. Sharing the Pool Experiment 2 on off
42. Sharing the Pool Experiment 1 and Experiment 2 are active
43. Sharing the Pool • Swimlanes is a way to pre-partition • Experiments assigned to lanes • Allows for control
44. Sharing the Pool Experiment 1 off on unavailable unavailable
45. What’s better?
46. What’s better? Experiment 1 Experiment 2 Experiment 3 Experiment 4
47. Testing Gets harder • Experiments can now have another condition • Previous if statement breaks • Complexity -> more cases
48. if self.button_color_cohort == 'status_quo': button_color = 'orange' else: button_color = 'green' Testing Gets harder if self.button_color_cohort == ‘enabled’: button_color = 'green' else: button_color = 'orange'
49. Roll out Experiment Analyze Roll Out Implement & Test Gather Data Hypothesis
50. Autobots Roll Out
51. Analyze Results Analyze Roll Out Implement & Test Gather Data Hypothesis
52. • Green the Winner • What’s next?
53. • Left is better! • 9.3% increase • Secondary metrics heavily impacted
54. Summary • Understand How to do A/B Experiments • Harder but Doable! ◦ Complexity can be managed ◦ Swimlanes are our solution • After just 3 experiments saw 15% increase in sales
55. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp