How To Avoid Lying To Yourself With Statistics

Great teams don’t run experiments to prove they are right; they run them to answer questions. Since it’s human nature to want things to go your way, how can we guard against falling into the traps of wishful thinking and hidden biases as we strive to reach meaningful outcomes?

You can follow a handful of core principles in the design, execution, and analysis of experiments proven by teams that run hundreds or even thousands of experiments every month. If you do, you'll increase the chances of learning something truly useful and you reduce the odds of wasting your time and heading off in an unproductive direction due to false signals.

Whether you are an “old hand” at online experimentation, merely “experimentation curious,” or somewhere in between, you’ll tilt the odds of productive outcomes in your favor by reviewing these slides and/or watching the video and transcript posted at https://www.split.io/blog/how-to-avoid-lying-to-yourself-with-statistics/

Transcript

1. How To Avoid Lying To Yourself With Statistics A Shortlist of Core Principles for Productive Online Controlled Experiments
2. • Why Experiment? • Why Sweat the Details? • Core Experimentation Principles By Phase: • Design • Execution • Analysis • Wrap-Up + Q&A
3. Why Experiment?
4. * Source: HBR - The Surprising Power of Online Experiments by Ronny Kohavi, Stefan Thomke 80-90% of features shipped have negative or neutral impact on metrics they were designed to improve* 4 How Effective Are We At Moving The Needle?
5. Who Will Ultimately Decide?
6. Who Will Ultimately Decide? “Some designers and managers have attacked A/B testing because of a misconception that testing somehow replaces or diminishes design. Nothing could be further from the truth. Good design comes first.”
7. “The key is that it is the end users who are the final arbiters of design success Who Will Ultimately Decide? and A/B testing is the instrument that informs us of their judgment.”
8. • Why Experiment? • Why Sweat the Details? • Core Experimentation Principles By Phase: • Design • Execution • Analysis • Wrap-Up + Q&A
9. Why Sweat The Details When Using Experimentation?
10. We want to avoid the cost of false signals Remember this example and you’ll never confuse false positive & false negative again Type I Error False positive Type II Error False negative
11. We want to avoid the cost of human bias Here are 180+ forms of it. (It’s a big problem)
12. Controlled experimentation done right removes external influences. Metrics Change Your new feature All other changes in the world
13. Controlled experimentation can distinguish noise from a real signal. There is always noise or randomness in the data. If we flip a coin 1000 times - we’re unlikely to get exactly 500 heads and 500 tails (only a 3% chance) - that doesn’t mean the coin is biased. Experimentation statistics allow us to determine if a result is big enough to rule out normal variations and noise. There is always noise or randomness in the data. If we flip a coin 1000 times - we’re unlikely to get exactly 500 heads and 500 tails (only a 3% chance) - that doesn’t mean the coin is biased. Experimentation statistics allow us to determine if a result is big enough to rule out normal variations and noise.
14. 14  Statistically Significant: The result is unlikely to have occurred by chance alone
15. • Why Experiment? • Why Sweat the Details? • Core Experimentation Principles By Phase: • Design • Execution • Analysis • Wrap-Up + Q&A
16. Core Experimentation Principles By Phase Design Execution Analysis
17. Our example: Airbnb host sign-up page A no video B with video
18. Design 1. Have an Explicit Hypothesis 2. Choose Metrics with Care 3. Perform Power Analysis to Determine Time Needed To Hit Traffic Goal 4. Consider Business Cycles & Seasonality Why? To prevent after-the-fact fitting of results to wishful thinking or hidden biases.
19. Design # 1 Have an Explicit Hypothesis
20. Standardized Format, Easy & Repeatable Steps A good hypothesis has a benchmark, is clear, concise and understandable, testable and measurable. By Craig Sullivan
21. B video Airbnb hypothesis : Because we see potential hosts aren’t aware of their level of control, we expect that adding an informative video will provide reassurances leading to more sign-ups. We’ll measure this using the metric: proportion of users who sign up as hosts A no video B with video
22. Airbnb hypothesis : Because our survey showed potential hosts aren’t aware of their level of control, we expect that adding an informative video for Canadian users will provide reassurances leading to more sign-ups. We’ll measure this using the metric: proportion of users who visited the host homepage who sign up as hosts. We expect to see an increase of 5% in the host-page conversion rate over a period of 2 weeks B video A no video B with video
23. Design # 2 Choose Metrics With Care
24. # 2.1 Characteristics of Good Metrics Design: #2 Choose Metrics With Care
25. 25 Characteristics of Good Metrics Key properties: • Meaningful - The metric directly captures business value or customer satisfaction • Sensitive - The metric should change due to small changes in user satisfaction or business value so that the result of an experiment is clear within a reasonable time period. • Directional - If the business value increases, the metric should move consistently in one direction. If it decreases, the metric should move in the opposite direction. • Understandable - It should be easily understood by business executives.
26. # 2.2 Beware of focusing on the wrong metric. Design: #2 Choose Metrics With Care
27. 27 The lure of winning: poor choice of OEC What went wrong? queries run per user as the north star metric at bing
28. 28 The lure of winning: blatant gaming Chinese shoe company tricks people into swiping Instagram ad with fake strand of hair.
29. 29 Examples of Key Hypothesis Metrics Examples: • # Sessions / User - Google • # Tweets / User - Twitter • # Rides / User - Uber One way to recognize whether you have a good key metric is to intentionally experiment with a bad feature you know your users would not like.
30. Airbnb Key Metric : Proportion of users who visited the host homepage who sign up as hosts B video A no video B with video
31. # 2.3 Beware of measuring just one metric. Design: #2 Choose Metrics With Care
32. NYPD CompStat One Metric: Reduce Reported Crime
33. 33  As NYPD Assistant Commissioner Ronald J. Wilhelmy wrote in a November 2013 internal NYPD strategy document: [W]e cannot continue to evaluate personnel on the simple measure of whether crime is up or down relative to a prior period. Most importantly, CompStat has ignored measurement of other core functions. Chiefly, we fail to measure what may be our highest priority: public satisfaction. We also fail to measure quality of life, integrity, community relations, administrative efficiency, and employee satisfaction, to name just a few other important areas.
34. 34 Guardrail metrics Guardrails are metrics that should not degrade in pursuit of the key metric. Like the key metrics, they should be directional and sensitive, but not necessarily tie back to business value.
35. # 2.4 Beware of too many metrics. Design: #2 Choose Metrics With Care
36. Every comparison you make brings α% chance of a false positive. This applies to testing multiple - Metrics Treatments Segments More comparisons → more false positives!
38. Safe ways to measure many metrics: Use a stricter confidence level 1,2 Re-test significant metrics that weren’t your primary metric
39. # 3 Perform Power Analysis to Determine Time Needed To Hit Traffic Goal. Design
40. 40 Understanding the power of your experiment is vital. • Any test will only be able to measure differences larger than a given size • Larger sample size → can detect smaller differences Minimum Likely Detectable Effect (MLDE) “The smallest effect your test is likely to be able to detect”
41. Power : “The likelihood of detecting an effect of a given size” Minimum Likely Detectable Effect (MLDE) : “The smallest change your test is likely to be able to detect” α = 0.05
42. Power : “The likelihood of detecting an effect of a given size” Minimum Likely Detectable Effect (MLDE) : “The smallest change your test is likely to be able to detect” α = 0.05
43. # 4 Consider Business Cycles & Seasonality Design
44. 44 Business Cycles & Seasonality Usually different traffic on different days of the week → recommend running weekly cycles
45. Execution 1. Monitor For Data Flow, Randomization (SRM) 2. Monitor for Big Degradations 3. Don’t Peek/Cherry Pick Outcome
46. # 5 Monitor For Data Flow, Randomization (SRM) Execution
47. Is data flowing through?
48. Did randomization work as targeted? (Sample Ratio Mismatch)
49. # 6 Monitor For Big Degradations Execution
50. Catch obvious broken stuff
51. # 7 Don’t Peek or Cherry Pick Outcome Execution
53. Analysis 1. Observe Statistical Significance, Not Just Raw Deltas of Metrics 2. Don’t “Dig” for “Insights” except to Formulate a New Hypothesis To Test
54. # 8 Don’t Peek or Cherry Pick Outcome Analysis
55. p-value = how likely is it that you’d see a difference as big as this if there were no real difference Example: flip a coin 100 times to check if its ‘fair’ • Getting 45 heads, 55 tails is not that unlikely for a fair coin → p-value = 0.32 • Getting 10 heads, 90 tails is very unlikely! → p-value = 0.000000000000001 You can calculate the p-value yourself, or use an online calculator 5,6 32% chance of getting a difference this big with a fair coin Close to 0% chance of getting a difference this big with a fair coin
56. Airbnb example analysis A 1000 visitors, 30% (300) signed up B 1000 visitors, 25% (250) signed up
57. “ Not Statistically Significant ” Does not mean your treatment had no impact It means that your test wasn’t able to confidently say it did have an impact
58. # 10 Don’t “Dig” for “Insights” except to Formulate a New Hypothesis Analysis
59. Experimentation Results & Actions Statistically Significant Continue to ramp the winning experience to maximise its value
60. Experimentation Results & Actions Statistically Significant Statistically Significant Continue to ramp the winning experience to maximise its value Kill experiment and re-iterate on your implementation
61. Experimentation Results & Actions Statistically Significant Statistically Significant Statistically Inconclusive Continue to ramp the winning experience to maximise its value Kill experiment and re-iterate on your implementation This inconclusive result means we can't confidently say there was an impact.
62. • Why Experiment? • Why Sweat the Details? • Core Experimentation Principles By Phase: • Design • Execution • Analysis • Wrap-Up + Q&A
63. Takeaways.
64. Users are the final arbiter of your design decisions. Takeaway One.
65. Experimentation allows us to watch our users vote with their actions. Takeaway Two.
66. To achieve a meaningful outcome, we must know what we are testing upfront and invest time in choosing the right metrics Takeaway Three.
67. Any well designed, implemented and analyzed experiment is a successful experiment Takeaway Four.
68. References: Multiple Comparison Corrections 1Bonferroni correction : https://www.stat.berkeley.edu/~mgoldman/Section0402.pdf 2Benjamini Hochberg procedure : https://www.statisticshowto.datasciencecentral.com/benjamini-hochberg-procedure/ Online power analysis calculators 3For proportions: http://www.experimentationhub.com/hypothesis-kit.html,4For means: http://statulator.com/SampleSize/ss2M.html Online p-value calculators 5For proportions: https://www.socscistatistics.com/tests/chisquare, 6For means: http://www.statskingdom.com/140MeanT2eq.html
69. 69 Review Periods & Seasonality Usually different traffic on different days of the week → recommend running weekly cycles
70. Correlation is not Causation
71. 71 Business Cycles & Seasonality Usually different traffic on different days of the week → recommend running weekly cycles