Slide 18
Slide 18 text
12の落とし穴
1. Metric Sample Ratio
Mismatch
2. Misinterpretation of Ratio
Metrics
3. Telemetry Loss Bias
4. Assuming Underpowered
Metrics had no Change
5. Claiming Success with a
Borderline P-value
6. Continuous Monitoring and
Early Stopping
7. Assuming the Metric
Movement is Homogeneous
8. Segment Interpretation
9. Impact of Outliers
10.Novelty and Primacy
Effects
11.Incomplete Funnel Metrics
12.Failure to Apply Twyman's
Law