Information Retrieval Performance Measurement Using Extrapolated Precision (DataPhilly)

Performance measures like the F1-score make strong assumptions about the trade-off between recall and precision that are not a good fit for some contexts like e-discovery. This presentation advocates comparing performance at different recall levels, when necessary, by using a novel method for extrapolating a single precision-recall point to a different level of recall. With this approach, the constant-performance contours are a parameterized family of reference precision-recall curves.

Transcript

1. Information Retrieval Performance Measurement Using Extrapolated Precision or How to Extrapolate From One Data Point Bill Dimm Data Philly October 25, 2016
2. Precision-Recall Curves
3. Comparing Precision-Recall Curves
4. What if we only know one point?
5. Comparing Precision-Recall Points
6. F 1 Score
7. F 1 Comparison: Wrong Conclusion!
8. F 1 Depends Strongly on Recall
9. F 1 Contours
10. F 1 Contours
11. Danger Zone
12. Danger Zone
13. Want Contours Like P-R Curves
14. Less Cutting Across Contours
15. How to Quantify?
16. Precision's Relationship with Cost • Precision is meaningful – inversely proportional to number of docs to review: n = ρNR/P
17. Extrapolated Precision
18. X is Fairly Independent of R
19. Precision-Recall Curve Math
20. Actual Precision-Recall Curves
21. Actual Probability Curves
22. Model
23. Model Probability vs Actual
24. Model Precision vs Actual
25. Extrapolation Limitations • P < 0.99 • R < 0.99 • P >= 2ρ / (1 + ρ + R*(1 - ρ))
26. Summary • Proportionality dictates recall – Need performance measure less sensitive to recall • Extrapolate precision-recall point to target recall level using model curves • Model precision-recall curves are constant performance contours • When close to target recall, performance measure inversely proportional to review cost