Practical Tips for Reviewing Presentations of Machine Learning Healthcare Applications

There is a fast growing interest in the application of machine learning to the field of healthcare. Medical institutions, universities and software giants like Google and Amazon are dedicating increasing resources to this field. Medical informaticians are likely to be exposed to presentations of machine learning applications by researchers, external vendors, or just by browsing the scientific literature. This is a very exciting but relatively young field. However, best practices for methods and reporting of results are not yet fully established.

This presentation gives an overview of keys aspects of machine learning methods, along with pros and cons of the most important error metrics. The goal is to provide simple guidelines on what to look for and which questions to ask to assess the validity of claims of oral presentations and papers.

Transcript

1. Lorenzo Rossi, PhD Data Scientist, City of Hope Center for Informatics CI4CC, October 2018, New Orleans, LA Practical Tips for Reviewing Presentations of Machine Learning Healthcare Applications
2. Machine Learning for Healthcare Growing Fast
3. Machine Learning for Healthcare Growing Fast
4. Machine Learning for Healthcare Growing Fast
5. Best Practices not yet Well Established Towards Guidelines for ML in Health (8.2018, Stanford)
6. Cancer informaticians will be exposed to increasing numbers of ML presentations Sources: – vendors – researchers – literature
7. Cancer informaticians will be exposed to increasing numbers of ML presentations Sources: – vendors – researchers – literature Questions: How does it really perform? Would the application fit to my organization? Any red flags?
8. Learning Objectives • Performance Metrics and Reporting • Methods • Cohort Definition Huge subject!
9. Learning Objectives • Performance Metrics and Reporting • Methods • Cohort Definition Huge subject! This presentation covers only aspects of 1st topic
10. Performance Metrics and Reporting
11. Consider only binary prediction tasks for simplicity Actual answer only Yes (1) or No (0), e.g.: – 30 day readmission – 6 month mortality – sepsis detection Prediction algorithm gives score from 0 to 1 – E.g. close to 1 → high risk of readmission within 30 days
12. Consider only binary prediction tasks for simplicity Actual answer only Yes (1) or No (0), e.g.: – 30 day readmission – 6 month mortality – sepsis detection Prediction algorithm gives score from 0 to 1 – E.g. close to 1 → high risk of readmission within 30 days Trade-off between false alarms and missed targets
13. Example: ROC Curve Very high detection rate, but also high false alarm rate
14. Types of Performance Metrics 1. Measure trade-offs – (ROC) AUC – average precision / PR AUC 2. Measure error rate at a specific decision point – false positives, false negatives, … – precision and recall – F1 – accuracy
15. Types of Performance Metrics (II) 1. Measure trade-offs – AUC, average precision / PR AUC, – good for global performance characterization and (intra)- model comparisons 2. Measure error rate at a specific decision point – false positives, false negatives, …, precision, recall – possibly good for interpretation of specific clinical costs and benefits
16. Performance should be reported with both types of metrics • 1 or 2 metrics for trade-off evaluation – ROC AUC – average precision, log loss, • 1 metric for performance at clinically meaningful decision point – e.g. recall @ 90% precision
17. Performance should be reported with both types of metrics • 1 or 2 metrics for trade-off evaluation – ROC AUC – average precision, log loss, • 1 metric for performance at clinically meaningful decision point – e.g. recall @ 90% precision + Comparison with a known benchmark (baseline)
18. Examples and Tips
19. 1. Metrics in Stanford 2017 paper on mortality prediction: AUC, average precision, recall @ 90%
20. 2. Be skeptic if accuracy is the only reported metric!
21. 3. ROC AUC can be misleading
22. [Avati, Ng et al., Countdown Regression: Sharp and Calibrated Survival Predictions. ArXiv, 2018] 3. ROC AUC can be misleading
23. 3. ROC AUC (1 year) > ROC AUC (5 years), but PR AUC (1 year) < PR AUC (5 years)! Latter prediction task is easier. [Avati, Ng et al., Countdown Regression: Sharp and Calibrated Survival Predictions. ArXiv, 2018]
24. Benchmarks
25. 4. Main paper [Google, Nature, 2018] only reports deep learning results with no benchmark comparison
26. 4. Comparison only in supplemental online file (not on Nature paper): deep learning only 1-2% better than logistic regression benchmark
27. 5. Plot scales can be deceiving [Unnamed Vendor, 2017]!
28. 5. Same TP, FP plots rescaled
29. Final Tips (Binary Predictions) • Look at types of metrics being reported – ROC AUC alone could be misleading – Precision-Recall curve often more useful than ROC – Accuracy misleading unless target classes are balanced (e.g. ~50% of patient readmitted) • Look for comparison with known benchmark • Check scales of plots
30. Thank You! Email: lrossi@coh.org Twitter: @LorenzoARossi
31. Supplemental Material
32. Methods: Testing