From Predictions to Decisions

From Predictions to Decisions

Bringing Decision Theory to Healthcare Data Science

5779dc3bf8c8a0c9dadb7ff95c67e9e3?s=128

Corey Chivers

June 14, 2018
Tweet

Transcript

  1. From Predictions to Decisions Bringing Decision Theory to Healthcare Data

    Science Corey Chivers, PhD bayesianbiologist.com @cjbayesian
  2. None
  3. Is this a good model? (1 – Specificity) (Sensitivity)

  4. Is this a good model? FP FN TP TN

  5. "Good" is only meaningful relative to the "goodness" of the

    events being predicted and our ability to do something to change them.
  6. All models are wrong, but some are useful. - George

    Box
  7. FP FN TP TN

  8. How good tho? How good tho? How bad tho? How

    bad tho? FP FN TP TN
  9. Goodness can be measured in any units More adorbs Less

    adorbs
  10. The best decision is the one that leads to the

    most goodness More adorbs Less adorbs
  11. When outcomes are uncertain, the best decision is the one

    that has the highest expected goodness.
  12. When outcomes are uncertain, the best decision is the one

    that has the highest expected goodness. Machine Learning can only help you with this part!
  13. When outcomes are uncertain, the best decision is the one

    that has the highest expected goodness. Machine Learning can only help you with this part!
  14. Healthcare Example Predicted Sepsis Treated a true case (Potential to

    avoid bad outcome) Predicted Sepsis Treated a false case (unnecessary) Predicted No Sepsis Didn’t treat (all good) Predicted No Sepsis Failed to treat (Bad outcome)
  15. Healthcare Example

  16. Healthcare Example

  17. Healthcare Example • But everything just adds to the ‘badness’

    total. • How does that help us decide? • The key is to compare against the alternatives. • Treat everyone? <==> set decision threshold to zero • Treat no one? <==> set decision threshold to one • Some other strategy? <==> Random? Status quo? • Choose the alternative that is least bad • Or, if you prefer the J perspective, multiply everything by -1 and choose the one that’s the most good!
  18. Healthcare Example • But everything just adds to the ‘badness’

    total. • How does that help us decide? • The key is to compare against the alternatives. • Treat everyone? <==> set decision threshold to zero • Treat no one? <==> set decision threshold to one • Some other strategy? <==> Random? Status quo? • Choose the alternative that is least bad • Or, if you prefer the J perspective, multiply everything by -1 and choose the one that’s the most good!
  19. So, like, should we use this model or what? •

    There are still things we don’t know, like what all these C terms actually are. • Work backwards and ask: • In what range would they need to be for us to choose the model over an alternative strategy? • How much better would the model need to be before we would choose to use it?
  20. Treat none Cost of Intervention ($) Cost of event ($)

    Treat all Pregnancy Related Hypertension (PRH) is the leading cause of maternal morbidity and mortality in the U.S. High-risk patients à remote blood pressure monitoring https://healthcareinnovation.upenn.edu/projects/heart-safe-motherhood
  21. Assume, evaluate, refine, repeat • Decision theory requires making assumptions,

    as we’ve done here • Nothing is stopping us from making different assumptions: • “What if the false positives lead to additional harm?” • “What if I have other priorities, like fairness?” • Decision theory analysis is not infallible, but it allows us to: • Get the best possible answers • to the most precisely formulated questions, • when starting from a given set of assumptions
  22. Thanks! bayesianbiologist.com @cjbayesian