$30 off During Our Annual Pro Sale. View Details »

From Predictions to Decisions

From Predictions to Decisions

Bringing Decision Theory to Healthcare Data Science

Corey Chivers

June 14, 2018
Tweet

More Decks by Corey Chivers

Other Decks in Science

Transcript

  1. From Predictions to
    Decisions
    Bringing Decision Theory to Healthcare Data Science
    Corey Chivers, PhD
    bayesianbiologist.com
    @cjbayesian

    View Slide

  2. View Slide

  3. Is this a good model?
    (1 – Specificity)
    (Sensitivity)

    View Slide

  4. Is this a good model?
    FP
    FN TP
    TN

    View Slide

  5. "Good" is only meaningful relative to
    the "goodness" of the events being
    predicted and our ability to do
    something to change them.

    View Slide

  6. All models are
    wrong, but some
    are useful.
    - George Box

    View Slide

  7. FP
    FN TP
    TN

    View Slide

  8. How good tho?
    How good tho?
    How bad tho?
    How bad tho?
    FP
    FN TP
    TN

    View Slide

  9. Goodness can be measured in any units
    More adorbs Less adorbs

    View Slide

  10. The best decision is the one that leads to the
    most goodness
    More adorbs Less adorbs

    View Slide

  11. When outcomes are uncertain, the best decision is
    the one that has the highest expected goodness.

    View Slide

  12. When outcomes are uncertain, the best decision is
    the one that has the highest expected goodness.
    Machine Learning
    can only help you
    with this part!

    View Slide

  13. When outcomes are uncertain, the best decision is
    the one that has the highest expected goodness.
    Machine Learning
    can only help you
    with this part!

    View Slide

  14. Healthcare Example
    Predicted Sepsis
    Treated a true case
    (Potential to avoid
    bad outcome)
    Predicted Sepsis
    Treated a false case
    (unnecessary)
    Predicted No Sepsis
    Didn’t treat
    (all good)
    Predicted No Sepsis
    Failed to treat
    (Bad outcome)

    View Slide

  15. Healthcare Example

    View Slide

  16. Healthcare Example

    View Slide

  17. Healthcare Example
    • But everything just adds to the ‘badness’ total.
    • How does that help us decide?
    • The key is to compare against the alternatives.
    • Treat everyone? <==> set decision threshold to zero
    • Treat no one? <==> set decision threshold to one
    • Some other strategy? <==> Random? Status quo?
    • Choose the alternative that is least bad
    • Or, if you prefer the J perspective, multiply everything by -1 and choose the
    one that’s the most good!

    View Slide

  18. Healthcare Example
    • But everything just adds to the ‘badness’ total.
    • How does that help us decide?
    • The key is to compare against the alternatives.
    • Treat everyone? <==> set decision threshold to zero
    • Treat no one? <==> set decision threshold to one
    • Some other strategy? <==> Random? Status quo?
    • Choose the alternative that is least bad
    • Or, if you prefer the J perspective, multiply everything by -1 and choose the
    one that’s the most good!

    View Slide

  19. So, like, should we use this model or what?
    • There are still things we don’t know,
    like what all these C terms actually are.
    • Work backwards and ask:
    • In what range would they need to be for
    us to choose the model over an
    alternative strategy?
    • How much better would the model need
    to be before we would choose to use it?

    View Slide

  20. Treat none
    Cost of Intervention ($)
    Cost of event ($)
    Treat all
    Pregnancy Related Hypertension (PRH) is the leading
    cause of maternal morbidity and mortality in the U.S.
    High-risk patients à remote blood pressure monitoring
    https://healthcareinnovation.upenn.edu/projects/heart-safe-motherhood

    View Slide

  21. Assume, evaluate, refine, repeat
    • Decision theory requires making assumptions, as we’ve done here
    • Nothing is stopping us from making different assumptions:
    • “What if the false positives lead to additional harm?”
    • “What if I have other priorities, like fairness?”
    • Decision theory analysis is not infallible, but it allows us to:
    • Get the best possible answers
    • to the most precisely formulated questions,
    • when starting from a given set of assumptions

    View Slide

  22. Thanks!
    bayesianbiologist.com
    @cjbayesian

    View Slide