Notes on Vickers’ Decision Curve Analysis

Decision Curve Analysis [1] enables clinical decisions that account for model error, cost / benefit of interventions, and risk preferences.

With DCA, net benefit of models can be compared directly.

This brief tutorial includes content from Dr. Andrew Vickers and Dr. Karandeep Singh

[1] Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006 Nov-Dec;26(6):565-74.

Transcript

1. Notes on Vickers’
   Decision Curve
   Analysis
   Clinical decisions that account for
   model error, cost / benefit of
   interventions, and risk preferences
   Robin R. Chauhan
   https://twitter.com/robinc
   robin@pathwayi.com
   

   View Slide

2. Last talk:
   On the Evaluation of Binary Classifiers,
   Robin Chauhan
   https://speakerdeck.com/robinchauhan/on-t
   he-evaluation-of-binary-classifiers
   Wikipedia topic
   https://en.wikipedia.org/wiki/Evaluation_of_b
   inary_classifiers
   Vickers AJ, Elkin EB. Decision curve analysis: a novel method for
   evaluating prediction models. Medical Decision Making. 2006
   Nov-Dec;26(6):565-74.
   See http://decisioncurveanalysis.org/
   Model ->
   Prediction ->
   Probability -> Calibration Curves
   Threshold -> Threshold Selection: ROC, TOC
   Decision ->
   Outcome -> Accuracy, F1, MCC
   This Talk:
   Cost/Benefit + Risk preferences
   Decision Curve Analysis
   Decision Pipeline with DCA
   

   View Slide

3. Thanks to Dr. Andrew Vickers at Memorial Sloan Kettering Cancer Center, for the data and concept for this slide.
   Decision curve plot made via R package “dcurves”, the others with scikit-learn and matplotlib.
   Models A+B predict cancer
   Which model should we use to
   decide who gets a biopsy?
   Is either model beneficial?
   

   View Slide

4. Thanks to Dr. Andrew Vickers at Memorial Sloan Kettering Cancer Center, for the data and concept for this slide.
   Decision curve plot made via R package “dcurves”, the others with scikit-learn and matplotlib.
   Decision Curve
   # Data Format:
   cancer modela modelb
   0 0.01 0.11
   0 0.08 0.18
   1 0.01 0.01
   0 0.01 0.11
   0 0.04 0.06
   1 0.08 0.19
   0 0.13 0.25
   0 0.02 0.01
   ...
   # R:
   install.packages('dcurves')
   library(dcurves)
   …
   dca(cancer ~ modela + modelb, tb,
   thresholds=seq(0.05,0.20,by=0.01) )
   

   View Slide

5. Thanks to Dr. Andrew Vickers at Memorial Sloan Kettering Cancer Center, for the data and concept for this slide.
   Decision curve plot made via R package “dcurves”, the others with scikit-learn and matplotlib.
   Decision Curve
   # Data Format:
   cancer modela modelb
   0 0.01 0.11
   0 0.08 0.18
   1 0.01 0.01
   0 0.01 0.11
   0 0.04 0.06
   1 0.08 0.19
   0 0.13 0.25
   0 0.02 0.01
   ...
   # R:
   install.packages('dcurves')
   library(dcurves)
   …
   dca(cancer ~ modela + modelb, tb,
   thresholds=seq(0.05,0.20,by=0.01) )
   

   View Slide

6. Decision Curve
   Thanks to Dr. Andrew Vickers at Memorial Sloan Kettering Cancer Center, for the data and concept for this slide.
   Decision curve plot made via R package “dcurves”, the others with scikit-learn and matplotlib.
   

   View Slide

7. Questions answered by Decision Curve Analysis
   ● Is this model useful in practice?
   ● Which policies are best for patients
   with different risk preferences?
   “sensitivity, specificity, or AUC … do not tell
   us whether the model … would do more good
   than harm if used in clinical practice”
   - Net benefit approaches to the evaluation of prediction
   models, molecular markers, and diagnostic tests, Vickers
   et al 2015, https://www.bmj.com/content/352/bmj.i6
   Image: Fig 1 from:
   A simple, step-by-step guide to interpreting decision curve analysis
   Andrew J. Vickers, Ben van Calster & Ewout W. Steyerberg
   Diagnostic and Prognostic Research volume 3, Article number: 18 (2019)
   https://diagnprognres.biomedcentral.com/articles/10.1186/s41512-019-0064-7
   

   View Slide

8. Goals of Decision Curve Analysis
   ● Compare various decision rules and
   approximate decision models
   ● Account for costs, benefits, and model
   error …
   ● … Covering a range of risk thresholds /
   cost-benefit ratios
   ● Without additional data (often difficult to
   accurately quantify costs + benefits)
   See: Vickers in “Decision curve analysis: a discussion”,
   Steyerberg et al 2008,
   https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2577563/
   

   View Slide

9. Therapeutic decision making: a cost-benefit analysis, Pauker 1975
   Benefit => Benefit of True
   Positive
   correctly treating the diseased
   Cost => Cost of False Positive
   incorrectly treating the
   non-diseased
   

   View Slide

10. Cost/Benefit Exchange Rate
    Equivalently:
    The threshold model revisited, Djulbegovic et al 2018
    https://onlinelibrary.wiley.com/doi/10.1111/jep.13091
    Image + slide via Dr. Karandeep Singh
    https://twitter.com/kdpsinghlab/status/1434696280719118340/
    

    View Slide

11. ● Interpretation of Cost and Benefit: Pauker
    1975
    ○ Benefit of Treating diseased vs
    ○ Cost of Treating non-diseased
    ● Interpretation of phrase “Net Benefit”
    ○ Pauker 1975 version
    ○ Vickers Net Benefit version (normalized)
    ● {Cost,Benefit} pair have 2 separate but
    related relationships
    ○ Ratio: Cost/Benefit ratio on x-axis of DCA plot
    ○ Difference: ( Benefit - Cost )/Benefit Net
    Benefit (normalized) on y-axis of DCA plot
    ● Estimating Cost/Benefit from clinical policy
    ○ clinician willing to conduct 20 biopsies to find a
    high grade cancer, the probability threshold
    would be 5%
    ■ – Net benefit approaches to the evaluation of
    prediction models, molecular markers, and
    diagnostic tests, Vickers et al 2015
    Things that were unclear to me when first learning DCA
    ● Equivalence of:
    ○ Cost / Benefit ratio for clinical procedure ⇔ Model threshold :
    “Exchange rate”
    ○ Both shown on same x-axis
    ● Why are we so into varying threshold probability?
    ○ “a) there are insufficient data on which to calculate a rational
    threshold, or
    b) patients can reasonably disagree about the appropriate
    threshold, due to different preferences for alternative health states”
    – Vickers in “Decision curve analysis: a discussion”, Steyerberg et
    al 2008, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2577563/
    ● What data is needed to construct DCA plot
    ○ Y_true: True binary labels
    ○ For models: Y_score: Estimated probability from model (thresholded
    to determine treatment decision)
    ○ For rules: Treatment decision [0/1]
    ● Interpretation of x axis:
    ○ x=0.0 : no costs only benefits, x=1.0 : no benefits only costs
    ○ X is both Cost / Benefit ratio ⇔ Model threshold
    ● Interpretation of y axis:
    ○ y=0.0 : do nothing implies no net benefit (no treatment)
    ○ Y axis units: expectation of true-positive result, 1.0 is benefit of
    treating 1 diseased patient
    ○ Max y value: prevalence; ie. mean benefit of treating all diseased
    patients
    

    View Slide

12. Special thank you to:
    ● Dr Andrew Vickers at Memorial Sloan Kettering Cancer
    Center for the data, comments and patience
    https://twitter.com/VickersBiostats
    ● Dr Karandeep Singh at University of Michigan for your
    advice and patience https://twitter.com/kdpsinghlab
    Resources
    ● http://decisioncurveanalysis.org/
    ● Vickers AJ, Elkin EB. Decision curve analysis: a novel
    method for evaluating prediction models. Medical
    Decision Making. 2006 Nov-Dec;26(6):565-74.
    https://pubmed.ncbi.nlm.nih.gov/17099194/
    ● Therapeutic decision making: a cost-benefit analysis,
    Pauker et al 1975
    https://pubmed.ncbi.nlm.nih.gov/1143303/
    ● https://cran.r-project.org/web/packages/dcurves/
    Email: robin@pathwayi.com
    Twitter: @robinc
    Podcast: TalkRL.com
    

    View Slide

13. End
    

    View Slide