Risk stratification: the UK cardiothoracic experience

Presented at the PROMS 2.0 workshop, Manchester, UK

Transcript

1. Risk  stra)fica)on:  The  UK  
   cardiothoracic  experience  
   Graeme  L  Hickey1;  Stuart  W  Grant2;  Iain  Buchan1;  
   Ben  Bridgewater1,2  
    
   1Northwest  Ins.tute  of  BioHealth  Informa.cs,  Manchester  University  
   2Department  of  Cardiothoracic  Surgery,  University  Hospital  of  South  Manchester  
   

   View Slide

2. Background  
   •  Around  35,000  adult  cardiac  surgery  
   procedures  performed  each  year  in  UK  
   •  In-­‐hospital  mortality  rate  in  2010-­‐11  was  3.4%  
    
   

   View Slide

3. What’s  risk  stra)fica)on  used  for?  
   The Society for
   Cardiothoracic Surgery
   in Great Britain & Ireland
   Sixth
   National Adult Cardiac
   Surgical Database Report
   2008
   Demonstrating quality
   Prepared by
   Ben Bridgewater PhD FRCS
   Bruce Keogh KBE DSc MD FRCS FRCP
   on behalf of the Society for Cardiothoracic Surgery
   in Great Britain & Ireland
   Robin Kinsman BSc PhD
   Peter Walton MA MB BChir MBA
   Dendrite Clinical Systems
   Cardiac Surgery
   Governance   Decision-­‐making  
   

   View Slide

4. Mo)va)on  
   •  Total  cost  =  £1.48m/year  in  England  (<1%  of  
   the  total  NHS  spend  on  adult  cardiac  surgery)*  
   •  Associated  with  a  50%  reduc)on  in  risk  
   adjusted  mortality*  
   *Maintaining  Pa.ents’  Trust,  SCTS,  Henley-­‐on-­‐Thames:  
   Dendrite  Clinical  Systems  Ltd,  2011  
   

   View Slide

5. Infrastructure  
   Cardiac  surgery   Input  data  locally   Uploaded  periodically  
   to  central  database  
   Aim:  3  months  
   Reality:  1  year  
   Sta)s)cian  +  clinicians  
   The Society for
   Cardiothoracic Surgery
   in Great Britain & Ireland
   Sixth
   National Adult Cardiac
   Surgical Database Report
   2008
   Demonstrating quality
   Prepared by
   Ben Bridgewater PhD FRCS
   Bruce Keogh KBE DSc MD FRCS FRCP
   on behalf of the Society for Cardiothoracic Surgery
   in Great Britain & Ireland
   Robin Kinsman BSc PhD
   Peter Walton MA MB BChir MBA
   Dendrite Clinical Systems
   Cardiac Surgery
   CQC  website   Na)onal  audit  
   Aim:  <1  year  
   Reality:  3  years  
   

   View Slide

6. Monitoring  methodology  
   1.  Funnel  plot  
   – Fixed  )me  period  (e.g.  3  years)  
   – Iden)fy  ‘outlier’  units  
   – Doesn’t  address  whether  hospitals  are  gegng  
   worse  
   2.  Variable  life  adjusted  display  (VLAD)  plot  
   – Intui)ve  dynamic  summary  
   – Doesn’t  iden)fy  when  a  unit  is  an  outlier  
   

   View Slide

7. Funnel  plot    
   Number  of  cardiac  procedures  
   ●
   ●
   ● ●
   ● ● ●
   ● ●
   ● ● ●
   ● ●
   ● ● ●
   ●
   ● ● ● ●
   ●
   ● ● ●
   ●
   ●
   ●
   ● ●
   ●
   ● ● ●
   ●
   ●
   ● ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   mEuroSCORE (08/10)
   5000 1000 2000 3000 4000 5000
   Number of cardiac procedures
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ● ● ●
   ● ●
   ● ●
   ●
   ● ●
   ●
   ● ● ●
   ●
   ● ● ●
   ●
   ●
   ● ●
   ●
   ● ● ●
   ●
   ● ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ● ● ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ● ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ● ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   mEuroSCORE (04/07) mEuroSCORE (0
   0.00
   0.01
   0.02
   0.00
   0.01
   0.02
   0.03
   0.04
   0.05
   0.06
   1000 2000 3000 4000 5000 1000 2000 3000
   Number of cardiac procedures
   Mortality rate
   Risk-­‐adjusted  mortality  propor)on  
   All  elec)ve  &  urgent  cardiac  surgery  in  England  &  Wales  
   warrants  closer  
   inves)ga)on  
   Na)onal  average  
   ±2σ  
   ±3σ  
   

   View Slide

8. VLAD  plot  
   Variable Life-Adjusted Display plot for an individual surgeon
   Observed Predicted
   Predicted deaths - observed deaths
   Operation sequence
   3
   2
   1
   0
   -1
   -2
   -3
   0 100 200 300 400 500
   The intervention
   The bad run
   Maintaining patients’ trust: modern medical professionalism
   

   View Slide

9. Problems  to  overcome  
   1.  Systema)c  model  miscalibra)on  
   2.  Data  dissemina)on  
   3.  Pooled  vs.  separate  models  
   4.  Data  quality  
   5.  Gaming  
   6.  Subgroup  performance  
   7.  Ancillary  methodology  
   

   View Slide

10. Systema)c  miscalibra)on  
    What’s  wrong  with  this?  
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ● ●
    ● ●
    ● ● ●
    ● ●
    ● ● ●
    ●
    ● ● ● ●
    ●
    ● ● ●
    ●
    ●
    ●
    ● ●
    ●
    ● ● ●
    ●
    Logistic EuroSCORE
    /07) mEuroSCORE (08/10)
    All elective & urgent cardiac surgery in the UK
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ● ● ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    mEuroSCORE (04/07)
    0.00
    0.01
    0.02
    0.00
    0.01
    0.02
    0.03
    0.04
    0.05
    0.06
    1000 2000 3000 4000 5000 1000 2000
    Number of cardiac procedures
    Mortality rate
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ● ● ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    mEuroSCORE (04/07)
    0.00
    0.01
    0.02
    0.00
    0.01
    0.02
    0.03
    0.04
    0.05
    0.06
    1000 2000 3000 4000 5000 1000 2000
    Number of cardiac procedures
    Mortality rate
    All  elec)ve  &  urgent  cardiac  surgery  in  England  &  Wales  
    Number  of  cardiac  procedures  
    Risk-­‐adjusted  mortality  propor)on  
    

    View Slide

11. 2002 2004 2006 2008 2010
    0.02 0.04 0.06 0.08 0.10
    Time
    Mortality proportion
    Observed
    Expected
    Actual
    Overall average
    Trend
    •  Observed  mortality  is  
    decreasing  
    –  beqer  surgical  tools  
    –  improvements  in  post-­‐
    surgery  treatment  
    •  Predicted  mortality  is  
    increasing  
    –  increase  in  older  pa)ents  
    –  more  complex  procedures  
    •  Model  valida)on  
    essen)al!  
    Systema)c  miscalibra)on  
    Mortality    
    propor)on  
    

    View Slide

12. Dynamical  modeling  vs.  periodic  
    recalibra)on  vs.  doing  nothing  
    Unstable angina LV function: moderate LV function: poor
    Recent MI Pulmonary hypertension Emergency/salvage
    −0.2
    0.0
    0.2
    0.4
    0.6
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    1.0
    1.2
    1.4
    1.6
    0.6
    0.8
    0.6
    0.8
    1.2
    1.4
    nt
    Cardiac and operation−related factors
    Age (adjusted) Female Pulmonary disease
    Extracardiac arteriopathy Neurological dysfunction Previous cardiac surgery
    Serum creatinine > 200µmol/l Active endocarditis Critical pre−op
    0.05
    0.06
    0.07
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.0
    0.1
    0.2
    0.3
    0.4
    0.5
    0.4
    0.6
    0.8
    −0.5
    0.0
    0.5
    1.0
    0.6
    0.8
    1.0
    1.2
    1.2
    1.4
    0.8
    1.0
    1.2
    0.8
    1.0
    Coefficient
    Patient−related factors
    Extracardiac arteriopathy Neurological dysfunction Previous cardiac surgery
    Serum creatinine > 200µmol/l Active endocarditis Critical pre−op
    0.05
    0.06
    0.07
    0.1
    0.2
    0.3
    0.4
    0.5
    0.0
    0.1
    0.2
    0.3
    0.4
    0.4
    0.6
    0.8
    −0.5
    0.0
    0.5
    1.0
    0.6
    0.8
    1.0
    1.2
    0.6
    0.8
    1.0
    1.2
    1.4
    0.0
    0.2
    0.4
    0.6
    0.8
    1.0
    1.2
    0.2
    0.4
    0.6
    0.8
    1.0
    2002 2004 2006 2008 2010 2002 2004 2006 2008 2010 2002 2004 2006 2008 2010
    Time
    Coefficient
    Model: Model 1 Model 2 Model 3
    Model  coefficients  (log-­‐odds)  
    

    View Slide

13. Data  dissemina)on:  past  
    Abandoned  CQC  website   The  SCTS  ‘Blue  Book’  
    512  pages!  
    

    View Slide

14. Data  dissemina)on:  future  
    Mr Ben Bridgewater
    EuroSCORE series
    Date
    mEuroSCORE
    0.1
    0.2
    0.3
    0.4
    0.5
    2009 2010 2011
    Cumulative mortality
    Date
    Total number of deaths
    0
    2
    4
    6
    8
    2009 2010 2011
    VLAD (with date dispersion)
    Date
    Predicted − Observed
    2
    4
    6
    8
    2009 2010 2011
    Crude mortality funnel plot
    Number of cardiac procedures
    Mortality rate
    0.00
    0.05
    ●
    200 400 600 800
    Risk adjusted mortality funnel plot
    Number of cardiac procedures
    Mortality rate
    0.00
    0.05
    ●
    200 400 600 800
    Cummulative mEuroSCORE
    Cummulative Mortality
    VLAD
    5
    10
    15
    0
    2
    4
    6
    8
    10
    12
    14
    0
    2
    4
    6
    2008−07 2009−01 2009−07 2010−01 2010−07 2011−01
    Date
    Unit of Interest
    2386780
    2503756
    3166114
    3207776
    3226274
    3286898
    3451180
    3631845
    4002776
    4473204
    4486266
    4683551
    

    View Slide

15. Data  dissemina)on:  future  
    

    View Slide

16. Data  quality  
    Outlier  surgeon  ≠  rogue  surgeon  
    0 200 400 600 800 1000
    0 100 200 300 400
    Number of procedures
    Number of incomplete records
    •  Missing  data  
    •  Input  souware  
    errors  
    •  Registry  cleaning  
    errors  
    •  Imputa)on  
    •  Valida)on  
    

    View Slide

17. Pooled  vs.  separate  models  
    •  CABG  +  MVR  +  Tricuspid  repair  =  AVR?  
    •  Cardiac  surgery  is  a  ‘catch-­‐all’  term  
    •  We  could  have  risk  predic)on  models  for:  
    1.  all  procedures  (combina)ons)  
    2.  all  procedures  with  mul)ple  procedure  variables  
    3.  each  procedure  group  (e.g.  CABG,  Valve,  CABG  +  
    Valve,  …)  
    •  Decision  depends  on  applica)on.  
    

    View Slide

18. Gaming  (+  other  unexpected  
    extraneous  varia)on)  
    Rank
    10
    20
    30
    40 ●
    ●
    ●
    Hospital
    BAL. Barts and The London
    BAS. Basildon Hospital
    BHL. Liverpool Heart and Chest Hospital
    BRI. Bristol Royal Infirmary
    CHH. Castle Hill Hospital
    CHN. Nottingham City Hospital
    ERI. Royal Infirmary of Edinburgh
    FRE. Freeman Hospital
    GEO. St George's Hospital
    GJH. Golden Jubilee Hospital
    GRL. Glenfield Hospital
    HAM. Hammersmith Hospital
    HH. Harefield Hospital
    HHW. Wellington Hospital North
    HSC. Harley Street Clinic
    KCH. King's College Hospital
    LBH. London Bridge Hospital
    LGI. Leeds General Infirmary
    MOR. Morriston Hospital
    MRI. Manchester Royal Infirmary
    NCR. New Cross Hospital
    NGS. Northern General Hospital
    NHB. Royal Brompton Hospital
    PAP. Papworth Hospital
    PLY. Derriford Hospital
    QEB. Queen Elizabeth Hospital
    RAD. John Radcliffe Hospital
    RIA. Aberdeen Royal Infirmary
    RSC. Royal Sussex County Hospital
    RVB. Royal Victoria Hospital
    SCM. James Cook University Hospital
    SGH. Southampton General Hospital
    STH. St Thomas Hospital
    STM. St Marys Hospital Paddington
    STO. University Hospital of North Staffordshire
    UCL. University College Hospital
    UHW. University Hospital of Wales
    VIC. Victoria Hospital
    WAL. University Hospital Coventry
    WYT. Wythenshawe Hospital
    Hospitals  
    Distribu)on  of  
    ranks  of  risk  
    factor  
    prevalence  
    might  be  
    expected  to  
    homogenous  
    across  hospitals  
     
     
     
    Further  
    inves)ga)on  
    required  
    ?   ?  
    

    View Slide

19. Subgroup  performance    
    •  Stra)fica)on  does  not  ensure  good  model  
    performance  
    

    View Slide

20. Ancillary  methodology  
    •  Mul)ple  tes)ng  
    – correc)on  adjustments  (e.g.  Bonferroni)  
    •  Overdispersion  
    – mul)plica)ve  variance  infla)on    
    – random  effects  models  
    

    View Slide