Risk stratification: the UK cardiothoracic experience

Presented at the PROMS 2.0 workshop, Manchester, UK

1. Risk  stra)fica)on:  The  UK   cardiothoracic  experience   Graeme  L  Hickey1;  Stuart  W  Grant2;  Iain  Buchan1;   Ben  Bridgewater1,2     1Northwest  Ins.tute  of  BioHealth  Informa.cs,  Manchester  University   2Department  of  Cardiothoracic  Surgery,  University  Hospital  of  South  Manchester  
2. Background   •  Around  35,000  adult  cardiac  surgery   procedures  performed  each  year  in  UK   •  In-­‐hospital  mortality  rate  in  2010-­‐11  was  3.4%    
3. What’s  risk  stra)fica)on  used  for?   The Society for Cardiothoracic Surgery in Great Britain & Ireland Sixth National Adult Cardiac Surgical Database Report 2008 Demonstrating quality Prepared by Ben Bridgewater PhD FRCS Bruce Keogh KBE DSc MD FRCS FRCP on behalf of the Society for Cardiothoracic Surgery in Great Britain & Ireland Robin Kinsman BSc PhD Peter Walton MA MB BChir MBA Dendrite Clinical Systems Cardiac Surgery Governance   Decision-­‐making  
4. Mo)va)on   •  Total  cost  =  £1.48m/year  in  England  (<1%  of   the  total  NHS  spend  on  adult  cardiac  surgery)*   •  Associated  with  a  50%  reduc)on  in  risk   adjusted  mortality*   *Maintaining  Pa.ents’  Trust,  SCTS,  Henley-­‐on-­‐Thames:   Dendrite  Clinical  Systems  Ltd,  2011  
5. Infrastructure   Cardiac  surgery   Input  data  locally   Uploaded  periodically   to  central  database   Aim:  3  months   Reality:  1  year   Sta)s)cian  +  clinicians   The Society for Cardiothoracic Surgery in Great Britain & Ireland Sixth National Adult Cardiac Surgical Database Report 2008 Demonstrating quality Prepared by Ben Bridgewater PhD FRCS Bruce Keogh KBE DSc MD FRCS FRCP on behalf of the Society for Cardiothoracic Surgery in Great Britain & Ireland Robin Kinsman BSc PhD Peter Walton MA MB BChir MBA Dendrite Clinical Systems Cardiac Surgery CQC  website   Na)onal  audit   Aim:  <1  year   Reality:  3  years  
6. Monitoring  methodology   1.  Funnel  plot   – Fixed  )me  period  (e.g.  3  years)   – Iden)fy  ‘outlier’  units   – Doesn’t  address  whether  hospitals  are  gegng   worse   2.  Variable  life  adjusted  display  (VLAD)  plot   – Intui)ve  dynamic  summary   – Doesn’t  iden)fy  when  a  unit  is  an  outlier  
7. Funnel  plot     Number  of  cardiac  procedures   • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (08/10) 5000 1000 2000 3000 4000 5000 Number of cardiac procedures • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) mEuroSCORE (0 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 3000 Number of cardiac procedures Mortality rate Risk-­‐adjusted  mortality  propor)on   All  elec)ve  &  urgent  cardiac  surgery  in  England  &  Wales   warrants  closer   inves)ga)on   Na)onal  average   ±2σ   ±3σ  
8. VLAD  plot   Variable Life-Adjusted Display plot for an individual surgeon Observed Predicted Predicted deaths - observed deaths Operation sequence 3 2 1 0 -1 -2 -3 0 100 200 300 400 500 The intervention The bad run Maintaining patients’ trust: modern medical professionalism
9. Problems  to  overcome   1.  Systema)c  model  miscalibra)on   2.  Data  dissemina)on   3.  Pooled  vs.  separate  models   4.  Data  quality   5.  Gaming   6.  Subgroup  performance   7.  Ancillary  methodology  
10. Systema)c  miscalibra)on   What’s  wrong  with  this?   • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Logistic EuroSCORE /07) mEuroSCORE (08/10) All elective & urgent cardiac surgery in the UK • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 Number of cardiac procedures Mortality rate • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 Number of cardiac procedures Mortality rate All  elec)ve  &  urgent  cardiac  surgery  in  England  &  Wales   Number  of  cardiac  procedures   Risk-­‐adjusted  mortality  propor)on  
11. 2002 2004 2006 2008 2010 0.02 0.04 0.06 0.08 0.10 Time Mortality proportion Observed Expected Actual Overall average Trend •  Observed  mortality  is   decreasing   –  beqer  surgical  tools   –  improvements  in  post-­‐ surgery  treatment   •  Predicted  mortality  is   increasing   –  increase  in  older  pa)ents   –  more  complex  procedures   •  Model  valida)on   essen)al!   Systema)c  miscalibra)on   Mortality     propor)on  
12. Dynamical  modeling  vs.  periodic   recalibra)on  vs.  doing  nothing   Unstable angina LV function: moderate LV function: poor Recent MI Pulmonary hypertension Emergency/salvage −0.2 0.0 0.2 0.4 0.6 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1.0 1.2 1.4 1.6 0.6 0.8 0.6 0.8 1.2 1.4 nt Cardiac and operation−related factors Age (adjusted) Female Pulmonary disease Extracardiac arteriopathy Neurological dysfunction Previous cardiac surgery Serum creatinine > 200µmol/l Active endocarditis Critical pre−op 0.05 0.06 0.07 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.4 0.6 0.8 −0.5 0.0 0.5 1.0 0.6 0.8 1.0 1.2 1.2 1.4 0.8 1.0 1.2 0.8 1.0 Coefficient Patient−related factors Extracardiac arteriopathy Neurological dysfunction Previous cardiac surgery Serum creatinine > 200µmol/l Active endocarditis Critical pre−op 0.05 0.06 0.07 0.1 0.2 0.3 0.4 0.5 0.0 0.1 0.2 0.3 0.4 0.4 0.6 0.8 −0.5 0.0 0.5 1.0 0.6 0.8 1.0 1.2 0.6 0.8 1.0 1.2 1.4 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.2 0.4 0.6 0.8 1.0 2002 2004 2006 2008 2010 2002 2004 2006 2008 2010 2002 2004 2006 2008 2010 Time Coefficient Model: Model 1 Model 2 Model 3 Model  coefficients  (log-­‐odds)  
13. Data  dissemina)on:  past   Abandoned  CQC  website   The  SCTS  ‘Blue  Book’   512  pages!  
14. Data  dissemina)on:  future   Mr Ben Bridgewater EuroSCORE series Date mEuroSCORE 0.1 0.2 0.3 0.4 0.5 2009 2010 2011 Cumulative mortality Date Total number of deaths 0 2 4 6 8 2009 2010 2011 VLAD (with date dispersion) Date Predicted − Observed 2 4 6 8 2009 2010 2011 Crude mortality funnel plot Number of cardiac procedures Mortality rate 0.00 0.05 • 200 400 600 800 Risk adjusted mortality funnel plot Number of cardiac procedures Mortality rate 0.00 0.05 • 200 400 600 800 Cummulative mEuroSCORE Cummulative Mortality VLAD 5 10 15 0 2 4 6 8 10 12 14 0 2 4 6 2008−07 2009−01 2009−07 2010−01 2010−07 2011−01 Date Unit of Interest 2386780 2503756 3166114 3207776 3226274 3286898 3451180 3631845 4002776 4473204 4486266 4683551
15. Data  dissemina)on:  future  
16. Data  quality   Outlier  surgeon  ≠  rogue  surgeon   0 200 400 600 800 1000 0 100 200 300 400 Number of procedures Number of incomplete records •  Missing  data   •  Input  souware   errors   •  Registry  cleaning   errors   •  Imputa)on   •  Valida)on  
17. Pooled  vs.  separate  models   •  CABG  +  MVR  +  Tricuspid  repair  =  AVR?   •  Cardiac  surgery  is  a  ‘catch-­‐all’  term   •  We  could  have  risk  predic)on  models  for:   1.  all  procedures  (combina)ons)   2.  all  procedures  with  mul)ple  procedure  variables   3.  each  procedure  group  (e.g.  CABG,  Valve,  CABG  +   Valve,  …)   •  Decision  depends  on  applica)on.  
18. Gaming  (+  other  unexpected   extraneous  varia)on)   Rank 10 20 30 40 • • • Hospital BAL. Barts and The London BAS. Basildon Hospital BHL. Liverpool Heart and Chest Hospital BRI. Bristol Royal Infirmary CHH. Castle Hill Hospital CHN. Nottingham City Hospital ERI. Royal Infirmary of Edinburgh FRE. Freeman Hospital GEO. St George's Hospital GJH. Golden Jubilee Hospital GRL. Glenfield Hospital HAM. Hammersmith Hospital HH. Harefield Hospital HHW. Wellington Hospital North HSC. Harley Street Clinic KCH. King's College Hospital LBH. London Bridge Hospital LGI. Leeds General Infirmary MOR. Morriston Hospital MRI. Manchester Royal Infirmary NCR. New Cross Hospital NGS. Northern General Hospital NHB. Royal Brompton Hospital PAP. Papworth Hospital PLY. Derriford Hospital QEB. Queen Elizabeth Hospital RAD. John Radcliffe Hospital RIA. Aberdeen Royal Infirmary RSC. Royal Sussex County Hospital RVB. Royal Victoria Hospital SCM. James Cook University Hospital SGH. Southampton General Hospital STH. St Thomas Hospital STM. St Marys Hospital Paddington STO. University Hospital of North Staffordshire UCL. University College Hospital UHW. University Hospital of Wales VIC. Victoria Hospital WAL. University Hospital Coventry WYT. Wythenshawe Hospital Hospitals   Distribu)on  of   ranks  of  risk   factor   prevalence   might  be   expected  to   homogenous   across  hospitals         Further   inves)ga)on   required   ?   ?  
19. Subgroup  performance     •  Stra)fica)on  does  not  ensure  good  model   performance  
20. Ancillary  methodology   •  Mul)ple  tes)ng   – correc)on  adjustments  (e.g.  Bonferroni)   •  Overdispersion   – mul)plica)ve  variance  infla)on     – random  effects  models