Monitoring the performance of cardiac surgeons

Presented at the Tipping Points Workshop, Durham University, UK (September 2012)

Transcript

1. Monitoring the performance of
   cardiac surgeons
   Graeme L Hickey1; Stuart W Grant2; Camila Caiado3;
   Iain Buchan1; Ben Bridgewater1,2
   1Northwest Institute of BioHealth Informatics, Manchester University
   2Department of Cardiothoracic Surgery, University Hospital of South Manchester
   3Department of Mathematical Sciences, Durham University
   

   View Slide

2. Background
   • Around 35,000 adult cardiac surgery
   procedures performed each year in UK
   • Mortality rate in 2010-11 was 3.4%
   • Monitoring primarily focuses on in-hospital
   mortality
   

   View Slide

3. Cardiac surgery performance in the
   spotlight clinical trials of hypericum extract for different pharmaceutical
   companies. AD and MK are employees of Dr Willmar Schwabe
   Pharmaceuticals.
   Ethical approval: The protocol was approved by the participat-
   ing centres’ appropriate independent ethics committees.
   1 Linde K, Mulrow CD. St John’s wort for depression. Cochrane Database Syst
   Rev 2004;(4):CD000448.
   2 Harrer G, Hübner WD, Podzuweit H. Effectiveness and tolerance of the
   hypericum extract LI 160 compared to maprotiline: a multicenter
   double-blind study. J Geriatric Psychiatry Neurol 1994;7(suppl 1):S24-8.
   3 Philipp M, Kohnen R, Hiller KO. Hypericum extract versus imipramine
   or placebo in patients with moderate depression: randomised multicen-
   tre study of treatment for eight weeks. BMJ 1999;319:1534-8.
   4 Vorbach EU, Hübner WD, Arnoldt KH. Effectiveness and tolerance of the
   hypericum extract LI 160 in comparison with imipramine: randomized
   double-blind study with 135 outpatients. J Geriatric Psychiatry Neurol
   1994;7(suppl 1):S19-23.
   5 Wheatley D. LI 160, an extract of St. John’s wort, versus amitriptyline in
   mildly to moderately depressed outpatients—a controlled 6-week clinical
   trial. Pharmacopsychiatry 1997;30(suppl 2):77-80.
   6 Harrer G, Schmidt U, Kuhn U, Biller A. Äquivalenzvergleich
   Johanniskrautextrakt LoHyp-57 versus Fluoxetin. Arzneimittel-Forschung
   1998;49:3-10.
   7 Izzo AA. Drug interactions with St. John’s Wort (Hypericum perforatum):
   a review of the clinical evidence. Int J Clin Pharmacol Ther 2004;42:139-48.
   8 Montgomery SA. Clinically relevant effect sizes in depression. Eur
   Neuropsychopharmacology 1994;4:283-4.
   9 Committee for Proprietary Medicinal Products. Points to consider on
   switching between superiority and non-inferiority. London: European Agency
   for the Evaluation of Medicinal Products, 2000.
   10 Paykel ES. The classification of depression. Br J Clin Pharmacol
   1983;15(suppl 2):155-9S.
   11 Hypericum Depression Trial Study Group. Effect of Hypericum perfora-
   tum (St. John’s wort) in major depressive disorder. JAMA 2002;287:
   1807-14.
   12 Dunner DL, Dunbar GC. Optimal dose regimen for paroxetine. J Clin
   Psychiatry 1992;53(suppl):21-6.
   13 Bourin M, Chue P, Guillon Y. Paroxetine: a review. CNS Drug Rev
   2001;7:25-47.
   (Accepted 17 December 2004)
   doi 10.1136/bmj.38356.655266.82
   Mortality data in adult cardiac surgery for named
   surgeons: retrospective examination of prospectively
   collected data on coronary artery surgery and aortic valve
   replacement
   Ben Bridgewater on behalf of the adult cardiac surgeons of north west England
   Abstract
   Objectives To present named surgeon mortality for
   isolated first time coronary artery surgery and aortic
   valve surgery.
   Design Retrospective analysis of prospectively
   collected data.
   Setting All NHS hospitals undertaking adult cardiac
   surgery in north west England.
   Participants 25 consultant surgeons carrying out
   coronary artery surgery and aortic valve replacement
   between April 2001 and March 2004.
   Main outcome measures Mortality for both
   operations according to surgeon. EuroSCORE to
   stratify patients into low and high risk.
   Results 10 163 patients underwent surgery under 25
   surgeons. The average number of patients per
   surgeon was 363 for coronary artery surgery and 44
   for aortic valve replacement. Seventeen per cent of
   the patients undergoing coronary artery surgery and
   half of those undergoing aortic valve surgery were
   considered high risk. The average mortality was 1.8%
   Introduction
   Recent years have seen a move towards increased
   openness and transparency in healthcare delivery. This
   has been accelerated by a series of events, including the
   Bristol Royal Infirmary inquiry into paediatric cardiac
   surgery deaths.1 One recommendation of the inquiry
   was that patients must be able to see information about
   the relative performance of individual consultants
   operating within hospitals. The Society of Cardiotho-
   racic Surgeons of Great Britain and Ireland therefore
   published a study in 2004 of activity and performance
   of all consultants undertaking adult cardiac surgery in
   the United Kingdom.2 The history leading to this
   analysis and the underlying methods have been
   comprehensively described.3 The study was conducted
   on a single operation: first time isolated coronary
   artery surgery. Because of a lack of comprehensive data
   on which to perform a complete analysis that would
   allow adjustments to be made for differing case mix,
   the benchmarking was done on “crude” non-adjusted
   mortality data. The exact mortality for individual
   Papers
   South Manchester
   University Hospital,
   Manchester
   M23 9LT
   Ben Bridgewater
   consultant surgeon
   Correspondence to:
   B Bridgewater
   ben.bridgewater@
   smuht.nwest.nhs.uk
   BMJ 2005;330:506–10
   BMJ 2005; 330 doi: 10.1136/bmj.330.7490.506 (Published 3 March 2005)
   Cite this as: BMJ 2005;330:506
   

   View Slide

4. Today’s monitoring programme
   • National and local-level audits
   • Revalidation
   • Dr. Foster Health
   • FOI requests
   The Society for
   Cardiothoracic Surgery
   in Great Britain & Ireland
   Sixth
   National Adult Cardiac
   Surgical Database Report
   2008
   Demonstrating quality
   Prepared by
   Ben Bridgewater PhD FRCS
   Bruce Keogh KBE DSc MD FRCS FRCP
   on behalf of the Society for Cardiothoracic Surgery
   in Great Britain & Ireland
   Robin Kinsman BSc PhD
   Peter Walton MA MB BChir MBA
   Dendrite Clinical Systems
   Cardiac Surgery
   

   View Slide

5. The framework
   Cardiac surgery Input data locally Uploaded periodically
   to central database
   Aim: 3 months
   Reality: 1 year
   Statistician + clinicians
   The Society for
   Cardiothoracic Surgery
   in Great Britain & Ireland
   Sixth
   National Adult Cardiac
   Surgical Database Report
   2008
   Demonstrating quality
   Prepared by
   Ben Bridgewater PhD FRCS
   Bruce Keogh KBE DSc MD FRCS FRCP
   on behalf of the Society for Cardiothoracic Surgery
   in Great Britain & Ireland
   Robin Kinsman BSc PhD
   Peter Walton MA MB BChir MBA
   Dendrite Clinical Systems
   Cardiac Surgery
   CQC website National audit
   Aim: <1 year
   Reality: 3 years
   

   View Slide

6. Risk adjustment
   • Not all surgeons do the same caseload
   • NHS surgeon
   • Does emergency surgery
   • Specialism in mitral valve
   repairs
   • Does a lot of private
   surgery
   • Mostly routine
   elective isolated
   bypass
   Crude mortality = 6.1% Crude mortality = 1.4%
   Risk-adjusted mortality = 3.1% Risk-adjusted mortality = 2.9%
   Need to risk adjust!
   

   View Slide

7. Modelling
   • Model for 1999-2012: EuroSCORE
   – Logistic regression
   – estimated using prospectively collected during
   1995
   – In-hospital mortality (binary) outcome
   – Adjustment for 18 risk factors
   • Replaced in 2012 by a contemporary model:
   EuroSCORE II
   

   View Slide

8. Monitoring methodology
   • Funnel plot
   – Fixed summary over 3 years of data
   – Helps identify ‘outlier’ units
   – Doesn’t address whether hospitals are getting
   worse
   • Variable life adjusted display (VLAD) plot
   – Intuitive dynamic summary
   – Doesn’t identify when a unit is an outlier
   

   View Slide

9. Funnel plot
   Number of cardiac procedures
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ● ● ●
   ● ●
   ● ● ●
   ● ●
   ● ● ●
   ●
   ● ● ● ●
   ●
   ● ● ●
   ●
   ●
   ●
   ● ●
   ●
   ● ● ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   4/07) mEuroSCORE (08/10)
   4000 5000 1000 2000 3000 4000 5000
   Number of cardiac procedures
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ●
   ● ●
   ●
   ●
   ● ●
   ●
   ●
   ●
   ● ● ●
   ● ●
   ●
   ●
   ●
   ●
   ●
   ● ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ● ● ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   ●
   mEuroSCORE (04/07) m
   0.00
   0.01
   0.02
   0.00
   0.01
   0.02
   0.03
   0.04
   0.05
   0.06
   1000 2000 3000 4000 5000 1000 2000
   Number of cardiac procedures
   Mortality rate
   Risk-adjusted mortality proportion
   All elective & urgent cardiac surgery in England & Wales
   warrants closer
   investigation
   

   View Slide

10. VLAD plot
    Variable Life-Adjusted Display plot for an individual surgeon
    Observed Predicted
    Predicted deaths - observed deaths
    Operation sequence
    3
    2
    1
    0
    -1
    -2
    -3
    0 100 200 300 400 500
    The intervention
    The bad run
    Maintaining patients’ trust: modern medical professionalism
    

    View Slide

11. Outlier ≠ poor performance
    • An ‘outlier’ surgeon or hospital (good or bad)
    might be explainable
    0 200 400 600 800 1000
    0 100 200 300 400
    Number of procedures
    Number of incomplete records
    • Missing data
    • Input software
    errors
    • Registry cleaning
    errors
    

    View Slide

12. Question: What’s wrong with this?
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ● ●
    ● ●
    ● ● ●
    ● ●
    ● ● ●
    ●
    ● ● ● ●
    ●
    ● ● ●
    ●
    ●
    ●
    ● ●
    ●
    ● ● ●
    ●
    ●
    Logistic EuroSCORE
    /07) mEuroSCORE (08/10)
    All elective & urgent cardiac surgery in the UK
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ● ● ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    mEuroSCORE (04/07)
    0.00
    0.01
    0.02
    0.00
    0.01
    0.02
    0.03
    0.04
    0.05
    0.06
    1000 2000 3000 4000 5000 1000 2000
    Number of cardiac procedures
    Mortality rate
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ●
    ● ●
    ●
    ●
    ● ●
    ●
    ●
    ●
    ● ● ●
    ● ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ● ● ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    ●
    mEuroSCORE (04/07)
    0.00
    0.01
    0.02
    0.00
    0.01
    0.02
    0.03
    0.04
    0.05
    0.06
    1000 2000 3000 4000 5000 1000 2000
    Number of cardiac procedures
    Mortality rate
    All elective & urgent cardiac surgery in England & Wales
    Number of cardiac procedures
    Risk-adjusted mortality proportion
    

    View Slide

13. Answer: Model over predicts mortality
    Predicted mortality
    2.7x greater than
    observed mortality!
    

    View Slide

14. 2002 2004 2006 2008 2010
    0.02 0.04 0.06 0.08 0.10
    Time
    Mortality proportion
    Observed
    Expected
    Actual
    Overall average
    Trend
    • Observed mortality is
    decreasing
    – better surgical tools
    – improvements in post-
    surgery treatment
    • Predicted mortality is
    increasing
    – increase in older
    patients
    – more complex
    procedures
    Dynamics of cardiac surgery
    Mortality
    proportion
    

    View Slide

15. Dynamics of risk
    North-South divide?
    

    View Slide

16. Patching the problem: short-term
    • Update the standard model
    – recalibration
    – model re-fit
    – shrinkage
    • Adopt new model
    – requires validation
    • Subgroup analyses
    – should some procedures be excluded from
    monitoring?
    – a statistical + political grey-zone
    

    View Slide

17. Patching the problem: long-term
    • Better modelling approaches
    • Online ‘live’ audit reporting + database
    querying
    

    View Slide

18. Dynamical modelling
    • Problem: need a model that stays
    contemporary
    • Some possible solutions:
    – do nothing
    – develop a new model every x-years
    – a moving window recalibration or re-fit
    – dynamic generalized linear modelling (DGLM)
    

    View Slide

19. DGLM
    Extracardiac arteriopathy Neurological dysfunction Previous surgery
    Creatinine > 200mumol/l Active endocarditis Critical pre−op
    0.050
    0.055
    0.060
    0.065
    0.2
    0.3
    0.4
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    −0.4
    −0.2
    0.0
    0.2
    0.4
    0.6
    0.7
    0.8
    0.9
    1.0
    1.1
    1.2
    0.7
    0.8
    0.9
    1.0
    1.1
    1.2
    1.3
    0.0
    0.2
    0.4
    0.6
    0.8
    0.4
    0.6
    0.8
    1.0
    2004 2006 2008 2010 2004 2006 2008 2010 2004 2006 2008 2010
    Time (months)
    Coefficient
    Model Model 1: Recalibration (single) Model 2: Piecewise recalibration Model 3: Window Model 4: DGLM
    Age (adjusted) Female Pulmonary disease
    Extracardiac arteriopathy Neurological dysfunction Previous surgery
    Creatinine > 200mumol/l Active endocarditis Critical pre−op
    0.050
    0.055
    0.060
    0.065
    0.070
    0.075
    0.2
    0.3
    0.4
    0.5
    0.1
    0.2
    0.3
    0.4
    0.4
    0.5
    0.6
    0.7
    0.8
    −0.4
    −0.2
    0.0
    0.2
    0.4
    0.6
    0.7
    0.8
    0.9
    1.0
    1.1
    1.2
    1.0
    1.1
    1.2
    1.3
    0.4
    0.6
    0.8
    0.6
    0.8
    1.0
    Coefficient
    Patient−related factors
    Model coefficients (log-odds)
    Re-fit Piecewise re-fit
    

    View Slide

20. DGLM
    • The largest change is
    coming from intercept
    • Latent variables
    affecting risk?
    • Doing nothing is not
    an option
    • Acting periodically
    might be to late
    −5.8
    −5.6
    −5.4
    −5.2
    2004 2006 2008 2010
    Time (months)
    Coefficient
    Model
    Model 1: Recalibration (single)
    Model 2: Piecewise recalibration
    Model 3: Window
    Model 4: DGLM
    Intercept
    Intercept
    Re-fit
    Piecewise re-fit
    Coefficient
    

    View Slide

21. Online tools
    National
    Audits
    Monitoring
    Tools
    Database
    Querying
    e-lab
    

    View Slide

22. Summary
    • Current infrastructure incompatible with
    developments in healthcare monitoring &
    surveillance
    • Real-world data is not perfect which increases
    uncertainty + potentially a source of bias
    • Modelling for contemporary cardiac surgery
    needs to adapt to changing dynamics
    

    View Slide

23. This research was partly funded by Heart Research UK
    (Grant number RG2583)
    Any questions?
    

    View Slide