Monitoring the performance of cardiac surgeons

Presented at the Tipping Points Workshop, Durham University, UK (September 2012)

Transcript

1. Monitoring the performance of cardiac surgeons Graeme L Hickey1; Stuart W Grant2; Camila Caiado3; Iain Buchan1; Ben Bridgewater1,2 1Northwest Institute of BioHealth Informatics, Manchester University 2Department of Cardiothoracic Surgery, University Hospital of South Manchester 3Department of Mathematical Sciences, Durham University
2. Background • Around 35,000 adult cardiac surgery procedures performed each year in UK • Mortality rate in 2010-11 was 3.4% • Monitoring primarily focuses on in-hospital mortality
3. Cardiac surgery performance in the spotlight clinical trials of hypericum extract for different pharmaceutical companies. AD and MK are employees of Dr Willmar Schwabe Pharmaceuticals. Ethical approval: The protocol was approved by the participat- ing centres’ appropriate independent ethics committees. 1 Linde K, Mulrow CD. St John’s wort for depression. Cochrane Database Syst Rev 2004;(4):CD000448. 2 Harrer G, Hübner WD, Podzuweit H. Effectiveness and tolerance of the hypericum extract LI 160 compared to maprotiline: a multicenter double-blind study. J Geriatric Psychiatry Neurol 1994;7(suppl 1):S24-8. 3 Philipp M, Kohnen R, Hiller KO. Hypericum extract versus imipramine or placebo in patients with moderate depression: randomised multicen- tre study of treatment for eight weeks. BMJ 1999;319:1534-8. 4 Vorbach EU, Hübner WD, Arnoldt KH. Effectiveness and tolerance of the hypericum extract LI 160 in comparison with imipramine: randomized double-blind study with 135 outpatients. J Geriatric Psychiatry Neurol 1994;7(suppl 1):S19-23. 5 Wheatley D. LI 160, an extract of St. John’s wort, versus amitriptyline in mildly to moderately depressed outpatients—a controlled 6-week clinical trial. Pharmacopsychiatry 1997;30(suppl 2):77-80. 6 Harrer G, Schmidt U, Kuhn U, Biller A. Äquivalenzvergleich Johanniskrautextrakt LoHyp-57 versus Fluoxetin. Arzneimittel-Forschung 1998;49:3-10. 7 Izzo AA. Drug interactions with St. John’s Wort (Hypericum perforatum): a review of the clinical evidence. Int J Clin Pharmacol Ther 2004;42:139-48. 8 Montgomery SA. Clinically relevant effect sizes in depression. Eur Neuropsychopharmacology 1994;4:283-4. 9 Committee for Proprietary Medicinal Products. Points to consider on switching between superiority and non-inferiority. London: European Agency for the Evaluation of Medicinal Products, 2000. 10 Paykel ES. The classification of depression. Br J Clin Pharmacol 1983;15(suppl 2):155-9S. 11 Hypericum Depression Trial Study Group. Effect of Hypericum perfora- tum (St. John’s wort) in major depressive disorder. JAMA 2002;287: 1807-14. 12 Dunner DL, Dunbar GC. Optimal dose regimen for paroxetine. J Clin Psychiatry 1992;53(suppl):21-6. 13 Bourin M, Chue P, Guillon Y. Paroxetine: a review. CNS Drug Rev 2001;7:25-47. (Accepted 17 December 2004) doi 10.1136/bmj.38356.655266.82 Mortality data in adult cardiac surgery for named surgeons: retrospective examination of prospectively collected data on coronary artery surgery and aortic valve replacement Ben Bridgewater on behalf of the adult cardiac surgeons of north west England Abstract Objectives To present named surgeon mortality for isolated first time coronary artery surgery and aortic valve surgery. Design Retrospective analysis of prospectively collected data. Setting All NHS hospitals undertaking adult cardiac surgery in north west England. Participants 25 consultant surgeons carrying out coronary artery surgery and aortic valve replacement between April 2001 and March 2004. Main outcome measures Mortality for both operations according to surgeon. EuroSCORE to stratify patients into low and high risk. Results 10 163 patients underwent surgery under 25 surgeons. The average number of patients per surgeon was 363 for coronary artery surgery and 44 for aortic valve replacement. Seventeen per cent of the patients undergoing coronary artery surgery and half of those undergoing aortic valve surgery were considered high risk. The average mortality was 1.8% Introduction Recent years have seen a move towards increased openness and transparency in healthcare delivery. This has been accelerated by a series of events, including the Bristol Royal Infirmary inquiry into paediatric cardiac surgery deaths.1 One recommendation of the inquiry was that patients must be able to see information about the relative performance of individual consultants operating within hospitals. The Society of Cardiotho- racic Surgeons of Great Britain and Ireland therefore published a study in 2004 of activity and performance of all consultants undertaking adult cardiac surgery in the United Kingdom.2 The history leading to this analysis and the underlying methods have been comprehensively described.3 The study was conducted on a single operation: first time isolated coronary artery surgery. Because of a lack of comprehensive data on which to perform a complete analysis that would allow adjustments to be made for differing case mix, the benchmarking was done on “crude” non-adjusted mortality data. The exact mortality for individual Papers South Manchester University Hospital, Manchester M23 9LT Ben Bridgewater consultant surgeon Correspondence to: B Bridgewater ben.bridgewater@ smuht.nwest.nhs.uk BMJ 2005;330:506–10 BMJ 2005; 330 doi: 10.1136/bmj.330.7490.506 (Published 3 March 2005) Cite this as: BMJ 2005;330:506
4. Today’s monitoring programme • National and local-level audits • Revalidation • Dr. Foster Health • FOI requests The Society for Cardiothoracic Surgery in Great Britain & Ireland Sixth National Adult Cardiac Surgical Database Report 2008 Demonstrating quality Prepared by Ben Bridgewater PhD FRCS Bruce Keogh KBE DSc MD FRCS FRCP on behalf of the Society for Cardiothoracic Surgery in Great Britain & Ireland Robin Kinsman BSc PhD Peter Walton MA MB BChir MBA Dendrite Clinical Systems Cardiac Surgery
5. The framework Cardiac surgery Input data locally Uploaded periodically to central database Aim: 3 months Reality: 1 year Statistician + clinicians The Society for Cardiothoracic Surgery in Great Britain & Ireland Sixth National Adult Cardiac Surgical Database Report 2008 Demonstrating quality Prepared by Ben Bridgewater PhD FRCS Bruce Keogh KBE DSc MD FRCS FRCP on behalf of the Society for Cardiothoracic Surgery in Great Britain & Ireland Robin Kinsman BSc PhD Peter Walton MA MB BChir MBA Dendrite Clinical Systems Cardiac Surgery CQC website National audit Aim: <1 year Reality: 3 years
6. Risk adjustment • Not all surgeons do the same caseload • NHS surgeon • Does emergency surgery • Specialism in mitral valve repairs • Does a lot of private surgery • Mostly routine elective isolated bypass Crude mortality = 6.1% Crude mortality = 1.4% Risk-adjusted mortality = 3.1% Risk-adjusted mortality = 2.9% Need to risk adjust!
7. Modelling • Model for 1999-2012: EuroSCORE – Logistic regression – estimated using prospectively collected during 1995 – In-hospital mortality (binary) outcome – Adjustment for 18 risk factors • Replaced in 2012 by a contemporary model: EuroSCORE II
8. Monitoring methodology • Funnel plot – Fixed summary over 3 years of data – Helps identify ‘outlier’ units – Doesn’t address whether hospitals are getting worse • Variable life adjusted display (VLAD) plot – Intuitive dynamic summary – Doesn’t identify when a unit is an outlier
9. Funnel plot Number of cardiac procedures • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 4/07) mEuroSCORE (08/10) 4000 5000 1000 2000 3000 4000 5000 Number of cardiac procedures • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) m 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 Number of cardiac procedures Mortality rate Risk-adjusted mortality proportion All elective & urgent cardiac surgery in England & Wales warrants closer investigation
10. VLAD plot Variable Life-Adjusted Display plot for an individual surgeon Observed Predicted Predicted deaths - observed deaths Operation sequence 3 2 1 0 -1 -2 -3 0 100 200 300 400 500 The intervention The bad run Maintaining patients’ trust: modern medical professionalism
11. Outlier ≠ poor performance • An ‘outlier’ surgeon or hospital (good or bad) might be explainable 0 200 400 600 800 1000 0 100 200 300 400 Number of procedures Number of incomplete records • Missing data • Input software errors • Registry cleaning errors
12. Question: What’s wrong with this? • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Logistic EuroSCORE /07) mEuroSCORE (08/10) All elective & urgent cardiac surgery in the UK • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 Number of cardiac procedures Mortality rate • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • mEuroSCORE (04/07) 0.00 0.01 0.02 0.00 0.01 0.02 0.03 0.04 0.05 0.06 1000 2000 3000 4000 5000 1000 2000 Number of cardiac procedures Mortality rate All elective & urgent cardiac surgery in England & Wales Number of cardiac procedures Risk-adjusted mortality proportion
13. Answer: Model over predicts mortality Predicted mortality 2.7x greater than observed mortality!
14. 2002 2004 2006 2008 2010 0.02 0.04 0.06 0.08 0.10 Time Mortality proportion Observed Expected Actual Overall average Trend • Observed mortality is decreasing – better surgical tools – improvements in post- surgery treatment • Predicted mortality is increasing – increase in older patients – more complex procedures Dynamics of cardiac surgery Mortality proportion
15. Dynamics of risk North-South divide?
16. Patching the problem: short-term • Update the standard model – recalibration – model re-fit – shrinkage • Adopt new model – requires validation • Subgroup analyses – should some procedures be excluded from monitoring? – a statistical + political grey-zone
17. Patching the problem: long-term • Better modelling approaches • Online ‘live’ audit reporting + database querying
18. Dynamical modelling • Problem: need a model that stays contemporary • Some possible solutions: – do nothing – develop a new model every x-years – a moving window recalibration or re-fit – dynamic generalized linear modelling (DGLM)
19. DGLM Extracardiac arteriopathy Neurological dysfunction Previous surgery Creatinine > 200mumol/l Active endocarditis Critical pre−op 0.050 0.055 0.060 0.065 0.2 0.3 0.4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 −0.4 −0.2 0.0 0.2 0.4 0.6 0.7 0.8 0.9 1.0 1.1 1.2 0.7 0.8 0.9 1.0 1.1 1.2 1.3 0.0 0.2 0.4 0.6 0.8 0.4 0.6 0.8 1.0 2004 2006 2008 2010 2004 2006 2008 2010 2004 2006 2008 2010 Time (months) Coefficient Model Model 1: Recalibration (single) Model 2: Piecewise recalibration Model 3: Window Model 4: DGLM Age (adjusted) Female Pulmonary disease Extracardiac arteriopathy Neurological dysfunction Previous surgery Creatinine > 200mumol/l Active endocarditis Critical pre−op 0.050 0.055 0.060 0.065 0.070 0.075 0.2 0.3 0.4 0.5 0.1 0.2 0.3 0.4 0.4 0.5 0.6 0.7 0.8 −0.4 −0.2 0.0 0.2 0.4 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.0 1.1 1.2 1.3 0.4 0.6 0.8 0.6 0.8 1.0 Coefficient Patient−related factors Model coefficients (log-odds) Re-fit Piecewise re-fit
20. DGLM • The largest change is coming from intercept • Latent variables affecting risk? • Doing nothing is not an option • Acting periodically might be to late −5.8 −5.6 −5.4 −5.2 2004 2006 2008 2010 Time (months) Coefficient Model Model 1: Recalibration (single) Model 2: Piecewise recalibration Model 3: Window Model 4: DGLM Intercept Intercept Re-fit Piecewise re-fit Coefficient
21. Online tools National Audits Monitoring Tools Database Querying e-lab
22. Summary • Current infrastructure incompatible with developments in healthcare monitoring & surveillance • Real-world data is not perfect which increases uncertainty + potentially a source of bias • Modelling for contemporary cardiac surgery needs to adapt to changing dynamics
23. This research was partly funded by Heart Research UK (Grant number RG2583) Any questions?