An error in a web-based Thoracoscore calculator

Presented at the Society for Cardiothoracic Surgery in Great Britain and Ireland Annual Meeting 2012.
By Kenrick Turner, Andrew Ho, Gunnar Rø, Martin Turner, & Filip Van Tornout.
Full code at http://github.com/kenners/SFAR-thoracoscore

Transcript

1. An error in a web-based Thoracoscore calculator Kenrick Turner Andrew Ho Gunnar Rø Martin Turner Filip Van Tornout
2. This is not about Thoracoscore being wrong.
3. It’s about a systematic error in the web’s most popular Thoracoscore calculator http://www.sfar.org/scores2/thoracoscore2.php
4. …and more widely, the risks posed by medical calculators Medical Calculator
5. •  Routine audit at the Norfolk & Norwich •  Difference observed between Thoracoscores calculated in Excel & by SFAR web calculator •  Explored the discrepancy by exhaustively calculating all 1,152 possible Thoracoscores with SciPy and the SFAR calculator. Some background
6. Some example data SFAR Thoracoscore SciPy Thoracoscore Absolute Error 0.11 0.06 0.0473 0.14 0.10 0.0416 0.16 0.11 0.0451 0.17 0.12 0.0452 52.44 52.39 0.0477 56.45 56.40 0.0468 58.35 58.30 0.0454 62.22 62.18 0.0446 … Mean absolute error: 0.0449 (SD 0.0029)
7. Is it them or us? Pretty confident we were right. So where is the error? To the source code!
8. Function  Fmt(x)  {    var  v;    if  (x  >=  0)  {      v  =  ‘’  +  (x  +  0.05);    }  else  {      v  =  ‘’  +  (x  –  0.05);    }    return  v.substring(0,  v.indexOf(‘.’)  +  3);   Adds 0.05 to all scores and then shortens to 2 decimal places. This is not a proper rounding function. Buggy rounding
9. •  The observed systematic error is the addition of 0.05 to all Thoracoscores. •  Small absolute error: actual Thorascore is marginally lower for a given patient than the score calculated. •  Unlikely to have had an impact on clinical decision making. •  Lucky. Clinical points
10. So what? Scoring systems are subject to formal peer review and rigorous analysis. But we don’t use them directly.
11. Because calculating scores by hand is laborious: So we use medical calculators to do the heavy lifting... Score = e−7.3737+Age+Sex+ASA+Performance+Dyspnoea+Priority+Procedure+Diagnosis+Cormobidity 1+e−7.3737+Age+Sex+ASA+Performance+Dyspnoea+Priority+Procedure+Diagnosis+Cormobidity
12. The problem with medical calculators ACME Medical Calculator Parameters Results! We treat them as Black Boxes that give us the correct results.
13. A call to arms •  Break open the Black Box: source code must be open sourced. •  Calculators must be subject to the same standard of peer review as the scoring systems. •  Ultimately the responsibility lies with the clinician using the calculator i.e. you!
14. Thank you Contact: kenrickturner@nhs.net http://kenners.org Full code: https://github.com/kenners/SFAR-thoracoscore