An error in a web-based Thoracoscore calculator

44b9d320267a3204e78af152243303c1?s=47 Kenrick Turner
April 15, 2012
350

An error in a web-based Thoracoscore calculator

Presented at the Society for Cardiothoracic Surgery in Great Britain and Ireland Annual Meeting 2012.
By Kenrick Turner, Andrew Ho, Gunnar Rø, Martin Turner, & Filip Van Tornout.
Full code at http://github.com/kenners/SFAR-thoracoscore

44b9d320267a3204e78af152243303c1?s=128

Kenrick Turner

April 15, 2012
Tweet

Transcript

  1. An error in a web-based Thoracoscore calculator Kenrick Turner Andrew

    Ho Gunnar Rø Martin Turner Filip Van Tornout
  2. This is not about Thoracoscore being wrong.

  3. It’s about a systematic error in the web’s most popular

    Thoracoscore calculator http://www.sfar.org/scores2/thoracoscore2.php
  4. …and more widely, the risks posed by medical calculators Medical

    Calculator
  5. •  Routine audit at the Norfolk & Norwich •  Difference

    observed between Thoracoscores calculated in Excel & by SFAR web calculator •  Explored the discrepancy by exhaustively calculating all 1,152 possible Thoracoscores with SciPy and the SFAR calculator. Some background
  6. Some example data SFAR Thoracoscore SciPy Thoracoscore Absolute Error 0.11

    0.06 0.0473 0.14 0.10 0.0416 0.16 0.11 0.0451 0.17 0.12 0.0452 52.44 52.39 0.0477 56.45 56.40 0.0468 58.35 58.30 0.0454 62.22 62.18 0.0446 … Mean absolute error: 0.0449 (SD 0.0029)
  7. Is it them or us? Pretty confident we were right.

    So where is the error? To the source code!
  8. Function  Fmt(x)  {    var  v;    if  (x  >=

     0)  {      v  =  ‘’  +  (x  +  0.05);    }  else  {      v  =  ‘’  +  (x  –  0.05);    }    return  v.substring(0,  v.indexOf(‘.’)  +  3);   Adds 0.05 to all scores and then shortens to 2 decimal places. This is not a proper rounding function. Buggy rounding
  9. •  The observed systematic error is the addition of 0.05

    to all Thoracoscores. •  Small absolute error: actual Thorascore is marginally lower for a given patient than the score calculated. •  Unlikely to have had an impact on clinical decision making. •  Lucky. Clinical points
  10. So what? Scoring systems are subject to formal peer review

    and rigorous analysis. But we don’t use them directly.
  11. Because calculating scores by hand is laborious: So we use

    medical calculators to do the heavy lifting... Score = e−7.3737+Age+Sex+ASA+Performance+Dyspnoea+Priority+Procedure+Diagnosis+Cormobidity 1+e−7.3737+Age+Sex+ASA+Performance+Dyspnoea+Priority+Procedure+Diagnosis+Cormobidity
  12. The problem with medical calculators ACME Medical Calculator Parameters Results!

    We treat them as Black Boxes that give us the correct results.
  13. A call to arms •  Break open the Black Box:

    source code must be open sourced. •  Calculators must be subject to the same standard of peer review as the scoring systems. •  Ultimately the responsibility lies with the clinician using the calculator i.e. you!
  14. Thank you Contact: kenrickturner@nhs.net http://kenners.org Full code: https://github.com/kenners/SFAR-thoracoscore