Jump to comment:

• Concern regarding the recently published sepsis prediction model
  Christopher V. Cosgriff, Leo Anthony Celi
  Published on: 19 September 2020

• Published on: 19 September 2020

  Concern regarding the recently published sepsis prediction model

  • Christopher V. Cosgriff, Resident Physician Hospital of the University of Pennsylvania
  • Other Contributors:
    • Leo Anthony Celi, Intensive Care Physician, Research Scientist

  To the editor:

  We read the recent study from Burdick and colleagues with great interest as few machine learning models have been assessed prospectively, and even fewer have shown benefits on clinically-relevant outcomes such as length of stay or mortality.1 However, we are concerned about some of the reported data. The model they deploy, termed InSight across various publications, predicts the development of sepsis based on the systemic inflammatory response syndrome (SIRS) criteria, which has been replaced by the Sepsis-3 guidelines.2,3 Their model which is a gradient boosted classification tree built with XGBoost is trained by looking back at vital sign data and predicts if a patient will meet SIRS criteria in the next four hours. In the present study they define outcomes cohort as “sepsis-related” by including any patient who met 2/4 of the SIRS criteria. Using this definition, they report a pre-intervention in-hospital mortality of 3.86%.4 However, this is far below the expected sepsis related mortality, regardless of the criteria used. Furthermore, by reporting outcomes in patients who met SIRS >2, rather than everyone screened by the algorithm, the relative harm done to patients for whom this model falsely predicted sepsis would not be factored into the equation assessing the algorithm’s value. They report an estimate of the cost reduction, but this again does not account for the costs that could be associated with the adverse effects of unnecessarily tre...

  Show More

  To the editor:

  We read the recent study from Burdick and colleagues with great interest as few machine learning models have been assessed prospectively, and even fewer have shown benefits on clinically-relevant outcomes such as length of stay or mortality.1 However, we are concerned about some of the reported data. The model they deploy, termed InSight across various publications, predicts the development of sepsis based on the systemic inflammatory response syndrome (SIRS) criteria, which has been replaced by the Sepsis-3 guidelines.2,3 Their model which is a gradient boosted classification tree built with XGBoost is trained by looking back at vital sign data and predicts if a patient will meet SIRS criteria in the next four hours. In the present study they define outcomes cohort as “sepsis-related” by including any patient who met 2/4 of the SIRS criteria. Using this definition, they report a pre-intervention in-hospital mortality of 3.86%.4 However, this is far below the expected sepsis related mortality, regardless of the criteria used. Furthermore, by reporting outcomes in patients who met SIRS >2, rather than everyone screened by the algorithm, the relative harm done to patients for whom this model falsely predicted sepsis would not be factored into the equation assessing the algorithm’s value. They report an estimate of the cost reduction, but this again does not account for the costs that could be associated with the adverse effects of unnecessarily treating patients who did not have sepsis (false positives). We are additionally concerned that the results are reported only as relative risk reductions as opposed to absolute risk reductions. In doing so the values appear more impressive. Based on the data presented, we calculate the absolute risk reduction to be 0.0152, and a number needed-to-treat (NNT) of 65.8. These numbers are likely an overestimate given that the denominator did not include all patients who were screened. Equally relevant to report is the number needed-to-harm (NNH) which pertains to patients who falsely triggers the algorithm and receive unnecessary, and potentially deleterious overtreatment. Most importantly, we are troubled that the results of this study may promote a false impression that algorithms can be sold and implemented without validation and re-calibration using local data. Readers should be constantly reminded that the accuracy of algorithms is bound by space and time.5 There must be processes in place to continuously monitor the performance of an algorithm before it is deployed by a health system.

  Christopher V. Cosgriff, M.D., M.P.H
  Leo Anthony Celi, M.D., M.S., M.P.H

  1. Burdick, H., et al. Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals. BMJ Health & Care Informatics 27, e100109 (2020).
  2. Mao, Q., et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 8, e017833 (2018).
  3. Singer, M., et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 315, 801-810 (2016).
  4. Johnson, A.E.W., et al. A Comparative Analysis of Sepsis Identification Methods in an Electronic Database. Critical care medicine 46, 494-499 (2018).
  5. Davis, S.E., Lasko, T.A., Chen, G. & Matheny, M.E. Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality. AMIA ... Annual Symposium proceedings. AMIA Symposium 2017, 625-634 (2018).

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top