Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

Introduction

Despite a high associated mortality1 2 and high costs of treatment,2 3 severe sepsis remains notoriously difficult to diagnose and treat. The healthcare costs of sepsis in the USA in 2013 reached nearly US$24 billion, roughly 6% of the nation’s total hospital bill, while sepsis patients represented only 3.6% of all hospital stays.4 Prior research has emphasised the importance of timely sepsis recognition to both improving patient outcomes and reducing costs associated with treatment.5–7 New definitions intended to improve the clinical recognition of sepsis have recently been proposed,8 9 as the previous use of screening based on systemic inflammatory response syndrome (SIRS) criteria has been found to be nonspecific.10 Evidence from the medical literature has shown that accurate early diagnosis and treatment can reduce the risk of adverse patient outcome from severe sepsis and septic shock.11–13 Therefore, earlier detection of sepsis and more accurate recognition of patients at high risk of developing severe sepsis or septic shock is essential for effective sepsis treatment.

Screening tools used in clinical settings for the identification of decompensating patients include the Sequential Organ Failure Assessment (SOFA),14 the SIRS criteria15 and the Modified Early Warning Score (MEWS).16 These systems have been used to recognise severe sepsis due to their ability to both identify systemic inflammation as a sign of infection, and to detect possible organ dysfunction. The utility of such systems for the identification of septic patients has been studied at length in recent literature.17–22 However, systems, such as MEWS, SOFA and SIRS, were originally designed as generalised screening tools as opposed to explicitly identifying sepsis, and their efficacy in sepsis diagnosis is limited. For example, SOFA has been reported to be not widely applicable outside of the intensive care unit (ICU), and it often requires use of laboratory values that are not rapidly available.17 SIRS has been reported to be nonspecific17–23 and also may yield up to one in eight false negatives in detecting patients with organ failure and infection.17–24 Despite their limitations, these scoring systems have established performance metrics, and serve as important comparators for newly developed severe sepsis detection and prediction systems and their effect on clinical outcomes.25–28

Improvement in sepsis care and adoption of electronic health record (EHR) systems have been incentivised by the Centers for Medicare & Medicaid Services in recent years.29 30 Currently, 96% of hospitals in the USA have an EHR federally tested and certified for the government's incentive programme.31–33 A number of methods have been developed to monitor patient EHR data for severe sepsis, but few provide predictive capabilities to enable early intervention and improve patient outcomes. Although they represent fairly new additions to the field of sepsis care, machine learning algorithms (MLAs) have the potential to significantly improve patient outcomes through advance warning of impending sepsis onset. Sepsis prediction MLAs may also serve to empower clinicians to have confidence in their sepsis diagnosis in a variety of ambiguous cases, including instances when positive culture results are not available,34 and in cases of atypical clinical presentation among older patients who comprise a majority of sepsis cases.35 36 Machine learning-based decision support systems, therefore, represent an important area of investigation for sepsis research.37 38

The MLA used in this study has been described in previous peer-reviewed publications both retrospectively and prospectively,39–45 but has not been evaluated for its effect on clinical outcomes on multicentre diverse hospital settings. In this study, performance of our MLA for severe sepsis prediction and detection was evaluated using real-world data from patient EHRs at nine diverse hospitals from the northeast, southern, midwestern and western regions of USA, spanning academic centres to community hospitals. A clinical outcomes analysis was performed to evaluate the effect of the algorithm on in-hospital patient mortality, hospital length of stay (LOS) and 30-day readmissions.

Methods

Results

Aggregated patient demographic data from the nine participating hospitals in this study are presented in table 1. Seventeen per cent of patients were included for the baseline analysis period and 83% of patients were included for the MLA analysis period. Vital sign averages and SD were not significantly different between the baseline and MLA analysis periods. Among those patients analysed by the MLA, the mean age was 45 years (41.9% male vs 58.1% female). Patients included for clinical outcomes analysis were generally representative of those at risk of developing severe sepsis in terms of gender and racial/ethnic distribution.1 52

Table 2 shows the variation in hospital size, location and type for the hospitals included in this analysis. The wide range of geographical and population distribution demonstrates a diverse range of hospital types included for clinical outcomes determination.

Clinical outcomes were measured for all patients over 18 years who met two or more SIRS criteria at any point during their stay, in order to ensure that those patients most at risk for sepsis were included in our final analysis. The subsequent outcomes analysis was performed in order to determine if use of the MLA had significant effects on in-hospital patient mortality, hospital LOS and/or 30-day readmissions. We emphasise that while the SIRS criteria were used to determine which patients should be included in the outcomes analysis, the MLA in this study uses only patient vital signs to predict severe sepsis.

The sepsis-related outcomes after MLA implementation were a 39.50% reduction of in-hospital mortality (p<0.001), a 32.27% reduction of LOS (p<0.001) and a 22.74% reduction in 30-day readmission (p<0.001; table 3, figure 1). These results include sites where data from the period preceding implementation of the MLA were not available, in which case the baseline period used was the month immediately following implementation.

Request permission

The analysis was repeated for a subset of 3 hospitals with at least 1 month of baseline (pre-MLA implementation) data, with a total of 52 487 patients. This resulted in 3951 patients in the baseline period (971 included as sepsis related, as defined in the Study Design section), and 48 536 patients (10 646 included as sepsis related) in the MLA analysis period. The outcomes for this patient subset were a 42.50% reduction of in-hospital mortality (p<0.05) and a 23.82% reduction in LOS (p<0.05).

Results indicate that our machine learning algorithm for severe sepsis prediction can be successfully used to improve clinical outcomes of in-hospital mortality, LOS and 30-day readmission rates.

Discussion

In this clinical outcomes study, we tested the hypothesis that use of an MLA for severe sepsis detection and prediction would result in reductions of adverse sepsis-related clinical outcomes. The design of this study involved minimal to no risk of patient harm, but offered potential benefits to both patients and providers. In particular, the algorithm’s ability to identify patients with severe sepsis prior to onset provided a significant opportunity for early intervention. Prior studies have shown that early detection or prediction of sepsis and severe sepsis, respectively, can lead to a decrease in the time to administration of antibiotics,40 53 and early intervention has been shown to reduce rates of patient mortality.54–56 Use of the MLA in this study was associated with a 39.5% reduction of in-hospital mortality (p<0.001), a 32.3% reduction in LOS (p<0.001) and a 22.7% reduction in 30-day readmissions (p<0.001).

Improvements in clinical outcomes were calculated by comparing outcomes before algorithm implementation with outcomes after implementation. Not all data fields were available for abstraction at all nine participating hospitals. In cases where pre-implementation measurements were not available, the first month of clinical implementation was used as an approximate baseline. During this initial period, the MLA alert sensitivity, specificity and clinical response were undergoing evaluation and development, and therefore, did not represent the final state of the MLA alert and response. However, including the use of this period as a baseline may result in an underestimation of the effect of the MLA, compared with the pre-implementation period.

Results from the clinical outcomes analysis indicate that the algorithm has a more significant effect on improving clinical outcomes than other screening tools such as MEWS, SOFA and SIRS.25–28 For example, in a prospective comparative analysis of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sepsis, discrimination of in-hospital mortality using the SIRS score was reported to be significantly less than that of the qSOFA score, with an overall in-hospital mortality rate of 19%.25 A pre-implementation and post-implementation study evaluating the effect of an SIRS-based sepsis early warning system that monitored SIRS criteria along with signs of organ dysfunction (based on systolic blood pressure and serum lactate thresholds), found that while the tool prompted more timely sepsis care, there was no significant reduction in mortality.53 In a comprehensive review of peer-reviewed literature to evaluate the effect of MEWS on improving clinical outcomes, limited data and no clinical trials which linked use of MEWS scoring systems to ‘robust’ outcomes were found.26 An analysis of a variety of disease severity scoring systems for the prognostic assessment of septic patients revealed that SOFA and MEWS showed only moderate discrimination in predicting 28-day mortality rates.28 Beyond the simple heuristics of rules-based scoring systems such as MEWS, SOFA, qSOFA and SIRS, several machine learning approaches have been retrospectively evaluated for the detection and prediction of incipient sepsis.37 38 57–66 They include dynamic Bayesian networks,60 support vector machines,57 survival-analytical models (TREWScore, Artificial Intelligence Sepsis Expert),61 62 smoothed disease severity score learning,63 hierarchical switching linear dynamical systems,64 autoregressive hidden Markov models,65 free-text models38 and random-forest models.57 These tools contribute notably to the field of sepsis detection because they offer generalisability, are scalable, and can be updated as new information is acquired.58 However, many do not use information about measurement trends or correlations,67 or do so ineffectively. Most machine learning approaches have been evaluated only on retrospective data as proof-of-concept.37 57 58 60 62 64–66 There remains an ongoing need for research which evaluates the clinical utility of sepsis prediction models in prospective and real-world settings.

Towards this end, Nelson et al conducted a prospective trial of a real-time electronic surveillance system to expedite early care of severe sepsis.67 Outcome measures were rate and timeliness of sampling of blood lactate and blood cultures, performance of chest radiography and provision of antibiotics; however, only time to blood culture was significantly improved. The primary limitation of the trial was cited as the inability to detect severely septic cases before caregivers. Umscheid et al conducted a real-world pre-implementation and post-implementation study of an early warning and response system (EWRS) for sepsis outside of the ICU.68 The EWRS identified at risk patients with a sensitivity of 16% and a specificity of 97%. Compared with a control period, the EWRS activated in the post-implementation period resulted in an increase in ICU transfer <6 hours after alert (p=0.06). However, additional outcome measures of hospital LOS (p=0.92), ICU transfer ﹤24 hours after alert (p=0.20), renal replacement therapy ≤6 hours after alert (p=0.51) and all-patient mortality reductions (p=0.45) failed to reach statistical significance.68 Austrian et al performed a time-series study which evaluated an electronic surveillance system on mortality and LOS on emergency department patients with severe sepsis or septic shock,69 finding a modest decrease in LOS (16%) that did not reach statistical significance, with no difference in-hospital mortality or other intermediate outcome measures. Alert fatigue due to low positive predictive value (PPV) (0.146) was proposed as the primary contributor to these results, and researchers noted that more sophisticated approaches to early sepsis identification are needed to consistently improve patient outcomes. Importantly, the study supports the principle that high PPV is critical for effective clinical decision support interventions.69 The early and accurate alerting system introduced in our study is associated with an LOS reduction of 32.3%, a mortality reduction of 39.50% and obtains a PPV of approximately 40% for sepsis prediction as demonstrated in prior work.44

In addition to clinically improving patient outcomes, the sepsis prediction tool analysed in this study also provides economic advantages. The cost of severe sepsis has been reported to extend ‘well beyond’ patient impact, as a large part of the sepsis economic burden is incurred after discharge and during rehospitalisation.70 Administration of timely treatment is therefore crucial to reducing costs, reducing rates of readmission and improving treatment outcomes. This clinical outcomes study provides a prospective analysis of machine learning algorithm performance in the sepsis care domain. To the extent possible, we also calculate a first order approximation of cost reductions incurred through reductions in LOS from the use of the algorithm. The average LOS reduction was found to be 1.56 days. At an average per diem cost of care of US$2271 with 343 patients included per month at the nine locations, the reduction of LOS translates to approximately US$14.5 million of annual cost savings across all nine hospitals included in this analysis. These findings on post-marketing real-world data confirm pre-marketing randomised clinical trial results.40 Previous research has shown that early detection and treatment of sepsis can improve patient outcomes and reduce hospital costs.12 13 71 72

Our real-world data analysis has several limitations. We cannot guarantee the usefulness of resulting alerts to clinicians, which indicates a need for future studies which include qualitative analysis of algorithm utility (ie, clinician surveys or interviews). Future work including clinician surveys would also help to determine how clinicians responded to the alerts, including any diagnostics tests or treatment interventions that were ordered. This level of detail would be helpful in assessing the potential means through which the observed positive impacts on patient outcomes were achieved. Ideally, this future work would also include the clinical adjudication of sepsis onset times, instead of defining onset times in terms of gold standard criteria, so that the extent to which alerts were accurate and early could be determined. Further, although machine learning systems have made significant advances in the healthcare domain over the past decade, it is important to consider the unintended ways in which they impact clinical practice. Unintended consequences of machine learning in medicine include an over-reliance on the capabilities of automation; a lack of contextual information which may lead to diagnostic misinterpretation; and observer variability affecting the accuracy and reliability of machine learning performance.73 However, it should be noted that risks of machine learning are minimised by screening or ‘sniffer’ algorithms such as this MLA, which are designed to increase clinician oversight for high-risk cases, and not to replace expert clinical judgement and standards of care. Other limitations of our study include variation in clinician and team responses to patients at possible risk for sepsis. Only adults in US hospitals were included in the study. While nine diverse hospitals were included in the analysis, these hospitals may not be representative of all US hospitals or international hospital settings. Data were not available from all hospitals for all months and outcome measurements. Baseline data were not available for all hospitals and the first month of MLA data was used as an approximation in these cases. This may lead to an underestimation of the effect of the MLA at these sites. However, the analysis was repeated on a subset of three hospitals with at least 1 month of baseline pre-MLA implementation data and outcomes were similar. This study did not follow patient mortality after hospital discharge. We cannot eliminate the possibility that implementation of a sepsis algorithm raised general awareness of sepsis within a hospital, which may lead to higher recognition of septic patients, independent of algorithm performance.

Conclusion

This study evaluates the effect of a machine learning algorithm for severe sepsis detection and prediction on clinical outcomes. In an analysis of the algorithm across nine hospitals, use of the MLA was associated with a 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital LOS and a 22.7% reduction in 30-day readmissions. These results support that the implementation of an accurate machine learning algorithm for early sepsis recognition may lead to improved patient outcomes, and by extension may serve to reduce the financial burden to the US healthcare system. In future studies, we will continue to analyse the algorithm’s impact on patient outcomes in other care settings.