Physician performance scores used to predict emergency department admission numbers and excessive admissions burden

Background

Overcrowding in hospital wards is associated with a panoply of adverse consequences for patients and staff alike.1–4 Internal medicine wards in Israeli hospitals are notorious for being overcrowded, with an annual nationwide average occupation of 97% in 2018 and 2019.5 These wards are struggling with patients in the corridor and team burnout, which is reflected in reduced quality of care to the most vulnerable patients, and the increasing challenge in recruiting high quality and motivated medical staff. The key to improvement is a reduction in the occupancy of the wards, by minimising unnecessary emergency department (ED) admissions, employing rapid workup and early discharge with emphasis on ambulatory treatment when possible. These approaches result in shorter inpatient stays, reduced inpatient mortality and increased patient and staff satisfaction.6 7

The appropriateness of admissions-related decisions in the ED has a direct effect on overcrowding, and excess admissions are specifically associated with poorer clinical and organisational outcomes.8 9 Multiple non-medical factors can influence the decision to admit an ED attendee, including the workload of the ED, hospital bed occupancy rates, the hour and day of week, the date and its correspondence to public events, individual staff performance, and even the weather.10 11

By applying statistical inference techniques, it is possible to create models for the prediction of admission caseload, based on social, institutional, staffing and stochastic factors. Hitherto dozens of models have been published that attempt to forecast ED demand,12 13 as well as predicting admission likelihood for individual patients.14 Only a few make any attempt to forecast overall admission numbers.15–19 No study hitherto has attempted to include all available factors, and particularly estimates of staff performance, in modelling admissions. No study has used such models as tools for optimisation of staffing policies in order to reduce admissions. Furthermore, very few of the published studies of any type have specifically involved the Israeli hospital system which, as with any locale, has its own unique set of demographic and administrative challenges.

The aim of this paper is to demonstrate the creation and testing of a model for the prediction of admission numbers, to confirm the hypothesis that measures of staff performance are important inputs into such models and to demonstrate the feasibility of using such models as tools to optimise staff allocation and reduce admissions.

For the current study, we use 5 years’ of ED case records at Meir Medical Center, a busy secondary-level general hospital in central Israel. Using this data, we construct metrics for individual staff performance, as well as time series for historic arrival and admission numbers, ED caseload and other factors. Based on these, we construct models to predict both daily ED arrivals and daily admissions from the ED to the internal medicine wards. We also derive a tool that can produce an alert when any given day is likely to see an excessive number of admissions.

Study population and data set

Meir Medical Center is a busy, secondary-level general hospital in Kefar Sava in central Israel. It serves a demographically diverse population of close to one million. Adult patients seeking urgent care are seen at triage, and either sent to the internal medicine ED or to one of several specialist surgical EDs. Starting in March 2020, due to the incipient COVID-19 epidemic, the internal medicine ED was split into two sections, one for patients for whom there is a high suspicion of COVID-19 and one for the rest. The determination as to which section a patient should attend was made at triage.

Our database comprises every attendance at the internal medicine ED between 1 February 2016 and 31 March 2021. The basic anonymised data set comprised the following: the date and time of ED presentation, which unit was attended (the COVID ED or the regular ED), the date and time of the patient’s release from the ED, the type of disposal (admission, discharge, self-discharge against advice, etc) and the name of the physician who signed the disposal. Additionally, for patients who were admitted: the name of the inpatient unit, and the date and time of subsequent discharge. Records for multiple attendances by the same individual were linked. The following derived data were calculated from the basic data and stored alongside it: ED occupancy at the time of ED receipt, ED occupancy at the time of ED release, whether the patient returned to the ED within 7 days of discharge and whether the patient was admitted for less than 24 hours (a zero-day admission).

For the 61 months covered by the study, there were 232 246 attendances by 128 570 individual patients, who were treated by 886 individual physicians. For each day covered by the study, hour-by-hour weather conditions for Kefar Sava were fetched from a commercial provider. All statistical analysis was carried out in R 4.1.2 64-bit.20 A list of R packages used is available in the supplementary material. This study received ethical approval from the local Helsinki committee. There was no patient or public involvement in the design of this study.

Measuring physician performance

In order to comprise an effective tool for predicting admission numbers, we require measures of physician performance that are stable, reproducible, simple and comparable. The most direct measure of an ED physician’s proclivity to admit patients is her or his historical admission rate. Other hypothesised measures of performance that we consider include the physician’s rate of zero-day admissions, as a proxy for inappropriate admissions, the rate of a physician’s discharges who return to the ED within 7 days, as a proxy for inappropriate discharges and the speed at which a physician works, measured in terms of chart signatures per hour, as a general performance correlate.

Since any physician’s performance is expected to vary with experience, it is not sufficient to calculate these statistics as a simple average over all time. Instead, each score is computed as an exponential weighted average, which favours more recent performance over older data. Separate scores are calculated for work in the COVID ED and the regular ED, since working conditions and hence performance differ dramatically between them. In order to compare directly between each of the different performance measures, and in order to combine performance scores from the COVID and regular EDs, every performance measure is converted to a peer-referenced t-score, in which a score of zero represents performance equal to the mean of a physician’s peers, and a score of ±1 represents performance one SD above or below the mean. The performance score in any given category for a physician who works in both the regular and COVID EDs is a weighted average of the separate scores for each ED, reflecting his/her relative workload in each location.

For the purposes of relating admission numbers to the historical performance of the admitting team, we must construct a weighted average over that team, for each of these performance t-scores, for each day. The weights for this average are chosen such that the team-performance score for any given day reflects the predicted contribution of each individual team member to the day’s work, based on posted hours and the individual’s rate of seeing patients. Full details of this calculation are provided in online supplemental appendix S1.

Modelling ED arrival and admission numbers

For both arrival and admission numbers, we construct a predictive model based on a negative-binomial generalised linear regression. The independent variables for each model are listed in table 1. As in previous studies, the inclusion of historic arrivals and admissions data provides a key boost to model performance.17 The most accurate predictions are made using arrivals data as recent as the hours leading up the prediction being made.

Redundant independent variables were removed from each model by an iterative procedure. At each step, the least significant variable is removed, and a new model generated, until all remaining variables have a statistically significant effect on model output. The final form for each model was shown to have similar predictive power to the initial model. In order to improve fitting performance and predictivity at the extremes of the range, at the expense of a slight loss in predictivity at the centrum of the data, weighting was applied which favoured data for days with a number of arrivals that was far from the mean. Full details of the arrivals model and the fitting procedure are provided in online supplemental appendix S2, and full details of the admissions model in online supplemental appendix S3.

Predicting arrival numbers

Figure 1 details the performance of the arrivals model. The actual number of admissions is shown to be well correlated with the predicted number, across the full range of values. The mean average percentage error (MAPE) is 6.85%.

Request permission

To test the ability of the model to retain predictive power when confronted by data on which it has not been trained, an alternative model was created, which is identical to the full model in all respects, except that it was trained on a randomly selected subset of 80% of the full data. Predictions were generated for the remaining 20% of the data set, on which the model was not trained. The performance of the model remained unchanged.

Predicting admission numbers

Figure 2 shows the performance of the admissions model. The actual number of admissions is shown to be well correlated with the predicted number, across the full range of values. The MAPE is 10.6%.

Request permission

Of all the physician performance metrics considered in ‘Measuring physician performance’ section, only the team admission rate metric was included in the final analysis. Although the zero-day admissions rate and the physician speed are indeed correlated with admission numbers, they have no correlation independent of team admission rate and therefore provide no extra predictive value to the model, though we note that clinician speed remains indirectly incorporated into the team admission rate metric as described in the ‘Measuring physician performance’ section. ED returnee rate was not found to correlate with admission numbers at all. Likewise, the amount of rainfall was not found to have an independent influence on admissions and was not included in the final model. Examination of model diagnostics shows that the negative binomial approximation is a good fit to the data, and there is no evidence for non-linearity or collinearity in the model variables. As with the arrivals model, the performance of the admissions model remained unchanged when confronted by data on which it had not been trained. To demonstrate the utility of clinician performance as an input variable, the same model was generated without including any performance variables: the MAPE is degraded to 11.1%.

It is possible to use the model as a tool to estimate the effect of prospective staffing changes on the number of admissions, by hypothetically varying the composition of the ED team, and observing the effect on output predictions. A precursory analysis shows that simply by swapping physicians between shifts in a single week, it is possible to strengthen the team performance sufficiently on those days predicted to be busiest—Sundays and days after public holidays—to provide a 12% across the board reduction in admissions, and a 40% reduction in the variance in admission number.

Predicting crises in admission numbers

The response variable of the model from the ‘Modelling crises in admission numbers’ section is a probability for encountering a crisis on the given day in question. A crisis is predicted when this probability exceeds some prespecified value, and sensitivity of the test is fixed by choosing this value. The resulting receiver operating characteristic curve (ROC) for this test is shown in the left panel of figure 3, and has an area under curve (AUC) of 0.903.

Request permission

The dashed line in the right-panel of figure 3 shows the sensitivity versus positive predictive value (PPV) for this test, when it is applied indiscriminately to every single day. Despite the high intrinsic performance, we see that the practical performance of this test is rather poor. For instance, if a false-positive rate of at most 0.25 (ie, PPV of 0.75) is required, then the sensitivity falls to around 0.2. This occurs because the pretest probability of a crisis is low (by definition 0.1) when the test is used in this way.

The practical performance of the test can be improved by applying the test discriminately to those days with a higher likelihood of hosting a crisis. Our data show that Sunday (which is the first day of the working week in Israel) and any weekday following a public holiday were found to significantly increase the odds of a crisis. Indeed, half of all crises occur on these days alone. Conversely, almost no crises occurred at weekends or during public holidays.

The solid line in the right-panel of figure 3 shows the sensitivity versus PPV for the test when applied only to Sundays or weekdays following public holidays, for which the pretest probability of a crisis is 0.35. Again considering a false-positive rate of at most 0.2, the sensitivity of the test is now improved to 0.75. The test thus becomes a practical tool for enabling same-day interventions to either directly reduce, or to mitigate the effects of, increased admissions on days at heightened risk of a crisis.

As before, we can examine the effect on model predictions of hypothetical changes to ED team composition. By swapping shifts to strengthen the busiest days at the expense of quieter ones, we find a dramatic reduction of 87% in the number of crises on the busiest days, and an overall reduction of around 33%.