Development and validation of prediction rules to target care intensification in veteran patients with diabetes

Introduction

Diabetes mellitus is an epidemic affecting 30.3 million people in the USA in 2015; the Department of Veterans Affairs (VA) has almost three times the prevalence of diabetes compared with the general US population, at approximately 25% and 75 mmol/mol, respectively.1 2 Glycaemic control is known to reduce microvascular complication risk for these patients.1 For instance, investigators from the UK Prospective Diabetes Study found participants with glycated haemoglobin (HbA1c) values of 75–86 mmol/mol had twice the incidence of microvascular events compared with participants with values of 64–75 mmol/mol.3 The American Diabetes Association (ADA) guidelines recommending HbA1c<64 mmol/mol in most patients underscore that those with an HbA1c≥75 mmol/mol are at greatly increased risk.4 Unfortunately, many patients remain above this.

Primary care has always been charged with balancing the patient’s acute care issues with chronic diseases despite the acute issue frequently being the cause of the clinic visit.5 As such, a substantial proportion of medication regimens and blood draws for HbA1c occur in the primary care environment. In the USA, the VA is the largest healthcare provider and has the most integrated electronic health record (EHR) system.6 In the current state of patient care in the VA, the primary care providers (PCPs) have little to no patient-specific information about the likelihood of success of a treatment decision. For instance, although intended to help, clinical reminders are non-specific. Estimated at approximately 60 per day, this information overload has been shown to contribute to overlooking results, leading to delayed patient care.7 8

Prediction rules could be incorporated into already-existing tools to simplify the clinical cognitive load by pointing to the care most likely to provide the best outcome, at that time in that particular patient. Through longitudinal analysis of patient data, these prediction algorithms can provide population health information about patients who were in the same circumstance as the current patient–provider interaction, contributing to the National Academy of Medicine’s concept of the learning healthcare organisation.9 Identifying patients with diabetes and HbA1c≥75 mmol/mol with higher probability of ever attaining a value <75 mmol/mol, as well as those with higher probability of maintaining HbA1c values of <75 mmol/mol through 2 years of follow-up could provide a risk stratification method targeting care intensification to inform resource allocation. The success of this strategy is dependent on models that accurately discriminate between patients who will have subsequent HbA1c values of <75 mmol/mol and those who will not. Based on a review of the literature, it appears only one other group has developed a prediction rule for HbA1c among patients with diabetes. We sought out to develop and validate each prediction rule by randomly assigning patients to derivation and validation samples. Validation samples are used to confirm model fit based on the derivation sample for each of the outcomes.

Methods

Results

General findings

Our population consisted of 145 659 patients; 90.81% were retained. Figure 1 depicts the population of interest and resultant patient flow. Baseline characteristics of the derivation and validation samples are provided in table 1.

Table 1

View inline|View popup

Baseline characteristics of the derivation and validation samples

Request permission

Figure 1

Patient flowchart. HbA1c, glycated haemoglobin.

The derivation sample comprised 80 435 patients followed up for a mean±SD of 1.76±0.46 years. The validation sample had 51 831 patients followed for 1.75±0.46 years. All univariate analyses of candidate predictor variables were significant for each outcome, in each sample (p<0.001 each, data not shown).

Discussion

This study provides knowledge not otherwise obtainable about patients with diabetes. First, we were surprised that we identified only one other study that developed and/or validated a prediction rule related to HbA1c among patients with diabetes. Second, without the rule, the provider does not know a given patient’s likelihood of having an HbA1c<75 mmol/mol within or by the end of the next 2 years.

While sensitivity and specificity provide the probability of the model showing attainment of HbA1c<75 mmol/mol, or lack thereof, in a patient who has such a measurement, the predictive value provides the probability of actual attainment of HbA1c<75 mmol/mol, given the model showing, or not showing, such attainment. The potential contribution of these prediction rules to a learning healthcare organisation is dependent on accurate differentiation between patients who will, and will not, continue to have HbA1c measurements ≥of 75 mmol/mol. As seen in tables 3 and 4, the predicted probabilities compared with the observed are fairly accurate for each of the tertiles for each of the outcomes.

If aggregate data from the healthcare system were available to providers, they would know 55% of these patients maintain HbA1c values<75 mmol/mol at 2 years (ie, the pretest probability). The usefulness of a prediction rule is based on the proportion of patients whose probability has substantially shifted from the pretest probability. When predicting who will maintain HbA1c values of <75 mmol/mol, the majority of patients’ probabilities shifted. This did not occur for predicting who will ever have an HbA1c<75 mmol/mol; of the two, maintenance is more imperative clinically.

Most importantly, this rule identifies patients for whom an alternative approach is more appropriate. Resources should be reallocated for the 15% of patients with predicted probabilities of 0–0.4 and who are interested in meeting treatment goals: they have less than one-third chance of maintaining HbA1c values of <75 mmol/mol using the approach employed by PCPs (table 3). This implies a substantial shift in workload as it represents multiple visits currently occurring between PCPs and patients addressing treatment intensification. Our model suggests the presence of underlying drivers of poor glycaemic control that operate over the course of years. These factors may be psychosocial and should be addressed before such treatment is again attempted; our recommendation is psychosocial screening. In this way, patients are also being offered treatment intensification: once these issues have been addressed, they may be more likely to attain an HbA1c value of <75 mmol/mol.14 By identifying the problem from a population health perspective, we are recognising a gap in care and naturally risk-stratifying patients, so the greatest benefit goes to those with the most need, with the resources available. The application of this rule across the VA could result in reduced costs for diabetes care and increased costs for psychosocial support. Future studies should be done to identify such factors, assessing whether their management results in better outcomes than routine primary care.

When incorporated into our already-existing reporting mechanism, providers can receive the model results specific to each patient during, or in preparation of, the clinic visit. With these predictions, a provider can assess the patient’s values and motivation in treatment intensification. If interested, they can set a goal based on the patient’s age and comorbidities, discussing treatment options. If the patient is still above goal, then psychological, social and cultural assessment can be performed to identify treatment barriers. If psychological screening indicates a psychiatric or personality disorder or substance abuse, the provider should refer the patient to behavioural health. Alternatively, if social assessment indicates an occupational problem or homelessness, the provider should refer the patient to social service. If a cultural barrier is identified, the patient should be referred to a support group for culturally sensitive recommendations.

As evidenced by the models retaining time HbA1c≥75 mmol/mol, consecutive number of HbA1c values of ≥75 mmol/mol, baseline HbA1c value and receiving insulin treatment, it appears intermediate and short-term factors play an important part on whether a patient with an HbA1c≥75 mmol/mol will have subsequent HbA1c values of <75 mmol/mol. Fortunately, modifiable factors such as diet, exercise, treatment regimen and recent change in health status (eg, pneumonia) are captured among these, highlighting their importance. Since time-weighted average HbA1c entered the model predicting HbA1c measurement ever <75 mmol/mol, but not the model predicting final HbA1c measurement <75 mmol/mol, another predictor variable potentially is more statistically, but not clinically, significant. Also noteworthy, minority status changed directions across the models. Our cohort has 4738 (3.58%) patients whose qualifying HbA1c during baseline was their first documented HbA1c in the VA. Future studies should be conducted to further elucidate these relationships.

In our review of the literature, we found only one other paper reporting a prediction rule for HbA1c in patients with diabetes.15 Investigators of that study followed up 684 European patients with type 2 diabetes mellitus 12 months from clinical appointments; their dependent variable was ‘an observed increase in terms of HbA1c %≥0.5’. While we considered higher-risk patients to have baseline HbA1c≥75 mmol/mol, their patients had microvascular complications with a median (range) HbA1c of 55 mmol/mol (48–72 mmol/mol). The authors of that paper do not report sensitivity, specificity, positive predictive value or negative predictive value to allow comparison of model performance to ours.

Conclusions

Our project simultaneously addresses population health and patient-centred care. Similar observed and predicted tertile probabilities were found within each rule; the majority of patients experienced probability shifts for maintaining HbA1c values of <75 mmol/mol, showing its usefulness in a learning healthcare organisation. We recommend psychosocial screening for one in every seven patients for whom there is about one-third chance of maintaining HbA1c values of <75 mmol/mol. We plan to leverage the richness of the national databases while testing different aspects of the technology, including assessment of human factors, in a graduated manner before any intention of national implementation.