Background

A major justification for the clinical adoption of electronic health records (EHRs) was the promise that it would improve the quality of medical care.1,2 Cross-sectional studies of the impact of EHRs on quality have shown inconsistent results.3–12 No randomized clinical trial or longitudinal study using a direct measure of clinical quality has been conducted to test this assumption. We performed a longitudinal retrospective study using hemoglobin A1c, a recognized clinical quality measure directly related to diabetes outcomes,13 to assess clinical quality. We hypothesized that the use of an EHR for five years would lower the A1c values of type 2 diabetic patients.

Methods

We performed a five-and-one-half-year multicentre longitudinal retrospective study. An independent contractor blinded to the study selected the notes of 537 type 2 diabetic outpatients who were treated by physicians at five U.S. Military Health System (MHS) outpatient primary care clinics. The MHS’s commercial EHR is similar to most EHRs and the MHS population is similar to the general population.14–17 The patients had to have been seen on at least three occasions: once approximately six months prior to EHR adoption (before-EHR), once approximately six months after EHR adoption (after-EHR) and once approximately five years after EHR adoption (five-years), for a total of 1611 notes.

The means and confidence intervals(CIs) for the A1c values at each time point were calculated. Patients were also stratified into one of two groups based on whether their charts contained a missing A1c value. The ‘complete’ group consisted of the patients where all three notes contained an A1c value. The ‘partial’ group consisted of the patients where at least one of their notes contained an A1c value, but all three notes did not contain A1c values. Patients missing all three A1c values were not included in the study. In addition, patients were stratified into tertiles based on their A1c values and the tertile’s mean (CI) A1c value for each time point was determined. The mean A1c values were compared across time intervals by Student’s t-test, significance p < .05, two-sided (SAS 9.7, SAS Institute Inc., Cary, North Carolina).

Results

The overall mean (CI) A1c values for the before-EHR, after-EHR and five-years were 7.07 (6.91 – 7.23), 7.33 (7.14 – 7.52) and 7.19 (7.06 – 7.32), respectively, labelled All in Figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1 Mean A1c values by time point and missing data

There was a small but significant increase in A1c values between before-EHR and after-EHR, p = .04, but there were no significant differences between before-EHR and five-years and between after-EHR and five-years.

The effect of missing data on A1c values is shown in Figure 1. The All group contains all the patients with A1c values, the complete group contains the patients with an A1c value for each of the three time points, and the partial group contains the patients with an A1c value for either one or two time points but not all three time points. For the complete group, the before-EHR, after-EHR and five-year mean (CI) A1c values for the complete group were 7.16 (6.84 – 7.38), 7.25 (7.02 – 7.48) and 7.25 (7.03 – 7.47), respectively. For the partial group, they were 6.96 (6.75 – 7.17), 7.43 (7.10 – 7.76) and 7.15 (7.00 – 7.30), respectively. The only significant difference was in the partial group, between the before-EHR and after-EHR, p = .02. There were no other significant differences between partial groups and there were no significant differences between complete groups. Twenty-nine patients (5%) were missing all three A1c values. The notes missing an A1c value (including the patients missing all three values) decreased from 42% before-EHR and after-EHR to 16% five-years (p < .001).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2 Mean (CI) and A1c values by tertile

The A1c tertiles are shown in Figure 2. The lowest tertile’s before-EHR, after-EHR and 5-year mean A1c values were 5.93 (5.88 – 5.99), 6.07 (6.00 – 6.13) and 5.98 (5.92 – 6.03), respectively. There was a small but significant increase in A1c values between before-EHR and after-EHR, p = 0.001, and there was a small but significant decrease in A1c values between after-EHR and five-years, p = 0.04, but there was no significant difference between before-EHR and five-years. The middle tertile’s before-EHR, after-EHR and five-year mean A1c values were 6.77 (6.72 – 6.82), 6.92 (6.88 – 6.97) and 6.82 (6.78 – 6.86), respectively. There was a small but significant increase in A1c values between before-EHR and after-EHR, p < 0.001, and there was a small but significant decrease in A1c values between after-EHR and five-years, p = 0.001, but there was no significant difference between before-EHR and five-years. The highest tertile’s before- EHR, after-EHR and five-year mean A1c values were 8.55 (8.26 – 8.84), 9.09 (8.71 – 9.49) and 8.9 (8.4 – 9.4), respectively. There was a significant increase in A1c values between before-EHR and after-EHR, p = 0.03, and there was a significant decrease in A1c values between after-EHR and five-years, p = 0.049, but there was no significant difference between before-EHR and five-years.

We expected to observe a significant decrease in A1c values primarily in the highest tertile, over five years of EHR use. We followed the same diabetic patients for five years after the introduction of the EHR, but we did not find a significant decrease in A1c values. In other words, based on their A1c values, the EHR use did not improve the clinical quality of diabetic care in six months and five years after EHR adoption, either across all patients or in the highest A1c tertile. Missing A1c values did not affect these results. We did find that the EHR significantly reduced the number of missing A1c values.

The increase and then decrease in mean A1c values observed at the before and after time points are consistent with complexity theory, which states that during the process of moving from one fitness peak to another, an organization experiences a decrease in performance that can last as long as a year.18 The finding that missing A1c values did not affect the A1c results suggests that the existence or absence of an A1c value is not a measure of clinical quality. The observation that EHRs reduced the frequency of missing A1c values can be explained by either increased testing, resulting in increased test results available for documentation, or the increased documentation of test results. Since there was usually only a slight increase, usually 3% – 4%, in A1c testing attributable to EHR use,19 the reduction in missing A1c values was probably due to the increased documentation engendered by the automated population of test results.

The observed inability of EHRs to improve clinical quality is congruent with physician perceptions. A 2014 national survey of 967 practicing physicians found that only 35% (CI = 3.2%) believed that their EHR had significantly or somewhat improved the quality of their patient care and 65% believed that either it had not changed or it had made worse the quality of their patient care.20

No randomized clinical trial has assessed the ability of EHRs to improve medical care and it is unlikely that one will be conducted since it would require half of the physicians in the study to return to handwritten notes. The relationship between EHRs and clinical quality has been investigated in cross-sectional studies,3–10 in studies that used claims data5, in studies that contained process measures3,5,6,8–10 and in studies that assessed the ability of alerts and reminders to improve outcomes.11,12 The results have been mixed. No study has demonstrated a benefit across all its measures. Some studies have shown a partial benefit,6,7,10,11 while others have not demonstrated that EHRs have a significant clinical impact.3–5,8,9,12 In a recent examination of the impact of the patient centred medical home (PCMH) on ten Healthcare Effectiveness Data and Information Set measures over one year, investigators found that the odds of receiving recommended care in the PCMH group were 7% higher for the physicians who had wrote their notes but only 6% higher for those physicians who used an EHR.20 There have been calls for longitudinal studies,10 but, to our knowledge, none has been performed using a direct measure of clinical quality over an appropriate time period.

The American Recovery and Reinvestment Act of 200921,22 has increased physician adoption of EHRs; the Centers for Disease Control and Prevention reported that 71.8% of office-based physicians were using an EHR in 2012.23 But the electronic medical record systems may not, by themselves, be able to improve the quality of care. A recent RAND report stated that, ‘the current state of EHR technology significantly worsened professional satisfaction in multiple ways. Poor EHR usability, time-consuming data entry, interference with face-to-face patient care, inefficient and less fulfilling work content, inability to exchange health information between EHR products, and degradation of clinical documentation were prominent sources of professional dissatisfaction’.24 Finally, shortcomings in the design and implementation of health information technology systems have caused physicians to complain that current EHRs do not deliver sufficient clinical value to compensate for their difficulty and expense.20 A recent review concluded that we should rethink the definition of meaningful use, reduce EHR difficulty and improve their clinical utility.25 In other words, EHRs may be necessary but not sufficient for increasing the quality of medical care. Clinical decision support systems, which are federally mandated as part of the meaningful use, have potential to add significant clinical value to EHRs.26

A study limitation is that it only assessed one diabetic quality measure, namely, A1c. But this measure has been used extensively to measure clinical quality3,5,8,10,11,27 and to assess EHR use.13

Conclusions

In summary, we assessed the clinical justification for the adoption of the EHR, namely, that it would improve the quality of medical care. In this five-and-one-half-year multicentre, longitudinal retrospective study did not find an improvement in clinical quality, as measured by an improvement in A1c values in six months and five years after the adoption of an EHR. We believe that further functionality must be added to EHRs, for example, clinical decision support systems, if they are to improve the quality of care. To our knowledge, this is the first longitudinal study to directly assess the relationship between the use of an EHR and clinical quality.