Original Research••

Evaluating a post-implementation electronic medical record training intervention for diabetes management in primary care

Abstract

Objective This study evaluated the potential for electronic medical record (EMR) video tutorials to improve diabetes (type 1 and 2) care processes by primary care physicians (PCP) using OSCAR EMR.

Design A QUAN(qual) mixed methods approach with an embedded design was used for the overall research study. EMR video tutorials were developed based on the chronic care model (CCM), value-adding EMR use, best practice guidelines for designing software video tutorials and clinician-led EMR training.

Results In total, 18 PCPs from British Columbia, Canada, participated in the study. The video EMR intervention elicited a statistically significant increase in EMR advanced feature use for diabetes care, with a large effect size (ie, F(1,51)=6.808, p<0.001, partial η2=0.286).

Conclusion This small-scale efficacy study demonstrates the potential of CCM-based EMR video tutorials to improve EMR use for chronic diseases, such as diabetes. A larger-scale effectiveness study with a control group is needed to further validate the study findings and determine their generalisability. The demonstrated efficacy of the intervention suggests that EMR video tutorials may be a cost-effective, sustainable and scalable strategy for supporting EMR optimisation and the continuous learning and development of PCPs. Health informatics practitioners may develop video tutorials for their respective EMR/electronic health record software based on theory and best practices for video tutorial design. For patients, EMR video tutorials may lead to improved tracking of processes of care for diabetes, and potentially other chronic conditions.

Summary

What is already known?

  • Current electronic medical record (EMR) use by primary care physicians (PCPs) is suboptimal, especially for supporting chronic care in Canada and the USA.

  • Many PCPs receive little effective or adequate end-user-support to use the advanced features of their EMR.

  • Primary research studies on the use of video tutorials for electronic health record/EMR training of PCPs or other practicing physicians have not been published to date.

What are the new findings?

  • This study has examined the potential of EMR video tutorials that are based on the chronic care model, clinician-led training and best practices for designing software video tutorials.

  • This small-scale efficacy study demonstrates the potential of EMR video tutorials to improve EMR use for chronic diseases, such as diabetes.

How might this impact on clinical practice?

  • The study identifies a potential low cost way to influence user behaviour and clinical practice following EMR implementation.

  • Through use of EMR video tutorials for PCPs, it is anticipated that processes of care and clinical indicators for chronic diseases such as diabetes may improve.

Introduction

According to the International Diabetes Federation, diabetes is one of the most challenging health problems in the twenty-first century.1 Currently, there are big, persistent gaps between the evidence-based guidelines for diabetes care and actual clinical practice around the world,2 including Canada.3

Almost 80% of diabetes care is provided in primary care by primary care physicians (PCPs).3 To bridge the gaps between current practices and optimal standards, the redesign of primary care has been proposed4 using the chronic care model (CCM).5 The CCM is ‘the best known and most influential’ organisational model for chronic care.6 The overall aim of the CCM is to develop well-informed, activated patients interacting with a practice team that is proactive and prepared for them with the end goal of improving outcomes. The CCM has been adopted and adapted for use in different countries, such as the UK, Denmark, Russia, China, Australia, New Zealand and Canada.7 The CCM includes six elements that are inter-related and designed to strengthen the patient-provider relationship and improve health outcomes: (1) delivery systems design, (2) self-management support, (3) decision support, (4) clinical information systems, (5) the community and (6) health systems. According to the CCM, clinical information systems such as an electronic medical record (EMR), and electronic health record (EHR) are often used interchangeably. In Canada, where this study has been conducted, an EMR is a health record under the custodianship of PCPs, whereas an EHR is used in secondary and tertiary care (hospital) settings. They can play a key role in facilitating improved capture, organisation and presentation of patient information. For diabetes care, EMRs can help PCPs to (1) identify patients with diabetes, (2) assess whether a patient is due for recommended tests or screening procedures and (3) determine which patients have not achieved evidence-based clinical goals for key measures.8

In general, current EMR use by PCPs is suboptimal, especially for supporting chronic care in Canada and the USA.9 10 Previous research highlights the need to support PCPs in the advanced use of their EMRs for diabetes care.11 12 This is referred to as ‘value-adding use’ and includes additional use by the user to increase output or effectiveness.13 The literature widely suggests that end-user-support (EUS) is a critical success factor for increasing value-adding EMR use.14 15 EUS is defined as ‘any information or activity that is intended to help physicians solve problems with, and better utilise, the system’.16 However, many PCPs receive little effective17 or adequate EUS,17 18 especially following the implementation of an EMR. In particular, post-implementation EMR training is an important form of EUS that the majority of PCPs currently lack, especially for using advanced features such as creating recalls/reminders and reports.19–21 Training provides end-users with ‘the conceptual and procedural knowledge necessary to put the technology to effective use’.22

The literature on educational interventions for information systems identifies seven methods of software training: (1) tutorial, (2) course/lecture/seminar, (3) computer-aided instruction, (4) interactive training manual, (5) resident expert, (6) help component and (7) external training.23

In the 1990's, video tutorials were introduced as a new type of tutorial for training end users of information systems. Video tutorials are visual demonstrations of how to accomplish tasks using software,24 and are a simple, affordable tool to produce authentic, situated and motivational instructional material.25 The benefits of video tutorials for learning include: (1) The development of a better mental model of the software26, (2) Faster initial learning and better comfort than using static versions of instructions25 and (3) Increased control and autonomy.25

However, the limitations of using video tutorials include the ‘mimicry model’ (ie, memorising and copying steps without internalising the task)27 28 and lack of an inferential step, which may lower retention of information. Despite these challenges, video tutorials can reduce cognitive processing and allow users to immediately practice the skills they have acquired.25 They may also serve as a cost-effective and scalable tool for software vendors and institutions to provide training to end-users.

In the context of training medical residents and nursing students on EMRs, He et al from the University of Massachusetts (USA) had used two video tutorials in EHR training for nursing students.29 The authors identified several changes to improve their video tutorials (eg, including hands-on practices and review questions with answers). In another study, Thiyagarajan et al30 from Eastern Virginia Medical School (USA) examined the use of EHR video tutorials to orient medical students to the EHR.30 Using retrospective chart review, the authors found that there was a significant improvement in completion of the past medical history and smoking status fields in the EHR after three video tutorials were introduced, although it was not possible to link the post-implementation charts with the medical students who had watched the video tutorials.30 Similarly, Zoghbi et al31 conducted a pre-post study to examine the effects of video tutorials on EMR use for a group of general surgery residents at the Perelman School of Medicine, University of Pennsylvania (USA). The study found statistically significant positive effects of the video tutorials on residents' confidence in carrying out EMR tasks, clinical scores on emergency simulations, as well as decreasing their time required to perform essential EMR tasks without the video tutorial training. However, primary research studies on the use of video tutorials for EHR/EMR training of practicing PCPs or other practicing physicians have not been published to date. Further, to our knowledge, research examining the use of EMR video tutorials has not been published to date outside of the USA. This study adds to this literature by examining the impact of video tutorials on PCPs’ use of advanced EMR features for diabetes care as a continuing medical education-like intervention; it also studies the use of video tutorials by PCPs in Canada. The study addressed the following research questions:

  1. To what extent does a CCM-based video tutorials demonstrate the potential to improve type 1 and type 2 diabetes care processes, including (a) use of a diabetes registry, (b) use of diabetes recalls/reminders, (c) ordering/viewing a patient's haemoglobin A1c every 3 to 6 months and (d) recording a patient's blood pressure every 3 to 6 months?

  2. To what extent do individual PCP characteristics relate to EMR use for diabetes care?

Methods

Overall research design

A QUAN(qual) mixed methods approach with an embedded design was used for the overall research study. In this paper, we only report the findings from the quantitative, experimental portion of the study as the qualitative portion focused on barriers and facilitators and video intervention design. For this part, a quasi-experimental design, with a one-group pre-test post-test using a double pre-test and an additional post-intervention measurement was conducted. Data were collected 1 month before, immediately before, 3 months after and 6 months after the intervention described below.

Setting and sample

The study was set in British Columbia (BC), Canada. Study participants included PCPs who use the OSCAR EMR, which is one of the major EMRs used in BC. An a-priori sample size calculation in the G*Power computer program32 indicated that a total sample of 24 participants would be needed to detect medium effects (f(U)=0.25) with 80% power using repeated measures analysis of variance (rANOVA) with alpha at 0.05. Given the barriers to engaging PCPs in research (especially lack of funding to compensate for physicians’ time), the study used a convenience sample. All PCPs in BC who use the OSCAR EMR (n=984) were invited to participate in the study. This potential sample was identified by a physician EMR champion in BC. Study participants were recruited through the Divisions of Family Practice and OSCAR BC Users’ Group using an invitation letter and a YouTube video that explained the research study.33 The Divisions of Family Practice advertised the study opportunity to their division members through emails, newsletters, division meetings and word of mouth. Eligible PCPs who expressed interest in participating in the study signed and submitted an informed consent form through email or fax. All PCPs who returned signed consents and met the study inclusion criteria were included in the study. PCPs were included if they were: (a) a service family physician (full-service family physician), (b) a member of a Division of Family Practice, (c) used the OSCAR EMR, (d) worked in a solo or group practice, (e) had diabetic patients in their practice and (f) were interested in implementing and applying the intervention in their practice, including using the advanced features of their EMR for diabetes care management. Study recruitment took place between July 2017 and October 2017.

Intervention

The study intervention, referred to as Management of Diabetes Post-Implementation EMR Training (MD-PET), was composed of a series of four short online video tutorials34 that targeted four EMR features for diabetes care, similar to the EMR use instructions and applications recently published by Singer and Ivers.35 In general, OSCAR EMR had been voluntarily described as ‘not user-friendly’ by several study participants during the course of the study. The OSCAR EMR functionality studied is considered hard/difficult for users to access, requiring use of multiple screens in different areas of the EMR. Further, OSCAR EMR exemplifies the current poor availability of EMR documentation support tools36 that are designed to achieve high EMR data quality. Hence, this functionality indicates an EMR usage maturity level of four on a five-level model developed by the Physician Information Technology Office in BC,37 and is considered advanced in Canada, as many physicians currently use their EMR as ‘electronic paper’ for free-text charting.35 For this reason, many PCPs need special training to learn how to use chronic disease registries, recalls and reminders to measure and follow guideline-informed care.

Specific learning contents covered in the intervention included how to: (1) code diabetic patients to create a diabetes registry and add the diabetic label to the cumulative patient profile, (2) create diabetes recalls using OscarMsg (internal email) and using Ticklers (scheduled reminders), (3) add diabetes-related indicators to the diabetes flowsheet and (4) order lab work using a smart lab requisition in the patient’s eForm library. It should be noted that the OSCAR EMR does not provide automated computer-generated reminders (or other computer aids) for the aforementioned functions; these must be created manually in the system. Although this advanced functionality requires access to many different parts of the EMR, the same method can be used for other chronic diseases. Thus, the training was designed to teach transferable skills that can be applied to multiple chronic diseases. The video tutorials were designed based on:

  1. The CCM, which served as the conceptual framework for the EMR training intervention. The intervention focused on the ‘clinical information systems’ and ‘delivery systems design’ components of the CCM.

  2. van der Meij and van der Meij's eight guidelines for designing video tutorials for software applications,38 namely (1) provide easy access, (2) use animation with narration, (3) enable function interactivity, (4) preview the task,5 provide procedural rather than conceptual information, (6) make tasks clear and simple, (7) keep videos short and (8) strengthen demonstration with practice.38

  3. Clinician-led EMR training, which aligns with the homophily characteristic of EUS and learning through modelling, as per Bandura's Social Learning Theory.39 Homophily in EUS is ‘the degree of similarity between the support source and the end-user, and the degree to which the support source demonstrates understanding of the day-to-day work of the user’.16 To that end, a physician champion was involved in the design and recording of the video tutorials.

Data collection

Data were collected from July 2017 to May 2018. To answer research questions 1 and 3, a demographic survey was conducted 1 month prior, and a Diabetes Care Questionnaire (DCQ) (See Appendix) was completed at all four time points. The DCQ measures the percentage of patients for which a PCP has (a) used a diabetes registry, (b) used diabetes recalls/reminders, (c) ordered/viewed haemoglobin A1c every 3 to 6 months and (d) recorded a patient’s blood pressure every 3 to 6 months. The process measures in the DCQ were combined to create a composite variable40 called mean composite EMR use (MCEU) at each time point by calculating the average score of the above measures a to d. The composite variable had good reliability (Cronbach’s α=0.808).

Data analysis

Descriptive statistics (frequencies) from the PCP Demographic Survey and DCQ were used to describe the participant characteristics. The means, medians, modes, ranges and SD were also calculated for data from the DCQ.

To study the relationships between various PCP characteristics, Kendall’s tau-b correlation coefficients were calculated. To examine relationships between PCP characteristics and PCPs' baseline use of the EMR for diabetes care, Pearson’s and Kendall’s tau-b correlation coefficients were used for interval and ordinal variables, respectively.

Differences in baseline use of the EMR between male and female participants, health authorities and levels of prior post-implementation EMR training of PCPs were tested using independent sample t-tests. To further examine the effects of PCP characteristics on PCPs’ use of EMR features for diabetes care at baseline, multiple linear regression was used. To decrease multicollinearity, only selected PCP characteristics were included in the linear regression model if they correlated with other PCP characteristics. Finally, rANOVA with post-hoc tests using the Bonferroni correction was employed to determine the difference between the pre-intervention and post-intervention observations. Quantitative data analysis was performed using IBM SPSS Statistics V.24.41

Results

Participants

Eighteen PCPs participated in the study, and their characteristics are outlined in table 1 below. The majority of participants were women (n=12; 67%), 45 to 64 years old (n=14; 78%), from Vancouver Island, BC, (n=12; 67%), worked in a multi-physician clinic (n=16; 89%) and had been practicing medicine for over 20 years (n=12; 67%). Almost half (n=8; 44%) of participants had used OSCAR EMR for 5 to 9 years, while 39% (n=7) had used it for 3 to 4 years. Most participants (n=7; 38%) had experience using two EMRs in total, while a third (n=6; 33%) of participants had only used one EMR (ie, OSCAR). The average number of EMRs used was two.

Table 1
|
Participant characteristics

Effects of physician characteristics on EMR use for diabetes care at baseline

A multiple regression was run to predict EMR use for diabetes care from three variables: years of practice, EMR skills and prior post-implementation training. Interval variables were included if they showed good correlation with MCEU at 1 month prior, while categorical variables were included if there were significant difference in MCEU between groups or a large effect size. Interval variables were excluded if there was small correlation with MCEU at 1 month prior, whereas categorical variables were excluded if there were no difference between groups in MCEU at 1 month prior. For this purpose, we used Cohen’s conventions of 0.1, 0.3 and 0.5 for small, medium and large correlation, respectively.42 Because years of practice was correlated with age (τb=0.658, p<0.01) and EMR skills was correlated with computer skills (τb=0.702, p<0.01), only years of practice and EMR skills were included in the regression model. Prior post-implementation training was included because of its large effect size (d=0.65). The multiple regression model significantly predicted EMR use for diabetes care, F3 14=3.80, p=0.04, R2=0.45. Only years of practice contributed statistically significantly to the prediction, p=0.05. Regression coefficients and SEs are presented in the table 2 below.

Table 2
|
Summary of multiple linear regression

Effects of the intervention on EMR use for diabetes care

MCEU scores for diabetes care increased from baseline (M=68.78, SD=14.05) to immediately before (M=72.33, SD=11.25), 3 months after (M=79.44, SD=11.66) and 6 months after (M=80.17, SD=14.39), in that order (figure 1). The rANOVA showed that there were statistically significant differences in use of diabetes-related EMR features over time, F(3,51)=6.808, p<0.001, with large effect size (partial η2=0.28642 43).

Figure 1
Request permissionrequest permission icon
Figure 1

EMR use for diabetes care across data collection time points with 95% CI. EMR, electronic medical record.

As indicated in figure 1, only the differences in MCEU scores between baseline and 3 months after and baseline and 6 months after were statistically significant in post-hoc tests. (95% CI 19.69 to 1.65, p=0.015) and 95% CI 24.09 to 1.37, p=0.021, respectively).

It is also valuable to view the trends in the data across process measures both in tabular and graphical form. The means, medians, modes, SD and ranges for the individual process measures, which make up the MCEU score, across the study time points can be seen in tables 3–11, and the graphs can be seen in figure 2. In the tables and graphs, it is interesting to note the increase in mean use of EMR features over time. In general, several process measures (ie, use of the diabetes-related EMR features) increase in a positive direction from baseline to 6 months after. Specifically, the trends suggest a marked increase in PCPs' (a) use of a diabetes registry, (b) creation of recalls and (c) ordering of lab work for both stable and unstable patients with diabetes. However, PCPs’ ordering of lab work for patients with diabetes appeared to slightly decrease in the present study from baseline to 6 months post-intervention.

Table 3
|
Primary care physicians’ average use of registry for all patients with diabetes over time
Table 4
|
Primary care physicians’ average use of registry for stable diabetic patients over time
Table 5
|
Primary care physicians’ average use of registry for unstable patients with diabetes over time
Table 6
|
Primary care physicians’ average use of recalls for stable patients with diabetes over time
Table 7
|
Primary care physicians' average use of recalls for unstable patients with diabetes over time
Table 8
|
Primary care physicians' ordering of lab work for stable patients with diabetes over time
Table 9
|
Primary care physicians' ordering of lab work for unstable patients with diabetes over time
Table 10
|
Primary care physicians' recording of blood pressure for stable patients with diabetes over time
Table 11
|
Primary care physicians' recording of blood pressure for unstable patients with diabetes over time
Figure 2
Request permissionrequest permission icon
Figure 2

Use of individual electronicmedical record features for diabetes care over time. BP, blood pressure.

Discussion

CCM-based video tutorials were investigated to determine their potential to improve processes of care for type 1 and type 2 diabetes. Although video tutorials have recently been applied in a few studies to support EHR training for medical students, medical residents and nursing students,29–31 their potential to support PCPs or practicing physicians in general has not been widely investigated. Further, the potential of EMR video tutorials that are based on the CCM, clinician-led training and best practices for designing software video tutorials has rarely been examined. Yet, given that video tutorials have been around for almost 30 years, their general use is not entirely new to PCPs, especially with the emergence of platforms such as YouTube. For example, a couple of PCPs reported watching video tutorials to learn how to fix a printer or bake a cake. However, the use of EMR video tutorials to teach EMR functionality to PCPs may be a newer development, as PCPs often receive pre-implementation and post-implementation training in-person18 20 44 through educational interventions, such as classroom-based training.

Previous studies by Thiyagarajan et al30 and Zoghbi et al31 report positive effects of video tutorials for medical students and medical residents; however, the authors used different study designs: a retrospective chart review and a single-institution prospective study, respectively, and did not report the effect size of their video tutorial interventions. In contrast, the quasi-experimental approach of the efficacy study presented here has allowed for a more detailed examination of the video tutorials intervention (MD-PET).

The present study demonstrates a statistically significant increase in EMR features use for diabetes care, with a large effect size. Graphical trends also indicated a positive increase in PCPs' use of almost all the EMR features for diabetes care. Ordering lab work is the EMR feature that appears to have decreased from baseline to 6 months after. External programme and initiatives designed to reduce unnecessary lab testing for haemoglobin A1c (eg, Choosing Wisely Canada) may explain the reduced use of ordering lab work for patients with diabetes. However, this requires further study. The present study suggests that EMR video tutorials may help to improve diabetes care for patients with diabetes, including (a) use of a diabetes registry, (b) use of diabetes recalls/reminders, (c) ordering/viewing a patient's haemoglobin A1c every 3 to 6 months and (d) recording a patient's blood pressure every 3 to 6 months. However, there was an increase in MCEU from the first pre-test at baseline to the second pre-test (immediately prior), suggesting a possible observer (Hawthorne) effect.45 There was also an additional increase in MCEU following the intervention (3 and 6 months after), which was significant compared with baseline but not to the immediately prior measurement. This may mean that although there is an added value to the use of video tutorials (ie, they are necessary as a training intervention), video tutorials alone may not be sufficient to support value-adding EMR use.

The role of the CCM (ie, the model that underpins the MD-PET intervention) in supporting PCPs' change in MCEU scores for diabetes care over time is also of interest in the present study. The CCM brings together two professional interventions that have been found to be effective in changing healthcare professionals' behaviour1: audit and feedback and2 reminders.46 In this present study, the MD-PET intervention teaches PCPs how to create a diabetes patient registry, which serves as an ‘audit and feedback’ tool, as the diabetes registry provides a ‘summary of clinical performance of healthcare over a specified period of time’ and is obtained from the patient's medical record.46 Similarly, the MD-PET intervention instructs PCPs how to create recalls/reminders for diabetes care, which serves as ‘specific information designed or intended to prompt a health professional to recall information or perform or avoid some action to aid individual patient care’.46 Johnson and May's systematic review46 suggests that interventions based on action (ie, audit and feedback, as well as reminders) are more likely to change healthcare professional behaviour. In this way, the action-oriented topics and skills taught in the intervention itself may have influenced the positive results in the present study.

A combination of video tutorials and audit and feedback could be more effective to support improvement in MCEU. This is supported by the systematic review by Renders et al,47 which found that multi-faceted interventions for healthcare professionals (as observed in the present study) facilitate the structured and regular review of diabetes processes of care. As per Ivers et al's48 meta-regression and cumulative analysis of audit and feedback interventions in healthcare, audit and feedback interventions can be more effective if they (a) are delivered by a supervisor or respected colleague, (b) are presented frequently, (c) feature specific goals and action plans, (d) aim to decrease the targeted behaviour and (e) focus on a problem where there is lower baseline performance. These are important and practical elements to consider in designing audit and feedback and video tutorials for EMR use that can supplement each other.

It should also be noted that the EMR features studied here may already be routine practices in countries such as the UK, which established routine chronic care and facilitated EMR system design support after the introduction of the Quality and Outcomes Framework in 2004. However, with the ongoing concerns surrounding the usability of EMRs in other countries like the USA and Canada,49–51 there is a need for immediate, interim support and training to use existing EMR systems while health informatics researchers, practitioners and vendors work on improving the usability of current and future EMRs. EMR video tutorials may be one educational intervention to support this effort.

For policymakers, the demonstrated efficacy of MD-PET suggests that EMR video tutorials may be a cost-effective, sustainable and scalable strategy for supporting EMR optimisation. However, this will require further research. For physicians, this study has demonstrated the efficacy of an emerging learning strategy to support the continuous learning and development of PCPs. Vendors may develop video tutorials for their respective EMR/EHR software based on theory and best practices for video tutorials’ design. For patients, EMR video tutorials may lead to improved tracking of processes of care for diabetes, which may have some effects on patient outcomes. However, although EMRs have significant potential to directly improve care processes and their measures (eg, PCP's recording and tracking of a blood pressure in the EMR) for chronic diseases, this does not necessarily suggest that the EMR directly influences outcomes for chronic diseases, including intermediate or surrogate measures (eg, haemoglobin A1c). It is important to note that the relationship between healthcare processes and patient outcomes is complex and likely non-linear. As Donabedien52 suggests, improvements in process measures do not necessarily result in improved patient outcomes, and vice versa, as there may be many potential intervening variables. This may explain the mixed effects of EMRs on care quality that are currently found in the literature.53 54 Similarly, studies examining the effects of EMRs on process and outcome measures for diabetes care are also inconclusive.55 56 However, given the positive empirical findings surrounding the effects of previous CCM-based interventions on process and outcome measures for diabetes care,5 47 the present study demonstrates how an educational EMR intervention can positively affect the process of diabetes care.

Limitations and directions for future research

The study was a within-group quasi-experiment without a control group. It employed a small convenience sample, had a short post-intervention follow-up period and relied on self-reported measures of EMR features use. Further, the study examined the performance of PCPs who had volunteered for the study and likely knew the value of the video tutorials being assessed. As such, this may be a considerable motivation for PCPs to learn from the video tutorials and change their EMR use behaviour. Hence, the study results should be seen in this context and may not be applicable to everyday practice.

While the study demonstrates the potential of video tutorials to improve processes of diabetes care, more research is needed to ascertain the effectiveness of video tutorials for improving EMR use. In the future, a larger-scale effectiveness study should be conducted using objective measures of EMR usage data and a control group to ensure the validity of the study findings, as well as to determine their generalisability. Measurement of changes in non-diabetes chronic care may also be a useful control to examine any observed changes relate to better use of the OSCAR system or more motivation to complete the diabetes management tasks as a result of participation.

Overview

Article metrics
Altmetricopen-url
Dimensionsopen-url