You are here

Article Text

Article menu
Original research
Interventions to improve the use of EMRs in primary health care: a systematic review and meta-analysis
  1. Noura Hamade1,
  2. Amanda Terry2 and
  3. Monali Malvankar-Mehta3
  1. 1Centre for Global Health, Dalla Lana School of Public Health, Toronto, Ontario, Canada
  2. 2Centre for Studies in Family Medicine, Department of Family Medicine, Schulich Interfaculty Program in Public Health, Department of Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada
  3. 3Department of Ophthalmology, Department of Epidemiology and Biostatistics, Schulich School of Medicine & Dentistry, The University of Western Ontario, London, Ontario, Canada
  1. Correspondence to Noura Hamade; nourahamade{at}gmail.com

Abstract

Background Electronic medical record (EMR) adoption in primary care has grown exponentially since their introduction in the 1970s. However, without their proper use benefits cannot be achieved. This includes: 1) the complete and safe documentation of patient information; 2) improved coordination of care; 3) reduced errors and 4) more involved patients. The use of EMRs is defined by practitioners using EMRs and their features to perform daily practice functions.

Objective The purpose of this systematic review was to identify interventions aimed at improving EMR use in primary healthcare settings.

Methods Ten online databases were searched to identify studies conducted in primary healthcare settings aimed at implementing interventions to observe the use of EMRs and directly measure the use of EMR functions or outcomes effected by the use of EMR functions.

Results Of 2098 identified studies, 12 were included in the review. Results showed that interventions focused on the use of EMR functions, including referrals, electronic communication, reminders, use of clinical decision support systems and workflow management support functions, were five times more likely to show improvements in EMR use compared with controls. Interventions focused on data quality were five and a half times more likely to show improvements in EMR use compared with controls.

Conclusions Individuals in primary healthcare settings aiming to improve EMR use would benefit from implementing interventions focused on EMR feature add-ons such as clinical decision support systems and customised referral templates, and provisions of educational materials, or financial incentives targeted at improving the use of EMR functions and data quality.

  • electronic medical records
  • primary healthcare
  • benefits realisation
  • health information technologies
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjhci-2019-000023

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Summary

    What is already known?

    • The electronic medical record (EMR) system was developed originally in the early 1970’s as a means to store patient health information. Over time, and with the improvements in technology, EMRs are now capable of using stored patient health information to assist in the daily care provisions primary health care personnel provide to patients. This is done with the hopes of improving patient health care through creating higher quality patient data and improving primary health care center processes.

    • However, whether EMR use has been successful in improving the provision of patient care has not been made clear in the literature. Due to the importance of improving the use of EMRs with regard to patient outcomes, there has been recent interest on the part of organizations and governments to provide guidelines to improve EMR use. Improving EMR use requires targeted interventions aimed at the areas in which EMRs were created to function. There have only been a few studies in the literature that have been aimed at studying the effect of certain types of intervention on EMR use.

    What does this paper add?

    • However, due to the high heterogeneity between those studies a meaningful meta-analysis was not feasible until this point where the synthesis of the results was possible through: the categorization of interventions using the EPOC taxonomy of interventions and the identification of possible intervention target areas to improve EMR use. This allowed for the meaningful grouping of the studies resulting in the ability to conduct the first meta-analysis in this field. This increases the power of the results and conclusions drawn from those results giving us a better understanding of the types of interventions that could be used to improve the use of EMRs.

    Introduction

    The past few decades have seen an expansion in the role of technology in healthcare reflected in the introduction of information technologies into the healthcare system. Electronic medical records (EMRs) are one form of information technology which can impact patient health outcomes.1 2 EMRs are computerised patient records introduced in the early 1970s as a way to organise, secure, complete and improve the quality of patient healthcare records.3 Around the turn of the century, EMRs gained attention because of the perceived and expected benefits they could offer the healthcare system which included: organisation of patient healthcare information, improving coordination of care as well as easier electronic access to medical information and expert opinion.4 5 With their creation and introduction into primary healthcare, EMRs were expected to have a positive impact on the quality of healthcare. This was expected to be realised through the use of EMRs to improve data quality through the recording of patient information and perform primary healthcare functions. This drove organisations and governments to create programme to promote the adoption of EMRs into the healthcare system.4

    The distinction between EMR adoption and use is not clearly defined in the literature. However, for the purposes of this review, adoption of EMRs is defined as simply the introduction of EMRs into primary healthcare practice. The use of EMRs is the second step following adoption, involving practitioners using EMRs and their features to perform daily primary healthcare practice functions. A national survey in 2015 showed that the adoption of EMRs into primary healthcare practices is on the rise in Canada while EMR use is still low in comparison.6 7 The US Centers for Medicare and Medicaid Services (CMS) defined meaningful use as: ‘Using (EMRs) to: improve quality, safety, efficiency, and reduce health disparities. Engage patients and family. Improve care coordination, and population and public health. Maintain privacy and security of patient health information’.8 For the purposes of this review, improved EMR use is defined as using EMRs according to the above definition. Some studies suggest that to achieve noticeable improvements in patient health outcomes following adoption, improving the use of EMRs is necessary.9–12 Therefore, improving the use of EMRs to achieve desirable health outcomes has attracted recent attention.13 The mechanisms to improving EMR use however, have not yet been determined. Therefore, this systematic review aims to review the literature to identify interventions and their effect on improving EMR use in primary healthcare settings in hopes of bridging the gap between adoption and use of EMRs.

    Methods

    Search strategy

    The search strategy was developed with limits to only include studies with human subjects conducted after 1970, the following databases were searched: MEDLINE Ovid, EMBASE Ovid, CINAHL, Cochrane Library and Web of Science. The grey literature was also searched through the following databases: Clinical Trials, NDLTD, CADTH, International Clinical Trials Registry, AHRQ. Finally, after applying the search strategy to all the mentioned databases and collecting the identified studies, snowballing was used as a supplementary search strategy. The full search strategy for all databases is listed in online supplementary appendix A.

    Supplemental material

    Inclusion criteria

    Research articles conducted after 1970 and published in the English language regardless of the location of the study. In addition, the following eligibility criteria were used to identify studies for inclusion:

    1. Study focus: included studies were those that specifically focused on actual use of EMRs in primary healthcare, not simply earlier stages of implementation.

    2. Intervention: studies with a clear intervention that was implemented or observed for the purpose of studying use or use patterns of EMRs were included.

    3. Setting: included studies were only those conducted in a primary healthcare setting.

    4. Outcome of interest: included studies had to have an outcome of interest related to measurements of use of EMR functions (number of uses, duration of use) as well as outcomes effected by EMR use such as number of referrals and completeness of patient records.

    No restrictions based on study design or comparator groups were used. Opinion pieces, editorials and publications without an abstract were excluded, along with conference abstracts. After conducting the database searches, EPPI Reviewer V.4.0 (by EPPI-Centre, Social Science Research Unit, the Institute of Education, the University of London, UK) was used to automatically remove duplicates; subsequently, a manual search was conducted to remove any missed duplicates.14 The abstracts were screened independently by two reviewers, NH and MH, based on a list of screening questions derived from the eligibility criteria described above. Two reviewers, NH and AT, then independently conducted the full-text screening of the included studies. All screening questions are listed in online supplementary appendix B.

    Supplemental material

    Data extraction

    The first author’s name, year of the study and setting (location and country) were extracted to be used as study citation information. Information on the study population and participant composition was also extracted. In addition, extracted from each study were: intervention name, intervention type and a brief description of the intervention. In terms of outcomes, the outcome measured and a description of the outcome were also extracted. Lastly, information was extracted to allow for assessment of individual study bias.

    Details of study interventions

    A system was adopted in this review to categorise the wide variety of possible interventions that could be implemented to improve EMR use. Interventions for this systematic review were categorised using the Effective Practice and Organisation of Care (EPOC) taxonomy of interventions, which was published in the Cochrane Review by the EPOC Group in 2015.15 Interventions included in this review were placed into one of the following categories and are shown in figure 1:

    Figure 1
    Figure 1

    Possible categories of interventions identified in this review. Recreated from: Effective Practice and Organisation of Care (EPOC), EPOC taxonomy, 2015. Available at: https://epoc.cochrane.org/epoc-taxonomy

    1. Professional interventions: defined by EPOC as an intervention implemented with the goal of educating or furthering the knowledge of the target group in a specific area with the purpose of creating change.

    2. Organisational interventions: defined by EPOC as interventions that target workflow, as well as those that create changes in an organisation’s framework. Therefore, included are any interventions that cause changes to the workflow of the primary healthcare practice through the healthcare professionals or structurally through changes to the EMR such as feature add-ons would be included.

    3. Financial interventions: according to the EPOC definition, interventions were considered to be financial interventions if they provided an incentive for action.

    Intervention target areas

    This study is the first of its kind in this area of research (the first meta-analysis to study the effect of interventions on the use of EMRs). Outcomes for this study were grouped into target areas identified from the definition of EMR use which focuses on the importance of practitioners using EMRs and their features to carry out daily practice function and to maintain a level of patient record quality. These outcomes were measured in a similar population with similar interventions at similar settings and due to those similarities were grouped into target areas using a technique aimed at combining units of the outcomes.16 17 This technique has been used to group complex interventions in the literature and was adopted to be used in this review.17 18 For the purposes of this study, the areas targeted for change were called target areas and were made up of the identified units based on the defined purpose of EMRs to create a logical and meaningful way to group the complex outcomes identified from the included studies. Two target areas were identified in the included studies:

    1. Use of EMR functions: describes the use of EMR functions directly in relation to duration and frequency of use. Examples of the functions include referrals, electronic communication, reminders triggered, use of clinical decision support systems as well as workflow management support functions.

    2. Data quality: studies that described the level of data completeness for basic patient information including diagnostic, laboratory and prescription management information were included in this group.

    The outcomes presented in the included studies were grouped by the target area of the intervention into either: 1) use of EMR functions or 2) data quality.

    Statistical analysis

    Using the above target areas as a guide, the effect sizes from individual studies were combined to create a summary effect size. Studies in which the interventions were aimed at the use of EMR functions all measured the use of those functions through reporting durations and frequency of use allowing for a meaningful summary effect estimate to be created. Similarly, studies with interventions aimed at improving data quality all described the levels of patient data completeness and were therefore grouped to create a single summary effect size. The outcome measures in each individual study were transformed into ORs to be included in the analysis using the data presented by the studies to generate 2-by-2 tables. Studies with multiple outcomes related to the same intervention target area were combined to be included into the analysis. Following that, the studies were separated by intervention target area into two different forest plots to create a meaningful meta-analysis. The statistical analysis including forest and funnel plots was completed using STATA V.13.0 (STATA, College Station, Texas, USA).19 All results were presented in forest plots and expressed in log-ORs because of the categorical nature of the outcomes of interest, using 95% CIs. Frequencies of outcomes along with the total number of participants were extracted. Some studies presented outcomes using regression coefficients. In this case, these coefficients were converted to ORs using the ln function.20 In addition, the ORs of the included studies were presented with their SEs in funnel plots to assess publication bias.

    The random-effects model was used to conduct the meta-analysis due to its ability to account for in between study variation that arises from differences in study target population, study intervention and presentation of outcomes. It does that by assuming the true effect estimate varies between studies. Therefore, the random-effects model using the DerSimonian and Laird methods was used in STATA to create the forest plots.21

    Risk of bias assessment

    The Downs and Black tool was used to assesses risk of bias for individual studies included in this review. It is made up of 27 questions divided into subsections based on reporting, external validity, internal validity (bias and confounding) and power.22 The breakdown of the four subsections and a brief explanation of their importance are listed in online supplementary appendix C. The Downs and Black assessment scale was applied to the selected studies to determine the reliability, validity and power of the study. The Downs and Black checklist for bias assessment is presented in online supplementary appendix D. Scores were then calculated and combined into a risk of bias bar graph, as suggested by the Cochrane Handbook for Systematic Reviews, used to indicate the level of bias in each study.16

    Supplemental material

    Supplemental material

    Results

    After searching the databases in October 2015, 2098 abstracts were identified. From these 2098 abstracts, 659 duplicates were removed. This left 1439 titles for abstract screening. Following abstract screening, 19 studies were identified for full-text screening. Full-text screening was performed on the 19 retrieved studies. Twelve were identified that fit the previously specified inclusion criteria.23–34 The PreferredReporting Items for Systematic Reviews and Meta-Analyses flow chart was used to map out the study selection process as shown in figure 2. 35All 12 studies identified in this review were identified from initial electronic database screening.

    Figure 2
    Figure 2

    Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart of study selection. EMR, electronic medical record.

    Study characteristics

    Of the identified studies more than half (n=7) were conducted in the USA.23 26–28 30 31 33 The remaining five were set in the UK (n=2),24 25 Finland (n=2)29 34 and Canada (n=1).32 All the included studies were conducted in a primary healthcare setting. In terms of study population size, the 12 included studies targeted 1564 primary care providers in 132 primary healthcare practices and included 578 071 patients. The characteristics of the included studies and the target population are listed in table 1. Of the 12 included studies, 4 were classified as purely organisational interventions27–30 and 5 were classified as professional interventions.23 24 32 34 The remaining three studies implemented mixed interventions, including at least two of the following: organisational, professional and financial.24 30 33 The studies, interventions and a brief description are listed in table 2. Eight of the included studies reported on the use of EMR functions, a more in-depth description of the EMR function and the outcome reported, along with an outcome measurement description is presented in table 3. All the studies in the data quality group focused on the effect of an intervention on improving the recorded data using an EMR.24–27 30 33 34 The completeness and accuracy of patient information are some of the markers used to measure data quality and were the areas most targeted by the included studies.35 A more in-depth description of the data quality area reported on and the outcome along with an outcome measurement description is presented in table 4.

    View this table:
    Table 1

    Study characteristics

    View this table:
    Table 2

    Interventions and intervention description

    View this table:
    Table 3

    Outcome measurement description of studies reporting on the use of EMR functions

    View this table:
    Table 4

    Outcome measurement description of studies reporting on data quality

    The study by de Lusignan et al24 was excluded from the meta-analysis because the results of the study were presented using regression coefficients and were not consistent with the other included studies.24 In addition, Kruse et al30 did not present preintervention scores and was therefore excluded from the meta-analysis.31 All other studies included in the analysis presented enough data to calculate ORs and SEs to assess publication bias.

    Two forest plots were generated by STATA using the ORs. The first forest plot represents studies focused on the use of EMR functions as the intervention target area, displayed in figure 3. The overall effect estimate was a significant log-odds of 1.66 (95% CI 1.43 to 1.88; absolute OR=e1.66=5.26, rounded to 5). Therefore, personal, organisational and financial interventions directed at the use of EMR functions had a significant and favourable effect on improving EMR use. More specifically, interventions targeted at the use of EMR functions were five times more likely to show improvements in EMR use compared with the controls (figure 3). The second forest plot represents studies focused on data quality as the intervention target area and is presented in figure 4. All the studies depicted in this forest plot favoured the intervention. The overall effect estimate was a significant log-odds of 1.71 (95% CI 0.01 to 3.41; absolute OR=e1.71=5.53, rounded to 5.5). Therefore, personal, organisational and financial interventions directed at data quality had a significant and favourable effect on improving EMR use five and a half times more than the controls.

    Figure 3
    Figure 3

    Log odds with associated 95% CIs showing the effect of interventions on use of electronic medical record functions.

    Figure 4
    Figure 4

    Log odds with associated 95% CIs showing the effect of interventions on data quality.

    To evaluate the publication bias, two separate funnel plots for the use of EMR functions and data quality were produced using STATA (see figures 5 and 6, respectively) and then visually assessed for symmetry. Both funnel plots showed that studies were clustered at the top with only one each at the base of the funnel plot. This asymmetry in the funnel plot could be the result of publication bias. However, due to the small number of studies it is difficult to confidently exclude the presence of publication bias.

    Figure 5
    Figure 5

    Funnel plot showing the spread of included studies targeted at use of electronic medical record.

    Figure 6
    Figure 6

    Funnel plot showing the spread of included studies targeted at data quality.

    The risk of bias for individual studies showed a high reporting of results scores and low internal validity scores. The mean score for risk of bias in individual studies is 64% with an IQR of 60%. This shows a moderate risk of bias in the included studies as expressed in figure 7.

    Figure 7
    Figure 7

    Risk of bias assessment of individual studies.

    Discussion

    The systematic review and meta-analysis were conducted to identify possible interventions focused on improving EMR use in primary healthcare settings. This review indicates that significant improvements in EMR use can be realised in primary healthcare settings where interventions targeting the use of EMR functions or data quality have been implemented. However, due to the possibility of publication bias, these results should be interpreted with caution. The findings of this review draw attention to four main themes in this area of study. Those themes are listed and discussed below.

    Number of identified studies

    In this review, only 12 studies of interventions focused on improving EMR use in primary healthcare were identified. Primary healthcare settings directly influence the majority of Canadians’ health outcomes.36 The importance of a well-functioning primary healthcare system was not reflected in the literature. Compared with the impact of this area on the health of the general population, the number of identified studies is surprisingly lacking. The deficiency in studies in the area of EMR use is possibly due to the focus in the field being on the adoption of EMRs. Even though studies have shown that adoption alone is not enough to access the EMR’s full potential, the shift to focus on improving EMR use is slow.8 11 12 36 In conclusion, one of the main hopes of this review is to draw attention to this gap in the literature. There should be a greater focus in the area of studies that can connect EMR availability to positive patient outcomes through improving EMR use with targeted interventions.

    Lack of consistency

    The area of EMR use is deficient in terms of available literature, and in the usability of this literature due to its lack of consistency in the information provided. Studies on the topic of EMR use vary in terms of interventions and approaches to assessing EMR use. Due to this being a relatively new field of study, there has been no standardisation of implementing interventions to improve EMR use established. This creates difficulties in synthesising those studies to create a useful meta-analysis.37 A standardised form of testing interventions to improve EMR use could create studies that are homogeneous enough to provide conclusions with greater power. In addition, there is no generally accepted evaluation method when discussing EMR use. In the future, studies would benefit from standardised interventions and a clearly defined way of evaluating meaningful use of EMRs.

    Nature of the interventions

    The predominant intervention type identified in this review used educational material, seminars and guidelines to target EMR use (professional interventions), which were identified in 8 of the 12 studies. This focus on professional interventions was found to be consistent with the literature given that the only other systematic review in this area,37 only included studies with educational interventions.37 However, perceived barriers to EMR use include lack of both financial incentives and useful EMR features.38 39 To address those barriers, the implementation of financial and organisational interventions is required. Therefore, there is a need for future studies to consider the other categories of interventions (organisational and financial) in the area of improving EMR use.

    Focus of interventions

    Both use of EMR functions and data quality received equal attention as target areas for interventions to improve EMR use. Even though the studies collected for this review represent two important areas for interventions to target in order to improve EMR use, the literature was found to be lacking in other areas that could be targeted to improve use areas such as: communication, workflow, knowledge/skills and technological support. The field of interventions and intervention target areas aimed at improving EMR use is still lacking in well-designed studies that cover all areas that effect EMR function and use.

    Strengths and limitations

    This review is aimed at a new and developing field. This is one of only two systematic reviews conducted in the area of improving EMR use.37 However, due to high heterogeneity in this area, previous reviews were unable to conduct a meaningful meta-analysis.37 In this review, a synthesis of the results was possible through: the categorisation of interventions using the EPOC taxonomy of interventions and the identification of possible intervention target areas to improve EMR use. A limitation of this study was that due to the new and wide geographic spread of information technology use in the health field, EMRs are identified differently in different countries, making it impossible to identify all the studies with one search term. In an attempt to learn all the possible terms that are used to refer to an EMR, a search was performed prior to the creation of the search strategies. Using those newly found terms, a search strategy was then created to be as inclusive as possible without straying from the inclusion/exclusion criteria.

    Conclusion

    This review reveals a lack of attention given to interventions aimed at improving EMR use in primary healthcare. This is also reflected in the absence of a generalised method to evaluate EMR use, as well as guidelines to implement interventions to improve this use. After an intensive and inclusive search of the literature, this systematic review found a relatively small number of included studies with high heterogeneity. However, it is still worth noting that the results of this meta-analysis indicate that it is beneficial for primary healthcare practice to implement organisational, professional and financial interventions. This can be achieved through investing in EMR feature add-ons, educational materials and financial incentives to improving EMR use.

    References

    1. 1.↵
      1. Delpierre C,
      2. Cuzin L,
      3. Fillaux J, et al
      . A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach? Int J Qual Health Care 2004;16:407–16.doi:10.1093/intqhc/mzh064
      OpenUrlCrossRefPubMedWeb of Science
    2. 2.↵
      1. Chaudhry B,
      2. Wang J,
      3. Wu S, et al
      . Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144:742–52.doi:10.7326/0003-4819-144-10-200605160-00125
      OpenUrlCrossRefPubMedWeb of Science
    3. 3.↵
      1. Weed LL
      . Medical records that guide and teach. N Engl J Med Overseas Ed 1968;278:593–600.doi:10.1056/NEJM196803142781105
      OpenUrl
    4. 4.↵
      1. Tripathi M
      . EHR evolution: policy and legislation forces changing the EHR. Journal of AHIMA 2012;83:24–9.
      OpenUrl
    5. 5.↵
      1. Chang F,
      2. Gupta N
      . Progress in electronic medical record adoption in Canada. Canadian Family Physician 2015;61:1076–84.
      OpenUrlAbstract/FREE Full Text
    6. 6.↵
      1. Collier R
      . National Physician Survey: EMR use at 75%. Canadian Medical Association Journal 2015;187:E17–E18.doi:10.1503/cmaj.109-4957
      OpenUrlFREE Full Text
    7. 7.↵
      1. The College of Family Physicians of Canada, Canadian medical Association, Royal College
      . 2014 survey: national physician survey, 2014. Available: http://nationalphysiciansurvey.ca/surveys/2014-survey/
    8. 8.↵
      1. Canadian Medical Association
      . How can Canada achieve enhanced use of electronic medical records? 2014. Available: https://www.cma.ca/Assets/assets-library/document/en/advocacy/Enhanced-Use-of-EMRs-Discussion-Paper-Final-May-2014.pdf
    9. 9.↵
      1. Menachemi N,
      2. Collum TH
      . Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy 2011;4:47–55.doi:10.2147/RMHP.S12985
      OpenUrlCrossRefPubMed
    10. 10.↵
      1. Lau F,
      2. Price M,
      3. Boyd J, et al
      . Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Decis Mak 2012;12.doi:10.1186/1472-6947-12-10
    11. 11.↵
      1. Linder JA,
      2. Ma J,
      3. Bates DW, et al
      . Electronic health record use and the quality of ambulatory care in the United States. Archives of Internal Medicine 2007;167:1400–5.doi:10.1001/archinte.167.13.1400
      OpenUrlCrossRefPubMedWeb of Science
    12. 12.↵
      1. Graetz I,
      2. Reed M,
      3. Shortell SM
      . The next step towards making use meaningful: electronic information exchange and care coordination across clinicians and delivery sites. Medical Care 2014;52:1037–41.
      OpenUrlPubMed
    13. 13.↵
      1. HealthIT.gov
      . Meaningful use definition and meaningful use objectives of EHRs: providers and professionals, 2015. Available: https://www.healthit.gov/providers-professionals/meaningful-use-definition-objectives
    14. 14.↵
      1. EPPI: Centre Home. UCL Institute of Education
      . Social science research unit, 2015. Available: http://eppi.ioe.ac.uk/cms/
    15. 15.↵
      1. Cochrane Effective Practice and Organisation of Care Review Group (EPOC)
      . Data collection checklist: EPOC Taxanomy, 2015. Available: http://epoc.cochrane.org/epoc-taxonomy
    16. 16.↵
      1. The Cochrane Collaboration
      Higgins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5, 2011.
    17. 17.↵
      1. Melendez-Torres GJ,
      2. Bonell C,
      3. Thomas J
      . Emergent approaches to the meta-analysis of multiple heterogeneous complex interventions. BMC Med Res Methodol 2015;15.doi:10.1186/s12874-015-0040-z
    18. 18.↵
      1. Lorenc T,
      2. Felix L,
      3. Petticrew M,
      4. Melendez GJ, et al
      . Meta-analysis, complexity, and heterogeneity: a qualitative interview study of researchers’ methodological values and practices. Syst Rev 2016;5.doi:10.1186/s13643-016-0366-6
    19. 19.↵
      1. StataCorp
      . Statistical software: release 13, 2015.
    20. 20.↵
      1. Borenstein M,
      2. Hedges L,
      3. Higgins J
      . Introduction to Meta-Analysis. United Kingdom: John Wiley & Sons Ltd, 2009.
    21. 21.↵
      1. DerSimonian R,
      2. Laird N
      . Meta-analysis in clinical trials. Control Clin Trials 1986;7:177–88.doi:10.1016/0197-2456(86)90046-2
      OpenUrlCrossRefPubMedWeb of Science
    22. 22.↵
      1. Downs SH,
      2. Black N
      . The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377–84.doi:10.1136/jech.52.6.377
      OpenUrlAbstract
    23. 23.↵
      1. de Lusignan S,
      2. Stephens PN,
      3. Adal N
      . Does feedback improve the quality of computerized medical records in primary care? Journal of the American Medical Informatics Association 2002;9:395–401.doi:10.1197/jamia.M1023
      OpenUrlCrossRefPubMed
    24. 24.↵
      1. de Lusignan S,
      2. Hague N,
      3. Brown A, et al
      . An educational intervention to improve data recording in the management of ischaemic heart disease in primary care. J Public Health 2004;26:34–7.doi:10.1093/pubmed/fdh104
      OpenUrlCrossRefPubMedWeb of Science
    25. 25.↵
      1. Pan Q,
      2. Meng Y-X,
      3. Gibbons GH, et al
      . Effectiveness of an intervention to improve the documentation required for diagnosis of metabolic syndrome in clinics serving African-American patients. Qual Prim Care 2009;17:191–6.
      OpenUrlPubMed
    26. 26.↵
      1. Baer HJ,
      2. Schneider LI,
      3. Colditz GA, et al
      . Use of a web-based risk appraisal tool for assessing family history and lifestyle factors in primary care. J Gen Intern Med 2013;28:817–24.doi:10.1007/s11606-013-2338-z
      OpenUrlPubMed
    27. 27.↵
      1. Maviglia SM,
      2. Yoon CS,
      3. Bates DW, et al
      . KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs. J Am Med Inform Assoc 2006;13:67–73.doi:10.1197/jamia.M1861
      OpenUrlCrossRefPubMed
    28. 28.↵
      1. Kortteisto T,
      2. Raitanen J,
      3. Komulainen J, et al
      . Patient-specific computer-based decision support in primary healthcare--a randomized trial. Implement Sci 2014;9.doi:10.1186/1748-5908-9-15
    29. 29.↵
      1. Nemeth LS,
      2. Ornstein SM,
      3. Jenkins RG, et al
      . Implementing and evaluating electronic standing orders in primary care practice: a PPRNet study. J Am Board Fam Med 2012;25:594–604.doi:10.3122/jabfm.2012.05.110214
      OpenUrlAbstract/FREE Full Text
    30. 30.↵
      1. Kruse GR,
      2. Kelley JHK,
      3. Linder JA, et al
      . Implementation of an electronic health record-based care management system to improve tobacco treatment. J Gen Intern Med 2012;27:1690–6.doi:10.1007/s11606-012-2174-6
      OpenUrlCrossRefPubMed
    31. 31.↵
      1. Maddocks H,
      2. Stewart M,
      3. Thind A, et al
      . Feedback and training tool to improve provision of preventive care by physicians using EMRs: a randomised control trial. Inform Prim Care 2011;19:147–53.
      OpenUrlPubMed
    32. 32.↵
      1. Davis AM,
      2. Cannon M,
      3. Ables AZ, et al
      . Using the electronic medical record to improve asthma severity documentation and treatment among family medicine residents. Fam Med 2010;42:334–7.
      OpenUrlPubMedWeb of Science
    33. 33.↵
      1. Sweeney J,
      2. Mchugh S,
      3. Perry IJ
      . Implementation and evaluation of a clinical data management programme in a primary care centre. Irish Medical Journal 2012;107:323–6.
      OpenUrl
    34. 34.↵
      1. Moher D,
      2. Liberati A,
      3. Tetzlaff J, et al
      . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097. doi:10.1371/journal.pmed.1000097
    35. 35.↵
      1. Hogan WR,
      2. Wagner MM
      . Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997;4:342–55.doi:10.1136/jamia.1997.0040342
      OpenUrlCrossRefPubMed
    36. 36.↵
      1. Himmelstein DU,
      2. Woolhandler S,
      3. Hope WS
      . Hope and hype: predicting the impact of electronic medical records. Health Aff 2005;24:1121–3.doi:10.1377/hlthaff.24.5.1121
      OpenUrlAbstract/FREE Full Text
    37. 37.↵
      1. Goveia J,
      2. Van Stiphout F,
      3. Cheung Z, et al
      . Educational interventions to improve the meaningful use of electronic health records: a review of the literature: BEME guide No. 29. Med Teach 2013;35:e1551–60.doi:10.3109/0142159X.2013.806984
      OpenUrl
    38. 38.↵
      1. Gagnon M-P,
      2. Ghandour EK,
      3. Talla PK, et al
      . Electronic health record acceptance by physicians: testing an integrated theoretical model. J Biomed Inform 2014;48:17–27.doi:10.1016/j.jbi.2013.10.010
      OpenUrlCrossRefPubMed
    39. 39.↵
      1. Boonstra A,
      2. Broekhuis M
      . Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res 2010;10.doi:10.1186/1472-6963-10-231
    40. 40.
      1. Jerome RN,
      2. Bettinsoli Giuse N,
      3. Rosenbloom ST, et al
      . Exploring clinician adoption of a novel evidence Request feature in an electronic medical record system. J Med Libr Assoc 2008;96:34–41.doi:10.3163/1536-5050.96.1.34
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    Footnotes

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.