Article Text
PDF
Abstract
Introduction Many clinical education programmes have not incorporated the use of the electronic health record (EHR) into their curriculum. It is important to incorporate technologies that will be used in real-world settings to better prepare students for clinical practice.
Objectives To undertake a review of literature to identify a training evaluation framework; to conduct a self-completion survey, pretraining and post-training, to determine students’ perceptions on the benefit of using EHR training system.
Setting Nursing School, University, North West England, UK; University Ethic Committee Approval Received.
Participants Registered nurses undertaking a validated return to practice course; 24 participants for the first cohort who completed pretraining questionnaire and 23 for the second post-training cohort.
Results The statistical results show that the students perceived that the training improved their capability in employing digital systems with statistically significant difference in the assessed preproficiency and post proficiency in the use of digital clinical systems (premedians and post medians are 2 and 5 on 10-point Likert scale, p=0.041). There was also an indication of an improvement in the knowledge of EHR systems although not statistically significant. Most students perceived it increased their knowledge on digital systems.
Conclusion Students perceived an increase in proficiency with the EHR. There was evidence of improvement in confidence in the use of the EHR, but this confidence would be enhanced by additional use of the system. Some desire to increase confidence further and to develop knowledge of digital systems was expressed.
- BMJ Health Informatics
- nursing
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary
What is already known?
During the last decade, the use of electronic health records (EHRs) in clinical settings has risen. Many clinical education programmes have not incorporated the use of electronic documentation into their curriculum. It is important to incorporate technologies that will be used in real-world settings into educational clinical simulations to better prepare students for clinical practice and promote patient safety.
What does this paper add?
As first of type, this research differs from earlier studies presenting new empirical data on students’ perceptions on their use of an EHR within a designated nursing educational programme.
The statistical results show that the students perceived an increase in proficiency in the use of EHRs, within clinical skills session.
Introduction
Since 2017, the University of Central Lancashire, a University in the North West (NW) of England, UK and a National Health Service (NHS) clinical systems and solution supplier, Egton Medical Information Systems Limited (EMIS) Health, have collaborated to introduce healthcare students to an electronic healthcare record (EHR), to build proficiency in using digital technology in a safe and controlled environment, prior to work placement and practice.1–3
Informatics, the use of information and technology to support quality care, is an essential part of the modern clinicians’ job.4 In the UK, this is supported by a range of strategy and policy documents, starting with Information for Health in 1998.5 Learning to Manage Health Information (LtMHI) was first developed in 1999 to establish a common framework in health informatics for clinical professionals at preregistration and postregistration level6; Learning to Manage Health Information: Moving Ahead was published in 2002, providing an updated edition with additional guidance and interpretation to take account of important developments since the original publication (in information, its management and educational approaches).7 Digital literacy is a key strand of the English National Information Board’s Building a Digital Ready Workforce.8 Higher Education England, working with professional bodies that include the UK Royal College of Nursing (RCN), tasked with delivering this work stream. RCN aims include ‘every nurse to be an e-nurse’.9 Current English policy includes the Topol review, which states (p81)10 ‘Educational providers must ensure that students gain an appropriate level of digital literacy at the outset of their study for their prospective career pathway’. UK government drivers, supported by professional organisations, suggest the imperative for clinical education providers to ensure that students acquire informatic knowledge and skills.
The research question that this paper sets out to answer: ‘Do nurses perceive benefit from the introduction of digital technology (use of EHR system) into the Return to Practice Programme?’
To achieve this aim, study objectives include:
To undertake a review of literature to identify what is known about:
nurses’ attitudes toward use of digital record systems;
incorporating informatic knowledge and skills into educational programmes: Is there an issue?
Published study research strategies to establish a theoretical base to the study;
To identify a training evaluation framework;
To conduct a self-completion survey to determine students’ perceptions, pre-EHR and post-EHR training session with registered nursing students on designated programme of study;
The relationship of this pilot study to previous research in this field is considered next.
Literature review
The Preferred Reporting Items for Systematic Review and Meta-Analyses statement,11 and Statement on Reporting of Evaluation Studies in Health Informatics,12 framed literature search protocol used in this study, shown in table 1.
Literature search protocol
A review of literature, using literature search protocol shown in table 1, was undertaken to determine what is known about (1) nurses’ attitudes towards use of digital record systems; (2) incorporating informatics into clinical education programmes: is there an issue? The overall aim of the review to identify a gap, if any, to justify study; to identify training evaluation model and inform choice of research strategy, relevant to the aims and objectives of this study. Published date range for the articles was between January 1950 and August 2019 to ensure established models and studies, and the most current, were captured and reviewed. The articles were published in English.
Articles were searched via e-databases including PubMed; Web of Science; CINAHL (Cummulated Index to Nursing and Allied Health Literature) Complete; Emerald Insight; Science Direct; using a list of key words: training evaluation, nursing education, electronic health record, health informatics, nursing informatics, pilot study, higher education, with ‘AND’ and ‘OR’ as a search strategy. Manual searches based on the reference lists and bibliographies of articles, reports and books considered relevant to this study were also performed. With the search filters applied, 12 397 articles were identified. Articles were screened across all results to remove duplicates. Following a review of the titles and abstracts removed further articles. The exclusion criteria were papers that did not mention a training evaluation model and papers with insufficient information about methods, resulting in the inclusion of 36 articles for the purposes of the review. Selected papers revealed various study designs that included case studies (n12), descriptive, qualitative, quasi-experimental strategies and details about data collection techniques including survey, interviews and focus groups. Geographical locations—USA; UK; Canada; Iran; Australia; Korea and rest of the World. An extract of selected literature review analysis is shown in table 2.
Extract of literature review results
Issue
Digital literacy of nurses has been identified as a significant factor influencing positive attitudes towards the use of electronic record systems in practice. Some studies identified a strong correlation between nurses’ previous computer use and positive attitudes towards the use of electronic record systems.13 14 Yet, the Research into Health Informatics Education study15 concluded that health informatics learning objectives had not been fully incorporated into formal educational programmes. In a study that built on this work, it was reported that students recognised that they would “…value more access to, and training on, digital clinical information systems” (p558)16; Academic staff reported limitations in their levels of expertise in the informatics topic area and generally assumed that student training occurred within clinical practice due to the wide variety of electronic record systems available. The findings highlighted that clinical systems training was not necessarily available for students in placement; where training was provided in practice, students were not prioritised; 61% of students asked for further training.
Several studies identified a problem with the provision of informatics within educational programmes.17–19 Many authors suggested a gap in nurses informatic knowledge and skills,20–24 need for assessment of learning outcomes,25–27 and assert the need for training to be evaluated. The aim to ensure that each iteration produces an evidence base, to identify what went well and what could be improved or changed; to look at whether an educational activity accomplished what it was supposed to; and to explore results of the participants’ learning, or to set goals for the future. Furthermore, evaluation should be planned from the start using a robust model. It is suggested that the findings from review of selected literature in this section show an issue with nurses informatic training, which justifies study.
Selection of training evaluation model
Some of the key influential models of training evaluation devised since the 1950s, the earliest mention in the literature, will be reviewed next to inform selection of training evaluation model. Kirkpatrick defined evaluation in 1959 as determining the effectiveness of a training programme.28 The Kirkpatrick model identified four levels at which trainings can be evaluated, summarised in table 3.
Kirkpatrick’s four-level model
Vizeshfar et al,27 in the context of a pilot study conducted within healthcare context, argue that Kirkpatrick’s model:
is one of the best-known models for evaluating the effectiveness of training courses, provides a comprehensive, simple and practical approach for use in many training situations and is known as a benchmark in the field.
Several authors highlight the importance of the Kirkpatrick model, which has been critiqued, refined and adapted for various purposes, including evaluations of workplace training, and nursing programmes delivered in higher education institutions.25 27 29–32
The selected literature shows a lack of consistency in the ways in which evaluation theory has been conceptualised. Several authors29 33 34 present the notion of an issue between positivist and phenomenological approaches to evaluation, and articulate the dichotomies apparent in the field, to clarify basic beliefs, and associated methodological approaches. Basic beliefs associated with a positivist approach include: the world is external and objective; suggesting that researchers should focus on facts; look for causality and reduce phenomena to simplest elements. It is suggested that Kirkpatrick’s four level model reflects a positivist world view. A phenomenological approach views the world as socially constructed and subjective; suggesting that researchers need to focus on meanings, to understand what is happening. Some authors suggest that rather than view these models as competing alternatives, advantage can be achieved from viewing them as complementary and amalgamating the strengths of each.29 34
In 2016, Kirkpatrick30 and Kirkpatrick41 revised the four-level model in which the complexity of the behavioural change increases as evaluation strategies ascend to each higher level, adding five principles described in table 4, which incorporates both positivist and phenomenological approaches.
Kirkpatrick and Kirkpatrick’s (2016) five principles
Shinners review, focused on principles above, suggests the value of Kirkpatrick and Kirkpatrick’s evaluation model in continuing nursing education.34
This pilot study is part of an overall University strategy to embed Health Informatics across educational programmes, underpinned by a collaborative relationship with a commercial EHR company. The Kirkpatrick and Kirkpatrick revised model was selected because the literature supports that it provides a comprehensive, simple and practical approach to training evaluation, within an overall purpose orientated approach that supports the principle of beneficence (do no harm), applicable to both nursing, and maintenance of collaborative commercial relationship, contextual environment that this study was conducted within. Moreover, the selected model provides scope to continue evaluation should the pilot be rolled out to include other preregistration and postregistration programmes delivered by the University.
Choice of research strategy
The research strategy selected for this pilot study is summarised next, followed by the argument for the choices made. The aim to ensure rigour in all aspects of this study to achieve trustworthy findings and conclusions. In this context, it is argued that the choice of an appropriate research strategy provides an approach to the collection and analysis of data in a way that ensures validity.
Informed by the research strategies used in several studies shown in table 2, an overall case study strategy was selected, being appropriate to the integration of data from different sources, limitations in one method counterbalanced by strength from an alternative data collection technique.35 For example, a strength associated with review of literature is that it is a relatively straightforward; and an efficient method to determine what is already known about a subject, allowing identification of issues, authoritative text, key characteristics and known models. A strength of using survey technique is to obtain a limited amount of information, at a given point in time; surveys allow anonymity. Limitations associated with survey techniques include limited information, low response rates; and do not allow detection of misunderstandings arising from questions. Data collection through survey is appropriate in seeking attribute data, that is relating to attitudes, perceptions and opinions of participants. For example, the use of Likert scales allows survey respondents to select from categories indicating their strength of agreement, or disagreement with statements.36 Open questions allow respondents to respond in their own terms, allowing unusual responses to be derived.
Methods
Setting
Faculty of Health and Well-being, Nursing School, University, NW England, UK.
Participants
All students registered on the Return to Practice (RTP) course as at 1 September 2018 were eligible for this pilot study. Participation was voluntary. One student declined to participate. Twenty-four participants for the first questionnaire (pretraining) and 23 for the second (post-training).
Ethics
This pilot study was designed and conducted applying principles of professional research practice; University Ethics Committee Approval received.37
Reasonable attempt to minimise any bias was made. All RTP registered students were invited to participate in self-completion online questionnaire, issuing a letter to each. All questionnaires were anonymous to protect the student’s identity. Statistical analysis of survey findings was supported by University Research Support Team to minimise any bias.
Data collection
A student survey was conducted using self-completion questionnaire. The survey tool was developed, piloted with University Staff. The first questionnaire was given to the students at the start of the course, prior to teaching sessions, to determine students’ perceptions about their existing knowledge and experience of EHRs, if any. Following completion, teaching sessions were delivered that included a theoretical session incorporating the concepts of the EHR, students’ personal and professional responsibilities. A further practical session was delivered during which students were given the opportunity to access and interrogate the EHR training system to answer specific clinical questions. The second questionnaire was conducted to determine how useful students perceived the benefit of using EHR training system. The first questionnaire had seven questions (three free text and four Likert Scale). The second questionnaire had 18 questions (5 free text and 13 Likert Scale). Both surveys consisted of a Likert style questionnaire from 1 to 10 with 1 being strongly disagree and 10 being strongly agree with an open-ended comments section for students wishing to add more information. The second questionnaire consisted of four main questions to be compared with the questions from the first questionnaire: about previous knowledge of and proficiency in digital clinical systems, about pretraining personal use of social media and its link with the professional use. The rest of the questions asked students how useful they perceived the teaching. Students were asked to complete the second questionnaire at the end of the Course. Questionnaire 1 took 10–15 min. Questionnaire 2 took between 10 and 20 min to complete.
Data analysis
Pilot Student survey: SPSS package (V.25) was used to perform the statistical analysis of the data from the survey and thematic analysis to report the findings from open-ended questions.38 Significance level used in statistical analysis was 0.05.
The questionnaires were completed entirely anonymously; no case-by-case comparison could be conducted. Answers to four questions related to participant’s pretraining and post-training experience were considered for comparison. The Likert Scale data were summarised using medians and IQRs of response to each question. The research suggests that Likert Scale with number of items close to 10 can be considered as an interval scale continuous measurement.38 However, the Shapiro-Wilkes test did not support the normality assumption for the samples, so the Mann-Whitney U test, a non-parametric test, was used for comparison.
Results
The statistical results show that the students perceived that the training improved their capability in employing digital systems with statistically significant difference in the assessed preproficiency and postproficiency in the use of digital clinical systems (premedians and postmedians are 2 and 5 on 10-point Likert Scale, p=0.041). There was also an indication of an improvement in the knowledge of EHR systems although not statistically significant. Most students perceived it increased their knowledge on digital systems. Descriptive statistics of Likert type scores for all questions used in the survey are presented in table 5.
Descriptive statistics (median and IQR) for pretraining and post-training questionnaires
Braun and Clarke’s39 thematic analysis was used to analyse the data from open-ended question responses, which consisted of familiarisation of the data; generation of codes; searching for themes; reviewing of themes; defining and naming themes and producing the report. Analysis of free text/open-ended questions revealed four themes: low self-belief in knowledge, the desire to know more about digital systems, the desire to increase capability using digital systems and an existing awareness of responsibilities of using social media. The results of the comparative analysis of the presurvey and postsurvey are shown in table 6.
Results of the comparative analysis of the presurvey and postsurvey
The median on the questions requesting survey respondents to rate their own knowledge and proficiency in using digital systems in a clinical setting in the pretest questionnaire were lower than the median for other questions (Mdn=3 and Mdn=2 pretraining; Mdn 5 post-training), suggesting that nurses did not rate their knowledge or capabilities in the use of digital systems in a clinical setting highly. Respondents perceived capabilities in the use of digital clinical systems did show a statistically significant improvement in the Mann-Whitney U test between pre and post training results (p=0.041, U=182.0). The respondent’s perceived knowledge of digital clinical systems after training indicated an increase, it was not statistically significant, though the p value was just above the significance level (p=0.052, U=186.0).
Theme: lack of confidence in the use of the technology
Open ended questions in the pretraining questionnaires revealed many students reported the need to become more confident in digital technology. Survey respondents expanded their notion of confidence in the post-training questionnaire that showed students strongly agreeing that they perceived more confidence using digital health systems (Q3d Mdn (IQR)=7 (3)). Open ended comments included: ‘Confidence will come after use of the systems in practice, although the initial session was very helpful’. ‘One to one training to increase confidence’.
Theme: the desire to increase knowledge and capability using digital systems
Those students who already had some awareness of the systems, in the open-ended questions reported the wish to reinforce or expand their existing knowledge. Open-ended survey responses included: ‘The desire to gain more knowledge and become more proficient’. ‘Digital is the future of healthcare so it is vital to become familiar and competent in its use and access’.
The post-training questionnaires revealed an expansion of what students perceived as knowledge and proficiency. Participants agreed with deeper understanding of the advantages of and the problems encountered with healthcare technology, Q2c and Q2d Mdn (IQR)=8 (1) and the need to use the system in practice with their ability to apply theory to practice being developed (Q3c Mdn (IQR)=7 (3)). Many reported that although being taught the system was useful and relevant it was limited as the system was not necessarily the EHR they would use in practice. The majority strongly agreed that the technology they will use in practice may be a different system (Mdn (IQR)=9 (3)). Open-ended question responses included: ‘It increased my understanding and I know that if I used the same system in a hospital it would be beneficial’.
Theme: an existing awareness of responsibilities of using social media
The students understanding of personal use of social media increased although this was not statistically significant (U=204.5, p=0.12). The understanding of how personal use of digital media links to professional use also increased post training but was not statistically significant (U=216.0, p=0.27). All students reported using digital technology socially although some stated this was only on the use of sending emails while others used a large variety of social media. Majority strongly agreed that they understand how personal use of digital media may link in with professional use (Q2b Mdn (IQR)=9 (2)).
Discussion
The application of the Kirkpatrick and Kirkpatrick evaluation model to this pilot study underpinned the foundation, and development, of a unique educational programme, to provide students with an opportunity to use an EHR in a safe and controlled environment. The programme was designed in partnership with stakeholders, which included an NHS clinical system solution supplier; Faculty Executive Dean, Heads of School and Course Leaders. The five principles associated with selected evaluation model (see table 4) were applied throughout the 12-month duration of the study; for example, the initial planning stage involved identification of a professional skill gap, and issue with embedding informatics in educational programmes, informed by review of literature. Stakeholder support for agreed programme objectives, evaluation and research study design was obtained. It is suggested that partnership working with stakeholders helped to prepare participants for the training on EHR systems, which impacted positively on survey response rates, which aligns to principle 3. Nursing student feedback, regarding perceptions to the EHR training, align to selected evaluation model Level 1, Reaction level, discussed next.
It is expected that students would have lower median scores on the pretest questionnaire on their knowledge of digital systems in clinical setting as the students were on a course to refresh their skills in healthcare. Despite the median scores being low, there was an increase in perceived knowledge of digital clinical systems post-training. Students perceived an increase in proficiency in the use of the EHR. There was evidence of improvement in confidence in the use of the EHR; this confidence would be enhanced by additional use of the system. Some desire to increase confidence further and to develop knowledge of digital systems was expressed. This may be attributable to the students developing a deeper awareness of, and confidence in, their own skill base and learning needs that requires further investigation.
The findings show that some students reported that it would be useful to gain experience on the system they would be using in practice. EHR systems currently vary between NHS Trusts. The justification for providing access to selected EHR in this pilot being that all NHS providers across the UK County geographical context of this study “have plans in place to improve their digital maturity through the implementation of EHR. Whilst each acute provider has a different solution, the provision of primary care and community systems is less complex, with EMIS Health Solutions in use across all GP practices, half of the county’s community services and its planned use as the county-wide child health information system (CHIS). Medicines management and e-prescribing are also largely delivered through EMIS solution” (p15).40
This small pilot study demonstrates measurable, and in some instances, statistically significant benefit of the EHR training/learning experience, which establishes a chain of evidence aligned to evaluation model, principle 5. Results may help to inform future developments, and instance of embedding use of EHR systems, across preregistration and postregistration clinical courses.
Limitations
This 12-month pilot study has limitations that include the questionnaire being short and not comprehensive, as well as the reported findings based on use of EHR in clinical skills session limited to registered RTP nurses in a University based Nursing School, NW England, which may not represent the general population of pre and post-registration healthcare professionals.
Recommendations
Expedite development of relevant clinical scenarios: educational modules, chosen by the university, to be reviewed by a multiprofessional team from system supplier; aligning all components of a module(s) to system functionality and Health Informatics theory;
EHR system to be used on PCs due to laptop screen resolution issues;
Roll-out access to Medical School that plan to use EHR system with year 5 students; potential to develop multidisciplinary approach to clinical scenarios;
Potential to increase student placements—university students familiar with EHR prior to placements should prove an advantage;
Continue to evaluate first of type, build evidence—what works, what could be improved, identify behaviour changes, impact and publish results.
Conclusion
As first of type, this EHR system deployment proved challenging due to university requiring a non-NHS network solution. The university’s instance of EHR system was on a separate environment and new processes had to be developed in system suppliers Hosted, Deployment and Support departments.
The statistical results reported in this paper show that the students felt they were more proficient in the use of the digital EHR systems and the difference in pretraining and post-training level was statistically significant. There was also an improvement in the knowledge of clinical systems although not statistically significant. Emergent themes from the comments in the questionnaires were that most students perceived it increased their knowledge on digital systems although some would have preferred more 1:1 time with it, and some stated they would have liked training on the system they were going to use in practice. Some stated it increased their confidence in using digital technology.
Acknowledgments
The authors wish to thank Editorial Board and reviewers for their constructive feedback.
References
Footnotes
Contributors JS and ST, Research Support Team, analysed the data. All authors discussed the results and contributed to the final manuscript.
Funding The University Faculty of Health and Wellbeing provided the funding for the pilot study, and Research Support Team to analyse survey findings.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval University of Central Lancashire Ethics Committee, granted approval 1 October 2018 of proposal application ‘Do nurses benefit from the introduction of digital technology into the Return to Practice Programme?’. Application Unique Reference Number STEMH 935, Preston: UCLan, 2018.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information. Further information available upon reasonable request.