Article Text
PDF
Abstract
Objective Inadequate adherence to resuscitation for non-crying infants will have poor outcome and thus rationalise a need for real-time guidance and quality improvement technology. This study assessed the usability, feasibility and acceptability of a novel technology of real-time visual guidance, with sound and video recording during resuscitation.
Setting A public hospital in Nepal.
Design A cross-sectional design.
Intervention The technology has an infant warmer with light, equipped with a tablet monitor, NeoBeat and upright bag and mask. The tablet records resuscitation activities, ventilation sound, heart rate and display time since birth. Healthcare providers (HCPs) were trained on the technology before piloting.
Data collection and analysis HCPs who had at least 8 weeks of experience using the technology completed a questionnaire on usability, feasibility and acceptability (ranged 1–5 scale). Overall usability score was calculated (ranged 1–100 scale).
Results Among the 30 HCPs, 25 consented to the study. The usability score was good with the mean score (SD) of 68.4% (10.4). In terms of feasibility, the participants perceived that they did not receive adequate support from the hospital administration for use of the technology, mean score (SD) of 2.44 (1.56). In terms of acceptability, the information provided in the monitor, that is, time elapsed from birth was easy to understand with mean score (SD) of 4.60 (0.76).
Conclusion The study demonstrates reasonable usability, feasibility and acceptability of a technological solution that records audio visual events during resuscitation and provides visual guidance to improve care.
- Decision Support Systems, Clinical
- Decision Support Techniques
- Artificial intelligence
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
There have been advances in low-cost technology and equipment to improve newborn resuscitation in low-resource settings.
Recent developments in machine learning suggest resuscitation activities recorded by tablet with analysis of the events provide the next step of resuscitation. This opens up new possibilities for real-time guidance and quality improvement. This study aims to assess the usability, feasibility and acceptability of a novel technology during newborn resuscitation by healthcare providers (HCPs) in a low-resource setting hospital.
WHAT THIS STUDY ADDS
In terms of usability of technology, the overall usability (ranging 1–100) was good with the mean score (SD) of 68.4 (10.4). The participant preferred to use the technology more frequently with the mean score (SD) of 4.52 (0.87).
In terms of feasibility of the technology (range 1–5), the participants received support from their supervisor to use the technology with, the mean score (SD) of 4.28 (1.24); however, they perceive of not receiving adequate support from the hospital administration while using the technology, with mean score (SD) of 2.44 (1.56).
In terms of acceptability of the technology (range 1–5), the information provided on the monitor, that is, time elapsed from birth, was easy to understand with the mean score (SD) of 4.60 (0.76). The access to NeoBeat and upright bag and mask in the infant warmer was easy with the mean score (SD) of 4.60 (0.91).
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
We are in a process of developing a novel technology based on automatic video and audio analysis as well as activity recognition using deep learning models.
The study demonstrates that this current (pilot) version of the application was usable, feasible and acceptable by the HCPs, providing rationale to continue the development of machine learning technology.
Introduction
Globally, every year, of the 140 million newborns born, 10–15 million do not cry after birth and resuscitation is required for these newborn to accomplish spontaneous breathing.1–4 Despite progress and efforts to reduce newborn deaths, over 90% of these deaths still occur in low-income and middle-income countries (LMICs), and most intrapartum deaths can be prevented with effective resuscitation.5 Anxiety and fear among healthcare providers (HCPs), difficulties in assessing the newborn’s condition and providing appropriate clinical response usually delay the initiation of bag and mask ventilation.6
To improve the competency on newborn resuscitation in LMICs, Helping Babies Breathe (HBB) training has been rolled out since 2010.7 8 Following the implementation of HBB training, there have been improvement in HCPs skill competence in newborn resuscitation.9 10 However, there is a rapid skill decay in skill competence of newborn resuscitation over a period of time.11 To tackle this problem, simulated short-term training sessions, such as structured skill drill in newborn simulator, have shown to maintain and retain skill competence on resuscitation.12 13 Despite maintenance of skill competency, implementation in clinical care have been low.14 HCPs have failed to translate their skills into clinical performance, and as a result infants who require resuscitation do not receive timely ventilation15 (figure 1A,B).
Need for development of automated feedback during resuscitation. HBB, Helping Babies Breathe; HCP, healthcare provider.
To improve the clinical performance, a periodic reviewing method using Plan–Do–Study–Act (PDSA) have been implemented.16 Reviewing newborn resuscitation procedures have shown to be highly beneficial for maintaining and improving skills17–19 and reduce mortality.20 21 However, review of resuscitation procedure is done after intervention and not during intervention. Therefore, during resuscitation, HCPs depend on their cognitive skills, memory and posted visual reminders for actions to be taken. To mitigate this problem, we are currently in the process to develop an automatic guidance to HCPs during resuscitation with the use of deep learning model, MAchine Learning Application (MALA)22 (figure 1C,D).
MALA will be a tablet-based MALA, which will use video and audio activities recorded by a tablet in analysing the event and will guide for next step of resuscitation through visual display and audio prompts in real time during resuscitation. The development of MALA will require a large number of video and audio recordings to train the MALA and currently a preversion of MALA has been developed which records video and audio activity. This current (pilot) version of the application provides a visual display of time on the tablet monitor mounted onto the infant warmer.
To guide the research team on further development of MALA application, acceptability of video and audio recordings as well as the current version of the application is needed.23–25 Therefore, this study aims to assess the usability, feasibility and acceptability of the current version of technology with visual time guidance, video and audio recordings, during newborn resuscitation.
Methods
Study design
This is a cross-sectional survey of assessing the usability, feasibility and acceptability of the novel technology. The survey was conducted between 19 and 26 January 2022.
Study setting
The study was conducted at Bharatpur Hospital, a referral hospital in Nepal. There are more than 13 000 annual deliveries and 23% by caesarean sections. The delivery unit has in total 21 beds (3 for admissions, 10 for waiting the onset of labour, 5 for labour, 3 for delivery) and 3 newborn resuscitation corners. There are 30 HCPs working in the maternity ward in the hospital, all of them received training on HBB as a part of quality improvement project. Before the introduction of technology, HCPs had been doing skill drills on newborn simulator and biweekly reviewing the resuscitation performance through PDSA approach.
Study participants
All HCPs working in maternity unit with the experience of performing resuscitation as well as who received training on technology was included during the pilot period were eligible. Information about duration of their experience on resuscitation was collected with the background information.
Intervention package
The intervention package consisted of a technology, its installation in the local context and training to HCPs. The technology was developed by Laerdal Medical in Stavanger, Norway, with design input from Golden Community (GC) National Research Institute team in Nepal.
Technology
Technology consisted of an infant warmer (Phoenix Medical Systems, Chennai, India), equipped with a tablet for sound and video recording, which provided visual guidance in elapsed time since birth. The tablet monitor was faced towards the bed of infant warmer to record resuscitative activities. HCPs could also see the time from birth in the monitor and continue resuscitation activities (figure 2).
Technology infant warmer equipped with a tablet computer with a camera for sound and video recording. The tablet monitor provides real-time video recordings and time elapsed since birth. The time of birth is based on input from an observer using the Liveborn app. Technology is also equipped with NeoBeat (newborn heart rate metre that provides heart rate and motion data), a manual suction device, and Upright bag with PEEP. PEEP, positive end-expiratory pressure.
The infant warmer is also equipped with a newborn heart rate monitor (NeoBeat, Laerdal Medical, Stavanger, Norway), a manual suction device, Laerdal Upright bag with PEEP (Positive end-expiratory pressure) functionality and a tube for recording of air pumped through upright bag while ventilating the newborn (figure 2). When ‘baby born’ was clicked on the Liveborn application at the time of birth, video recording got automatically started by the tablet mounted onto the infant warmer. Newborn resuscitation was observed and annotated in the Liveborn application. And when no further resuscitative care was provided by HCPs, observation was ended in the Liveborn application, and then the video recording got automatically stopped. After that, the annotations along with the recorded video were uploaded to a highly secured data storage system. If a newborn did not need any resuscitative care after birth, the already initiated observation was cancelled in the Liveborn application, which automatically stopped and deleted the video recording.
Technology installation to the local context
Laerdal team demonstrated installation and use of technology step by step including the use of two tablets: one for Liveborn application and the other as video recorder. The training was provided to database manager, research coordinator and planning monitoring evaluation manager of GC. The Liveborn and video application systems were repeatedly practised on neonatalie Live by different users at GC office. Feedback on the application system and performance provided to the Laerdal team helped to refine the interoperability between video recorder and Liveborn applications.
Technology training and facilitation
At Bharatpur Hospital, the technology was first introduced to the hospital management on 22 November 2021. The following day, orientation on the system was provided to the doctors and nursing team on (a) installing the technology on infant warmer (bag and mask container, tablet case, NeoBeat charger along with upright bag and mask with PEEP functionality and tube for sound recording of ventilation quality), (b) installation of Liveborn and Video recorder application in the tablet, (c) linking Liveborn application with Video recorder application. Following this, the data collectors practised a demo video recording in real infant. If no resuscitation was needed, the started observation in Liveborn was cancelled and the video recording was automatically stopped and deleted. The doctors and nurses could see the time from birth in the tablet mounted and conduct resuscitation on the infant warmer (figure 3).
Installation and training of data collection team and healthcare providers on the technology.
Experience with the technology and participants’ involvement
Average number of newborn resuscitations per month at Bharatpur hospital was 50. We designed the study in which the technology can be used in both real and simulation cases, such that all HCPs have adequate hands-on experience with the technology. After 8 weeks of hands-on experience with the technology, HCPs participated in the survey.
Development of the provider survey tool
The questionnaire included demographic questions (position in the hospital, years of experience, and education level), skill (newborn resuscitation and NeoNatalie Live skill drill in the last 1 month), technology usage (computer at home, smart phone, app use) and questions related to technology usability, feasibility and acceptability.
Usability
The usability of the technology was assessed using the System Usability Scale (SUS), which is the most widely used standardised questionnaire for the assessment of perceived usability.26 27 The SUS consists of 10 statements with 5-point Likert scale for each statement, that it provides a global view of subjective assessment of a system usability. Among 10 statements in SUS, 5 of them are positively formulated (items with odd numbers) and the other 5 statements are negatively formulated (items with even numbers). After cognitive testing of the translated tool among few HCPs other than the study population, wordings of some of the statements had to be simplified for more clear and better understanding of those statements (online supplemental file 1). The originality of the scale was maintained after translation and adaptation.
Supplemental material
Feasibility and acceptability
In addition, HCPs completed a self-administered 15 questions assessing perceived feasibility and acceptability of the technology using a Likert scale of 1–5, where 1 represents strongly disagree and 5 presents strongly agree. Feasibility and acceptability-related questions were developed by the research team based on the seven constructs from the ‘unified theory of acceptance and use of technology’.28 The finalised questionnaire was translated into Nepali language and no adaptations in the questionnaires were required based on the cognitive testing.
Data collection
After at least 8 weeks of experience with the technology, data collectors provided usability, feasibility and acceptability questionnaires to the HCPs. Data collection was conducted for a week, from 19 to 26 January 2022. Data collectors collected the questionnaire from the HCPs, which were then entered into the database system. The entered data was extracted into SPSS Software (IBM SPSS Statistics for Windows, V.23.0) and reviewed by the study team.
Data analysis
To calculate the overall SUS Score, the following formula was applied: items 1, 3, 5, 7 and 9 (positive statements) were subtracted by 1 from their scale position and items 2, 4, 6, 8 and 10 (negative statements) were subtracted by 5 from the scale position. The sum of these item scores was then multiplied by 2.5 to obtain the overall value of SUS. The overall SUS scores ranged from 1 to 100 with 1 indicating not at all usable and 100 indicating perfect usability. For stratified analysis (online supplemental file 2), p values were calculated using one-way ANOVA (Analysis of Variance) for the ‘Newborn resuscitation and/or NeoNatalie Live skill drill’ or independent samples t-test for other variables (years of experience, education and position/role).
Supplemental material
Written consent was taken from HCPs for the survey (online supplemental file 3).
Supplemental material
Results
Among the 30 HCPs (nurse and midwives) working in the labour and delivery room, 25 (83.3%) of them consented to participate in the survey. All of them were female and 16/25 (64%) of them were nurses. Overall, 11/25 (44%) of them had 2–5 years of experience working, 9/25 (36%) of them had 6–10 years of experience and 4/25 (16 %) had more than 10 years of experience in labour and delivery room. During the last month, 10/25 (40%) of them had performed both at least 1 newborn resuscitation and practised skill drill in Neonatalie Live, 6/25 (24%) had performed newborn resuscitation only and 5/25 (20%) had performed skill drill only. Of the participants, 19/25 (76%) had computer at home and all of them had smart phone. Overall, 24/25 (96%) use mobile-based application in their smart phone on a daily basis (table 1).
Characteristics of the healthcare providers
In terms of the usability of the technology, the overall usability (ranging 1–100) was good with the mean overall score (SD) of 68.4% (10.4). The participants wanted to use the technology (range 1–5) more frequently with the mean score (SD) of 4.52 (0.87) and did not find the technology unnecessary complex with the mean score (SD) of 2.04 (1.37). They perceive that the person using the technology (range 1–5) needs to be good in technology with the mean score (SD) of 4.28 (1.28). The participant did not perceive that there was lot of inconsistency or mismatch between the components of the technology (range 1–5) with the mean score (SD) of 2.44 (1.45). Participants perceived that they could use and operate the technology quickly with the mean score (SD) of 4.40 (0.65). Participants also perceived that they need to learn more and get continuous education to use the technology (range 0–5) in daily routines with the mean score (SD) of 4.52 (0.65) (figure 4).
Healthcare providers usability scores (N=25). a1=strongly disagree, 2=somewhat disagree, 3=neutral or no opinion, 4=somewhat agree, 5=completely agree. bFor items 1, 3, 5, 7 and 9, the converted score is the mean score minus 1. For items 2, 4, 6, 8 and 10, the converted score is 5 minus the mean score. cThe average SUS Score is 68. General guideline on the interpretation of SUS Score: >80.3 (A)=excellent, 68–80.3 (B)=good, 68–58 (C)=okay, 51–58 (D)=poor, <51 (F)=awful. Reference https://measuringu.com/sus/.
When the usability of the technology was stratified by participated HCPs’ newborn resuscitation and/or NeoNatalie skill drill, years of experience, education and position/role, participants who had bachelor’s degree believed stronger that a person needs to be technology friendly for using the technology than participants who were undergraduates (mean score of 4.88 (0.35) vs 4.00 (1.46); p value=0.03). Participants who had less than 7 years of experience working in labour and delivery also believed stronger that a person needs to be good in technology for using the technology, compared with participants who had 7 or more years of experience (mean score (SD) of 4.82 (0.53) vs 3.13 (1.64); p value=0.02). Participants who had bachelor’s degree thought that there is more inconsistency among the component of the technology, compared with undergraduates (mean score (SD) of 3.38 (1.19) vs 2.00 (1.37); p value=0.02) (online supplemental table 2).
In terms of feasibility of the technology, the participants responded that they need support from their colleague (range 1–5) to use the upright bag and mask and NeoBeat in the infant warmer, mean score (SD) of 2.48 (1.66). While the participants received support from their supervisor for use of the technology, mean score (SD) of 4.28 (1.24), they did not receive support from the hospital administration for use of the technology, mean score (SD) of 2.44 (1.56) (figure 5). The participants who were working in the labour room as a nurse thought that the system (range 1–5) was more compatible for use within the existing clinical service system, compared with participants in other roles (mean score (SD) of 4.00 (1.03) vs 2.56 (1.42); p value=0.02) ((online supplemental file 3)).
Healthcare providers feasibility and acceptability scores (N=25). a1=strongly disagree, 2=somewhat disagree, 3=neutral or no opinion, 4 somewhat agree, 5 completely agree.
In terms of acceptability of the technology, the participants reported that the information provided in the monitor that is, time elapsed from the birth, was easy to understand and the access to NeoBeat and upright bag and mask was easy with the mean score of 4.60 for both. The participants felt comfortable with the video recording of the health workers performing newborn resuscitation with the mean score (SD) of 4.36 (0.86) and wanted the video recording to be continued (mean score (SD) of 4.60 (0.76)) after the pilot study. The participants were relatively comfortable with audio recording during newborn resuscitation with the mean score (SD) of 4.20 (1.00) and they perceived the possibility of getting more realistic guidance during newborn resuscitation would reassure them while taking care of the newborn and performing the resuscitation (mean score (SD) of 4.64 (0.86)) (figure 5).
Discussion
This pilot study evaluated the usability, feasibility and acceptability of a novel technology, which included video and audio recordings of newborn resuscitation, visual guidance of time elapsed from birth and resuscitation equipment (NeoBeat, a manual suction device, and upright bag and mask). Our study showed that the HCPs found the technology useful. HCPs wanted to use the technology more frequently and found the system simple and consistent. Although they found some issues of interoperability between the components of the technology, HCPs perceived that a person familiar with the use of smart phone application use can operate the system better. HCPs thought that the system was compatible to use within the existing clinical service system; however, they required support from their colleagues to use the upright bag and NeoBeat in the infant warmer. As the operators that is, HCPs felt they needed support from another team member to use the technology, a continuous capacity building of the HCPs is required to use the application to make the system compatible in a low-resource environment. Currently, research team provides continuous quality improvement support to build the capacity of all HCPs (n=30) to use the application.
In our study, HCPs felt relatively comfortable regarding audio and video recording during newborn resuscitation and felt that the audio and video recording for resuscitation should be continued. However, they also felt that if the technology had provided more realistic guidance during resuscitation, they would be even more confident to take care of the newborns. A similar study by Aude Le Bris and team showed that HCPs considered video to be useful and acceptable under certain prerequisite when assured with robust data protection and limiting potential negative impacts on healthcare professionals.29
During the study period, the HCPs felt that they received adequate support from their supervisors, while relatively less support was received from the hospital administration. The potential support that HCPs might have expected from the hospital administration could be providing proper infrastructure for setting-up the newborn resuscitation corner, support from medical technicians and staff who could best operate the technology, independently without any external support from the implementing partner. Hospital administration has a central role in the implementation success of any newly introduced intervention.30 31 A multidisciplinary approach to engage the hospital administrator in introduction of new technology will help facilitate introduction of new intervention.32
A previous study done at a tertiary hospital in Nepal found that the staff did not adhere to newborn resuscitation guidelines.33 During resuscitation, HCPs generally underestimate or are unable to track the passage of time, which might lead to ineffective resuscitation as they are unaware of the amount of time they have wasted in unnecessary drying, suctioning or stimulating.34 35 Incorporating the MALA system in resuscitation can possibly demonstrate the potential benefits of technology as well as access of all necessary resuscitation equipment for improving the care . In 2021, the WHO provided a Standards-based, Machine-readable, Adaptive, Requirements-based and Testable (SMART) guidelines of using technological solution for improving service delivery and measurement of care.36 The MALA technology development is in line with SMART guideline for improving service delivery and measurement.
There are number of limitations of this study. First, the feasibility and acceptability questionnaires were developed for this study, but their psychometric properties were studied. Second, most of the participants were nursing staffs working in the same unit of the hospital, which might have influenced their opinions and perceptions regarding the technology. Third, our results could not be generalised to other institutions or countries as this was a single-site study with a small sample size in a low-resource setting. Lastly, since this pilot study evaluated usability, feasibility and acceptability of the first phase of the technology without real-time automated feedback, further iterative studies of the complete MALA system are warranted.
Conclusion
This study showed that the technology was acceptable, feasible and usable by the HCPs, providing rationale for continued development of the MALA system with the aim to provide automated real-time feedback based on machine learning technique. Further improvement in the technology for more advanced guidance is required through codesigning the technology together with HCPs and hospital managers. Availability of MALA technology with real-time guidance will reduce the cost of training, boost up the confidence of HCPs in performing and hence, improve the quality of care for newborns.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants. The study was approved by institutional review committee Bharatpur hospital with the ID 12-2021. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
We would like to thank technical advisory group of the project. We would like to thank all the data collectors, health workers and parents involved in the study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors AKC, HMyklebust and ØM-B conceptualised the study. SHH, ØM-B and ØG developed the technology. OB, HMalla and PB installed the technology in the hospital. RG, PB, YNB, OP, HS, OB and HMalla supervised the project implementation. SYJK developed the acceptability and feasibility tool and adapted the SUS tool. AA provided input to finalise the tool. PB and OB collected the health worker survey tool. AKC and PB developed the first draft. All other provided comment to the first version. AKC acceptes full responsibility for the work and had access to the data and controlled the decision to publish. All others agreed to the final version.
Funding The piloting of the project was funded by Laerdal Medical, Stavanger, Norway.
Competing interests HMyklebust, ØM-B, SHH, ØG and SYJK are employed at Laerdal Medical.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.