Building a house without foundations? A 24-country qualitative interview study on artificial intelligence in intensive care medicine
•,,.
...
Abstract
Objectives To explore the views of intensive care professionals in high-income countries (HICs) and lower-to-middle-income countries (LMICs) regarding the use and implementation of artificial intelligence (AI) technologies in intensive care units (ICUs).
Methods Individual semi-structured qualitative interviews were conducted between December 2021 and August 2022 with 59 intensive care professionals from 24 countries. Transcripts were analysed using conventional content analysis.
Results Participants had generally positive views about the potential use of AI in ICUs but also reported some well-known concerns about the use of AI in clinical practice and important technical and non-technical barriers to the implementation of AI. Important differences existed between ICUs regarding their current readiness to implement AI. However, these differences were not primarily between HICs and LMICs, but between a small number of ICUs in large tertiary hospitals in HICs, which were reported to have the necessary digital infrastructure for AI, and nearly all other ICUs in both HICs and LMICs, which were reported to neither have the technical capability to capture the necessary data or use AI, nor the staff with the right knowledge and skills to use the technology.
Conclusion Pouring massive amounts of resources into developing AI without first building the necessary digital infrastructure foundation needed for AI is unethical. Real-world implementation and routine use of AI in the vast majority of ICUs in both HICs and LMICs included in our study is unlikely to occur any time soon. ICUs should not be using AI until certain preconditions are met.
What is already known on this topic
Existing research on intensive care professionals’ views about artificial intelligence remains limited and only includes participants from four high-income countries.
What this study adds
This is one of the largest qualitative studies to date to examine the views of intensive care professionals regarding the use and implementation of AI technologies in intensive care units, involving 59 participants from 24 countries. It shows that the vast majority of ICUs neither have the technical capability to capture the necessary data or run AI algorithms, nor the staff with the right knowledge and skills to use the technology as designed.
How this study might affect research, practice or policy
Pouring massive amounts of resources into developing AI without first building the necessary digital and knowledge infrastructure foundation needed for AI is unethical and needs to change.
Introduction
Intensive care medicine has long been at the forefront of efforts to use routinely collected digital health data to improve patient care,1–3 and it is seen to be particularly well positioned to use the advances in artificial intelligence (AI) given the amount of data typically generated in intensive care units (ICUs).4 It is expected that applications of AI in ICUs will primarily be focused on machine learning to assist in disease identification, prediction of disease progression, disease phenotyping, recognising unique patterns within complex data and guiding clinical decision-making.5–7 Other potential applications include algorithms taking a physically embodied presence, such as in smart autonomous ventilators or infusion pumps.8 9 Despite the anticipated benefits AI technology, a large ‘implementation gap’ between what has been developed and what is used in clinical practice continues to grow, with most developed ICU AI models remaining in testing and prototyping.10 11 Challenges for the successful development and implementation of AI tools in ICUs have been increasingly researched and discussed in recent years,4–17 including: (1) various technological challenges around obtaining high-quality data; ICU data is often heterogeneous and noise-prone, and de-identifying, standardising, cleaning and structuring the data can be difficult4–6 11 14; (2) a number of general ethical, legal and regulatory issues, particularly around data protection and sharing5 6 18; (3) the vast majority of ICU AI models are not robust or ready for clinical use; they have been developed using retrospective data, without external validation or prospective evaluation6 15; and (4) obtaining the trust and acceptance of clinicians and other stakeholders.5 6 Indeed, it is important to better understand intensive care professionals’ views and acceptance of AI to help identify key barriers and facilitators to AI technology being implemented and adding value to intensive care medicine. At the time of designing and initiating this study, there was a lack of empirical studies on intensive care professionals’ views about AI. However, in the past 2 years, a few quantitative and qualitative studies have been published.13 14 19 20 These studies have found general positive attitudes and expectations of ICU professionals towards the use of AI, but also primarily identified technical barriers to the implementation of AI in ICUs. In addition, they identified some non-technical factors (a lack of AI knowledge among ICU professionals, high clinical workload, no clear AI policy, a lack of funding for digitalisation and a culture of doctor-knows-best). However, these studies have consisted of three small survey studies involving one centre13 20 or two centres19 from the Netherlands or the USA, and an interview study including participants from the USA and three European countries (the Netherlands, Belgium and the UK).14 Existing research on intensive care professionals’ views about AI therefore remains limited and only includes participants from four high-income countries (HICs). Furthermore, HICs have so far dominated the discussion over AI and related ethical issues.21 In an era of increasing global collaborative health research efforts, this imbalance is problematic. Lower-to-middle-income countries (LMICs) are also increasingly using healthcare data science and AI.22–25 This study therefore aims to explore the views of intensive care professionals in both HICs and LMICs regarding the use and implementation of AI technologies in ICUs.
Methods
This study is presented in accordance with the Consolidated Criteria for Reporting Qualitative Research reporting guideline.26 See online supplemental information 1 for additional details on methods used in the study. Intensive care professionals were primarily selected through purposive sampling to ensure that participants were from different backgrounds and regions.27 The classification of a country as an HIC or an LMIC was taken from the Statistical Annex of the World Economic Situation and Prospects 2022.28 Additional participants were identified using snowball sampling.29 59 intensive care professionals (physicians, nurses, pharmacists, physical therapists) from 24 countries agreed to participate. Interviews were held via telephone or video call between December 2021 and August 2022. All interviews were conducted in English, except for seven interviews which were held in Spanish. A researcher-developed semi-structured interview guide was developed to guide the discussion (see online supplemental information 2). It should be noted that the interviews were conducted prior to the release of ChatGPT and other chatbots powered by large language models (LLMs).30 Interviews were audio recorded and transcribed and were analysed in their original language using conventional content analysis with the assistance of the qualitative software MAXQDA (VERBI Software).31
Results
Among the 59 intensive care professionals who participated in the study, 69.5% were physicians (41/59), 18.6% were nurses (11/59), 6.8% were pharmacists (4/59) and 3.4% were physical therapists (2/59). Overall, 23.7% of participants were from Europe (14/59), 16.9% were from Asia (10/59), 15.3% were from North America (9/59), 13.6% were from South America (8/59), 11.9% were from the Middle East (7/59), 10.2% were from Australasia (6/59) and 6.8% were from sub-Saharan Africa (4/59). Furthermore, 66.1% (39/59) of participants were male presenting (table 1).
Table 1
|
Participants demographics
Status quo—patient data collection, documentation and utilisation
Most participants described a pervasive lack of digital data collection and documentation, and a chronic underutilisation of patient data in ICUs in both HICs and LMICs. In relation to patient data collection and documentation, most ICUs were reported to be paper-based or partially digitalised. Although patient data may be being collected with electronic monitors in these ICUs, it is typically documented manually either in paper-based records or in electronic health records. Consequently, the amount of available digital data was reported to be limited in most ICUs. With regard to the use of patient data for purposes other than patient care, although most ICUs are using data for national quality benchmarking data sets, the secondary use of patient data was reported to be extremely limited or non-existent by most participants.
Only a few participants working in a small number of large tertiary hospitals in HICs reported that patient data in their ICUs were primarily being automatically collected and documented digitally and being extensively used for secondary purposes. However, these were outliers and participants reporting that most other ICUs within the same country or even city as these fully digitalised ICUs were only paper-based or partially digitalised. Furthermore, even in most fully digitalised ICUs, it was reported that data is still required to be manually verified at regular intervals due to regulatory requirements to ensure data validity. Nevertheless, participants noted that in practice large amounts of data would often be confirmed without detailed verification. A minority of participants reported that verification is not required in their ICU; they want the raw data and did not think that nurses at the bedside were best placed to check data validity and that their time would be best spent on other tasks (table 2).
Table 2
|
Status quo—patient data collection, documentation and utilisation
Views about using AI in ICUs
Perceived opportunities
Although there were large variations in knowledge of AI among participants, and the vast majority are currently not using AI technology in practice, all participants in both HICs and LMICs had a generally positive view of AI. Participants saw huge potential for the technology to be very helpful and improve patient outcomes in the ICU, although not all participants had a clear idea of what or how benefits would happen. Many participants, however, highlighted the potential benefits of AI in relation to their workload given the number of patients they needed to simultaneously look after and the impossibility of keeping track of all the information being constantly generated in the ICU. AI was seen as a tool to support intensive care professionals deal with this data overload and to do their jobs more effectively and efficiently; by providing an early warning system for patients deteriorating, predicting which patients are at greatest risk and reducing errors. Many participants also noted the potential for AI to improve workflows, such as helping to manage ICU bed capacity or improving the accuracy of documentation (table 3).
Table 3
|
Views about using AI in ICUs
Concerns about use
Most intensive care professionals, however, also held some well-known concerns about the use of AI in clinical practice. There were no important differences regarding the concerns expressed by participants from HICs and LMICs. Five key concerns emerged from the interviews:
Validity
A major concern raised by participants was regarding the risk of AI technology being biased and not generalisable. Participants were very concerned about AI applications not being applicable in real life to the majority of patients, particularly in ICU where there is such a heterogeneous group of patients. Participants were also concerned that AI technology would not work as well with minorities who are already disadvantaged (eg, Indigenous communities or those with limited healthcare access) if those groups are not sufficiently present in the training data set.
Explainability
Some participants thought explainable AI was necessary as they always needed to understand exactly why they were doing something when working with critically ill patients, and that a lack of understanding could generate fear and undermine the trust of clinicians and patients. However, most participants were not concerned about ‘blackbox’ AI applications and thought that evidence that an application was helpful and safe was far more important than explainability. These participants noted that they did not understand how many other technologies used in the ICU worked and that clinical judgement should not be based purely on an algorithm but should combine a range of patient information and professional expertise.
Responsibility
Most participants saw the issue of responsibility being dependent on how AI was used. If AI was used in place of a clinician, making changes to patient care independently, then the question of who would be responsible if things went wrong was seen as very problematic by many. However, if AI was used as just another tool to help clinical decision-making, then participants thought that there was no significant problem and responsibility would remain with the clinician.
Dependency
Many participants raised concerns about clinicians becoming too dependent and trusting of AI technology in the ICU and not using their own clinical judgement or skills. Participants saw this as part of a wider problem related to increasing digitalisation. Although this technology potentially has benefits, participants reported many junior staff becoming too reliant on technology, which was leading to (1) deskilling of staff, who can no longer do certain tasks themselves (eg, calculate dosages) because the system is down, and (2) a dehumanisation of care, with staff spending too much time looking at the computer screen to the detriment of personal care of the patient.
Disparity
Some participants were also concerned that there will be large disparities in the application and use of AI technology in ICUs, which is going to widen the gap between richer and poorer settings.
Barriers and challenges to implementing AI in ICUs
Three overarching barriers to implementing AI in ICUs emerged (table 4):
Digital infrastructure: Participants from both HICs and LMICs identified the current digital infrastructure of institutions as a major barrier. Most participants reported that their institution has neither the technical capability (hardware and software) to capture the necessary data or run the algorithms, nor the staff with the right knowledge and skills to use the technology. Some participants in LMICs reported not even having a stable electricity supply. This pointed to ongoing structural problems in the organisation and delivery of healthcare, and many participants in both HICs and LMICs described how they worked in broken healthcare systems where funds were limited to varying degrees, and investing in digitalisation and AI is not a priority. They suggested that many decision-makers either did not understand the value of digital technologies for improving patient care or were too burdened by the existing financial strain on their health system.
Knowledge and understanding: Participants also identified a lack of knowledge and understanding about AI and the clinical context these tools will be implemented in as a significant barrier. Participants felt that this affected professionals’ and patients’ acceptance and willingness to use AI, and that the disconnect between clinicians and technical partners too often leads to non-optimal AI tools. Indeed, one participant described most AI applications as ‘solutions looking for problems’ that do not exist in the view of clinicians. Participants also reported that some colleagues’ views about data ownership and competition led them to be unwilling to share data, which was also reported to be a substantial challenge that undermines AI implementation.
Regulatory: Large variations in regulations regarding data protection within and across countries were also highlighted by participants as an important barrier. Some institutions and countries were reported to be significantly stricter than others with regard to data sharing and the secondary use of data. Although participants all agreed that protecting patient privacy was essential, they also felt that the current situation could potentially harm patients because it is undermining research and their ability to improve care.
Table 4
|
Barriers and challenges for implementing AI in ICUs
Facilitators for implementing AI in ICUs
Three key suggestions for facilitating and improving the implementation of AI in ICUs emerged (table 5):
Demonstrating the value/limits of AI
Table 5
|
Facilitators for implementing AI in ICUs
Participants thought that clear and consistent evidence from robust research studies confirming the utility and reliability of AI applications would be the most important facilitator for increasing the acceptance of and willingness to use AI applications in ICUs. Participants also saw a need for a clear explanation of the strengths/weaknesses and advantages/disadvantages of each application.
Closing the gap of understanding
Participants made two main suggestions for improving the current gap of understanding between clinicians and technical partners, to increase the acceptance and the clinical utility of AI in ICUs:
Training and education: Many participants noted the need to improve training and education of both clinicians and data scientists, so clinicians have a better understanding of AI concepts and data scientists understand the clinical context better. Although participants saw the need to improve all intensive care professionals’ knowledge and skills in this area, some participants also advocated for a new (sub)specialty where clinicians are trained in AI as it was unrealistic to think that all clinicians could be trained to the required level.
More inclusion of clinicians: Participants strongly felt that there needed to be more consultation and involvement of clinicians from the beginning in the design and development of AI applications for ICUs, to improve the connection between clinicians and developers and the resulting product.
Improving ecosystems
Participants also saw the need to improve the wider ecosystem, including: ensuring that there is a proper system of data collection and documentation, that funding bodies are aware of bottlenecks so funding is directed to efforts to translate research into practice rather than just generating more accurate prediction models, that grant panels have the right expertise to evaluate multidisciplinary research and enhance the potential for academic/commercial partnerships.
Discussion
This is one of the largest qualitative studies to date to examine the views of intensive care professionals regarding the use and implementation of AI technologies in ICUs, involving 59 participants from 24 countries, including countries from Europe, Asia, North America, South America, Middle East, Australasia and sub-Saharan Africa. This study found general agreement among participants’ views regarding the use and implementation of AI in ICUs, which were largely in line with existing empirical research with ICU professionals.13 14 19 20 Participants had generally positive views about the potential use of AI in ICUs but identified important technical and non-technical barriers to the implementation of AI. A key finding of this study, however, was important differences between ICUs regarding their current readiness to implement AI. It was striking that these differences were not primarily between HICs and LMICs as might be expected. Rather, the key difference was between a small number of ICUs in large tertiary hospitals in HICs, which were reported to have the necessary digital infrastructure for AI, and nearly all other ICUs in both HICs and LMICs, which were reported to neither have the technical capability to capture the necessary data or run AI algorithms, nor the staff with the right knowledge and skills to use the technology. Although technical barriers to implementing AI in ICUs have been widely discussed,4–6 11 14 intensive care medicine needs to be careful not to gloss over the importance of the current readiness of ICUs to implement and use AI, otherwise it will risk building a house of cards. Pouring massive amounts of resources into developing AI without first (or in parallel) building the necessary digital and knowledge infrastructure foundation needed for AI is unethical.32 We do not see the possibility of real-world implementation and routine use of AI in the vast majority of ICUs in both HICs and LMICs included in our study any time soon, and we do not think this ‘last mile’ of implementation33 will be reached unless the necessary digital and knowledge infrastructures are built first. We are of the view that ICUs should not be using AI until certain preconditions are met. Intensive care societies from around the world need to come together and reach a consensus on what these preconditions should be.
Limitations
This is a qualitative study that did not collect statistically representative data. However, we included a range of intensive care professionals from 24 HICs and LMICs, which makes it likely that this study has captured key aspects of a multisided issue. A bias might exist toward the reporting of socially desirable attitudes,34 however, given our results that are rather critical of current practice, we believe that such a bias is limited. The study was carried out across 24 countries, and there may be some regional and country-specific differences that might limit the generalisability. Nevertheless, many of the key issues are associated with aspects that are common in all countries (eg, limited digital data collection and documentation, and an underutilisation of patient data in ICUs), these findings are likely to be of wider international interest. There is currently no established definition of what constitutes AI, and a definition of AI in medicine was not provided to participants. As noted in the results section there were large variations in knowledge of AI among participants, and concrete examples were provided where needed. However, this may have affected the ability of some participants with limited knowledge of AI to answer some questions. The study was also undertaken before the explosion of interest in the use of LLMs and the chatbots that they power. The AI discussed in this manuscript therefore does not include LLMs.
