TY - JOUR T1 - Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment JF - BMJ Health & Care Informatics JO - BMJ Health Care Inform DO - 10.1136/bmjhci-2022-100549 VL - 29 IS - 1 SP - e100549 AU - Paul Festor AU - Yan Jia AU - Anthony C Gordon AU - A Aldo Faisal AU - Ibrahim Habli AU - Matthieu Komorowski Y1 - 2022/07/01 UR - http://informatics.bmj.com/content/29/1/e100549.abstract N2 - Objectives Establishing confidence in the safety of Artificial Intelligence (AI)-based clinical decision support systems is important prior to clinical deployment and regulatory approval for systems with increasing autonomy. Here, we undertook safety assurance of the AI Clinician, a previously published reinforcement learning-based treatment recommendation system for sepsis.Methods As part of the safety assurance, we defined four clinical hazards in sepsis resuscitation based on clinical expert opinion and the existing literature. We then identified a set of unsafe scenarios, intended to limit the action space of the AI agent with the goal of reducing the likelihood of hazardous decisions.Results Using a subset of the Medical Information Mart for Intensive Care (MIMIC-III) database, we demonstrated that our previously published ‘AI clinician’ recommended fewer hazardous decisions than human clinicians in three out of our four predefined clinical scenarios, while the difference was not statistically significant in the fourth scenario. Then, we modified the reward function to satisfy our safety constraints and trained a new AI Clinician agent. The retrained model shows enhanced safety, without negatively impacting model performance.Discussion While some contextual patient information absent from the data may have pushed human clinicians to take hazardous actions, the data were curated to limit the impact of this confounder.Conclusion These advances provide a use case for the systematic safety assurance of AI-based clinical systems towards the generation of explicit safety evidence, which could be replicated for other AI applications or other clinical contexts, and inform medical device regulatory bodies.Data are available in a public, open access repository. The MIMIC-III data set is openly available and requires applying formally for access. Access to our computer code used in this research is available by request to the corresponding author. ER -