TY - JOUR T1 - Evaluation framework to guide implementation of AI systems into healthcare settings JF - BMJ Health & Care Informatics JO - BMJ Health Care Inform DO - 10.1136/bmjhci-2021-100444 VL - 28 IS - 1 SP - e100444 AU - Sandeep Reddy AU - Wendy Rogers AU - Ville-Petteri Makinen AU - Enrico Coiera AU - Pieta Brown AU - Markus Wenzel AU - Eva Weicken AU - Saba Ansari AU - Piyush Mathur AU - Aaron Casey AU - Blair Kelly Y1 - 2021/10/01 UR - http://informatics.bmj.com/content/28/1/e100444.abstract N2 - Objectives To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components.Methods To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert.Results TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system.Discussion One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems.Conclusion The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.Data sharing not applicable as no datasets generated and/or analysed for this study. ER -