RT Journal Article SR Electronic T1 Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials JF BMJ Health & Care Informatics JO BMJ Health Care Inform FD BMJ Publishing Group Ltd SP e100466 DO 10.1136/bmjhci-2021-100466 VO 28 IS 1 A1 George C M Siontis A1 Romy Sweda A1 Peter A Noseworthy A1 Paul A Friedman A1 Konstantinos C Siontis A1 Chirag J Patel YR 2021 UL http://informatics.bmj.com/content/28/1/e100466.abstract AB Objective Given the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).Methods We searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.Results We found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).Conclusions We found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.All data relevant to the study are included in the article or uploaded as online supplemental information. Not applicable.