Machine learning in medical education: a survey of the experiences and opinions of medical students in Ireland
==============================================================================================================

* Charlotte Blease
* Anna Kharko
* Michael Bernstein
* Colin Bradley
* Muiris Houston
* Ian Walsh
* Maria Hägglund
* Catherine DesRoches
* Kenneth D Mandl

*   artificial intelligence
*   machine learning

## Introduction

Leading figures in biomedical informatics advocate education in digital health for the healthcare workforce.1 2 In healthcare, artificial intelligence/machine learning (AI/ML)-enabled tools increasingly play a role by informing patient triage decisions, clinical decision support systems, and healthcare resource management3 – advances that are undoubtedly set to grow.4 Tens of thousands of healthcare apps are available for download by consumers, promising a range of services, from symptom tracking to diagnostic and treatment advice.

To date, surveys of medical professionals reveal divergent views about the value and impact of AI/ML on their job with many physicians sceptical about the potential scope for technological innovations on medical tasks.5–7 Furthermore, surveys consistently find limited evidence of formal teaching in medical education about AI/ML. Only a few studies – conducted in Europe, the US and South Korea – have explored the formal education and familiarity of medical or healthcare students with respect to digital advances in healthcare, and much of this work consists of single site studies.8–14 To better understand and engage with discussion about the benefits, limitations, and ethical dilemmas presented by these tools, today’s medical students will need to become more digitally savvy. Equally, as patients make increasing use of healthcare and well-being algorithms, medical students will need to become better prepared to offer patients advice, and to have knowledge about, the robustness of these tools including when algorithms are safe to use.

In the present study, we built on this research by assessing the experiences and opinions of final year medical students throughout Ireland about their exposure to AI/ML during their entire degree programme.

## Methods

A paper-based, cross-sectional survey was administered to final year medical students at four of Ireland’s seven medical schools. Institutions were selected in each of the country’s four geographical provinces. The study team devised an original survey instrument to investigate the familiarity, formal exposure to, and opinions of medical students about ML/AI in medicine. We developed the survey instrument in consultation with Irish, British, and American physicians and piloted the survey with physicians in Ireland and the UK (n=6) and final year medical students in the UK (n=5) to ensure face validity. The survey explored students’ experiences and opinions about the teaching of AI/ML in their medical degree programme to date (see Section E of online supplemental appendix 1, and table 1 for survey items). Using ‘yes’ or ‘no’ responses, the survey asked whether students had heard of the term ‘machine learning’, were familiar with “big data analytics”, and whether they had read any academic articles on AI/ML in medicine. Students were requested to estimate both how many hours their instructors or lecturers had spent, and will spend, discussing AI/ML during their degree. In addition, selecting from ‘yes’, ‘no’ or ‘maybe’ responses, the survey inquired whether students planned to learn about how AI/ML as it pertains to medicine. Finally, using a 6-point Likert scale, students were requested to rate their level of agreement with the statement ‘Discussion about AI/ML should be part of medical training.’

### Supplementary data

[[bmjhci-2021-100480supp001.pdf]](pending:yes) 

View this table:
[Table 1](http://informatics.bmj.com/content/29/1/e100480/T1)

Table 1 
Familiarity and opinions of medical students about Artificial Intelligence/Machine learning in their medical degree

The institutional review boards at University College Cork [protocol # 2018–188], National University of Ireland Galway [protocol # 19-Dec-15], Queen’s University Belfast [protocol # 19.28], and University College Dublin [protocol # LS-19–89] approved the study protocol at their respective sites. Between April 2019 and March 2020, the anonymous survey was distributed by lecturers after compulsory final year classes at each institution to increase responses. Participation was voluntary and all students who decided to participate provided written consent. After survey collection, responses were entered into Excel, and descriptive statistics and analysis were carried out using JASP (0.9.2) and SPSS v 27.

## Results

A total of 252 of 585 (43%) of final year students across three medical schools responded. Data collection at one medical school (University College Dublin) was terminated in March 2020 because of teaching disruption due COVID-19, and survey data from this site was excluded from the analysis. Of all respondents, 157 of 251 (62.6%) were female, and 223 of 246 (90.7%) were born in 1992 or later. Among respondents, 66.5% reported zero hours of teaching on AI/ML during their degree with 62.4% anticipating zero hours during the remainder of the degree programme, 43.4% (95% CI, 37.1% to 49.6%) had not heard of the term ‘machine learning’, and 80.6% (95% CI, 75.6% to 85.6%) had not read any academic journal articles on AI/ML. Asked about whether they intended to learn about AI/ML in medicine 41.1% (95% CI, 34.9% to 47.3%) reported ‘yes’ and 46.5% (95% CI, 40.2% to 52.8%) responded ‘maybe.’ However, 78.6% agreed that discussion about AI/ML should form part of their training. Results are reported in table 1.

Descriptive data were analysed for differences according to gender and birth year. Male respondents were more likely than females to report having heard about ML (69.7% v. 48.7%), χ2(1)=10.05, p=0.002. Participants who heard about ML, on average, had an earlier birth year than those who had not, t(234)=2.193, p=0.029. Willingness to learn about AI/ML was recoded to reflect the ordinal nature of the data (yes=1, maybe=2, and no=3) so that inferential statistics could be run. There was a trend towards younger participants being less likely to plan to learn about AI/ML, rho=-.109, p=0.095. Based on the results of a Mann-Whitney U test, male respondents were more likely to plan to learn about AI/ML than female participants, Z=2.25, p=0.025.

## Discussion

This is the first study to explore the experiences and opinions of Irish medical students about AI/ML in their medical degree programme. Medical students reported limited awareness and education on AI/ML. Notably, around four in ten of survey respondents had not heard of the term ‘machine learning’. Around two in three respondents reported no time spent learning about AI/ML during their whole medical degree. Although a minority of students did report some formal teaching on AI/ML, it is unclear whether this was part of their compulsory medical curriculum or (for example) via elective medical courses or guest lectures. Perhaps reflecting training gaps or lack of confidence on the topic, few students reported reading any academic articles on AI/ML in medicine. Relatedly, students were divided about their plans to fill educational gaps, with almost half of students reporting some uncertainty about whether they would undertake additional learning on these topics. Contrary to our expectations, younger participants were less likely to have heard of ML; however, the majority of participants were typically young adults: 91% had a birth year between 1992–1999. Conceivably, with greater variance in ages of participants we might have observed different findings. Finally, while the majority of students reported a lack of formal instruction on AI/ML in medicine, considerably fewer students seemed to approve of the status quo. In common with other surveys,8 9 12–14 the majority of medical students considered learning about AI/ML should form part of their formal medical degree.

To help address education deficits, we suggest medical schools consider developing short, cross-disciplinary courses in digital health, including an understanding of augmented intelligence, to empower students to keep abreast of technological advances. Indeed, the need for further education on these topics may also apply to allied health professional training including nursing, pharmacy, clinical psychology, and physiotherapy. Because technology changes rapidly, we recommend that training and education encompass critical thinking skills so that students are well equipped to appraise new technologies. For example, courses in evidence-based medicine might incorporate discussion about evaluation of clinical decision support systems, the potential for algorithmic biases in data sets, and challenges associated with the explainability of AI/ML decisions. Medical ethics courses might usefully incorporate topics related to patient privacy with the use of digital devices and apps, and the potential for AI/ML-tools to mitigate or exacerbate digital divides in healthcare. Finally, we caution that without solid curricular advances, medical students and health professionals may rely too heavily on hype or inflated media reportage to inform their views, leading to negative consequences for healthcare. For example, surveys in Canada and the UK suggest that, under the misguided view that radiology will be imminently replaced as a field by AI/ML, students are more likely to rule out this specialty as a career choice.12 15 

This study has some strengths and limitations. A strength was soliciting the views of students from institutions in geographically distinctive regions of the country. However, the moderate response rate (43%) raises questions about representativeness. Response biases could also have influenced our findings depending on whether students most enthusiastic or those inclined to view AI/ML negatively answered the survey. While our aim was to gauge the general awareness of medical students about these topics, some survey items, such as ‘familiarity with big data analytics’ might be challenged as vague and open to interpretation. We recommend that qualitative research methods might provide more nuanced findings on students’ opinions and awareness about AI/ML in medicine. In addition, we suggest future studies might usefully explore the opinions and familiarity of medical faculty about AI/ML in medical education, and/or evaluate medical curricula course content to assess where, if at all, students acquire learning on these topics. Finally, the survey was administered prior to the COVID-19 pandemic which has overseen considerable developments and attention given to the role of AI/ML-enabled tools including in digital epidemiology and public health. Conceivably, as a result, had the survey been undertaken today we might have found increased awareness or familiarity about these topics among medical students. However, we emphasise it remains to be seen whether this heighted attention translates into tangible curricular developments. Furthermore, no surveyed medical school has since modified their curriculum to include education about AI/ML.

We close by noting, in recent years Ireland has gained recognition as a global technology hub with the fastest growing tech workforce in Europe.16 Despite these advances, we cannot help but observe the risk of digital education in healthcare lagging behind. Improvements in digital education will help prepare tomorrow’s doctors to lead policy and practice advances on the role of AI/ML-enabled tools in the health professions and in patientcare.

## Data availability statement

Data are available upon reasonable request.

## Ethics statements

### Patient consent for publication

Not applicable.

### Ethics approval

This study involves human participants and was approved by Ethics approval was granted by the institutional review boards at University College Cork (#2018-188), National University of Ireland Galway (#19-Dec-15), Queen’s University Belfast (#19.28), and University College Dublin (#LS-19-89). Participants gave informed consent to participate in the study before taking part.

## Acknowledgments

CB thanks Dr Cliona McGovern for assisting with data gathering prior to the termination of the study at UCD due to COVID-19, and Dr John Halamka for early discussions about the content of the survey.

## Footnotes

*   CD and KDM are joint senior authors.

*   Twitter @crblease, @AnnaKharko

*   Contributors Conceptualization: CB. Data curation: CB & AK. Formal analysis: AK, MB, CB. Survey Design: CB, CD, KDM, CBr. Survey Administration: CB, CBr, IW, MH. Writing – original draft: CB. Writing – review & editing: CB, CBr, IW, MH, CD, KDM, AK, MB, MH.

*   Funding Open access fees were paid by FORTE—the Swedish Research Council for Health, Working Life and Welfare through the research project “Beyond Implementation of eHealth” (2020-01229) awarded to MH.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

*   © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.   Topol E . The Topol review: preparing the healthcare workforce to deliver the digital future, 2019. Available: [https://topol.hee.nhs.uk/](https://topol.hee.nhs.uk/) [Accessed 6 Jun 2021].
    
    

2.   Obermeyer Z , Lee TH . Lost in Thought - The Limits of the Human Mind and the Future of Medicine. N Engl J Med 2017;377:1209–11.[doi:10.1056/NEJMp1705348](http://dx.doi.org/10.1056/NEJMp1705348) pmid:http://www.ncbi.nlm.nih.gov/pubmed/28953443 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1056/NEJMp1705348&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

3.   Cerrato P , Halamka J . Reinventing clinical decision support: data analytics, artificial intelligence, and diagnostic reasoning. Taylor & Francis, 2020.
    
    

4.   Blease C , Kharko A , Locher C , et al . Us primary care in 2029: a Delphi survey on the impact of machine learning. PLoS One 2020;15:e0239947. [doi:10.1371/journal.pone.0239947](http://dx.doi.org/10.1371/journal.pone.0239947) pmid:http://www.ncbi.nlm.nih.gov/pubmed/33031411 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

5.   Blease C , Bernstein MH , Gaab J , et al . Computerization and the future of primary care: a survey of general practitioners in the UK. PLoS One 2018;13:e0207418. [doi:10.1371/journal.pone.0207418](http://dx.doi.org/10.1371/journal.pone.0207418) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30540791 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

6.   Doraiswamy PM , Blease C , Bodner K . Artificial intelligence and the future of psychiatry: insights from a global physician survey. Artif Intell Med 2020;102:101753. [doi:10.1016/j.artmed.2019.101753](http://dx.doi.org/10.1016/j.artmed.2019.101753) pmid:http://www.ncbi.nlm.nih.gov/pubmed/31980092 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

7.   Blease C , Locher C , Leon-Carlyle M , et al . Artificial intelligence and the future of psychiatry: qualitative findings from a global physician survey. Digit Health 2020;6:205520762096835.[doi:10.1177/2055207620968355](http://dx.doi.org/10.1177/2055207620968355) pmid:http://www.ncbi.nlm.nih.gov/pubmed/33194219 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

8.   Pinto Dos Santos D , Giese D , Brodehl S , et al . Medical students' attitude towards artificial intelligence: a multicentre survey. Eur Radiol 2019;29:1640–6.[doi:10.1007/s00330-018-5601-1](http://dx.doi.org/10.1007/s00330-018-5601-1) pmid:http://www.ncbi.nlm.nih.gov/pubmed/29980928 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1007/s00330-018-5601-1&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

9.   Wood EA , Ange BL , Miller DD . Are we ready to integrate artificial intelligence literacy into medical school curriculum: students and faculty survey. J Med Educ Curric Dev 2021;8:238212052110240.[doi:10.1177/23821205211024078](http://dx.doi.org/10.1177/23821205211024078) pmid:http://www.ncbi.nlm.nih.gov/pubmed/34250242 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

10.  Blease C , Kharko A , Annoni M , et al . Machine learning in clinical psychology and psychotherapy education: a mixed methods pilot survey of postgraduate students at a Swiss university. Front Public Health 2021;9:273.[doi:10.3389/fpubh.2021.623088](http://dx.doi.org/10.3389/fpubh.2021.623088) 
    
    

11.  Yüzbaşıoğlu E . Attitudes and perceptions of dental students towards artificial intelligence. J Dent Educ 2021;85:60–8.[doi:10.1002/jdd.12385](http://dx.doi.org/10.1002/jdd.12385) pmid:http://www.ncbi.nlm.nih.gov/pubmed/32851649 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

12.  Sit C , Srinivasan R , Amlani A , et al . Attitudes and perceptions of UK medical students towards artificial intelligence and radiology: a multicentre survey. Insights Imaging 2020;11:14. [doi:10.1186/s13244-019-0830-7](http://dx.doi.org/10.1186/s13244-019-0830-7) pmid:http://www.ncbi.nlm.nih.gov/pubmed/32025951 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

13.  Cho SI , Han B , Hur K , et al . Perceptions and attitudes of medical students regarding artificial intelligence in dermatology. J Eur Acad Dermatol Venereol 2021;35:e72–3.[doi:10.1111/jdv.16812](http://dx.doi.org/10.1111/jdv.16812) pmid:http://www.ncbi.nlm.nih.gov/pubmed/32852856 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

14.  Machleid F , Kaczmarczyk R , Johann D , et al . Perceptions of digital health education among European medical students: mixed methods survey. J Med Internet Res 2020;22:e19827. [doi:10.2196/19827](http://dx.doi.org/10.2196/19827) pmid:http://www.ncbi.nlm.nih.gov/pubmed/32667899 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

15.  Gong B , Nugent JP , Guest W , et al . Influence of artificial intelligence on Canadian medical students' preference for radiology specialty: ANational survey study. Acad Radiol 2019;26:566–77.[doi:10.1016/j.acra.2018.10.007](http://dx.doi.org/10.1016/j.acra.2018.10.007) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30424998 
    
    [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.acra.2018.10.007&link_type=DOI) 
    
    [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F29%2F1%2Fe100480.atom) 

16.  Hannon P . This Economy Grew Faster Than China’s Thanks to Big Tech, Pharma. The Wall Street Journal, 2021. Available: [https://www.wsj.com/articles/this-economy-grew-faster-than-china-thanks-to-big-tech-pharma-11614951060](https://www.wsj.com/articles/this-economy-grew-faster-than-china-thanks-to-big-tech-pharma-11614951060) [Accessed 11 Jul 2021].