Adoption of clinical risk prediction tools is limited by a lack of integration with electronic health records ============================================================================================================= * Videha Sharma * Ibrahim Ali * Sabine van der Veer * Glen Martin * John Ainsworth * Titus Augustine * BMJ health informatics * health care * information management * information science * medical informatics ## Introduction Prognostic risk prediction models aim to estimate the risk of a future outcome based on available clinical parameters.1 There has been an increase in the development of such models given the move towards personalised and precision medicine, since they provide individualised risks for patients.2 3 They can help convey risks and benefits more succinctly and promote shared decision-making. Despite their benefits, risk prediction models in front-line clinical practice remain underutilised and their potential impact on care outcomes has not been fullfilled.4 A recent systematic review of clinical decision support systems by Kwan *et al* published in the *BMJ* demonstrated only a poor to moderate improvement of care and highlighted the importance of designing models and tools that critically consider care processes and patient outcomes.5 Ongoing challenges include poor methodological development and lack of external validation of models.6 7 However, where robust models have been externally validated, an underappreciated barrier to their adoption in clinical practice is the lack of integration with electronic health records (EHRs). ## Lack of integration as a barrier to use Clinical risk prediction models have clear potential to influence clinical decision-making and enhance the quality of care delivered to patients.8 9 However, developing a successful model is a rigorous process with many pitfalls, such as incomplete training data, risk of bias and failure to address clinical need. There are further challenges to externally validate and calibrate a model across different patient groups before being accepted for clinical use.10 As a result, although there is a large body of literature on the development of risk prediction models, the evidence of successful clinical adoption and impact on care outcomes is largely absent.11 Risk prediction models are primarily developed using routinely collected clinical data, increasingly retrieved from EHRs.12 13 Thus, the variables selected and assessed during model development are those available in electronic data repositories, such as demographics, diagnostic results, medical history or drug history. Some models that were robustly validated and gained international recognition were converted to online tools and made available through web-interfaces or mobile applications. An example of such a model is the CHA(2)DS(2)-VASc score, which is used to predict the risk of stroke in patients with atrial fibrillation (AF) and thus guide anticoagulation.14 It has successfully achieved clinical impact and is the gold-standard risk prediction model for AF management as recommended by the National Institute of Health and Care Excellence.15 To use the model however, a healthcare professional is required to access a website or open an app, manually complete data fields with the patients’ details and receive a risk score to guide clinical decision-making. Though this task may seem trivial compared with the potential added benefit of greater quality decision-making, the practicalities and time constraints of clinical practice form a significant barrier to usage. This is compounded with the potential of manual transcription errors, which form a hazard of receiving incorrect results. Another example of this is the kidney failure risk equation (KFRE) developed by Tangri *et al*, which is similarly available as an online tool.16 This model uses routinely collected clinical data including patient age, gender, estimated glomerular filtration rate and urinary albumin:creatinine ratio, to provide a 2-year and 5-year risk of progression to kidney failure for patients with chronic kidney disease. The KFRE has been validated internationally, and is generally reviewed positively.17 However, its widespread adoption is limited by the dependence on externally accessing the tool online and manually transcribing the variables into data fields before a risk score is presented. This impractical process has been shown to contribute as a barrier to clinical impact in primary care settings.18 A number of initiatives have attempted to increase the usability of risk prediction tools by developing user-friendly interfaces. An example of this is MDCalc(c), which is a medical calculator available via a website and a mobile application. By making the content easy to navigate and using an intuitive visual design it aims to enhance the user experience. However, the fundamental barrier of accessing the interface as an external application and manually completing fields is yet to be overcome. This is particularly a challenge as many healthcare institutions still lack interoperable EHRs and store clinical data across multiple digital systems. This means that a healthcare professional wishing to use a risk prediction tool may have to access multiple electronic sources to gather the required data to complete the fields and obtain a risk. As healthcare providers increasingly turn to unified EHRs, the success of risk prediction models will be dependent on the integration of tools within these systems. Usability barriers may be mitigated if clinicians can access risk prediction tools, pertinent to their practice, within their local EHR and have a risk score presented automatically as fields are populated with relevant data from within the system. This intuitively simple concept would create a paradigm shift for the practical daily use of such tools and translate to patient benefit (figure 1). Risks may be presented graphically over a period of time to illustrate the impact of risk-addressing therapies and thus promote compliance. By improving accessibility in this way, it will also have an impact on future academic research evaluating these tools’ performance and impact on clinical outcomes. Currently, research into usability of risk prediction tools, as standalone interfaces, or within EHRs is largely absent. User experience is a significant part of successful product development in areas outside of healthcare and formal methodology for evaluation in other fields has been established. Recognising the importance of this as part of model development is crucial to achieve value out of future solutions.19 ![Figure 1](http://informatics.bmj.com/https://informatics.bmj.com/content/bmjhci/28/1/e100253/F1.medium.gif) [Figure 1](http://informatics.bmj.com/content/28/1/e100253/F1) Figure 1 Current and proposed use of risk prediction tools. EHRs, electronic health records. ## Future concepts For risk prediction models that have undergone rigorous validation and assessment of clinical impact, integrating tools into EHRs will likely overcome a major barrier to use. Unfortunately, this practical implication has not been widely explored and new tools continue to appear as web-interface solutions risking non-adoption and thus failure to impact care. An example of such a recent model is the iPREDICTLIVING (2019) developed to predict risks around kidney donation to better inform renal transplant decision-making.20 In the context of a sensitive and complex clinical decision as kidney donation, detracting the clinician from the human interaction by a time-consuming on-screen process will likely impact the patient experience. Digital health interventions should be centred around improving the quality of care delivered to patients, which includes better decisions, but also enhancing the patient–clinician relationship by providing clinicians the time to consult patients. To realise a more streamlined workflow, a change in how we think about clinical risk prediction models is required. Front-line usability should be part of the initial exploration of the proposed model. This means involving clinicians (end-users) at the outset as part of research projects to understand how the tool would be practically used and the impact it would have on clinical encounters. The usability of such interventions plays a crucial role in preventing clinician fatigue and improving uptake.21 Technical challenges revolve around non-standardised coding of health data across EHR providers.22 This means that even if a risk prediction model is made available as a standalone software, which can be integrated, misaligned clinical terminology may limit implementation. Involving EHR vendors early in the development of risk prediction models and imploring greater alignment across the industry will mitigate barriers to implementation and subsequent scale-up of novel solutions. An example of successful tool integration is QRISK, which has been embedded within a number of primary care clinical management systems.23 The tool calculates individual cardiovascular risk and generates a score based on existing data. Not only has this impacted positively on front-line practice, regular use provides evolving data quality and completeness reflecting the changing population characteristics over time. This has allowed researchers to update and calibrate the tool for long-term accuracy.24 Similar implementation through hospital EHR vendors may bring such models into routine secondary care settings unifying and standardising practice. Another relevant example was the PREDICT software used in general practice in New Zealand, which automatically recorded patients’ risk profiles for cardiovascular disease and prospectively linked this to coded hospital and mortality databases. This allowed a risk prediction model to be developed that took in to account an area-based deprivation index and self-reported ethnicity alongside clinical parameters, resulting in greater personalised risk profiles for individual patients. The strength of this study was its prospective nature and ability to seamlessly collect healthcare data without additional intervention by clinicians delivering care.25 The tremendous potential of clinical risk prediction models mandates policy-makers to establish regulations to standardise the integration of tools into EHRs. Strategies to achieve this may be through EHR vendors working directly with data scientists to incorporate statistical models within their user interface, or alternatively provide non-proprietary application programming interfaces for third party developers to seamlessly integrate with. The potential success of this however, heavily relies on the engagement of front-line healthcare professionals who can provide the clinical context and workflow that a risk prediction model is intending to influence. Encouraging multidisciplinary research and development teams, which can appreciate the different facets of clinical context, statistical modelling and implementation science, supported by EHR vendors working to unified standards has the potential to bridge the current bench-to-bedside gap for clinical risk prediction models. ## Footnotes * Twitter @VidehaSharma * Contributors VS conceptualised the manuscript. VS and IA reviewed the literature and wrote the manuscript. GM reviewed the literature and edited the manuscript. SV, JA and TA reviewed and edited the manuscript. VS created the manuscript figures. * Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-­for-­profit sectors. * Competing interests None declared. * Patient consent for publication Not required. * Provenance and peer review Not commissioned; internally peer reviewed. * Received October 9, 2020. * Accepted February 3, 2021. * © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/) This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/). ## References 1. Steyerberg EW. Clinical prediction models. Springer, 2019. 2. Banning M. A review of clinical decision making: models and current research. J Clin Nurs 2008;17:187–95.[doi:10.1111/j.1365-2702.2006.01791.x](http://dx.doi.org/10.1111/j.1365-2702.2006.01791.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/17331095 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=17331095&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000251630600007&link_type=ISI) 3. Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med 2002;9:1184–204.[doi:10.1197/aemj.9.11.1184](http://dx.doi.org/10.1197/aemj.9.11.1184)pmid:http://www.ncbi.nlm.nih.gov/pubmed/12414468 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1197/aemj.9.11.1184&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=12414468&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000179203100019&link_type=ISI) 4. Ahmed I, Debray TPA, Moons KGM, et al. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Med Res Methodol 2014;14:3. [doi:10.1186/1471-2288-14-3](http://dx.doi.org/10.1186/1471-2288-14-3)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24397587 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1186/1471-2288-14-3&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=24397587&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 5. Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020;370:m3216.[doi:10.1136/bmj.m3216](http://dx.doi.org/10.1136/bmj.m3216)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32943437 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNzAvc2VwMTdfMy9tMzIxNiI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI4LzEvZTEwMDI1My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 6. Wynants L, Van Calster B, Collins GS, Bonten MM, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ 2020;369:m1328. [doi:10.1136/bmj.m1328](http://dx.doi.org/10.1136/bmj.m1328)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32265220 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjkvYXByMDdfMi9tMTMyOCI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI4LzEvZTEwMDI1My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 7. Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014;14:40. [doi:10.1186/1471-2288-14-40](http://dx.doi.org/10.1186/1471-2288-14-40)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24645774 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1186/1471-2288-14-40&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=24645774&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 8. Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014;2:3.[doi:10.1186/2047-2501-2-3](http://dx.doi.org/10.1186/2047-2501-2-3)pmid:http://www.ncbi.nlm.nih.gov/pubmed/25825667 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 9. Topol E. The Topol review: preparing the healthcare workforce to deliver the digital future. Health Educ J 2019. 10. Chen L. Overview of clinical prediction models. Ann Transl Med 2020;8:71.[doi:10.21037/atm.2019.11.121](http://dx.doi.org/10.21037/atm.2019.11.121)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32175364 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 11. Dekker FW, Ramspek CL, van Diepen M. Con: most clinical risk scores are useless. Nephrol Dial Transplant 2017;32:752–5.[doi:10.1093/ndt/gfx073](http://dx.doi.org/10.1093/ndt/gfx073)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28499028 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 12. Goldstein BA, Navar AM, Pencina MJ, et al. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc 2017;24:198–208.[doi:10.1093/jamia/ocw042](http://dx.doi.org/10.1093/jamia/ocw042)pmid:http://www.ncbi.nlm.nih.gov/pubmed/27189013 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1093/jamia/ocw042&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=27189013&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 13. Rothman B, Leonard JC, Vigoda MM. Future of electronic health records: implications for decision support. Mt Sinai J Med 2012;79:757–68.[doi:10.1002/msj.21351](http://dx.doi.org/10.1002/msj.21351)pmid:http://www.ncbi.nlm.nih.gov/pubmed/23239213 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1002/msj.21351&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=23239213&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 14. Lip GYH, Nieuwlaat R, Pisters R, et al. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the Euro heart survey on atrial fibrillation. Chest 2010;137:263–72.[doi:10.1378/chest.09-1584](http://dx.doi.org/10.1378/chest.09-1584)pmid:http://www.ncbi.nlm.nih.gov/pubmed/19762550 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1378/chest.09-1584&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=19762550&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000274612600006&link_type=ISI) 15. National Institute of Health and Care Excellence. Clinical guideline 180 (CG180): atrial fibrillation: management, 2014. 16. Tangri N, Stevens LA, Griffith J, et al. A predictive model for progression of chronic kidney disease to kidney failure. JAMA 2011;305:1553–9.[doi:10.1001/jama.2011.451](http://dx.doi.org/10.1001/jama.2011.451)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21482743 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1001/jama.2011.451&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21482743&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000289683000024&link_type=ISI) 17. Peeters MJ, van Zuilen AD, van den Brand JAJG, et al. Validation of the kidney failure risk equation in European CKD patients. Nephrol Dial Transplant 2013;28:1773–9.[doi:10.1093/ndt/gft063](http://dx.doi.org/10.1093/ndt/gft063)pmid:http://www.ncbi.nlm.nih.gov/pubmed/23645476 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1093/ndt/gft063&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=23645476&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 18. Major RW, Shepherd D, Medcalf JF, et al. The kidney failure risk equation for prediction of end stage renal disease in UK primary care: an external validation and clinical impact projection cohort study. PLoS Med 2019;16:e1002955. [doi:10.1371/journal.pmed.1002955](http://dx.doi.org/10.1371/journal.pmed.1002955)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31693662 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 19. Vermeeren AP, Roto V. User experience evaluation methods: current state and development needs. Proceedings of the 6th Nordic conference on human-computer interaction: Extending boundaries, 2010:521–30. 20. Haller MC, Wallisch C, Mjøen G, et al. Predicting donor, recipient and graft survival in living donor kidney transplantation to inform pretransplant counselling: the donor and recipient linked iPREDICTLIVING tool - a retrospective study. Transpl Int 2020;33:729–39.[doi:10.1111/tri.13580](http://dx.doi.org/10.1111/tri.13580)pmid:http://www.ncbi.nlm.nih.gov/pubmed/31970822 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 21. Sutton RT, Pincock D, Baumgart DC, et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020;3:1–10.[doi:10.1038/s41746-020-0221-y](http://dx.doi.org/10.1038/s41746-020-0221-y) 22. D'Amore JD, Mandel JC, Kreda DA, et al. Are meaningful use stage 2 certified EHRs ready for interoperability? findings from the smart C-CDA collaborative. J Am Med Inform Assoc 2014;21:1060–8.[doi:10.1136/amiajnl-2014-002883](http://dx.doi.org/10.1136/amiajnl-2014-002883)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24970839 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1136/amiajnl-2014-002883&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=24970839&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom) 23. Hippisley-Cox J, Coupland C, Vinogradova Y, et al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ 2008;336:1475–82.[doi:10.1136/bmj.39609.449676.25](http://dx.doi.org/10.1136/bmj.39609.449676.25)pmid:http://www.ncbi.nlm.nih.gov/pubmed/18573856 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMzYvNzY1OS8xNDc1IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpoY2kvMjgvMS9lMTAwMjUzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 24. Hippisley-Cox J, Coupland C, Brindle P. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ 2017;357:j2099. [doi:10.1136/bmj.j2099](http://dx.doi.org/10.1136/bmj.j2099)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28536104 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTcvbWF5MjNfNS9qMjA5OSI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI4LzEvZTEwMDI1My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 25. Pylypchuk R, Wells S, Kerr A, et al. Cardiovascular disease risk prediction equations in 400 000 primary care patients in New Zealand: a derivation and validation study. Lancet 2018;391:1897–907.[doi:10.1016/S0140-6736(18)30664-0](http://dx.doi.org/10.1016/S0140-6736(18)30664-0)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29735391 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(18)30664-0&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100253.atom)