Lack of integration as a barrier to use

Clinical risk prediction models have clear potential to influence clinical decision-making and enhance the quality of care delivered to patients.8 9 However, developing a successful model is a rigorous process with many pitfalls, such as incomplete training data, risk of bias and failure to address clinical need. There are further challenges to externally validate and calibrate a model across different patient groups before being accepted for clinical use.10 As a result, although there is a large body of literature on the development of risk prediction models, the evidence of successful clinical adoption and impact on care outcomes is largely absent.11

Risk prediction models are primarily developed using routinely collected clinical data, increasingly retrieved from EHRs.12 13 Thus, the variables selected and assessed during model development are those available in electronic data repositories, such as demographics, diagnostic results, medical history or drug history. Some models that were robustly validated and gained international recognition were converted to online tools and made available through web-interfaces or mobile applications. An example of such a model is the CHA(2)DS(2)-VASc score, which is used to predict the risk of stroke in patients with atrial fibrillation (AF) and thus guide anticoagulation.14 It has successfully achieved clinical impact and is the gold-standard risk prediction model for AF management as recommended by the National Institute of Health and Care Excellence.15 To use the model however, a healthcare professional is required to access a website or open an app, manually complete data fields with the patients’ details and receive a risk score to guide clinical decision-making. Though this task may seem trivial compared with the potential added benefit of greater quality decision-making, the practicalities and time constraints of clinical practice form a significant barrier to usage. This is compounded with the potential of manual transcription errors, which form a hazard of receiving incorrect results.

Another example of this is the kidney failure risk equation (KFRE) developed by Tangri et al, which is similarly available as an online tool.16 This model uses routinely collected clinical data including patient age, gender, estimated glomerular filtration rate and urinary albumin:creatinine ratio, to provide a 2-year and 5-year risk of progression to kidney failure for patients with chronic kidney disease. The KFRE has been validated internationally, and is generally reviewed positively.17 However, its widespread adoption is limited by the dependence on externally accessing the tool online and manually transcribing the variables into data fields before a risk score is presented. This impractical process has been shown to contribute as a barrier to clinical impact in primary care settings.18

A number of initiatives have attempted to increase the usability of risk prediction tools by developing user-friendly interfaces. An example of this is MDCalc(c), which is a medical calculator available via a website and a mobile application. By making the content easy to navigate and using an intuitive visual design it aims to enhance the user experience. However, the fundamental barrier of accessing the interface as an external application and manually completing fields is yet to be overcome. This is particularly a challenge as many healthcare institutions still lack interoperable EHRs and store clinical data across multiple digital systems. This means that a healthcare professional wishing to use a risk prediction tool may have to access multiple electronic sources to gather the required data to complete the fields and obtain a risk.

As healthcare providers increasingly turn to unified EHRs, the success of risk prediction models will be dependent on the integration of tools within these systems. Usability barriers may be mitigated if clinicians can access risk prediction tools, pertinent to their practice, within their local EHR and have a risk score presented automatically as fields are populated with relevant data from within the system. This intuitively simple concept would create a paradigm shift for the practical daily use of such tools and translate to patient benefit (figure 1). Risks may be presented graphically over a period of time to illustrate the impact of risk-addressing therapies and thus promote compliance. By improving accessibility in this way, it will also have an impact on future academic research evaluating these tools’ performance and impact on clinical outcomes. Currently, research into usability of risk prediction tools, as standalone interfaces, or within EHRs is largely absent. User experience is a significant part of successful product development in areas outside of healthcare and formal methodology for evaluation in other fields has been established. Recognising the importance of this as part of model development is crucial to achieve value out of future solutions.19

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Current and proposed use of risk prediction tools. EHRs, electronic health records.

Future concepts

For risk prediction models that have undergone rigorous validation and assessment of clinical impact, integrating tools into EHRs will likely overcome a major barrier to use. Unfortunately, this practical implication has not been widely explored and new tools continue to appear as web-interface solutions risking non-adoption and thus failure to impact care. An example of such a recent model is the iPREDICTLIVING (2019) developed to predict risks around kidney donation to better inform renal transplant decision-making.20 In the context of a sensitive and complex clinical decision as kidney donation, detracting the clinician from the human interaction by a time-consuming on-screen process will likely impact the patient experience. Digital health interventions should be centred around improving the quality of care delivered to patients, which includes better decisions, but also enhancing the patient–clinician relationship by providing clinicians the time to consult patients.

To realise a more streamlined workflow, a change in how we think about clinical risk prediction models is required. Front-line usability should be part of the initial exploration of the proposed model. This means involving clinicians (end-users) at the outset as part of research projects to understand how the tool would be practically used and the impact it would have on clinical encounters. The usability of such interventions plays a crucial role in preventing clinician fatigue and improving uptake.21

Technical challenges revolve around non-standardised coding of health data across EHR providers.22 This means that even if a risk prediction model is made available as a standalone software, which can be integrated, misaligned clinical terminology may limit implementation. Involving EHR vendors early in the development of risk prediction models and imploring greater alignment across the industry will mitigate barriers to implementation and subsequent scale-up of novel solutions. An example of successful tool integration is QRISK, which has been embedded within a number of primary care clinical management systems.23 The tool calculates individual cardiovascular risk and generates a score based on existing data. Not only has this impacted positively on front-line practice, regular use provides evolving data quality and completeness reflecting the changing population characteristics over time. This has allowed researchers to update and calibrate the tool for long-term accuracy.24 Similar implementation through hospital EHR vendors may bring such models into routine secondary care settings unifying and standardising practice. Another relevant example was the PREDICT software used in general practice in New Zealand, which automatically recorded patients’ risk profiles for cardiovascular disease and prospectively linked this to coded hospital and mortality databases. This allowed a risk prediction model to be developed that took in to account an area-based deprivation index and self-reported ethnicity alongside clinical parameters, resulting in greater personalised risk profiles for individual patients. The strength of this study was its prospective nature and ability to seamlessly collect healthcare data without additional intervention by clinicians delivering care.25

The tremendous potential of clinical risk prediction models mandates policy-makers to establish regulations to standardise the integration of tools into EHRs. Strategies to achieve this may be through EHR vendors working directly with data scientists to incorporate statistical models within their user interface, or alternatively provide non-proprietary application programming interfaces for third party developers to seamlessly integrate with. The potential success of this however, heavily relies on the engagement of front-line healthcare professionals who can provide the clinical context and workflow that a risk prediction model is intending to influence. Encouraging multidisciplinary research and development teams, which can appreciate the different facets of clinical context, statistical modelling and implementation science, supported by EHR vendors working to unified standards has the potential to bridge the current bench-to-bedside gap for clinical risk prediction models.