What characteristics of clinical decision support system implementations lead to adoption for regular use? A scoping review

Eligibility criteria

CDS systems were included if they met clear definitions (table 1). We focused on clinician-facing CDS systems due to their particular workflow challenges and excluded developmental or patient-facing systems.

Searches

We searched Web of Science, OpenGrey,25 the BMA26 and Trip databases27 in June 2022 with updates in April 2024. CDS systems and websites were suggested by expert colleagues and were searched in Google and Google Scholar for related publications or reports. Once a CDS was included, a further search of names and authors was conducted to deepen the analysis. See online supplemental appendix 2 for details.

Quality assessment

Included CDS projects were assessed against the Standards for Reporting Implementation Studies (STARI),28 which assesses interventions separately from implementation, recognising the benefits of studying and reporting both in healthcare settings. See online supplemental appendix 3 for details.

Analysis framework

The NPT framework15 was used to deductively analyse the implementation of included CDS systems,15 as it highlights issues critical to successful implementation and integration into routine work. It is recommended for use in designing complex interventions29 and commonly used as a qualitative data analysis framework.30

The NPT framework is split into four domains: Coherence, cognitive participation, collective action and reflexive monitoring (figure 1). The four domains are further subdivided within the NPT framework using questions to be answered during the analysis.15

Request permission

Data charting process and data items

The subquestions within the four NPT domains were answered during full-text review and data extraction, from each CDS’s associated publications, or designated ‘not reported’ where not described in the publications. Quotes to substantiate answers are given in the framework analysis (online supplemental appendix 4).

Synthesis of results

Results were analysed quantitatively by calculating the percentage of positive and not reported answers by NPT domain and by project. The NPT was used as the qualitative content analysis framework.

Quality assessment

Of the 22 CDS tools, 18 had evidence of over half the STARI checklist domains. Many publications associated with the projects focused exclusively on either implementation or intervention. See online supplemental appendix 2 for full scoring.

Synthesis of results

The questions in each NPT domain answered from the published information for each CDS (online supplemental appendix 4) were colour-coded with red for a ‘no’ or negative response; amber for a ‘maybe’, neutral or ambivalent response and green for a ‘yes’ or positive response (table 3). If there was insufficient information, this was deemed ‘not reported’ and colour-coded grey (figure 3). See online supplemental appendix 4 for full definitions of terms and synthesis explanation.

Request permission

Coherence

‘Is there a clear understanding of the new service?’ scored well with 64% positive ‘Yes’ reports (figure 3). Publications provide explanations that are clear to the reader, however, there is limited reporting of the explanation given to clinicians involved. Frequently, the CDS replicate published guidelines.

‘Do individuals have a shared understanding of the aims, objectives and benefits?’ was variable. It appeared that mandating use to circumvent this aspect has been used in some cases (F, Q, V, Z) . In others, the CDS was ‘communicated’ to staff (figure 4) but there was no evidence of understanding its purpose (A, H, UU). Training and targeted advertising appear to have been successful in many cases. Where training was used, the emphasis was on bespoke training by profession or clinical role.

Request permission

Only 36% of studies said ‘yes’ in response to ‘Do individuals have a clear understanding of their task or responsibility?’, and several studies reported ‘no’ that is, that during implementation individuals explicitly expressed a lack of understanding (A, F, MM). Understanding role-specific tasks and responsibilities was problematic at the outset of most of the implementation projects that reported on this (figure 4).

‘Do individuals understand the value, benefits and importance of the service?’ was variably reported. Where uptake of the CDS was low, implementers in some cases mandated use to achieve uptake. Learning from inappropriate requests was mentioned as a benefit among multiple CDS projects. Where clinicians were aware of improvements in costs, efficiency or compliance with guidelines, this was well received (A, B, S,KK).

Cognitive participation

There were 45% ‘yes’ responses to ‘Do individuals buy into the idea of the service?’ (figure 3). Five projects relied on mandated use of the CDS to achieve uptake, rather than encouraging buy-in (F, Q, V, Z, FF). There were significant barriers to engagement in many cases. Training and teamwork appeared to be facilitating factors (H,X) along with time-saving interventions embedded in the CDS pathway or workflow (JJ).

‘Can individuals sustain involvement?’ scored 59% ‘yes’ responses. Across the board, integration with existing electronic health record (EHR) was key to sustaining involvement (Figure 5) (B, K, GG).

Request permission

41% of CDS did not report any evidence for ‘Are key individuals willing to drive the implementation?’ Of those that mentioned key individuals, management support was key (Z, GG, MM) and having a small team of known team members to discuss and encourage usage (F, H, CC).

‘Do individuals feel it is right for them to be involved?’ was the least reported element of the framework (73% not reported). Of the five projects that reported against this question, two reported a negative response (Figure 5) (CC,OO). Only four projects contained any report on this domain (HH, MM, OO, UU). Where problems were identified by the team undertaking the implementation, it was clear they had a mandate to be involved.

Collective action

Organisational support for the CDS project was reported positively in 45% of projects (figure 3). Where it was not explicitly reported, indications of organisational support included enabling significant changes to a local care pathway (F). Support came from a wide range of sources, including clinical commissioning groups in the UK (F, CC, KK), insurance companies in the USA (Z) or professional bodies such as the Swedish Pharmacy Association (M). Organisational incentives to introduce a CDS were usually cost and efficiency savings (figure 6). These priorities did not necessarily align with those mentioned by the users: safety and improved patient care. Even where the CDS had no key individuals to drive the implementation (K, RR), the organisational support led to success.

Request permission

‘How does the innovation affect roles and responsibility or training needs?’ was positively reported in 41% of cases. There are examples of implementation leading to positive role change within a team (Figure 7). For example, a change in the pharmacists’ and microbiologists’ roles from being exclusively lab based to having a patient-facing role was reported (KK). Where the CDS replicated best practice guidelines, subsequent service audits demonstrated improvement in adherence to guidelines long term (K, KK, NN).

Request permission

There were only 32% ‘yes’ responses to ‘Does the service make people’s work easier?’. Additional time burden was a perceived disadvantage (Figure 7). Some factored in incremental changes, and therefore, the burden was mitigated in response to feedback. Examples of mitigation included increasing integration with the EHR,32 improved automation of inputs (B) or removing another burdensome task to make the CDS timesaving or time-neutral (JJ).

‘Do individuals have confidence in the new system?’ was answered ‘yes’ in 45% of projects, however, 50% was not reported. Where there was feedback from clinicians, the studies reported confidence in the new system almost universally (B, MM, OO, UU).

Reflexive monitoring

‘Do individuals try to alter the new service ?’ had 14% ‘yes’ answers (figure 3). Lack of flexibility for clinical judgement was a clear negative aspect of implementations; this led to circumventing (F), ‘gaming’ (A), or ignoring the CDS (M, CC). Nevertheless, adherence to guidelines and better patient care was frequently assessed as a benefit of the CDS. Where there was an additional, unmitigated administrative burden, CDS implementations were viewed negatively (N, Q, Z, HH). However, where mitigations were introduced, participants reported positive effects on knowledge and patient care (GG, UU).

There were 18% positive responses to: ‘How do individuals appraise the effects on them and their work environment?’. Effects on clinicians and clinical environment were often not reported (50%), and when it is, it is mostly negative—time burden (Z), alert fatigue (CC) and administrative tasks were identified downsides (GG).

‘How are benefits or problems identified or measured?’ was positively reported in 55% of CDS. Many of the projects either planned a qualitative element to their studies (H, Z, CC), or an iterative approach (B, F) to implementation that allowed for feedback. This meant that problems with a CDS could be identified once in practice and subsequently solved.

Summary of evidence

Despite extensive searches, there are few publications available for analysis regarding CDS implementations adopted into routine clinical practice. Even among those included, there are extensive areas of the NPT not explicitly addressed. CDS research studies typically focus on quantitative methods, reflected in extensive categories not reported. The NPT interrogates individuals’ responses to new technology, however, quantitative studies may only explore the aggregated numerical impact of a given measure. Projects with the most content against the NPT domains were those with a mixed-methods or primarily qualitative implementation paradigm (H, N, Z, CC, HH, MM, SS, VV).

Organisational support

Organisational prioritisation of CDS implementation was one of the most reported characteristics, and endorsement is key for CDS adoption, in turn, more likely if the best practice or national guidelines are included (A, O, V, FF, GG, OO). Having a financial or operational mandate to use a CDS from an insurance company or clinical commissioning group (F, Z, CC), or key supportive individuals leading the implementation (K, M) is an adoption promotor. The ongoing use of the CDS in these cases demonstrates the importance of organisational support in ensuring success. This was demonstrated during the COVID-19 pandemic where there was impetus to allow clinical services to continue despite rules preventing face-to-face contact; senior leadership allowed significant innovation where previously there had been stagnation.33 There were examples of mandating use when uptake was low (F, Q, V, Z, FF); it is unclear whether this was instead of or in addition to communicating importance and value. It is, therefore, difficult to draw conclusions about the impact this had. Where there was an organisational imperative to introduce the CDS, iterative approaches and staff feedback will likely have been deprioritised. This is evidenced by the two studies with the most negative reporting against the NPT (F, Z). This goes against the ISO standards for human-centred design,34 which recommends having users at the heart of implementation. One of the projects comments on clinical autonomy being compromised by mandating the use of the CDS (F). Mandating usage is a blunt tool which achieves the organisational aim but ignores the complexity of the patient-facing role. The evidence in this review shows improved engagement when clinicians are consulted for feedback and problem-solving instead of forced implementation. This is also reflected by Greenhalgh and Abimbola,7 recognising the adopter is not just the organisation, but all stakeholders involved in the system’s use. Golinelli et al33 highlighted the lack of legal frameworks surrounding CDS; mandating systems and reducing clinical autonomy without legal guidance may discourage adoption.

Shared value proposition

Understanding role-specific tasks was often reported neutrally or negatively . Conversely, key commonalities were having staff who understood the value proposition of the CDS and understood their role in using it. The manifestation of lack of understanding varied. Wilkinson et al (MM) reported an explicit lack of understanding by clinicians, whereas Hart et al (F) and Macpherson et al (B) reported the CDS was not being completed correctly or at all. This highlights the key sections of the NPT which have a positive impact are a clear understanding of the new service (CDS), individuals can sustain involvement, a positive value judgement of the CDS and the ability to identify and measure problems. This is echoed in the NASSS framework7 and has also been demonstrated in other case studies.35 36 Understanding that more effective care is delivered using CDS facilitates buy-in.35

Iterative, mixed-methods and the need for customisation

In studies with qualitative feedback or questionnaires, it is easier to explore how the technology has been normalised into service delivery (N, Z, CC, II, MM, UU), illustrated by having the lowest number of domains not reported. Novel health technology implementation is recognised as a complex intervention and as such requires a comprehensive approach to all aspects of system performance and usability.37 Mixed-methods approaches13 combined with human-centred design34 optimise the implementation and prevent circumventing or gaming the system.36 This illustrates the requirement for adaptability and customising of the system to meet clinicians’ needs in practice.36 Rolling with complexity is recommended in the NASSS framework,7 recognising the overlapping responsibilities of clinicians and organisations and accepts that there are some complexities which will not be solved by a CDS. As such an iterative implementation in response to feedback was pivotal here(B, F, MM) and is backed up in the literature.7 13 38

Strengths and limitations

The strength of this review lies in the exclusive inclusion of projects that are being used in regular clinical practice. This gives actionable insights relevant to many healthcare environments, for future planning of CDS implementation. Previous reviews focusing on digital adoption post pandemic have rightly highlighted the drive for innovation,33 however, the complexities of the healthcare system and reasons for previous stagnation are rapidly being realised.39 It is possible that the insights here could also be characteristic of failed projects since we have only considered those CDS with evidence that they are being used in routine clinical care.

Quality assessment is not strictly needed with a scoping review. However, the STARI checklist28 acknowledges several subtypes of implementation study and the need for flexibility in the approach to judging quality in implementation. Most projects scored over half marks in the assessment. Given that these are successfully adopted implementations, it could be that the quality of the implementation study(ies) was also a factor in their success. There was a distinct pattern of certain projects which had focused on the implementation as the focus of the publication, and others which had focused on the intervention (online supplemental appendix 1). As such there are elements of the checklist that the authors of the studies did not intend to address.

This review details extensive searches of the grey literature and websites, in addition to the peer-reviewed literature databases. Coupled with follow-up searches for included projects, this analysis gives a thorough picture of the CDS systems used in clinical practice where the implementation documentation is available. It is possible that examples were missed, especially where there is a lack of publications related to a particular CDS. The review has nevertheless been able to identify tangible and actionable strategies for future implementations (figure 8). Future work should explore patterns in adoption such as particular clinical areas, successful funding streams and the type and range of expertise that has proven successful.

Request permission

Deductive analysis of the NPT domains was relatively straightforward where there were multiple qualitative and quantitative outputs from an implementation. It is possible that NPT domains may have been addressed by the implementation project but were not reported in the associated outputs. By accessing as many sources as possible, we have attempted to minimise this eventuality.