NHS Scotland’s Decision Support Platform: a formative qualitative evaluation

Ethical approval

We obtained Institutional Review Board approval for the study from the Centre for Population Health Sciences at The University of Edinburgh, UK. We anonymised all places, names and organisations to ensure confidentiality. Individual participants provided written informed consent for participation.

Design

We conducted a formative qualitative evaluation consisting of a combination of face-to-face/telephone interviews and observations of workshops with representatives of different stakeholder groups including policy and strategy leads, system vendors and clinical opinion leaders (primarily general practitioners (GP) and pharmacists).

Sampling

Sampling was facilitated by the Scottish Government who helped to establish contacts with key stakeholder groups. We used purposeful sampling to select stakeholders that were involved in planning, commissioning, development and early testing/use of the Platform.17 We attempted to ensure maximum variation in terms of background (clinical, managerial, strategic) and geographical location.

We used a degree of respondent-driven sampling to maximise the chances of recruiting a maximum variation sample.18 In doing so, we selected clinical interviewees randomly from those attending the workshop to ensure a range of locations and professions.

Data collection

We used semistructured interviews as the main method of data collection. We also undertook non-participant observation of workshops where the Platform and applications were demonstrated to clinical leads to observe the dynamics of prospective user reactions in situ. These workshops were led and facilitated by a representative from Scottish Government in order to engage key clinical opinion leaders. Participants were presented with demonstrator scenarios showing how the Platform worked with the patient record, followed by interactive discussion of first impressions, expectations, drivers and potential challenges. Combining different methods enabled us to triangulate the data sources to validate emerging findings.19

The interviews consisted of open-ended questions about the experiences of existing solutions and expected outputs of a Scotland-wide system through exploring:

• Stakeholders’ attitudes.

• Implementation strategies.

• Measurements for the impact of DSS and how it could be achieved across settings.

• Potential challenges to using proposed solutions as well as ways to address these and ensure sustainability.

• Definitions of ‘success’ from various perspectives.

The interview guides were tailored to the roles and organisations of individuals. Each interview took between 20 min and 1 hour. Interviews with policymakers explored strategic drivers for DSS and how these fit within the wider political and supplier landscape within NHS Scotland. Interviews with clinicians explored their attitudes towards the DSS Platform, how it could impact their work and how it could integrate with their existing practices.

During workshops, the researcher (MC) recorded the layout of the room, the actors and their roles, how the Platform was perceived to be received, emerging attitudes and reactions and the researcher’s own impressions/feelings in relation to the observation.

With written consent from participants, interview data were digitally audio recorded and transcribed verbatim by a professional transcriber. The researchers (KC and MC) also recorded field notes for each meeting, interview and for observations.

Data analysis

Our analysis was informed by the sociotechnical approach.16 20 This focuses on exploring structural technological factors and associated social processes as well as the inter-relationship between the two during implementation and optimisation. The sociotechnical approach has been applied widely to understand, for example, how technological change can result in changes to work practices of healthcare professionals, and vice versa, and how users can shape technological designs.

Data were thematically analysed, initially within stakeholder groups in order to explore different viewpoints and then data sources were triangulated. We drew on our previous work in this area as a deductive coding framework, where possible. In addition, we inductively identified emerging themes.21 Themes were developed based on frequency of occurrence and salience among different stakeholder groups. Negative cases, that is, those that did not fit within the narrative, were explored, keeping potential implications for the national strategy in constant focus.

Findings across stakeholder groups were then compared in analysis meetings of the research team. This involved discussing commonalities and differences across data sources and participant groups, as well as exploring potential underlying explanations for differences and remaining tensions. Although we observed subtle differences across participant groups (which we explain in the Results section), our general findings were broadly comparable, so consensus was achieved though aggregating findings at higher analytical levels.

Widespread recognition of the potential value of DSS

All interviewees and workshop attendees agreed that, in line with existing empirical evidence, the strategic decision to implement DSS in NHS Scotland was the right way forward.

With good supportive decision-making, people have a better chance of getting the right care at the right time in the right place. Not the wrong care too late at excessive cost, with disabling consequences. (Participant 1, male, Policy)

Participants expected DSS to have a positive impact on safety, quality and efficiency. Areas and potential outcomes mentioned most frequently are summarised in table 2, although it is important to note that many of these related to the expected impact of digital health initiatives more generally rather than being associated with DSS. The only two areas that were specifically related to DSS were adherence to guidelines and availability of information.

The expected areas of impact frequently aligned with important policy drivers around guideline implementation and person-centred approaches to decision-making in health and social care settings.22 23 For example, stakeholders talked of giving patients more power to decide on their own treatment. A typical response was:

…I think hopefully it’s a better care for that patient and a more informed patient as well, so they know more about their treatment and why they’re on a treatment and therefore can help…making a decision in conjunction with the clinician. I think that’s going to be quite beneficial for patients as well to give them more empowerment… (Participant 22, male, Pharmacist)

However, the definition of outcome measures was perceived to be difficult and the long time to realise potential benefits was highlighted. Participants also acknowledged that outcomes would be dependent on the DSS application in question, and some interviewees were not entirely clear which application was included in the Platform.

I come across decision support in various different guises. Or for what I would consider decision support. (Participant 13, male, Clinical Lead)

Despite the overall positive attitudes among participants, we also observed concerns that expectations may not necessarily match system functionality. For example, many clinicians discussed a more general problem of information overload and the difficulty of navigating different sources of information. One GP stated that she often had open 10 tabs on her computer at a time, while another mentioned going in and out of different systems, and several stated carrying folders of paper-based records. Participants therefore highlighted evaluation of benefits as a priority area.

Leadership and strategy to implement DSS nationally

While no participant expressed an overt negative opinion on the concept of DSS in principle, early and ongoing engagement with clinicians and other healthcare staff was viewed as crucial during system development and implementation. Participants were positive towards the concept of DSS, but had some concerns about how this would work in practice. At one workshop, a male GP emphasised that his comments should not be seen as negative but ‘in the spirit of improving the system’.

So their forward travel seems to be in the right direction providing that…the final product will work as it’s supposed to work…we’ve got 10 min. That has a big impact on your time whereas if you’ve got something popping up that can do these things then it should work far more efficiently. (Participant 20, female, GP, Tayside)

Participants wanted to get ‘hands-on’ experience of using the system and be engaged in ongoing development to ensure that it worked in practice as well as in theory. Ongoing stakeholder engagement was also stated to be necessary to raise the profile of the Platform among the wider healthcare workforce.

The actual workshop…was a good step in the right direction, but I think for people who aren’t aware of these things, not just GPs but pharmacists, nursing and other kind of clinicians, I think a wee bit of work needs to be done to raise its profile. I really enjoyed looking at the screens and saw great potential. (Participant 27, female, GP)

We also observed a tension between a perceived lack of strong national-level leadership (which was considered to be needed for the successful implementation of the strategy) and staff changes within the eHealth Directorate of the Scottish Government. Several interviewees stated that these changes caused challenges surrounding strategic priorities and direction of travel.

…my preference is…we have a sense of where we’re going to prioritise the initial investment and that is all coordinated from the one position, rather than from a variety of different piecemeal pockets and funded…it needs the central coordinating function. (Participant 13, male, Clinical Lead)

While calling for central leadership and direction, participants also acknowledged that a variety of different projects had to be managed under a portfolio-based approach. Here, there appeared to be a tension between the perceived need to have a firm direction of travel beyond Platform implementation and the multitude of ongoing DSS initiatives across settings (including primary, secondary and social care). While leadership was seen to be required to ensure alignment of initiatives and avoid silos, there was also a perceived need to recognise that strategy was sufficiently agile to cope with changing demands and stakeholder experiences.

I think [Platform] is one of the potential ways…the fact that there’s a number of pilots which have been started to test and to see what learning we can take from that, to me feels that the right approach. But it might not be the only approach…we have to make sure that we look slightly beyond that to say, okay, and this is what we’ve got, this is the tools that we have in the here and the now. But we know that the world is changing…as technology’s concerned it’s changing really, really quickly. So, we need to be preparing ourselves for that next leap, as well. (Participant 12, female, Policy)

System usability and interoperability

The Digital Health and Care Strategy provided a mandate for integration of digital health systems in Scotland and the Platform (which adheres to OpenEHR and Fast Healthcare Interoperability Resources open standards) fits in well with this.23 24 However, it did at the time of our work not integrate with Egton Medical Information Systems and Vision, primary care EHR systems procured centrally in Scotland. Participants expressed hope that this integration would be possible in the future. In the interim, developers had created interfaces that allowed interfacing with the Platform but did not provide full integration.

I think it will have to [integrate], but how we get to that point, I'm not sure. So we do have a contractual requirement for [developers] to deliver, and interact with our clinical decision support systems. So any supplier would, notionally, at least, have to be able to do that. (Participant 14, male, Clinical Lead)

What we do know is we are using a demonstrator environment with the look and feel exactly like it will look in reality, but for the real integration we will most likely use a third-party developer in the first phase. (Participant 9, male, Developer)

System integration was also perceived to be crucial from a usability viewpoint and clinical interviewees expressed strong concerns that the Platform may not effectively integrate with other primary and secondary care systems (as well as their existing in-built DSS). This lack of integration, it was feared, may lead to slowing down of existing systems and parallel data entry (leading to duplication of data entry and data inconsistency).

We feel like we’ve got lots of systems that work separately. Now, they don’t talk to each other…so you end up having to put data into more than one system…so if they could talk to each other a bit better I think that would help. (Participant 25, male, Consultant)

Another perceived factor potentially affecting end-user usability was the degree of local customisation needed to tailor the number of pop-ups to local needs and avoid alert fatigue. Clinicians also mentioned that they were already presented with pop-ups in their own GP systems and that alerts from various sources needed to be aligned and thresholds for alerting set carefully in close consultation with them to ensure usability.

Pop-ups come up from all different places, so there’s the notes that practices put on the systems to try and remind you to do something…you’re potentially going to throw another set of pop-ups in there to have your clinical decision support… (Participant 21, male, GP)

Risk of unintended consequences emerging from implementations of DSS

As clinicians had only seen pilot versions of the software in test environments and some had previous negative experiences of using new digital systems, there were uncertainties around the use of the Platform in real-life settings. For example, clinicians mentioned the risk of duplication of work and increased workloads despite their desire for more time with patients.

… to introduce that amount of potentially new information into a consultation that’s very, very tight time wise is always going to be a challenge. (Participant 21, male, GP, Glasgow)

It was also viewed as important to avoid over-reliance on systems by those with insufficient clinical experience and to ensure that the system did not attempt to replace the expertise of clinicians.

It’s useful to see amount of information available, but it could lead you down the wrong pathway and be in charge rather than the doctor. They [GPs] have the background knowledge of the patient and want to use their expertise. (Glasgow Workshop, Table Feedback)

Several stakeholders in the workshops and in interviews also highlighted some potential tensions between decisions of clinicians and patients, with GPs generally being more risk averse than patients. Some therefore argued that DSS should be patient informed and not patient led, with the ultimate decision of treatment being in the hands of the clinician.

Patients take more risks than doctors but the clinician is responsible for them it should be patient-informed but not patient-led. (Participant 20, male, GP)

Another concern expressed at workshops and in interviews was that, for clinicians, the patient-centred approach may be in direct contrast to other policy drivers such as efficiency and patient outcomes. GPs described scenarios where patients might choose minimal intervention and therefore have a worse outcome.

You’re trying to reduce variability for drugs etc. but this can go against [patient centred approach] where you are giving personal choice. How can success be measured? (Glasgow Workshop, Table Feedback)

Strengths and limitations

This evaluation offered important opportunities for realising true impact through ongoing formative feedback to policymakers. In doing so, our written progress reports have informed a business case submitted to Scottish Government in October 2018 for further funding of the Platform beyond pilot sites. It also allowed to adjust to changing policy landscapes and emerging priorities through the timely delivery of formative feedback and contributed to building clinical and policy-level engagement.

The formative real-time nature of this work has, however, also posed some challenges for all involved. Changes in policy priorities and emerging findings necessitated flexible changes to our initially planned methodology over the course of our work in order to ensure that it remained relevant. For example, although we had initially planned to work with settings that had implemented systems as part of the Programme, only some functionality existed at the time we began our work. Such delays are common in large public initiatives. Interviews with policymakers further indicated a strong interest in the Platform, as a cloud-based CDS Platform based on open standards, which fits well with ongoing efforts to create a National Digital Platform that facilitates interoperability of health and care services.24

Moreover, our funders also acted as gatekeepers to participants and workshops. The sample may therefore not be representative of the wider stakeholder groups and reflecting the range of voices surrounding the DSS landscape. Participants consisted largely of those who were already engaged and might not reflect the views of wider stakeholders. As a result, our findings are likely to provide an incomplete picture of reality and should be interpreted with caution, despite (or precisely because of) fulfilling their purpose of delivering political impact. We have also not yet had sight of the business case, which our work has contributed to, so it difficult to position our work in the evolving policy landscape.

It is further important to recognise that this work presents only a snapshot of an early evaluation of the beginnings of a national programme. It by no means provides a complete picture and calls for more in-depth and longitudinal work to assess ongoing developments and use over time.25 For example, it was in some instances not clear to us if clinicians and policymakers really understood the implications of using the Platform in the ‘real world’ as the workshop demonstrations were based on idealised scenarios. Real challenges may only emerge/materialise when the system is tested and a later stage in development and when it is used in practice.

Integration of findings with the current literature

Emerging issues in this work reflect tensions present in many large-scale digital change initiatives in healthcare settings. While the rationale for change and the value proposition of the Platform was clear to all participants in this work, the leadership and governance model going forward was an important concern. One common theme was the agreement that there needed to be a mixture between ‘bottom-up’ and ‘top-down’ approaches to implementation, ensuring a degree of central leadership and direction while also allowing for local input in decision-making.26 This balance is crucial, as many existing ‘failed’ change initiatives such as the National Programme for Information Technology (NPfIT) have illustrated that national ‘top-down’ approaches alone are insufficient to realise the benefits associated with large-scale change in healthcare settings. In the NPfIT, ‘top-down’ strategies were superseded by locally driven strategies, and ultimately changed to a more ‘middle-out’ model, where national strategic direction aligns more closely with local strategy.27

Both clinical and policy stakeholders in our sample had a strong desire for systems to save time and improve safety, quality and efficiency of care. However, as the expected areas of impact in table 2 illustrate, the impact of the DSS may be overestimated which may lead to problems and frustration during implementation. Similarly, time savings stand in stark contrast to the often long and painful implementation experiences in healthcare settings where benefits of digital systems can take a long time to materialise, and where benefits are often not visible to those who take the additional burden associated with their implementation.28 29 This long pathway to benefits is often due to social dimensions surrounding the technology, including, for example, adverse impacts on work practices and increased workloads, the potential introduction of new errors, adverse impacts on time spent with patients and unrealistic stakeholder attitudes/expectations.30–32

While most participants were optimistic in relation to potential benefits of the Platform, there were also some major concerns. These related mainly to the would-be adopters interacting with technologies, and specifically usability and integration with existing workflows. Issues with system usability adversely affecting adoption patterns feature consistently in the empirical literature. In particular, the issue of alert fatigue in DSS and information overload among clinicians remains an unresolved area internationally.33–35 There is a trade-off between the number of alerts and the attention of clinicians, which has led some to advocate the use of a limited user-determined number of carefully designed pop-ups.36

An increasing number of interfaces can adversely affect system performance and usability and also lead to the introduction of new errors. Our work exploring the implementation of hospital electronic prescribing systems has shown that integration and interfacing problems could in some cases inhibit the effective transfer of information, leading to duplicate data entry and adverse effects on the availability of information.15 37 This highlights that therefore particular attention needs to be given to effective integration of the Platform with existing primary care systems.

Policy recommendations and implications for practice emerging from this work

Recognising the early stages of this evaluation, our work has some preliminary implications for policy to ensure long-term sustainable delivery of the Platform. Most importantly, sufficient time and resources need to be available to implement, facilitated by overall national ownership to support the implementation of the Platform within the eHealth Directorate. This work also needs to align with the development of the wider National Digital Platform (which is planned to connect all digital health and care services in Scotland) to facilitate interoperability and the wider Scottish Digital Health and Care Strategy to facilitate alignment with national strategic priorities.23 24

As technology landscapes and needs are constantly changing, future iterations of the Programme have to be sufficiently agile. Revisiting the original document at predefined time points informed by ongoing evaluation and stakeholder engagement is likely to facilitate these developments. This should also involve ongoing communications about the current strategy and associated timescales, stakeholder engagement opportunities and resulting actions, as well as ‘hands-on’ experience of system use.

In order to promote usability, there is a need to negotiate seamless integration with existing primary care systems, as interfaces as interim solutions can adversely affect user experiences. The negotiation with primary care system suppliers and promoting seamless integration through effective user interfaces is therefore key going forward.

There are further a number of potential ways to tackle tensions between local and national dynamics. These are summarised in box 3.

As this evaluation has focused on the very early stages of piloting, our findings are preliminary. We strongly recommend ongoing formative evaluation of the implementation of the Platform through longitudinal work to ensure effective alignment with wider strategy, and continuous stakeholder engagement. This work needs to include a designated strand exploring evolving policy and supplier landscapes and market management.