Article Text
PDF
Abstract
Context The Scottish Government has identified computerised decision support as a strategic priority in order to improve knowledge management in health and social care settings. A national programme to build a pilot Decision Support Platform was funded in 2015.
Aims We undertook a formative evaluation of the Decision Support Platform to inform plans for its national roll-out in primary care.
Methods We conducted a series of in-depth semistructured interviews and non-participant observations of workshops demonstrating decision support systems. Participants were policymakers and clinical opinion leaders from primary care. As the Platform was in its early stages of development at the time of data collection, we focused on exploring expectations and drivers of the pilot decision support system tested in primary care. Our methodological approach had to be tailored to changing circumstances and offered important opportunities for realising impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. We drew on sociotechnical principles to inform data analysis and coded qualitative data with the help of NVivo software.
Findings We conducted 30 interviews and non-participant ethnographic observations of eight stakeholder engagement workshops. We observed a strong sense of support from all stakeholders for the Platform and associated plans to roll it out across NHS Scotland. Strategic drivers included the potential to facilitate integration of care, preventive care, patient self-management, shared decision-making and patient engagement through the ready availability of clinically important information. However, in order to realise these benefits, participants highlighted the need for strong national eHealth leadership to drive a coherent strategy and ensure sustained funding, system usability (which stakeholders perceived to be negatively affected by alert fatigue and integration with existing systems) and ongoing monitoring of potential unintended consequences emerging from implementations (eg, increasing clinical workloads).
Conclusions and implications In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Platform. Potential local tensions could be addressed through allowing a degree of local customisation of systems and tailoring of alerts, and investing in a limited number of pilots that are carefully evaluated to mitigate emerging risks early.
- medical informatics
- primary healthcare
- information systems
- Qualitative
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary
What is already known?
Decision support systems have significant potential in improving the safety, quality and efficiency of care. However, empirical work has indicated that implementation can introduce new errors and have unintended consequences. Careful evaluation of the introduction of such technologies and their impact is therefore needed.
What does this paper add?
Our formative qualitative evaluation of a Scottish national programme to build a pilot Decision Support Platform indicated a strong sense of support from all stakeholders. However, in order to address potential tensions between national leadership and local usability as well as potential unintended consequences, there is a need to have overall national ownership to support the implementation of the Platform.
Potential local tensions could be addressed through allowing a degree of local customisation of systems and tailoring of alerts and investing in a limited number of pilots that are carefully evaluated to mitigate emerging risks before the Platform is rolled out nationally.
Introduction
In Scotland, there is a national strategic drive towards participatory, personalised, predictive and preventive medicine.1 Decision support systems (DSS) are key enablers in the delivery of such a vision. They are characterised as information systems that draw on an active knowledge base to support the decision-making of its user, and can be used globally by health and social care professionals as well as by citizens who have an interest in tracking and quantifying their health and activities (eg, in diaries, appointment reminders, wearables).2
Electronic health record (EHR) infrastructures and the drive to integrate additional data streams, ranging from administrative, social care, genetic and patient-generated wearable data, harbour the potential for a major step-change across the Scottish health and social care (henceforth care) landscapes. In order to capitalise on these, the Scottish Government launched the national Decision Support Programme for NHS Scotland in 2015, which sought to develop an evidence-based strategy and associated delivery framework for DSS.3
Clinical decision support (CDS) systems are a type of DSS that draw on real-time patient-specific information to generate individualised advice such as alerts (such as warnings and reminders) for clinicians tailored to the case at hand. They can be independent (apps or websites), interfaced (eg, to patient management systems or portals) or integrated (with EHRs). CDS to support prescribing is a key strategic priority area throughout much of the world including Scotland, as consistent empirical evidence points to a large proportion of potentially avoidable morbidity and mortality caused by medication-related harm.4–6
Although it is now commonly recognised that CDS systems can improve practitioner performance and also reduce some medication error rates,7 8 there are concerns that CDS implementation can introduce new errors and have unintended consequences.9–13 For example, Ash and colleagues have described how CDS content and presentation can lead to users ignoring potentially important alerts and promote errors (eg, through autocomplete functions).14 This illustrates a need for careful evaluation of the introduction of such technologies and their impact.15
We were commissioned to conduct a formative evaluation of the pilot DSS Platform for the Scottish National Decision Support Programme (see boxes 1 and 2), which combines an open standard-based algorithms editor and engine combined with bespoke decision support applications delivered as web and mobile products and integrated into primary care EHR systems. We designed our formative work to help inform the ongoing development of the Programme through identifying early drivers and implementation strategies, anticipated challenges, and identifying potential ways to address these.16
Summary of the Scottish National Decision Support Programme
The Roadmap to deliver the Programme was published in June 2015. It set out three key milestones:
Phase 1—18 months: Deliver clinical decision support (CDS) for antibiotic prescribing venous thromboembolism guidelines, pilot a Decision Support Web Gateway, define requirements for the knowledge base and CDS Platforms.
Phase 2—year 3: Deliver a Mobile Decision Support Toolbox, implement CDS for long-term conditions in primary care.
Phase 3—year 5: Deliver a suite of point of care mobile CDS, implement CDS with community systems, clinical portal and specialist systems.
Stakeholders: The implementation of the Roadmap was led by the Scottish Government and the Digital Health and Care Institute.
Objectives: By 2020, it was envisaged that CDS would be available to clinicians when needed on desktop computers and mobile applications. Although the Roadmap focused on CDS, there is a wider drive in Scotland to promote decision support systems (DSS) more generally, including citizen-facing functions.
Project organisation: It was envisaged that the Roadmap would be delivered by support roles including product management, knowledge management, technology design and delivery, and implementation support.
System architecture: The Roadmap proposed two key elements:
Small-scale demonstrators: to show that benefit can be delivered and to build clinical engagement and leadership.
The broader infrastructure including: a web place for users to access quality-assured web-based CDS and mobile apps, knowledge-based feeding algorithms, technology and content management tools, including but not exclusively focused on prescribing, including alerts and reminders.
Progress at time of writing: Built digital infrastructure for national delivery of CDS. Some successful early adopter projects use these tools.
Description of the pilot decision support system (DSS) Platform for the Scottish National Decision Support Programme
One part of the system architecture of the Scottish National Decision Support Programme.
Engine that enables writing of clinical decision support algorithms.
Can be plugged into any electronic health record (EHR) system and integrates patient-specific information with published guidelines and regulations.
Includes development tools for portable apps.
Built for open standard interfaces.
Cloud based (in the developer’s own cloud that uses application programming interfaces to allow third-party vendors to link their applications to the Platform).
Methods
Ethical approval
We obtained Institutional Review Board approval for the study from the Centre for Population Health Sciences at The University of Edinburgh, UK. We anonymised all places, names and organisations to ensure confidentiality. Individual participants provided written informed consent for participation.
Design
We conducted a formative qualitative evaluation consisting of a combination of face-to-face/telephone interviews and observations of workshops with representatives of different stakeholder groups including policy and strategy leads, system vendors and clinical opinion leaders (primarily general practitioners (GP) and pharmacists).
Sampling
Sampling was facilitated by the Scottish Government who helped to establish contacts with key stakeholder groups. We used purposeful sampling to select stakeholders that were involved in planning, commissioning, development and early testing/use of the Platform.17 We attempted to ensure maximum variation in terms of background (clinical, managerial, strategic) and geographical location.
We used a degree of respondent-driven sampling to maximise the chances of recruiting a maximum variation sample.18 In doing so, we selected clinical interviewees randomly from those attending the workshop to ensure a range of locations and professions.
Data collection
We used semistructured interviews as the main method of data collection. We also undertook non-participant observation of workshops where the Platform and applications were demonstrated to clinical leads to observe the dynamics of prospective user reactions in situ. These workshops were led and facilitated by a representative from Scottish Government in order to engage key clinical opinion leaders. Participants were presented with demonstrator scenarios showing how the Platform worked with the patient record, followed by interactive discussion of first impressions, expectations, drivers and potential challenges. Combining different methods enabled us to triangulate the data sources to validate emerging findings.19
The interviews consisted of open-ended questions about the experiences of existing solutions and expected outputs of a Scotland-wide system through exploring:
Stakeholders’ attitudes.
Implementation strategies.
Measurements for the impact of DSS and how it could be achieved across settings.
Potential challenges to using proposed solutions as well as ways to address these and ensure sustainability.
Definitions of ‘success’ from various perspectives.
The interview guides were tailored to the roles and organisations of individuals. Each interview took between 20 min and 1 hour. Interviews with policymakers explored strategic drivers for DSS and how these fit within the wider political and supplier landscape within NHS Scotland. Interviews with clinicians explored their attitudes towards the DSS Platform, how it could impact their work and how it could integrate with their existing practices.
During workshops, the researcher (MC) recorded the layout of the room, the actors and their roles, how the Platform was perceived to be received, emerging attitudes and reactions and the researcher’s own impressions/feelings in relation to the observation.
With written consent from participants, interview data were digitally audio recorded and transcribed verbatim by a professional transcriber. The researchers (KC and MC) also recorded field notes for each meeting, interview and for observations.
Data analysis
Our analysis was informed by the sociotechnical approach.16 20 This focuses on exploring structural technological factors and associated social processes as well as the inter-relationship between the two during implementation and optimisation. The sociotechnical approach has been applied widely to understand, for example, how technological change can result in changes to work practices of healthcare professionals, and vice versa, and how users can shape technological designs.
Data were thematically analysed, initially within stakeholder groups in order to explore different viewpoints and then data sources were triangulated. We drew on our previous work in this area as a deductive coding framework, where possible. In addition, we inductively identified emerging themes.21 Themes were developed based on frequency of occurrence and salience among different stakeholder groups. Negative cases, that is, those that did not fit within the narrative, were explored, keeping potential implications for the national strategy in constant focus.
Findings across stakeholder groups were then compared in analysis meetings of the research team. This involved discussing commonalities and differences across data sources and participant groups, as well as exploring potential underlying explanations for differences and remaining tensions. Although we observed subtle differences across participant groups (which we explain in the Results section), our general findings were broadly comparable, so consensus was achieved though aggregating findings at higher analytical levels.
Results
We conducted 30 one-to-one interviews and observed eight workshops between 10 May 2018 and 30 October 2018. Participants are summarised in table 1.
High-level interview participants
We identified four key themes which we explore in detail with illustrative quotes:
Widespread recognition of the potential value of DSS.
Leadership and strategy to implement DSS nationally.
The important role of usability and interoperability.
Risk of unintended consequences emerging from implementation of DSS.
Widespread recognition of the potential value of DSS
All interviewees and workshop attendees agreed that, in line with existing empirical evidence, the strategic decision to implement DSS in NHS Scotland was the right way forward.
With good supportive decision-making, people have a better chance of getting the right care at the right time in the right place. Not the wrong care too late at excessive cost, with disabling consequences. (Participant 1, male, Policy)
Participants expected DSS to have a positive impact on safety, quality and efficiency. Areas and potential outcomes mentioned most frequently are summarised in table 2, although it is important to note that many of these related to the expected impact of digital health initiatives more generally rather than being associated with DSS. The only two areas that were specifically related to DSS were adherence to guidelines and availability of information.
Expected areas of impact and potential outcomes of DSS
The expected areas of impact frequently aligned with important policy drivers around guideline implementation and person-centred approaches to decision-making in health and social care settings.22 23 For example, stakeholders talked of giving patients more power to decide on their own treatment. A typical response was:
…I think hopefully it’s a better care for that patient and a more informed patient as well, so they know more about their treatment and why they’re on a treatment and therefore can help…making a decision in conjunction with the clinician. I think that’s going to be quite beneficial for patients as well to give them more empowerment… (Participant 22, male, Pharmacist)
However, the definition of outcome measures was perceived to be difficult and the long time to realise potential benefits was highlighted. Participants also acknowledged that outcomes would be dependent on the DSS application in question, and some interviewees were not entirely clear which application was included in the Platform.
I come across decision support in various different guises. Or for what I would consider decision support. (Participant 13, male, Clinical Lead)
Despite the overall positive attitudes among participants, we also observed concerns that expectations may not necessarily match system functionality. For example, many clinicians discussed a more general problem of information overload and the difficulty of navigating different sources of information. One GP stated that she often had open 10 tabs on her computer at a time, while another mentioned going in and out of different systems, and several stated carrying folders of paper-based records. Participants therefore highlighted evaluation of benefits as a priority area.
Leadership and strategy to implement DSS nationally
While no participant expressed an overt negative opinion on the concept of DSS in principle, early and ongoing engagement with clinicians and other healthcare staff was viewed as crucial during system development and implementation. Participants were positive towards the concept of DSS, but had some concerns about how this would work in practice. At one workshop, a male GP emphasised that his comments should not be seen as negative but ‘in the spirit of improving the system’.
So their forward travel seems to be in the right direction providing that…the final product will work as it’s supposed to work…we’ve got 10 min. That has a big impact on your time whereas if you’ve got something popping up that can do these things then it should work far more efficiently. (Participant 20, female, GP, Tayside)
Participants wanted to get ‘hands-on’ experience of using the system and be engaged in ongoing development to ensure that it worked in practice as well as in theory. Ongoing stakeholder engagement was also stated to be necessary to raise the profile of the Platform among the wider healthcare workforce.
The actual workshop…was a good step in the right direction, but I think for people who aren’t aware of these things, not just GPs but pharmacists, nursing and other kind of clinicians, I think a wee bit of work needs to be done to raise its profile. I really enjoyed looking at the screens and saw great potential. (Participant 27, female, GP)
We also observed a tension between a perceived lack of strong national-level leadership (which was considered to be needed for the successful implementation of the strategy) and staff changes within the eHealth Directorate of the Scottish Government. Several interviewees stated that these changes caused challenges surrounding strategic priorities and direction of travel.
…my preference is…we have a sense of where we’re going to prioritise the initial investment and that is all coordinated from the one position, rather than from a variety of different piecemeal pockets and funded…it needs the central coordinating function. (Participant 13, male, Clinical Lead)
While calling for central leadership and direction, participants also acknowledged that a variety of different projects had to be managed under a portfolio-based approach. Here, there appeared to be a tension between the perceived need to have a firm direction of travel beyond Platform implementation and the multitude of ongoing DSS initiatives across settings (including primary, secondary and social care). While leadership was seen to be required to ensure alignment of initiatives and avoid silos, there was also a perceived need to recognise that strategy was sufficiently agile to cope with changing demands and stakeholder experiences.
I think [Platform] is one of the potential ways…the fact that there’s a number of pilots which have been started to test and to see what learning we can take from that, to me feels that the right approach. But it might not be the only approach…we have to make sure that we look slightly beyond that to say, okay, and this is what we’ve got, this is the tools that we have in the here and the now. But we know that the world is changing…as technology’s concerned it’s changing really, really quickly. So, we need to be preparing ourselves for that next leap, as well. (Participant 12, female, Policy)
System usability and interoperability
The Digital Health and Care Strategy provided a mandate for integration of digital health systems in Scotland and the Platform (which adheres to OpenEHR and Fast Healthcare Interoperability Resources open standards) fits in well with this.23 24 However, it did at the time of our work not integrate with Egton Medical Information Systems and Vision, primary care EHR systems procured centrally in Scotland. Participants expressed hope that this integration would be possible in the future. In the interim, developers had created interfaces that allowed interfacing with the Platform but did not provide full integration.
I think it will have to [integrate], but how we get to that point, I'm not sure. So we do have a contractual requirement for [developers] to deliver, and interact with our clinical decision support systems. So any supplier would, notionally, at least, have to be able to do that. (Participant 14, male, Clinical Lead)
What we do know is we are using a demonstrator environment with the look and feel exactly like it will look in reality, but for the real integration we will most likely use a third-party developer in the first phase. (Participant 9, male, Developer)
System integration was also perceived to be crucial from a usability viewpoint and clinical interviewees expressed strong concerns that the Platform may not effectively integrate with other primary and secondary care systems (as well as their existing in-built DSS). This lack of integration, it was feared, may lead to slowing down of existing systems and parallel data entry (leading to duplication of data entry and data inconsistency).
We feel like we’ve got lots of systems that work separately. Now, they don’t talk to each other…so you end up having to put data into more than one system…so if they could talk to each other a bit better I think that would help. (Participant 25, male, Consultant)
Another perceived factor potentially affecting end-user usability was the degree of local customisation needed to tailor the number of pop-ups to local needs and avoid alert fatigue. Clinicians also mentioned that they were already presented with pop-ups in their own GP systems and that alerts from various sources needed to be aligned and thresholds for alerting set carefully in close consultation with them to ensure usability.
Pop-ups come up from all different places, so there’s the notes that practices put on the systems to try and remind you to do something…you’re potentially going to throw another set of pop-ups in there to have your clinical decision support… (Participant 21, male, GP)
Risk of unintended consequences emerging from implementations of DSS
As clinicians had only seen pilot versions of the software in test environments and some had previous negative experiences of using new digital systems, there were uncertainties around the use of the Platform in real-life settings. For example, clinicians mentioned the risk of duplication of work and increased workloads despite their desire for more time with patients.
… to introduce that amount of potentially new information into a consultation that’s very, very tight time wise is always going to be a challenge. (Participant 21, male, GP, Glasgow)
It was also viewed as important to avoid over-reliance on systems by those with insufficient clinical experience and to ensure that the system did not attempt to replace the expertise of clinicians.
It’s useful to see amount of information available, but it could lead you down the wrong pathway and be in charge rather than the doctor. They [GPs] have the background knowledge of the patient and want to use their expertise. (Glasgow Workshop, Table Feedback)
Several stakeholders in the workshops and in interviews also highlighted some potential tensions between decisions of clinicians and patients, with GPs generally being more risk averse than patients. Some therefore argued that DSS should be patient informed and not patient led, with the ultimate decision of treatment being in the hands of the clinician.
Patients take more risks than doctors but the clinician is responsible for them it should be patient-informed but not patient-led. (Participant 20, male, GP)
Another concern expressed at workshops and in interviews was that, for clinicians, the patient-centred approach may be in direct contrast to other policy drivers such as efficiency and patient outcomes. GPs described scenarios where patients might choose minimal intervention and therefore have a worse outcome.
You’re trying to reduce variability for drugs etc. but this can go against [patient centred approach] where you are giving personal choice. How can success be measured? (Glasgow Workshop, Table Feedback)
Discussion
Overall, we observed a strong sense of support from all stakeholders for the concept of DSS and, if it worked as they expected and through sustained end-user engagement, associated plans to roll it out across NHS Scotland. As this was a first of type, and as the Platform, at the time of our evaluation, only existed in pilot settings, there were also some tensions. These included the need for strong national leadership and associated strategy allowing for a degree of local input, system usability associated with alert fatigue and integration with existing systems, and ongoing monitoring of potential benefits and unintended consequences emerging from implementations.
Strengths and limitations
This evaluation offered important opportunities for realising true impact through ongoing formative feedback to policymakers. In doing so, our written progress reports have informed a business case submitted to Scottish Government in October 2018 for further funding of the Platform beyond pilot sites. It also allowed to adjust to changing policy landscapes and emerging priorities through the timely delivery of formative feedback and contributed to building clinical and policy-level engagement.
The formative real-time nature of this work has, however, also posed some challenges for all involved. Changes in policy priorities and emerging findings necessitated flexible changes to our initially planned methodology over the course of our work in order to ensure that it remained relevant. For example, although we had initially planned to work with settings that had implemented systems as part of the Programme, only some functionality existed at the time we began our work. Such delays are common in large public initiatives. Interviews with policymakers further indicated a strong interest in the Platform, as a cloud-based CDS Platform based on open standards, which fits well with ongoing efforts to create a National Digital Platform that facilitates interoperability of health and care services.24
Moreover, our funders also acted as gatekeepers to participants and workshops. The sample may therefore not be representative of the wider stakeholder groups and reflecting the range of voices surrounding the DSS landscape. Participants consisted largely of those who were already engaged and might not reflect the views of wider stakeholders. As a result, our findings are likely to provide an incomplete picture of reality and should be interpreted with caution, despite (or precisely because of) fulfilling their purpose of delivering political impact. We have also not yet had sight of the business case, which our work has contributed to, so it difficult to position our work in the evolving policy landscape.
It is further important to recognise that this work presents only a snapshot of an early evaluation of the beginnings of a national programme. It by no means provides a complete picture and calls for more in-depth and longitudinal work to assess ongoing developments and use over time.25 For example, it was in some instances not clear to us if clinicians and policymakers really understood the implications of using the Platform in the ‘real world’ as the workshop demonstrations were based on idealised scenarios. Real challenges may only emerge/materialise when the system is tested and a later stage in development and when it is used in practice.
Integration of findings with the current literature
Emerging issues in this work reflect tensions present in many large-scale digital change initiatives in healthcare settings. While the rationale for change and the value proposition of the Platform was clear to all participants in this work, the leadership and governance model going forward was an important concern. One common theme was the agreement that there needed to be a mixture between ‘bottom-up’ and ‘top-down’ approaches to implementation, ensuring a degree of central leadership and direction while also allowing for local input in decision-making.26 This balance is crucial, as many existing ‘failed’ change initiatives such as the National Programme for Information Technology (NPfIT) have illustrated that national ‘top-down’ approaches alone are insufficient to realise the benefits associated with large-scale change in healthcare settings. In the NPfIT, ‘top-down’ strategies were superseded by locally driven strategies, and ultimately changed to a more ‘middle-out’ model, where national strategic direction aligns more closely with local strategy.27
Both clinical and policy stakeholders in our sample had a strong desire for systems to save time and improve safety, quality and efficiency of care. However, as the expected areas of impact in table 2 illustrate, the impact of the DSS may be overestimated which may lead to problems and frustration during implementation. Similarly, time savings stand in stark contrast to the often long and painful implementation experiences in healthcare settings where benefits of digital systems can take a long time to materialise, and where benefits are often not visible to those who take the additional burden associated with their implementation.28 29 This long pathway to benefits is often due to social dimensions surrounding the technology, including, for example, adverse impacts on work practices and increased workloads, the potential introduction of new errors, adverse impacts on time spent with patients and unrealistic stakeholder attitudes/expectations.30–32
While most participants were optimistic in relation to potential benefits of the Platform, there were also some major concerns. These related mainly to the would-be adopters interacting with technologies, and specifically usability and integration with existing workflows. Issues with system usability adversely affecting adoption patterns feature consistently in the empirical literature. In particular, the issue of alert fatigue in DSS and information overload among clinicians remains an unresolved area internationally.33–35 There is a trade-off between the number of alerts and the attention of clinicians, which has led some to advocate the use of a limited user-determined number of carefully designed pop-ups.36
An increasing number of interfaces can adversely affect system performance and usability and also lead to the introduction of new errors. Our work exploring the implementation of hospital electronic prescribing systems has shown that integration and interfacing problems could in some cases inhibit the effective transfer of information, leading to duplicate data entry and adverse effects on the availability of information.15 37 This highlights that therefore particular attention needs to be given to effective integration of the Platform with existing primary care systems.
Policy recommendations and implications for practice emerging from this work
Recognising the early stages of this evaluation, our work has some preliminary implications for policy to ensure long-term sustainable delivery of the Platform. Most importantly, sufficient time and resources need to be available to implement, facilitated by overall national ownership to support the implementation of the Platform within the eHealth Directorate. This work also needs to align with the development of the wider National Digital Platform (which is planned to connect all digital health and care services in Scotland) to facilitate interoperability and the wider Scottish Digital Health and Care Strategy to facilitate alignment with national strategic priorities.23 24
As technology landscapes and needs are constantly changing, future iterations of the Programme have to be sufficiently agile. Revisiting the original document at predefined time points informed by ongoing evaluation and stakeholder engagement is likely to facilitate these developments. This should also involve ongoing communications about the current strategy and associated timescales, stakeholder engagement opportunities and resulting actions, as well as ‘hands-on’ experience of system use.
In order to promote usability, there is a need to negotiate seamless integration with existing primary care systems, as interfaces as interim solutions can adversely affect user experiences. The negotiation with primary care system suppliers and promoting seamless integration through effective user interfaces is therefore key going forward.
There are further a number of potential ways to tackle tensions between local and national dynamics. These are summarised in box 3.
Recommendations to tackle potential local and national tensions
Developing a strategy allowing for a degree of local customisation of systems and managed tailoring of alerts. The devolving of responsibility to local level needs to be tempered by the legal requirement to comply with the Medical Device Regulations for Decision Support that are classified as medical devices.
Ongoing system development and optimisation with sustained stakeholder engagement including ‘hands-on’ experience for clinicians.
Local organisations need to recognise the value of systems—a limited number of pilots that are carefully evaluated will help to engage other organisations yet to implement and will help to identify and mitigate emerging risks early.
There is a need to develop a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.
As this evaluation has focused on the very early stages of piloting, our findings are preliminary. We strongly recommend ongoing formative evaluation of the implementation of the Platform through longitudinal work to ensure effective alignment with wider strategy, and continuous stakeholder engagement. This work needs to include a designated strand exploring evolving policy and supplier landscapes and market management.
Conclusions
Our work has laid the early foundations for a formative longitudinal evaluation of the DSS Programme in Scotland. It has helped to promote early engagement of key stakeholders and also informed policy planning by identifying key clinical and political drivers of system implementation, and challenges that are likely to warrant negotiation going forward. These include achieving a balance between national targets and local incentives, system usability and benefits realisation. Careful ongoing formative evaluation guided by the evaluation framework developed as part of this work will help systems to realise their maximum potential while minimising disruption to existing services.
Our formative methods have the potential for immediate impact on practice and we believe that international settings can learn from these early insights. We have shown that a relatively modest evaluation budget can help to actively shape strategic developments by highlighting challenges, identifying potential pain points and informing proactive decision-making that is likely to avoid many of the obstacles faced by digitally enabled change projects. Formative evaluation activities can therefore provide significant value as they help to gain insights into local challenges through contextualising wider empirical findings. Applying empirical insights without analysing local dynamics is likely to be of limited value.
Acknowledgments
We gratefully acknowledge all participants’ time and input, and Dr Ann Wales’ advice throughout the process of the evaluation. We also thank two anonymous peer reviewers for their helpful feedback which has helped us improve this work.
References
Footnotes
Contributors KC, HM and AS have conceived this work. KC and MC have collected the data and led on the analysis. KC has led on drafting the manuscript and all authors have commented critically on several iterations.
Funding This article has drawn on a programme of independent research funded by the Digital Health and Care Institute and the Scottish Government.
Disclaimer The views expressed are those of the author(s) and not necessarily those of the funders.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval Institutional Review Board from the Centre for Population Health Sciences at The University of Edinburgh, UK.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available upon reasonable request.