Driving digital health transformation in hospitals: a formative qualitative evaluation of the English Global Digital Exemplar programme ======================================================================================================================================= * Marta Krasuska * Robin Williams * Aziz Sheikh * Bryony Franklin * Susan Hinder * Hung TheNguyen * Wendy Lane * Hajar Mozaffar * Kathy Mason * Sally Eason * Henry Potts * Kathrin Cresswell ## Abstract **Background** There is currently a strong drive internationally towards creating digitally advanced healthcare systems through coordinated efforts at a national level. The English Global Digital Exemplar (GDE) programme is a large-scale national health information technology change programme aiming to promote digitally-enabled transformation in secondary healthcare provider organisations by supporting relatively digitally mature provider organisations to become international centres of excellence. **Aim** To qualitatively evaluate the impact of the GDE programme in promoting digital transformation in provider organisations that took part in the programme. **Methods** We conducted a series of in-depth case studies in 12 purposively selected provider organisations and a further 24 wider case studies of the remaining organisations participating in the GDE programme. Data collected included 628 interviews, non-participant observations of 190 meetings and workshops and analysis of 9 documents. We used thematic analysis aided by NVivo software and drew on sociotechnical theory to analyse the data. **Results** We found the GDE programme accelerated digital transformation within participating provider organisations. This acceleration was triggered by: (1) dedicated funding and the associated requirement for matched internal funding, which in turn helped to prioritise digital transformation locally; (2) governance requirements put in place by the programme that helped strengthen existing local governance and project management structures and supported the emergence of a cadre of clinical health informatics leaders locally; and (3) reputational benefits associated with being recognised as a centre of digital excellence, which facilitated organisational buy-in for digital transformation and increased negotiating power with vendors. **Conclusion** The GDE programme has been successful in accelerating digital transformation in participating provider organisations. Large-scale digital transformation programmes in healthcare can stimulate local progress through protected funding, putting in place governance structures and leveraging reputational benefits for participating provider organisations, around a coherent vision of transformation. * health information systems * program evaluation ### Summary #### What is already known? * There is currently a strong drive internationally towards creating digitally advanced healthcare systems through coordinated efforts at a national level but there is lack of knowledge on how to stimulate large-scale digitalisation. #### What does this paper add? * Large-scale digital transformation programmes in healthcare can stimulate local progress through protected funding, putting in place governance structures, and leveraging reputational benefits for participating provider organisations, around a coherent vision of transformation. ## Introduction There is currently a strong international drive towards creating digitally-enabled health systems and settings, with governments embarking on large-scale health information technology (HIT) change initiatives to improve quality, safety and efficiency of health and care.1 2 For example, in the USA, the Health Information Technology for Economic and Clinical Health (HITECH) initiative launched in 2009 combined over US$25.9 billion of central funding with development of a national set of standards for implementation of electronic health records (EHRs) to stimulate digital transformation of provider organisations.3 The German federal government’s 2020 Hospital Future Act committed over €3 billion across a 2-year period to stimulate digital transformation of hospitals.4 Government-led, large-scale HIT change programmes have also recently been initiated in Canada,5 Australia6 and New Zealand.7 However, historically, such national programmes often have failed to realise their ambitious digitalisation goals. For example, in England, the National Programme for Information Technology (NPfIT)—the largest ever national digitalisation programme with an initial budget of over £9.8 billion8 9—was discontinued in 2012 as it was perceived to not sufficiently cater for the needs of implementing organisations.10 The relative lack of success of many nationally-led, large-scale HIT change programmes may be attributed to limited current understanding of how such programmes work to help promote digital transformation locally.11 There is therefore now a growing need for evidence on how best to stimulate digital transformation of healthcare systems and settings through these kinds of initiatives. To address this gap, we here present findings from an independent, formative evaluation of the Global Digital Exemplar (GDE) programme—a flagship, national HIT change initiative aiming to stimulate digitalisation of English hospitals through creating a cohort of provider organisations that would act as exemplars of digital excellence (box 1).12 The programme was developed in response to an independent review that drew lessons from previous digital transformation initiatives in the UK and the USA.13 Given that funding available was not sufficient to allow all provider organisations to fully digitalise, this strategy adopted a phased approach with funding initially allocated to relatively digitally mature organisations. These were paired up with less mature partner organisations, with whom they were encouraged to share knowledge and thereby accelerate digitalisation. We aimed to address the following research question: How did the GDE programme promote digital transformation in participating provider organisations? Box 1 ### Overview of the Global Digital Exemplar programme The Global Digital Exemplar (GDE) programme is a large-scale health information technology (HIT) change programme launched by National Health Service (NHS) England aiming to stimulate the digital transformation of the English healthcare system. It had a total budget of over £385 million of central funding, a 5-year duration (2017–2021) and involvement of 51 individual provider organisations. The GDE programme was introduced in the aftermath of the English National Programme for IT—the largest national digitalisation programme worldwide with a budget of over £9.8 billion,8 which was discontinued in 2012 following a brief period of relatively uncoordinated digital transformation attempts across the healthcare system. The key strategy of the GDE programme, led by NHS England, was to stimulate digital transformation across English NHS healthcare providers and to form a central point for facilitating knowledge creation by creating ‘Global Digital Exemplars’ (GDEs)—local centres of digital excellence that could serve as examples of best practice. Provider organisations were selected to become GDEs, based on their relatively high levels of digital maturity (the extent to which organisations had digitally-enabled processes) and capability to undertake an innovative digital transformation programme. Each GDE provider organisation signed a funding agreement with NHS England to implement a detailed portfolio of HIT change projects over a period of 2–3.5 years and received £5–10 million of central funding (which had to be matched with the same level of internal funding). Additionally, GDE provider organisations were paired with one (and in two cases two) partner providers—referred to as Fast Followers (FFs). The FFs were not expected to be as digitally mature as their partner GDEs but to be sufficiently mature to be able to rapidly accelerate their digital transformation through knowledge transfer from their partner. The FFs were also asked to prepare a portfolio of digital transformation projects to be carried out during this period. FFs received half of the central funding that the GDE organisations received (ie, £5 million), which again had to be locally matched with the same amount. Twenty-three provider organisations took part as GDEs and 25 as FFs. All participating organisations were asked to establish a senior clinical digital leadership role in the form of a Chief Clinical Information Officer ahead of the start of the programme. The Healthcare Information and Management Systems Society (HIMSS) Electronic Medical Record Adoption Model classification tracking hospitals’ levels of digital maturity on a scale from Level 0 to 7,14 was used as a benchmark for digital excellence in the programme. Acute GDEs were expected to achieve HIMSS Level 6 with a view to 7 and mental health GDEs and FFs Level 5 by the end of the programme. In addition, the GDE programme supported coordinated learning including setting up learning networks for staff in participating organisations, organising networking events and other knowledge transfer activities including the production/circulation of Blueprints (documents capturing learning in implementing particular changes).15 ## Methods We undertook a longitudinal qualitative study of the GDE programme that aimed to explore digital transformation in participating provider organisations and the wider healthcare system.16 Our work had both formative and summative elements, but its defining characteristic was its formative nature, feeding back emerging findings to decision-makers and thereby shaping delivery of the programme. The detailed methodology is described in a separate published research protocol and in Appendix 1.17 The evaluation took place between January 2018 and March 2021. We followed the Consolidated Criteria for Reporting Qualitative Research in this paper.18 Formative, qualitative evaluations, conducted in real-time alongside change programmes, can help to explore the processes involved in seeking to stimulate digital transformation and can thereby inform future initiatives.19 20 This type of evaluation collects evidence on the processes involved in stimulating digital transformation through HIT change initiatives and on an array of emerging outcomes including consequences not anticipated/intended by programme architects. Such formative evaluations are well placed to inform decision-makers during the programme that is being evaluated. We conducted 628 interviews, observed 190 meetings and analysed 499 documents (see box 2 for an overview of the data set and Appendix 2 for a detailed description). This included an additional round of interviews performed in autumn 2020 in relation to the impact of COVID-19 on digital transformation. Interviews lasted 1 hour on average. Box 2 ### Description of sample (GDE, Global Digital Exemplar; FF, Fast Follower) #### In-depth case study sites (12 provider organisations; 8 GDEs: 6 acute, 2 mental health; 4 FFs: 3 acute, 1 specialist) * 309 interviews (39 senior managers; 65 clinical digital leaders; 47 non-clinical digital leaders; 46 GDE programme staff; 112 operational staff). * 104 documents. * 67 meetings observed. Interview periods: * Pilot interview: March 2018 (1 interview) * 1 GDE programme staff. * First round: May 2018 – February 2019 (137 interviews) * 16 senior managers. * 20 clinical digital leaders. * 11 non-clinical digital leaders. * 14 GDE programme staff. * 76 operational staff. * Second round: March 2019 – May 2019 (34 interviews) * 6 senior managers. * 10 clinical digital leaders. * 3 non-clinical digital leaders. * 12 GDE programme staff. * 3 operational staff. * Third round: June 2019 – March 2020 (101 interviews) * 11 senior managers. * 27 clinical digital leaders. * 26 non-clinical digital leaders. * 10 GDE programme staff. * 27 operational staff. * Fourth round: August 2020 – December 2020 (post-lockdown) (36 interviews) * 6 senior managers. * 8 clinical digital leaders. * 7 non-clinical digital leaders. * 9 GDE programme staff. * 6 operational staff. #### Broader case study sites (24 provider organisations; 15 GDEs: 10 acute, 5 mental health; 9 acute FFs) * 247 interviews (32 senior managers; 78 clinical digital leaders; 65 non-clinical digital leaders; 44 GDE programme staff; 28 operational staff). * 283 documents. * 19 meetings observed. #### Interview periods: * First round: 2018 (95 interviews). * Second round: 2019 (69 interviews). * Third round: 2020 (83 interviews). #### Other data * 72 interviews (61 policymakers; 3 vendors; 4 engagement leads and 4 other stakeholders). * Non-participant observations of 104 national meetings, workshops and conferences. * 112 documents. #### Interview periods: * First round: March 2018 – December 2018 (31 interviews). * Second round: January 2019 – November 2019 (20 interviews). * Third round: January 2020 – April 2020 (3 interviews). * Fourth round: July 2020 – February 2021 (18 interviews). ## Results Our analysis identified several sociotechnical dimensions associated with digital transformation. Many of these have already been extensively discussed in the literature (table 1) and we therefore focus here on exploring novel findings surrounding the wider macro-environmental dimensions associated with the GDE programme. View this table: [Table 1](http://informatics.bmj.com/content/28/1/e100429/T1) Table 1 Findings associated with sociotechnical dimensions of change confirming previous findings in the empirical literature20 28–44 The impact of the GDE programme in stimulating digital transformation locally is described in Appendix 3. ### Earmarked funding stimulated digital transformation locally Dedicated funding over a multiyear period, comprising both external funding (allocated from a central national budget) and matched funding from the provider organisation’s internal budget, was perceived to play a key role in accelerating digital transformation. Funding was used to support and bring forward major upgrades in digital information infrastructures (including renewing core EHR systems) together with a range of smaller-scale digital change projects such as implementation of electronic clinical observations systems or projects to support staff working remotely in the community. Many organisations reported that plans for these changes were already in place prior to the launch of the programme. > It enabled us to do things, because of the money, it enabled us to do things, that we would have done anyway, at twice the speed, (…) but there is something about scale and there is something about speed, which brings a value that is greater than achieving it in twice the time. (Site D, GDE, in-depth case study, GDE programme staff) The scope to secure external funding combined with a requirement for matched funding, also helped to secure local leadership buy-in and support. > [Central NHS funding through the GDE Programme] was enough money to make a case to our finance director and the acting chief executives that we should do it [GDE Programme], because it was money we wouldn’t get otherwise, for a thing we wanted to do anyway. (Site G, GDE, in-depth case study, clinical digital leader) Protected funding was especially important in driving digital transformation for smaller provider organisations with correspondingly smaller internal budgets. For the largest organisations, external GDE programme funding was modest in relation to their overall digital investments. In particular, some of the large provider organisations had substantial development capabilities and large technology budgets that had allowed them, in some cases, to begin planning and implementing comprehensive digital change, meaning that they had already achieved a certain momentum ahead of the programme. As a result, participating in the programme strengthened but did not per se transform the digital strategies and capabilities of these organisations in the dramatic way that could be observed in smaller and less digitally mature providers (which in many cases included Fast Followers (FFs)). Provider organisations described this support as accelerating the rate of change but not radically changing the direction of their prior digital journey. They were able to achieve more because of these additional resources. > My reflection on the GDE process is that I don’t think we would have done this without it. I think we always wanted to do it and it gave us the opportunity to do what we wanted to do anyway but we would not have been able to employ this people, we would not have been able to pay [Supplier] to deliver the extra functionality, we wound not have been able to pay me for two years to provide some clinical input. (Site G, GDE, in-depth case study, clinical digital leader) This momentum and ambition for change grew as a result and continued beyond the end of the programme. > So it has focused…just by the injection of money rather than anything else, the money has enabled us to buy products which when you start delivering them, you then can’t really stop, so although the £10m isn’t enough, it’s now made it an issue that we benefit from this if we did a bit more and we spent a bit more. (Site I, GDE, in-depth case study, clinical digital leader) Provider organisations perceived that the provision of national support primarily through capital funding, as opposed to revenue funding, affected local digital transformation initiatives, as it promoted investment in purchasing hardware and software. The administrative complexity of converting capital funding into revenue streams meant that investing in staff and third-party services to maintain, service, support, upgrade and optimise systems was somewhat inhibited. ### Prestige and reputational benefits helped to secure organisational buy-in and to negotiate with suppliers The prestige and reputational benefits obtained through taking part in a flagship national HIT change programme and competing for the status of being a ‘Global Digital Exemplar’, were instrumental in securing leadership buy-in and also helped to secure wider organisational support for digital transformation efforts. Although some of the organisations participating in the programme already considered themselves as national leaders, being a GDE involved projecting a claim not only of being nationally excellent but also of attaining internationally recognised standards of excellence. Other national programmes had not specifically targeted this already high-achieving segment of provider organisations. Those who were already ‘high-achievers’ were keen to be seen as international leaders and others saw this as putting their organisations into the limelight. In many cases, the ‘Global Digital Exemplar’ badge had been used to communicate the upcoming HIT change projects (eg, EHR upgrades, or implementation of electronic observations) across the organisation, for example, through posters and newsletters. > [The GDE Programme and its agenda] was helpful both from a reputation and to badge it all in a concept of…it gave people a…rallying cry around our direction of travel. (Site 12, FF, broader case study, clinical digital leader) The benefits of enhanced national visibility and status from participation in the Programme were less evident for organisations with a strong prior national or international profile (including many FFs). Smaller provider organisations with modest local profiles reported that taking part in the programme allowed them to be more visible and recognised locally. > Reputationally, we’re considered regionally as digitally mature, and that’s quite a battle to fight. Not necessarily with other mental health or community trusts but certainly with the larger acutes [acute care provider organisations],… you kind of have to earn your place. You do have to earn your place around the table and some of the things that we’ve done in GDE have enabled us, to use a very common expression at the moment, a more sort of level playing field. (Site E, FF, in-depth case study, GDE programme staff) Provider organisations further noted that the status associated with the programme increased their negotiating power with vendors. Large provider organisations (mainly GDEs) that were recognised nationally and internationally as leading centres were often invited to become reference sites for certain product implementations and thereby secured allocation of additional resources from vendors. Smaller, less prestigious provider organisations (including many mental health providers and FFs) in contrast often found themselves competing over vendor resources with other customers including other provider organisations taking part in the programme. > I think if you speak to our finance director… he would say it’s the [vendor] relationship that’s the most valuable part of the GDE… being part of the GDE process, he thinks, gives him much more leverage with [vendor] to actually deliver what they’ve promised. Cause quite frankly, if they don’t deliver it with us, then they won’t be able to sell to other organisations, ‘cause we will be their site, where everyone will come and see all their solutions together. (Site I, GDE, in-depth case study, GDE programme staff) Being labelled a ‘Fast Follower’ offered lower perceived status benefits than GDE. Some FF organisations felt that they were in some respects more advanced than their GDE and should therefore be labelled ‘partners’ instead of ‘followers’. ### Governance requirements supported establishment of project management structures, secured executive buy-in and strengthened clinical digital transformation leadership The funding agreement between provider organisations and the central funding body contained contractual obligations, which included the organisations’ digital strategy and an outline of HIT projects to be undertaken with timescales, funding milestones and a Statement of Planned Benefits. Provider organisations were thus required to prepare and then execute a portfolio of HIT change projects in a relatively short period. Further, although not a formal obligation, there was also an expectation for the provider organisations to set up a local GDE Programme Board to oversee deployment of the programme locally. These in turn supported the creation and expansion of change management and engagement structures within provider organisations to support the implementation of the HIT change projects outlined in the funding agreement. The requirement to meet the milestones set out in the funding agreement, combined with well-depicted digital transformation goals, helped to secure executive support and helped to make the transformation agenda more salient at the executive level. > I think one of the main parts that was really effective is the pace-setting element of the GDE. […] The pace setting as part of the Programme was a massive part of achievements. And I think the reason for that is it really focuses the board. Because you have essentially money attached to a deadline to achieve something, that’s extremely motivating. And in trusts where you have so many competing priorities […] I thought was very effective actually that we had to hit certain milestones with good quality and that then funding would be achieved. And I think that really helped focus the board. And because of that, we had a really, I think, strong functioning Digital Oversight Committee through the Programme and that’s one of the things that kept the momentum going. (Site 10, GDE, broader study, clinical digital leader) Provider organisations were required to report regularly on implementation progress and benefits achieved to the central funding body. However the reporting methods were perceived as burdensome, particularly as these reports were not always aligned with the reports that provider organisations had to submit to their own boards and for other health service reporting systems. Provider organisations reported that the burden of reporting diverted efforts from other key activities related to digital transformation. Although there was an attempt to simplify central reporting procedures as the programme progressed, with the adoption and refinement of a computerised reporting tool, little progress was made in harmonising reporting requirements among different parts of the health service (which had different established report requirements, deadlines and reporting periods). Another issue was that, although the funding agreements laid out a timetable of contractual commitments, over time as the programme progressed, context, technologies and local priorities changed. Some provider organisations had trouble in meeting the contractual obligations and milestones, given the dynamism and uncertainties surrounding digital transformation, and highlighted the rigidity of funding agreements. Although it was possible to renegotiate funding agreements, this process was seen as slow and time-consuming. > Yes, we can set milestones for 6 months or 12 months but trying to set a milestone for three years’ time when IT changes, the organisation changes so quickly. (Site D, GDE, in-depth case study, non-clinical digital leader) Another aspect of centrally introduced governance requirements was a mandatory requirement to appoint a Chief Clinical Information Officer (CCIO) ahead of the programme—a senior leadership role within provider organisations combining clinical and digital transformation expertise. This requirement was critical in helping organisations to build capacity to manage and lead digital transformation projects. The CCIOs also had a major role in securing and enhancing clinical engagement in the digital transformation process and in selecting and configuring the technological systems to ensure they would be fit for purpose in the clinical context. Further, they contributed to raising the awareness and priority of the digital transformation agenda within senior leadership. The appointment of a CCIO further promoted the creation of a number of related senior leadership positions combing clinical and digital expertise such as a Chief Nursing Information Officer (CNIO), Chief Medical Information Officer (CMIO) and deputy CCIOs responsible for specific subdisciplines (eg, cardiology, oncology). > We wouldn’t have had CCIOs [Chief Clinical Information Officers] if we weren’t a GDE really, I think the GDE opportunity coalesced in the IT department which was very IT-driven to actually, well, we need to engage clinicians in this, otherwise we won’t get this money [from the GDE Programme], we’ve got to show that we’ve got clinical involvement. (Site I, GDE, in-depth case study, clinical digital leader) The strengthening of digital informatics capabilities was reinforced by related changes in the whole sector including the establishment of the NHS Digital Academy—an NHS training programme that aimed to develop a new generation of clinical digital leaders to drive digital transformation. > ‘…going through… the Digital Academy has really helped in this kind of difficult phase where you’re looking at projects, programmes, organising, whole organisations around it. I mean I’m falling back on some of the stuff we did there quite a bit now actually and I was, I realise how inexperienced we were when we started.’ (Site E, GDE, in-depth case study, clinical digital leader) Finally, as part of the GDE requirements, participating provider organisations were expected to achieve high levels of performance under the Healthcare Information and Management Systems Society (HIMSS) Electronic Medical Record Adoption Model (EMRAM). Their ability to meet these ambitious targets within the relatively short timeframes of the GDE programme was greatly influenced by their choice of supplier. Some (US) vendors that had recently entered the UK market offered comprehensive ‘mega-suites’ already well-aligned with the wide range of functionality required to meet the HIMSS EMRAM accreditation criteria. Many GDE providers turned to these solutions in order to meet the ambitious aims of the programme. Other EHR adopters that stayed with their existing EHR supplier sought to bridge the gap by asking their vendor to extend their range of functionality or by procuring and integrating modules from other suppliers (a strategy labelled ‘Best-Of-Breed’). These provider organisations and their suppliers thereby embarked on an unpredictable journey that posed challenges for both sides. Some suppliers struggled to deliver the new functionalities required within the timeframe of the GDE programme. In addition, the growth in demand due to the programme was such that even some large suppliers were unable to provide the level of development support expected by individual provider organisations. ## Discussion ### Summary of key findings The GDE programme strategy of supporting relatively digitally mature healthcare provider organisations to become exemplars of digitally enabled transformation has resulted in rapid acceleration of transformation and promoted the visibility and priority of digital transformation plans in those organisations. The programme also contributed to the promotion of clinically focused digital change management capability and the emergence and strengthening of local clinical change leaders (ie, those planning and implementing local programmes, including CIOs, GDE programme managers, CNIOs, CMIOs and CCIOs). This has driven a visible culture shift among clinicians and leaders to a proactive expectation that digital solutions underpin care delivery and enable transformation. There has also been a concomitant increase in engagement and capability in the general workforce as organisations increasingly digitalised their organisational processes. Earmarked funding, the strengthening of local governance structures, digital project management capability and the reputational benefits associated with being included in the GDE programme have helped to ensure buy-in for digital transformation plans from both senior managers and frontline staff. This ensured that what was delivered was digital transformation rather than simply a technology implementation programme. However, it is important to keep in mind that while the GDE programme support imparted momentum and direction, some provider organisations were already on this trajectory of change and during the programme followed local digital transformation strategies that were already planned. ### Strengths and limitations We conducted a national, longitudinal, formative evaluation of a first-of-a-kind large-scale HIT change initiative to advance digital transformation in the English National Health Service. We collected a large, qualitative data set from participating provider organisations and from national actors over extended timescales. This allowed gaining comprehensive insights into the mechanisms of change promoted through the GDE programme and associated outcomes. Detailed limitations of our work can be found in box 3. Box 3 ### Limitations of our work Our findings on the digital transformation outcomes associated with the Global Digital Exemplar (GDE) programme should be interpreted with caution. Intended and unanticipated consequences were still emerging at the end of our evaluation work. Attribution of outcomes in large-scale digital transformation initiatives is not straightforward, as interventions are often multifaceted, stimulating digitalisation through a combination of enhancements in technological systems and organisational processes—as a result, outcomes take a long time to materialise and may not then be directly attributable to HIT.45 In addition, large-scale change programmes are situated within evolving wider policy and economic settings that may influence outcomes. Various local factors are also likely to have an impact. To address these complexities, our evaluation used a combination of in-depth case studies that allowed for detailed understanding of how the programme unfolded in a range of specific settings and wider case studies of other providers that involved capturing broader patterns and verifying findings from the in-depth case studies. Further, each participating provider organisation proposed a portfolio of digital innovations as part of the programme. Unfortunately, our methodology did not allow us to systematically appraise individual innovations and outcomes. However, a wide range outcomes were reported including many not initially anticipated improvements that were coming to the fore at the end of the evaluation period, sometimes in areas that were not directly related to the original area of implementation (eg, in shared care records across settings). We focused largely on the perspective of provider organisations, particularly local GDE programme managers and implementers. As a result, perspectives of individual healthcare staff within provider organisations are underrepresented. ### Integration of the findings with existing literature Previous findings surrounding the importance of sociotechnical dimensions of digitally-enabled change in the empirical literature have been confirmed in our work,20 28–44 but we have uncovered some important issues surrounding macro-environmental dimensions of change and how these can impact on technological, social and organisational dimensions. These include the role of wider incentives, prestige and governance requirements to stimulate local digitalisation efforts. We found that the GDE programme, as a large-scale digital transformation initiative, accelerated digital transformation in selected digitally advanced sites. Key to success was a combination of dedicated resources, governance frameworks, local ownership and vision. It began with a national review that took stock of previous national experiences and sought to learn from them, actively involving national and international experts, and laying out a vision and steps towards achieving digitally-enabled transformation.12 This stands in stark contrast to previous experiences in the NPfIT, which was, from the start, driven by an arguably unrealistic vision based on centralised procurement which created problems around technology choice and lack of organisational and clinical buy-in.46 The GDE programme allowed a new digital vision and we observed changes in staff attitudes towards digitalisation. This in turn facilitated staff engagement with digitally-enabled transformation activities. The impact of the programme was affected by the COVID-19 crisis that impeded organisational progress towards achieving HIMSS targets but which, by demonstrating the value of digital capabilities (notably in remote consultations), also encouraged more rapid uptake and acceptance and helped to accelerate digital transformation locally. The GDE programme has also helped to reconcile tensions surrounding local input in decision-making with national direction. Key here was setting national goals and monitoring progress, while allowing a degree of local freedom over how to achieve these goals.45 Experiences with other national initiatives reinforce the effectiveness of balancing goal-setting with local choice, a perspective that is supported by the notion of loose coupling where organisational subsystems function well if they can maintain their own identity and autonomy.47–49 The US HITECH initiative reinforces the important role of centrally allocated funding and goal setting in facilitating adoption.50 51 However, although resulting in dramatic increased computerisation of healthcare, HITECH has also illustrated that rapid adoption and mandating use without the cultural changes needed to support transformation can create unrealistic expectations and disillusion frontline clinical staff, a consequence that only became apparent after the programme had concluded.52 The emergence and strengthening of local clinical change leaders helping to promote clinical engagement and leadership-buy-in might help to mitigate risk.53 Throughout this journey, HIMSS served as a roadmap, allowing implementing organisations to plan changes in small steps and allowing national programme managers to benchmark and monitor progress.54 However, requiring providers to rapidly achieve particular benchmarks may restrict markets (favouring existing vendors whose products are already aligned with HIMSS EMRAM) and limit innovation as it leaves little room for experimentation and innovation around local priorities.55 Although characterising digital maturity was not the focus of this paper, these results, building also on existing literature and our previous work surrounding the definition of technological characteristics of digital excellence in hospitals,56 serve as a starting point to identify organisational characteristics of digital excellence in hospitals (box 4). Box 4 ### Organisational characteristics associated with digital maturity * Leadership focus on digitally-enabled transformation of services (rather than merely Information Technology deployment). * Digital transformation expertise at Board level. * Clinical engagement and dedicated intermediary roles between clinical and digital areas. * Activity surrounding envisioning benefits/targets and measuring progress. * Demonstrating benefits for individual users early on in the process. * Strong and experienced project management structures dedicated to digital transformation. * Willingness to share experiences and learn from others. * Open and transparent decision-making and communication across the organisation. * A conceptualisation of digital maturity as a continuous quality improvement process. ### Implications for policy and practice In contrast to recent heavily-funded technology procurement programmes that failed to deliver,2 8 52 the GDE programme has succeeded in promoting digital transformation across a significant tranche (20%) of provider organisations. The experience highlights how a coordinated approach with relatively modest funding can catalyse rapid and significant improvements in digital maturity in healthcare. At the time of writing (August 2021), four provider organisations had achieved HIMSS Level 6 and two had achieved HIMSS Level 7.57 Programme managers recognised that the most mature provider organisations (eg, those expected to meet targets in 2 years) had already begun their digital journey. Although a few organisations struggled to meet the ambitious programme goals, most achieved a substantial boost in terms of the pace and strategic direction of their digital transformation. In this sense the programme seems to have successfully targeted what welfare policymakers have described as the ‘Goldilocks zone’, minimising (wasteful) over servicing and (ineffective) underservicing.58 As this programme ends, there is a risk that the momentum created through the programme is lost. It is imperative to build on lessons learnt and exploit the valuable experience acquired in the programme through follow-on initiatives. Its immediate successor, the Digital Aspirant (DA) programme, currently underway in NHS England, addresses concerns that less mature providers might be left behind.59 Less digitally mature organisations are likely to require more support.60 Questions arise as to whether the DA programme will deliver similar successes to those seen in the GDE programme. The key drivers identified in this paper are somewhat weakened under the DA programme: Organisations participating in DA programme start with lower levels of digital maturity and will receive less funding than those that participated in the GDE programme. Programme governance arrangements are more limited than in the GDE programme, and some of the successful mechanisms to facilitate learning have not been carried forward (notably GDE/FF partnerships). The prestige associated with being a Digital Aspirant may also be lower. The policy agenda is however evolving. Having demonstrated an ability to create islands of excellence, the 2019 NHS Long Term Plan requires all providers to achieve a core level of digitalisation by 2024 to allow information exchange across regional ecosystems.61 Future efforts should focus on strengthening learning networks in order to ensure that lessons learnt are effectively and widely disseminated across the wider NHS. We have summarised the lessons for running digital transformation programmes emerging from our work in table 2. View this table: [Table 2](http://informatics.bmj.com/content/28/1/e100429/T2) Table 2 Lessons for running digital transformation programmes ## Conclusions The GDE programme helped to accelerate digital transformation in participating provider organisations and to establish the foundations for a digital health learning ecosystem. It appears to have achieved this through protected funding, putting in place governance structures and through harnessing reputational benefits for participating provider organisations. The GDE programme provides a template for successful digital transformation that was lacking after the failure of recent high profile heavily funded technology procurement programmes. It is now important that learning from this initiative is maximised in efforts to bridge the digital divide across provider organisations. ## Data availability statement Data are available upon reasonable request. The raw data are available upon reasonable request from the corresponding author. ## Ethics statements ### Patient consent for publication Not applicable. ### Ethics approval The work received institutional ethical approval from the School of Social and Political Science at The University of Edinburgh, UK (no reference number provided). Participants gave informed consent to participate in the study before taking part. ## Acknowledgments We gratefully acknowledge the input of the participants and the Steering Group of this evaluation. ## Footnotes * Contributors KC and RW conceived this paper. KC, MK and RW led the drafting of the manuscript and all authors commented on drafts of the manuscript. KC acts as the guarantor. * Funding This article has drawn on a programme of independent research supported by NHS England. The views expressed are those of the author(s) and not necessarily those of the NHS, NHS England or NHS Digital. This work was also supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. * Disclaimer The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Care. * Competing interests None declared. * Provenance and peer review Not commissioned; externally peer reviewed. * Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise. * Received June 11, 2021. * Accepted November 22, 2021. * © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/) This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/). ## References 1. 1.Sheikh A, Sood HS, Bates DW. Leveraging health information technology to achieve the "triple aim" of healthcare reform. J Am Med Inform Assoc 2015;22:849–56.[doi:10.1093/jamia/ocv022](http://dx.doi.org/10.1093/jamia/ocv022)pmid:http://www.ncbi.nlm.nih.gov/pubmed/25882032 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1093/jamia/ocv022&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=25882032&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 2. 2.Jha AK, Doolan D, Grandt D, et al. The use of health information technology in seven nations. Int J Med Inform 2008;77:848–54.[doi:10.1016/j.ijmedinf.2008.06.007](http://dx.doi.org/10.1016/j.ijmedinf.2008.06.007)pmid:http://www.ncbi.nlm.nih.gov/pubmed/18657471 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.ijmedinf.2008.06.007&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=18657471&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000261447100006&link_type=ISI) 3. 3.Washington V, DeSalvo K, Mostashari F, et al. The HITECH era and the path forward. N Engl J Med 2017;377:904–6.[doi:10.1056/NEJMp1703370](http://dx.doi.org/10.1056/NEJMp1703370)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28877013 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 4. 4.Lovell T. German hospitals to get €3 billion funding boost for digitalisation: The Hospital Future Act aims to modernise the country’s hospital system. In: Healthcare IT News, 2020. 5. 5.Infoway CH. A new day in healhcare is coming: annual report 2018-2019, 2019. 6. 6.Burton-Jones A, Akhlaghpour S, Ayre S. Changing the conversation on evaluating digital transformation in healthcare: insights from an institutional analysis. Information and Organization 2019;100255. 7. 7.Bowden T, Coiera E. Comparing New Zealand's 'Middle Out' health information technology strategy with other OECD nations. Int J Med Inform 2013;82:e87–95.[doi:10.1016/j.ijmedinf.2012.12.002](http://dx.doi.org/10.1016/j.ijmedinf.2012.12.002)pmid:http://www.ncbi.nlm.nih.gov/pubmed/23287413 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.ijmedinf.2012.12.002&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=23287413&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 8. 8.Review of the final benefits statement for programmes previously managed under the National programme for it in the NHS. Available: [https://www.nao.org.uk/wp-content/uploads/2013/06/10171-001\_NPfiT\_Review.pdf](https://www.nao.org.uk/wp-content/uploads/2013/06/10171-001_NPfiT_Review.pdf) [Accessed 16 May 2021]. 9. 9.The challenges in implementing digital change. Available: [https://www.nao.org.uk/wp-content/uploads/2021/07/The-challenges-in-implementing-digital-change.pdf](https://www.nao.org.uk/wp-content/uploads/2021/07/The-challenges-in-implementing-digital-change.pdf) [Accessed 14 Aug 2021]. 10. 10.Coiera EW. Lessons from the NHS National Programme for IT. Med J Aust 2007;186:3–4.[doi:10.5694/j.1326-5377.2007.tb00774.x](http://dx.doi.org/10.5694/j.1326-5377.2007.tb00774.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/17229021 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=17229021&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000244462400001&link_type=ISI) 11. 11.Greenhalgh T, Russell J, Ashcroft RE, et al. Why national eHealth programs need dead philosophers: Wittgensteinian reflections on policymakers' reluctance to learn from history. Milbank Q 2011;89:533–63.[doi:10.1111/j.1468-0009.2011.00642.x](http://dx.doi.org/10.1111/j.1468-0009.2011.00642.x)pmid:http://www.ncbi.nlm.nih.gov/pubmed/22188347 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1111/j.1468-0009.2011.00642.x&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=22188347&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000298357400001&link_type=ISI) 12. 12.NHS England. Global digital exemplars. Available: [https://www.england.nhs.uk/digitaltechnology/connecteddigitalsystems/exemplars/](https://www.england.nhs.uk/digitaltechnology/connecteddigitalsystems/exemplars/) [Accessed 16 May 2021]. 13. 13.Wachter RM. Making it work: harnessing the power of health information technology to improve care in England, report of the National Advisory group on health information technology in England, 2016. 14. 14.HIMSS continuity of care maturity model (CcmM). Available: [https://www.himssanalytics.org/ccmm](https://www.himssanalytics.org/ccmm) [Accessed 16 May 2021]. 15. 15.Williams R, Sheikh A, Franklin BD, et al. Using blueprints to promote interorganizational knowledge transfer in digital health initiatives-a qualitative exploration of a national change program in English hospitals. J Am Med Inform Assoc 2021;28:1431–9.[doi:10.1093/jamia/ocab020](http://dx.doi.org/10.1093/jamia/ocab020)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33706378 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 16. 16.Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods 2006;5:80–92.[doi:10.1177/160940690600500107](http://dx.doi.org/10.1177/160940690600500107) 17. 17.Cresswell K, Sheikh A, Dean Franklin B, et al. Formative independent evaluation of a digital change programme in the English National health service: study protocol for a longitudinal qualitative study. BMJ Open 2020;10:e041275. [doi:10.1136/bmjopen-2020-041275](http://dx.doi.org/10.1136/bmjopen-2020-041275)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33033100 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMzoiMTAvMTAvZTA0MTI3NSI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI4LzEvZTEwMDQyOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 18. 18.Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19:349–57.[doi:10.1093/intqhc/mzm042](http://dx.doi.org/10.1093/intqhc/mzm042)pmid:http://www.ncbi.nlm.nih.gov/pubmed/17872937 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1093/intqhc/mzm042&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=17872937&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000251963800004&link_type=ISI) 19. 19.Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 2009;6:e1000126. [doi:10.1371/journal.pmed.1000126](http://dx.doi.org/10.1371/journal.pmed.1000126)pmid:http://www.ncbi.nlm.nih.gov/pubmed/19688038 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1371/journal.pmed.1000126&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=19688038&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 20. 20.Cresswell K, Williams R, Sheikh A. Developing and applying a formative evaluation framework for health information technology implementations: qualitative investigation. J Med Internet Res 2020;22:e15068. [doi:10.2196/15068](http://dx.doi.org/10.2196/15068)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32519968 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 21. 22.Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010;19 Suppl 3:i68–74.[doi:10.1136/qshc.2010.042085](http://dx.doi.org/10.1136/qshc.2010.042085)pmid:http://www.ncbi.nlm.nih.gov/pubmed/20959322 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 22. 28.Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017;19:e367. [doi:10.2196/jmir.8775](http://dx.doi.org/10.2196/jmir.8775)pmid:http://www.ncbi.nlm.nih.gov/pubmed/29092808 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=doi:10.2196/jmir.8775&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=29092808&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 23. 29.Shaw J, Shaw S, Wherton J, et al. Studying scale-up and spread as social practice: theoretical introduction and empirical case study. J Med Internet Res 2017;19:e244. [doi:10.2196/jmir.7482](http://dx.doi.org/10.2196/jmir.7482)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28687532 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.2196/jmir.7482&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=28687532&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 24. 30.Sittig DF, Ash JS, Singh H. The safer guides: empowering organizations to improve the safety and effectiveness of electronic health records. Am J Manag Care 2014;20:418–23.pmid:http://www.ncbi.nlm.nih.gov/pubmed/25181570 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=25181570&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000339146800012&link_type=ISI) 25. 31.Greenhalgh T, Stramer K, Bratan T, et al. Introduction of shared electronic records: multi-site case study using diffusion of innovation theory. BMJ 2008;337:.[doi:10.1136/bmj.a1786](http://dx.doi.org/10.1136/bmj.a1786)pmid:http://www.ncbi.nlm.nih.gov/pubmed/18948344 26. 32.Doebbeling BN, Chou AF, Tierney WM. Priorities and strategies for the implementation of integrated informatics and communications technology to improve evidence-based practice. J Gen Intern Med 2006;21 Suppl 2:S50–7.[doi:10.1007/s11606-006-0275-9](http://dx.doi.org/10.1007/s11606-006-0275-9)pmid:http://www.ncbi.nlm.nih.gov/pubmed/16637961 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 27. 33.Beuscart-Zéphir MC, Anceaux F, Crinquette V, et al. Integrating users’ activity modeling in the design and assessment of hospital electronic patient records: the example of anesthesia. Int J Med Inform 2001;64:157–71.[doi:10.1016/S1386-5056(01)00210-6](http://dx.doi.org/10.1016/S1386-5056(01)00210-6) [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/S1386-5056(01)00210-6&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=11734383&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000172890600008&link_type=ISI) 28. 34.Keshavjee K, Bosomworth J, Copen J, et al. Best practices in EMR implementation: a systematic review. AMIA Annu Symp Proc 2006:982.pmid:http://www.ncbi.nlm.nih.gov/pubmed/17238601 29. 35.Rose AF, Schnipper JL, Park ER, et al. Using qualitative studies to improve the usability of an EMR. J Biomed Inform 2005;38:51–60.[doi:10.1016/j.jbi.2004.11.006](http://dx.doi.org/10.1016/j.jbi.2004.11.006)pmid:http://www.ncbi.nlm.nih.gov/pubmed/15694885 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.jbi.2004.11.006&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=15694885&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) [Web of Science](http://informatics.bmj.com/lookup/external-ref?access_num=000227197700006&link_type=ISI) 30. 36.Granlien MF, Hertzum M, Gudmundsen J. The gap between actual and mandated use of an electronic medication record three years after deployment. Stud Health Technol Inform 2008;136:419–24.pmid:http://www.ncbi.nlm.nih.gov/pubmed/18487767 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=18487767&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 31. 37.Bates DW, Ebell M, Gotlieb E, et al. A proposal for electronic medical records in U.S. primary care. J Am Med Inform Assoc 2003;10:1–10.[doi:10.1197/jamia.M1097](http://dx.doi.org/10.1197/jamia.M1097)pmid:http://www.ncbi.nlm.nih.gov/pubmed/12509352 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1197/jamia.M1097&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=12509352&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 32. 38.Aarts J, Doorewaard H, Berg M. Understanding implementation: the case of a computerized physician order entry system in a large Dutch University medical center. J Am Med Inform Assoc 2004;11:207–16.[doi:10.1197/jamia.M1372](http://dx.doi.org/10.1197/jamia.M1372)pmid:http://www.ncbi.nlm.nih.gov/pubmed/14764612 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1197/jamia.M1372&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=14764612&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 33. 39.Jaspers MWM, Peute LWP, Lauteslager A, et al. Pre-Post evaluation of physicians' satisfaction with a redesigned electronic medical record system. Stud Health Technol Inform 2008;136:303–8.pmid:http://www.ncbi.nlm.nih.gov/pubmed/18487748 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=18487748&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 34. 40.Singh H, Sittig DF. Measuring and improving patient safety through health information technology: the health it safety framework. BMJ Qual Saf 2016;25:226–32.[doi:10.1136/bmjqs-2015-004486](http://dx.doi.org/10.1136/bmjqs-2015-004486)pmid:http://www.ncbi.nlm.nih.gov/pubmed/26369894 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzQvMjI2IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpoY2kvMjgvMS9lMTAwNDI5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 35. 41.Clemmer TP. Computers in the ICU: where we started and where we are now. J Critical Care 2004;(4):201-207. Pendergast DK, Buchda VL. Charting the course. A quality journey. Nurs Admin Q 2003;27:330–5. 36. 42.Hendy J, Reeves BC, Fulop N, et al. Challenges to implementing the National programme for information technology (NPfIT): a qualitative study. BMJ 2005;331:331–6.[doi:10.1136/bmj.331.7512.331](http://dx.doi.org/10.1136/bmj.331.7512.331)pmid:http://www.ncbi.nlm.nih.gov/pubmed/16081447 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzEvNzUxMi8zMzEiO3M6NDoiYXRvbSI7czoyNToiL2JtamhjaS8yOC8xL2UxMDA0MjkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 37. 43.Morrison C, Jones M, Blackwell A, et al. Electronic patient record use during ward rounds: a qualitative study of interaction between medical staff. Crit Care 2008;12:R148.[doi:10.1186/cc7134](http://dx.doi.org/10.1186/cc7134)pmid:http://www.ncbi.nlm.nih.gov/pubmed/19025662 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1186/cc7134&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=19025662&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 38. 44.McGowan JJ, Cusack CM, Poon EG. Formative evaluation: a critical component in EHR implementation. J Am Med Inform Assoc 2008;15:297–301.[doi:10.1197/jamia.M2584](http://dx.doi.org/10.1197/jamia.M2584)pmid:http://www.ncbi.nlm.nih.gov/pubmed/18308984 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1197/jamia.M2584&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=18308984&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 39. 45.Cresswell K, Sheikh A, Franklin BD, et al. Theoretical and methodological considerations in evaluating large-scale health information technology change programmes. BMC Health Serv Res 2020;20:477. [doi:10.1186/s12913-020-05355-7](http://dx.doi.org/10.1186/s12913-020-05355-7)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32460830 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 40. 46.Toynbee P. NHS: the Blair years. BMJ 2007;334:1030–1.[doi:10.1136/bmj.39210.492188.AD](http://dx.doi.org/10.1136/bmj.39210.492188.AD)pmid:http://www.ncbi.nlm.nih.gov/pubmed/17510148 [FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMzQvNzYwMi8xMDMwIjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpoY2kvMjgvMS9lMTAwNDI5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 41. 47.Abraham C, Nishihara E, Akiyama M. Transforming healthcare with information technology in Japan: a review of policy, people, and progress. Int J Med Inform 2011;80:157–70.[doi:10.1016/j.ijmedinf.2011.01.002](http://dx.doi.org/10.1016/j.ijmedinf.2011.01.002)pmid:http://www.ncbi.nlm.nih.gov/pubmed/21292546 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=21292546&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 42. 48.Weick KE. Educational organizations as loosely coupled systems. Adm Sci Q 1976;21:1–9.[doi:10.2307/2391875](http://dx.doi.org/10.2307/2391875) [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.2307/2391875&link_type=DOI) 43. 49.Estonia’s digital transformation. Available: [https://library.oapen.org/bitstream/handle/20.500.12657/23594/9780198843719.pdf?sequence=1#page=158](https://library.oapen.org/bitstream/handle/20.500.12657/23594/9780198843719.pdf?sequence=1#page=158) [Accessed 16 May 2021]. 44. 50.Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health records. N Engl J Med 2010;363:501–4.[doi:10.1056/NEJMp1006114](http://dx.doi.org/10.1056/NEJMp1006114)pmid:http://www.ncbi.nlm.nih.gov/pubmed/20647183 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1056/NEJMp1006114&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=20647183&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 45. 51.Adler-Milstein J, Jha AK. HITECH act drove large gains in hospital electronic health record adoption. Health Aff 2017;36:1416–22.[doi:10.1377/hlthaff.2016.1651](http://dx.doi.org/10.1377/hlthaff.2016.1651)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28784734 [Abstract/FREE Full Text](http://informatics.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjM2LzgvMTQxNiI7czo0OiJhdG9tIjtzOjI1OiIvYm1qaGNpLzI4LzEvZTEwMDQyOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 46. 52.Halamka JD, Tripathi M. The HITECH era in retrospect. N Engl J Med 2017;377:907–9.[doi:10.1056/NEJMp1709851](http://dx.doi.org/10.1056/NEJMp1709851)pmid:http://www.ncbi.nlm.nih.gov/pubmed/28877012 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 47. 53.Ingebrigtsen T, Georgiou A, Clay-Williams R, et al. The impact of clinical leadership on health information technology adoption: systematic review. Int J Med Inform 2014;83:393–405.[doi:10.1016/j.ijmedinf.2014.02.005](http://dx.doi.org/10.1016/j.ijmedinf.2014.02.005)pmid:http://www.ncbi.nlm.nih.gov/pubmed/24656180 [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1016/j.ijmedinf.2014.02.005&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 48. 54.Cresswell K, Sheikh A, Krasuska M, et al. Reconceptualising the digital maturity of health systems. Lancet Digit Health 2019;1:e200–1.[doi:10.1016/S2589-7500(19)30083-4](http://dx.doi.org/10.1016/S2589-7500(19)30083-4)pmid:http://www.ncbi.nlm.nih.gov/pubmed/33323267 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 49. 55.Achieving a digital NHS lessons for national policy from the acute sector. Available: [https://www.nuffieldtrust.org.uk/files/2019-05/digital-report-br1902-final.pdf](https://www.nuffieldtrust.org.uk/files/2019-05/digital-report-br1902-final.pdf) [Accessed 16 May 2021]. 50. 56.Krasuska M, Williams R, Sheikh A, et al. Technological capabilities to assess digital excellence in hospitals in high performing health care systems: international eDelphi exercise. J Med Internet Res 2020;22:e17022. [doi:10.2196/17022](http://dx.doi.org/10.2196/17022)pmid:http://www.ncbi.nlm.nih.gov/pubmed/32808938 [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 51. 57.HIMSS. STAGE 6 & 7 Achievement. Available: [https://www.himssanalytics.org/europe/stage-6-7-achievement](https://www.himssanalytics.org/europe/stage-6-7-achievement) [Accessed 16 Aug 2021]. 52. 58.Leon N, Schneider H, Daviaud E. Applying a framework for assessing the health system challenges to scaling up mHealth in South Africa. BMC Med Inform Decis Mak 2012;12:1–2.[doi:10.1186/1472-6947-12-123](http://dx.doi.org/10.1186/1472-6947-12-123) [CrossRef](http://informatics.bmj.com/lookup/external-ref?access_num=10.1186/1472-6947-12-1&link_type=DOI) [PubMed](http://informatics.bmj.com/lookup/external-ref?access_num=22217121&link_type=MED&atom=%2Fbmjhci%2F28%2F1%2Fe100429.atom) 53. 59.Digital Aspirants. Available: [https://www.nhsx.nhs.uk/key-tools-and-info/digital-aspirants/](https://www.nhsx.nhs.uk/key-tools-and-info/digital-aspirants/) [Accessed 16 May 2021]. 54. 60.Cresswell K, Williams R, Sheikh A. Bridging the growing digital divide between NHS England’s hospitals. J R Soc Med 2020;1:141076820974998. 55. 61.Nhs long term plan. Available: [https://www.longtermplan.nhs.uk/](https://www.longtermplan.nhs.uk/) [Accessed 16 Aug 2021].