Lessons learned from a comprehensive electronic patient record procurement process—implications for healthcare organisations

Introduction

Great Ormond Street Hospital for Children NHS Trust (GOSH/the Trust) and the associated UCL Institute of Child Health represent a partnership of a large, specialist Children’s Hospital in London with extensive research and academic activity. As a component of a larger piece of work on an overarching digital strategy, GOSH has undergone an Official Journal of the European Union (OJEU) procurement process, with competitive dialogue. The procurement was managed under two ‘Lots’. The scope of Lot1 was an enterprise electronic patient/health record (EPR/EHR) system to replace many of the current clinical systems in place across the Trust. The scope of Lot2 was a dedicated secondary use data repository, and research and analytics platform (Digital research environment (DRE)), replacing numerous individual research databases.

At the outset of the process we intended to use learnings from published evidence, but while numerous peer-reviewed papers regarding healthcare implementation were available,1 there was a paucity of practically useful publications regarding details of the process and learnings from other centres regarding EPR systems. The use of clinical scenario simulation and evaluation for usability has been described,2 as have the use of usability questionnaires,3 but no information was available regarding practical lessons from other centres, such as which elements of the evaluation process were most discriminatory and whether there were specific issues detected during the procurement process which would have been useful to have been aware of initially. Therefore, following our EPR procurement process, the aim was to present our experience, which we hope may be useful for other organisations undergoing similar procurement processes.

The objectives of the current study are to describe details of the procurement process used, review existing published literature providing data in this area, present data regarding the methodology used and present the major lessons learned from the process, all of which may benefit other organisations considering a major EPR procurement process.

Methods: details of procurement process

In the United Kingdom National Health Service (NHS), there are existing detailed general guidelines and regulations governing all procurements, and similar structures exist elsewhere, in order that the organisation can satisfy both its own governance arrangements and legal requirements. Each organisation will have its own agreed Standing Financial Instructions (SFIs) which detail the process to be followed for each procurement depending on the value of the procurement. In the case of GOSH this states that all contracts with a value of over £1 million require board approval.

To satisfy the legal requirements all procurements above £100,000 are tendered through OJEU guidelines (2015; https://www.ojeu.eu/whatistheojeu.aspx), which govern the procedural and legal aspects (figure 1). There are methods implemented within the regulations to allow for the large-scale purchase of items such as hardware or more innovative partnerships for bespoke developments of new software. This can either be done through a previously OJEU tendered framework (http://www.lpp.nhs.uk/categories/technology-consultancy/clinical-and-digital-information-systems/) or through a separate OJEU procurement. The framework is an agreement with a provider or number of providers following Public Contracts Regulations that enables buyers to place contracts by direct award or following a mini-competition without running full tendering exercises. Frameworks will carry their own terms and conditions which can be restrictive, however, the process reduces the cost and time of procurement.

Request permission

Figure 1

Schematic illustrating timescales and elements of the procurement process.

When GOSH evaluated its option, some of the newer frameworks were not available, or did not contain a comprehensive list of suppliers, and GOSH opted for a new procurement. Procurement of an EPR, from the production of the detailed specification, legal limitations and scoring criteria generally require specialist expertise not available in an NHS Trust, therefore services of an experienced EPR procurement supplier were sought. The use of such specialist procurement advisors adds value, since they will have been involved in recent procurements and know the marketplace and the potential suppliers, in addition to being able to provide content on which to base the procurement. This aims of reducing time and cost, allows the organisation to customise the content and focus their specialised needs (as a tertiary children’s hospital in this case). The advisors led the Trust through the procurement process, producing documentation, crafting supplier communication and participating in dialogue sessions. While LOT1 and LOT2 procurements followed a similar process, we focus here on the LOT1 EPR system procurement only.

Following procurement launch, prequalifying questionnaires were received from nine suppliers from which three fulfilled the essential criteria and were invited to submit initial tenders. The trust had internally developed an extensive list of specifications required from the EPR system (output-based specification (OBS)). A pre-determined minimum compliance level of 70% was required for supplier invitation to the next round (dialogue) in order to optimise supplier and Trust time. There were >5000 rows within the OBS, of which>3400 items were individually scored, each on a scale of 0–6: 0 representing no evidence of compliance, 4 representing adequate compliance and 6 representing compliance with additional beneficial features. For each item score, the supplier was requested to self-score and the Trust generated a moderated score based on discussion of the information provided (figure 1).

The authors of the current manuscript were involved in the procurement process at all stages, from conception and requirement gathering, through specification selection, evaluation, dialogue and final selection. In addition, the authors were members of an EPR programme board, which reviewed and discussed the process and outcomes.

Results: scoring

There were two suppliers that reached the final stage of dialogue. For these suppliers, around 90% of their responses demonstrated adequate compliance with the OBS requirements and had similar supplier self-scores and moderated scores. Therefore, around 3000 elements of the OBS provided assurance regarding EPR product functionality, but were not discriminatory between suppliers. Overall, the moderated scores were greater than the supplier self-scores in around 0.5% of cases and lower in 7%–12% of cases. There was a significant difference in supplier-moderated discordance and ‘overscoring’ between suppliers (242/3432 vs 407/3432 respectively; Z=−7.0, P<0.0001). There were 80 (approximately 2%) specifications in which the moderated and self-scores differed by at least two marks, these therefore representing areas requiring extensive clarification during dialogue and being potentially discriminatory. These areas specifically related to: care pathways, patient flow, operational surveillance, consent, pharmacy and laboratory medicine.

The OBS scores were based on documentation provided by the suppliers, with subsequent clarification during dialogue. The total cost of ownership also formed a significant part of the overall final evaluation score, in addition to OBS scores, functionality, usability, such that an overall final score was applied to suppliers. A predetermined cut-off was used to ensure that any successful supplier could fulfil the essential requirements of the organisation, and for suppliers meeting that score, they were ranked based on the final score taking account of all aspects as above. However, the organisation did not eliminate any supplier purely on cost, but rather total cost of ownership vs potential benefits was evaluated.

Site visits were performed by a mixed group of technical, operational and clinical staff of multiple hospitals using the systems. The suppliers were encouraged to identify sites (including international sites) which best demonstrated the potential of their systems applied to a hospital broadly similar to the organisation. These were not scored in the same way as the OBS requirements, but rather, the findings were translated into a negative risk score which was then attributed to the other scored criteria. Thus the site visits evaluated whether the systems were being used in practice to the extent that the OBS score represented and if not, how much risk this constituted to the overall benefits case. If the functionality stated in the OBS could not be demonstrated in clinical practice, the procurement process took the view that although the systems could be shown to theoretically perform the function to their full potential then other factors such as human behaviours (and in particular, ‘usability’) were a factor. Therefore, site visits were used to provide a risk score regarding the likely delivery of specific elements based on both how the systems were being used at other site organisations in addition to the planned workflows at GOSH. Dialogue sessions were undertaken to clarify specific elements of functionality or questions regarding system modules which had been highlighted suring the objective scoring. The dialogue sessions themselves were not individually scored, rather the process of clarification was used to allow modification of the scoring elements as appropriate.

Results: lessons learned

Throughout the process the senior EPR team undertook numerous meetings to discuss the ongoing procurement through the EPR board, including discussion of supplier scores, functionality and feedback, in order to make the procurement decision. Following the process, the team also discussed and reflected on ‘lessons learned’ from the procurement. These ‘lessons’ were not formally scored in terms of importance but were collated and are presented below, presented by general theme but not in specific ‘ranked order’ of importance. However, we have attempted to present what we feel are the priority areas initially within the appropriate sections. The elements include both positive and negative aspects, which were highlighted as part of the organisational reflection on the EPR procurement. Where appropriate, mitigation strategies are also described.

Discussion

The findings of this study describe a UK procurement process for a comprehensive clinical EPR system for a specialist clinical academic institution. The main lessons learned, based on reflection of the entire process by the senior EPR team, include the importance of detailed preparation of organisational requirements/specifications and organisational ‘readiness’. Early staff involvement was beneficial, resulting in ownership of the selected system by the organisation as a key achievement. In contrast, the scoring process used required significant resource commitment by both suppliers and organisation but despite being extensive in scope, provided relatively poor distinction between suppliers in itself, despite significant variation in supplier self-scoring: whether such ‘overscoring’ was intentional or incidental cannot be determined from this data. Other aspects of the process, such as demonstrations and site visits, provided subjectively superior evaluation of functional abilities, and we suggest that the specification requirements should best be regarded as a threshold- setting process.

The inclusion of ‘real-world’ usability testing was a significant and useful component of evaluating systems. This is similar to findings from a previous study in which novice users of a system were asked to identify potential usability issues for early recommendations for correction. These users had greater satisfaction with the system compared with experienced users of another system, who did not undergo usability testing prior to implementation, and who subsequently had relatively high levels of dissatisfaction.4 In the present study, usability testing provided both important immediate feedback from many types of potential system users but also the widespread inclusion of staff groups encouraged involvement and ownership with the process, generating enthusiasm from such staff and their peers, regarding the anticipated system once implemented.

The Trust felt that the process and opportunity to negotiate with the successful bidder under the Public Contracts Regulations 2015 Competitive Dialogue process would help to establish a solid partnership with the chosen supplier and ensure that development and implementation plans were developed jointly. This allowed a wide selection of suppliers to participate while facilitating early exclusion of inappropriate suppliers-based criteria such as financial status and technical inability to deliver requirements. In choosing the ‘dialogue method’, throughout the course of procurement, the requirements and implementation plan could be developed and refined with suppliers to ensure the solution was optimal for the Trust. However, such a process is time consuming, likely to take up to 12 months to complete even with a well-organised dialogue period and the necessary resources allocated from the Trust. For some organisations, alternative processes may be more appropriate. For example, in the UK a ‘framework’ route might be equally successful, especially with limited scope of requirements, since it is likely to cost less. While in general, framework agreements are a useful and effective approach for smaller purchasing decisions, these are usually not indicated for major procurements such as comprehensive EPR systems.5 The reasons for this include the fact that not all EPR providers may be available through the framework process, thus reducing the scope of potential vendors, and frameworks are based on preselected criteria, which may be beneficial in specific circumstances but when considering enterprise EPR solutions, the scale and complexity of requirements may be incompatible with a framework approach. The previous UK National Programme for IT could be argued as an example of how the framework-type approach may not work well for complex programmes across varied NHS organisations.

In addition, it is important to emphasise that the aim of the process is to procure the best affordable system, and therefore the procurement process required, and specifications are constrained by the overall budget, including long-term total cost of ownership (TCO). EPR systems require significant ongoing workflow optimisation, upgrade management and other maintenance and it is important that TCO, including Trust resource input, is evaluated rather than simply ‘purchase cost’.

Compared with evaluations of implemented EPR systems, there are a paucity of published data examining EPR procurements and implementations, although the importance of well-defined strategies for planning and procurement are established.6 The concept of important preimplementation phases for EPR projects, including scoping, methods of assessment, strategic planning and method of choosing a system have been well described,7 but there is little evidence supporting specific scoring and comparative system evaluation aspects, supporting our approach of using established core principles with superimposed specific evaluation methods. Recently, a detailed framework for planning EPR usability evaluation has been described based on objectives, attributes and measures, which may provide the structure for determining usability elements in future selections.8

Recently, across the NNHS, procurement has been identified as a potential area for efficiency savings and a review of more than 70 procurement studies reported that efficient procurement processes were associated with good relationships with suppliers, development of skills to make good decisions and best use of technology. However, published evidence to support specific procurement approaches was either absent or poor quality, most being simple descriptive studies. It is therefore recognised that there is a need for research to assess optimal procurement approaches in many settings, with evaluation of practices, such as the present study.9 An NIHR study reviewed peer-reviewed published literature regarding buying behaviour, contracting economics, organisational relationships and supply chain management in the NHS and concluded that this was a complex process with a wide variety of practices and with significant impact of political aspects to most major procurement decisions.10 Finally, in addition to immediate requirements, the EPR platform should support future developments. Prediction of technological futures is fraught with difficulty, however, it is likely that cross-platform approaches, such as SMART on FHIR-compatibility, will be important for future integration of apps into the EPR system, and this approach has successfully been described.11

In conclusion, we have presented lessons learned from a comprehensive EPR system procurement for a specialist UK hospital, in addition to evaluation of the existing peer-reviewed literature in the field, which should be beneficial to other organisations undertaking similar projects. The main finding is that an EPR procurement process is a complex, pan-organisation issue, and evaluation should be optimised for the specific organisation, including multiple assessment methods from core specification through usability. Given the increasing implementation of comprehensive EPR systems there is paucity of published evidence regarding optimal scoring and evaluation methodologies and outcomes, and further work should focus on these areas. In addition, it would be useful to have publicly available, core documents regarding elements which are likely to be common across organisations, to be included in elements of the procurement and evaluation process, such as site visit and demonstration checklists and templates, which could be developed by the community.