Article Text
PDF
Abstract
Background This study describes learning from procurement of a comprehensive electronic patient record (EPR/electronic health record (EHR)), system for a specialist clinical academic institution.
Method Retrospective review of procurement process in addition to evaluation of peer-reviewed literature in the field.
Results Main lessons learned include the importance of detailed preparation of organisational requirements/specifications and organisational ‘readiness’. Early staff involvement, resulting in ownership of the selected system by the organisation was a key achievement. The scoring process used required significant resource commitment but, despite being extensive in scope, provided relatively poor distinction between suppliers, despite significant variation in supplier self-scoring. Other elements, such as demonstrations and site visits, provided superior evaluation of functional abilities, and specification requirements should be regarded as threshold evaluation.
Conclusion While principles should be followed, the procurement process must be modified to meet the needs of the specific organisation, in terms of its clinical activities, digital maturity, existing infrastructure and budget.
- electronic patient record
- procurement
- lessons
- information technology
- review
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0
Statistics from Altmetric.com
Summary
What is already known?
Standard procurement processes exist for EPR systems including numerous elements.
The discriminatory utility of components is poorly described
The process is resource heavy and costly and learning dissemination would be useful.
What does this paper add?
During EPR procurement, use of raw objective functional assessment provides poor discriminary value.
Demonstrations and site visits represent important discriminatory elements.
Evaluation processes should be modified the meet the specific needs of the organisation.
Introduction
Great Ormond Street Hospital for Children NHS Trust (GOSH/the Trust) and the associated UCL Institute of Child Health represent a partnership of a large, specialist Children’s Hospital in London with extensive research and academic activity. As a component of a larger piece of work on an overarching digital strategy, GOSH has undergone an Official Journal of the European Union (OJEU) procurement process, with competitive dialogue. The procurement was managed under two ‘Lots’. The scope of Lot1 was an enterprise electronic patient/health record (EPR/EHR) system to replace many of the current clinical systems in place across the Trust. The scope of Lot2 was a dedicated secondary use data repository, and research and analytics platform (Digital research environment (DRE)), replacing numerous individual research databases.
At the outset of the process we intended to use learnings from published evidence, but while numerous peer-reviewed papers regarding healthcare implementation were available,1 there was a paucity of practically useful publications regarding details of the process and learnings from other centres regarding EPR systems. The use of clinical scenario simulation and evaluation for usability has been described,2 as have the use of usability questionnaires,3 but no information was available regarding practical lessons from other centres, such as which elements of the evaluation process were most discriminatory and whether there were specific issues detected during the procurement process which would have been useful to have been aware of initially. Therefore, following our EPR procurement process, the aim was to present our experience, which we hope may be useful for other organisations undergoing similar procurement processes.
The objectives of the current study are to describe details of the procurement process used, review existing published literature providing data in this area, present data regarding the methodology used and present the major lessons learned from the process, all of which may benefit other organisations considering a major EPR procurement process.
Methods: details of procurement process
In the United Kingdom National Health Service (NHS), there are existing detailed general guidelines and regulations governing all procurements, and similar structures exist elsewhere, in order that the organisation can satisfy both its own governance arrangements and legal requirements. Each organisation will have its own agreed Standing Financial Instructions (SFIs) which detail the process to be followed for each procurement depending on the value of the procurement. In the case of GOSH this states that all contracts with a value of over £1 million require board approval.
To satisfy the legal requirements all procurements above £100,000 are tendered through OJEU guidelines (2015; https://www.ojeu.eu/whatistheojeu.aspx), which govern the procedural and legal aspects (figure 1). There are methods implemented within the regulations to allow for the large-scale purchase of items such as hardware or more innovative partnerships for bespoke developments of new software. This can either be done through a previously OJEU tendered framework (http://www.lpp.nhs.uk/categories/technology-consultancy/clinical-and-digital-information-systems/) or through a separate OJEU procurement. The framework is an agreement with a provider or number of providers following Public Contracts Regulations that enables buyers to place contracts by direct award or following a mini-competition without running full tendering exercises. Frameworks will carry their own terms and conditions which can be restrictive, however, the process reduces the cost and time of procurement.
Schematic illustrating timescales and elements of the procurement process.
When GOSH evaluated its option, some of the newer frameworks were not available, or did not contain a comprehensive list of suppliers, and GOSH opted for a new procurement. Procurement of an EPR, from the production of the detailed specification, legal limitations and scoring criteria generally require specialist expertise not available in an NHS Trust, therefore services of an experienced EPR procurement supplier were sought. The use of such specialist procurement advisors adds value, since they will have been involved in recent procurements and know the marketplace and the potential suppliers, in addition to being able to provide content on which to base the procurement. This aims of reducing time and cost, allows the organisation to customise the content and focus their specialised needs (as a tertiary children’s hospital in this case). The advisors led the Trust through the procurement process, producing documentation, crafting supplier communication and participating in dialogue sessions. While LOT1 and LOT2 procurements followed a similar process, we focus here on the LOT1 EPR system procurement only.
Following procurement launch, prequalifying questionnaires were received from nine suppliers from which three fulfilled the essential criteria and were invited to submit initial tenders. The trust had internally developed an extensive list of specifications required from the EPR system (output-based specification (OBS)). A pre-determined minimum compliance level of 70% was required for supplier invitation to the next round (dialogue) in order to optimise supplier and Trust time. There were >5000 rows within the OBS, of which>3400 items were individually scored, each on a scale of 0–6: 0 representing no evidence of compliance, 4 representing adequate compliance and 6 representing compliance with additional beneficial features. For each item score, the supplier was requested to self-score and the Trust generated a moderated score based on discussion of the information provided (figure 1).
The authors of the current manuscript were involved in the procurement process at all stages, from conception and requirement gathering, through specification selection, evaluation, dialogue and final selection. In addition, the authors were members of an EPR programme board, which reviewed and discussed the process and outcomes.
Results: scoring
There were two suppliers that reached the final stage of dialogue. For these suppliers, around 90% of their responses demonstrated adequate compliance with the OBS requirements and had similar supplier self-scores and moderated scores. Therefore, around 3000 elements of the OBS provided assurance regarding EPR product functionality, but were not discriminatory between suppliers. Overall, the moderated scores were greater than the supplier self-scores in around 0.5% of cases and lower in 7%–12% of cases. There was a significant difference in supplier-moderated discordance and ‘overscoring’ between suppliers (242/3432 vs 407/3432 respectively; Z=−7.0, P<0.0001). There were 80 (approximately 2%) specifications in which the moderated and self-scores differed by at least two marks, these therefore representing areas requiring extensive clarification during dialogue and being potentially discriminatory. These areas specifically related to: care pathways, patient flow, operational surveillance, consent, pharmacy and laboratory medicine.
The OBS scores were based on documentation provided by the suppliers, with subsequent clarification during dialogue. The total cost of ownership also formed a significant part of the overall final evaluation score, in addition to OBS scores, functionality, usability, such that an overall final score was applied to suppliers. A predetermined cut-off was used to ensure that any successful supplier could fulfil the essential requirements of the organisation, and for suppliers meeting that score, they were ranked based on the final score taking account of all aspects as above. However, the organisation did not eliminate any supplier purely on cost, but rather total cost of ownership vs potential benefits was evaluated.
Site visits were performed by a mixed group of technical, operational and clinical staff of multiple hospitals using the systems. The suppliers were encouraged to identify sites (including international sites) which best demonstrated the potential of their systems applied to a hospital broadly similar to the organisation. These were not scored in the same way as the OBS requirements, but rather, the findings were translated into a negative risk score which was then attributed to the other scored criteria. Thus the site visits evaluated whether the systems were being used in practice to the extent that the OBS score represented and if not, how much risk this constituted to the overall benefits case. If the functionality stated in the OBS could not be demonstrated in clinical practice, the procurement process took the view that although the systems could be shown to theoretically perform the function to their full potential then other factors such as human behaviours (and in particular, ‘usability’) were a factor. Therefore, site visits were used to provide a risk score regarding the likely delivery of specific elements based on both how the systems were being used at other site organisations in addition to the planned workflows at GOSH. Dialogue sessions were undertaken to clarify specific elements of functionality or questions regarding system modules which had been highlighted suring the objective scoring. The dialogue sessions themselves were not individually scored, rather the process of clarification was used to allow modification of the scoring elements as appropriate.
Results: lessons learned
Throughout the process the senior EPR team undertook numerous meetings to discuss the ongoing procurement through the EPR board, including discussion of supplier scores, functionality and feedback, in order to make the procurement decision. Following the process, the team also discussed and reflected on ‘lessons learned’ from the procurement. These ‘lessons’ were not formally scored in terms of importance but were collated and are presented below, presented by general theme but not in specific ‘ranked order’ of importance. However, we have attempted to present what we feel are the priority areas initially within the appropriate sections. The elements include both positive and negative aspects, which were highlighted as part of the organisational reflection on the EPR procurement. Where appropriate, mitigation strategies are also described.
Overall organisation of process
Development of a stakeholder map of key staff groups prior to commencement of the procurement process, along with securing protected time for these staff to be actively involved throughout resulted in excellent levels of clinical engagement with the process (>200 clinical and operational staff from across all areas of the hospital were involved to avoid EPR being envisaged as an ‘IT’ project). Poor staff inclusion has been previously reported as a factor associated with unsuccessful implementations.1
Clear articulation of the overall vision by the executive team, in terms of high-level outcomes describing new ways of working supported by the EPR system and associated benefits, resulted in early recognition by staff of the scale and importance of the project. At the procurement stage it is important that the entire organisation understands the expected outcomes and process.
Involvement of the procurement team in wider market testing, including liaising with staff at similar organisations who have completed a similar process resulted in awareness of likely practical areas of possible difficulty or delay. These ‘shop floor’ difficulties are hard to predict other than through experience.
Procurement process and scoring
The initial procurement requirements and vision should be modified based on findings from scoping and other organisations such that development of functional and non-functional requirements is focused on support for key outcomes for the specific organisation.
Time should be spent determining requirements based on the specific clinical activity of the organisation. For example, workshops and focus groups were completed to determine the OBS requirements across all areas.
Early identification of likely system biases allows their management and mitigation. For example, some members of the procurement team may have had experience of the specific systems being evaluated (either positive or negative). While previous experience can be useful, there are many reasons why a system may appear to work better or worse at different sites (such as underlying infrastructure, workflows, scope of procurement etc), and potential biases should be identified to ensure objective evaluation.
While detailed specifications provided assurance regarding fitness for purpose of the system, the data presented here demonstrate that most scores did not distinguish between potential suppliers, while representing a significant investment of time and resource from Trust and suppliers. In retrospect we suspect that a significantly reduced process, perhaps requiring checkboxes for compliance for many areas, may have resulted in a similar outcome with less resource.
Since all suppliers of major EPR systems can provide high-level similar functionality, the OBS may not clearly separate suppliers, rather objectively determining a ‘threshold’ level of functionality.
A reduced OBS scoring process could be related to provision of detailed example workflow objectives for system elements. This would represent a reduced burden on both organisation and suppliers, but with a focus on specific issues which are likely to be discriminatory.
Despite most scores being consistent, there was significant variation in supplier self-scoring against OBS requirements, hence organisational scoring is also required to reduce bias from supplier ‘overscoring’.
Despite using a scoring approach in which a score of 6 could be used to allow demonstration of exceptional system capability, overall aggregate scores poorly demonstrated such areas due to most scores meeting requirements, reducing the impact of this approach.
While being associated with shortcomings, the OBS exercise allowed the selection team to foster deep understanding of what was needed from such an enterprise level platform and helped create an active multiprofessional team from clinical, operational and support backgrounds.
Additional evaluation elements
The dialogue and demonstration process provided important clarification, since there may be several ways to fulfil a given specification. In written terms the specification may appear non-discriminatory but practical scored demonstrations allows clarification regarding whether the approach would fit within the organisation and workflows.
Increased focus on elements in addition to specification scoring, such as scripted demonstrations, usability assessments and site visits, provide highly valuable insights regarding how systems may fit into specific organisational teams and cultures, and could provide enhanced discrimination.
Practical site visits were of major importance for evaluating the real-world applicability of systems and risk scoring from the site visits was invaluable in providing real world value, and thus an indication ultimately of the total cost of ownership (TCO). There are several ways that a ‘specification’ may be met, which may markedly vary in compatibility with specific organisational activity. To this end, choice of sites to visit is of importance and should include ‘real world’ examples of similar organisations, in terms of size and type of clinical services, in addition to supplier nominated sites.
Procedural elements
Supplier demonstrations should be scripted based on real hospital workflows/patient scenarios, to provide evaluators a true view how a system may be used in practice. During such, suppliers should not be allowed to avoid demonstrating areas of the system that they may feel are weak. Highly polished demonstrations of pre-planned elements may not reflect future clinical usage.
Usability sessions as part of the scored demonstration element provides further insight into the practicalities of Trust staff using the system that has been demonstrated: this also provides an indication of ease of adoption. As many ‘user types’ as possible should evaluate the system, not only frontline clinical staff.
The use of procurement advisors was valuable in terms of providing insight into the marketplace, suppliers and provide balance to the process, but advisors should concentrate on technical procurement advice and should not become involved in system demonstrations, site visits or other ‘softer’ elements of the process since they may have unconscious bias from previous procurements and may not understand specifics of the organisation.
Technical and infrastructure requirements must be fully evaluated and included in the procurement process internally, even if not part of EPR supplier procurement itself. Evaluation of trust IT infrastructure enables early awareness of the ‘readiness’ of the organisation for EPR implementation. For example, it was recognised early that the networking/wireless capabilities of the organisation were inadequate for future demand and the upgrade process was planned well ahead of EPR implementation.
Engaging other stakeholders into the evaluation process (particularly patient representatives) provided an additional perspective on systems and is important for including functionality such as a patient portal. We found that patient/family involvement in the entire process ensures focus remains on a usable system that will deliver benefits to patients.
Discussion
The findings of this study describe a UK procurement process for a comprehensive clinical EPR system for a specialist clinical academic institution. The main lessons learned, based on reflection of the entire process by the senior EPR team, include the importance of detailed preparation of organisational requirements/specifications and organisational ‘readiness’. Early staff involvement was beneficial, resulting in ownership of the selected system by the organisation as a key achievement. In contrast, the scoring process used required significant resource commitment by both suppliers and organisation but despite being extensive in scope, provided relatively poor distinction between suppliers in itself, despite significant variation in supplier self-scoring: whether such ‘overscoring’ was intentional or incidental cannot be determined from this data. Other aspects of the process, such as demonstrations and site visits, provided subjectively superior evaluation of functional abilities, and we suggest that the specification requirements should best be regarded as a threshold- setting process.
The inclusion of ‘real-world’ usability testing was a significant and useful component of evaluating systems. This is similar to findings from a previous study in which novice users of a system were asked to identify potential usability issues for early recommendations for correction. These users had greater satisfaction with the system compared with experienced users of another system, who did not undergo usability testing prior to implementation, and who subsequently had relatively high levels of dissatisfaction.4 In the present study, usability testing provided both important immediate feedback from many types of potential system users but also the widespread inclusion of staff groups encouraged involvement and ownership with the process, generating enthusiasm from such staff and their peers, regarding the anticipated system once implemented.
The Trust felt that the process and opportunity to negotiate with the successful bidder under the Public Contracts Regulations 2015 Competitive Dialogue process would help to establish a solid partnership with the chosen supplier and ensure that development and implementation plans were developed jointly. This allowed a wide selection of suppliers to participate while facilitating early exclusion of inappropriate suppliers-based criteria such as financial status and technical inability to deliver requirements. In choosing the ‘dialogue method’, throughout the course of procurement, the requirements and implementation plan could be developed and refined with suppliers to ensure the solution was optimal for the Trust. However, such a process is time consuming, likely to take up to 12 months to complete even with a well-organised dialogue period and the necessary resources allocated from the Trust. For some organisations, alternative processes may be more appropriate. For example, in the UK a ‘framework’ route might be equally successful, especially with limited scope of requirements, since it is likely to cost less. While in general, framework agreements are a useful and effective approach for smaller purchasing decisions, these are usually not indicated for major procurements such as comprehensive EPR systems.5 The reasons for this include the fact that not all EPR providers may be available through the framework process, thus reducing the scope of potential vendors, and frameworks are based on preselected criteria, which may be beneficial in specific circumstances but when considering enterprise EPR solutions, the scale and complexity of requirements may be incompatible with a framework approach. The previous UK National Programme for IT could be argued as an example of how the framework-type approach may not work well for complex programmes across varied NHS organisations.
In addition, it is important to emphasise that the aim of the process is to procure the best affordable system, and therefore the procurement process required, and specifications are constrained by the overall budget, including long-term total cost of ownership (TCO). EPR systems require significant ongoing workflow optimisation, upgrade management and other maintenance and it is important that TCO, including Trust resource input, is evaluated rather than simply ‘purchase cost’.
Compared with evaluations of implemented EPR systems, there are a paucity of published data examining EPR procurements and implementations, although the importance of well-defined strategies for planning and procurement are established.6 The concept of important preimplementation phases for EPR projects, including scoping, methods of assessment, strategic planning and method of choosing a system have been well described,7 but there is little evidence supporting specific scoring and comparative system evaluation aspects, supporting our approach of using established core principles with superimposed specific evaluation methods. Recently, a detailed framework for planning EPR usability evaluation has been described based on objectives, attributes and measures, which may provide the structure for determining usability elements in future selections.8
Recently, across the NNHS, procurement has been identified as a potential area for efficiency savings and a review of more than 70 procurement studies reported that efficient procurement processes were associated with good relationships with suppliers, development of skills to make good decisions and best use of technology. However, published evidence to support specific procurement approaches was either absent or poor quality, most being simple descriptive studies. It is therefore recognised that there is a need for research to assess optimal procurement approaches in many settings, with evaluation of practices, such as the present study.9 An NIHR study reviewed peer-reviewed published literature regarding buying behaviour, contracting economics, organisational relationships and supply chain management in the NHS and concluded that this was a complex process with a wide variety of practices and with significant impact of political aspects to most major procurement decisions.10 Finally, in addition to immediate requirements, the EPR platform should support future developments. Prediction of technological futures is fraught with difficulty, however, it is likely that cross-platform approaches, such as SMART on FHIR-compatibility, will be important for future integration of apps into the EPR system, and this approach has successfully been described.11
In conclusion, we have presented lessons learned from a comprehensive EPR system procurement for a specialist UK hospital, in addition to evaluation of the existing peer-reviewed literature in the field, which should be beneficial to other organisations undertaking similar projects. The main finding is that an EPR procurement process is a complex, pan-organisation issue, and evaluation should be optimised for the specific organisation, including multiple assessment methods from core specification through usability. Given the increasing implementation of comprehensive EPR systems there is paucity of published evidence regarding optimal scoring and evaluation methodologies and outcomes, and further work should focus on these areas. In addition, it would be useful to have publicly available, core documents regarding elements which are likely to be common across organisations, to be included in elements of the procurement and evaluation process, such as site visit and demonstration checklists and templates, which could be developed by the community.
Footnotes
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.