Research Article•

Connecting medical records: an evaluation of benefits and challenges for primary care practices

Abstract

Background Implementation of systems to support health information sharing has lagged other areas of healthcare IT, yet offers a strong possibility for benefit. Clinical acceptance is a key limiting factor in health IT adoption.

Objectives To assess the benefits and challenges experienced by clinicians using a custom-developed health information exchange system, and to show how perceptions of benefits and challenges influence perceptions of productivity and care-related outcomes.

Methods We used a mixed methods design with two phases. First, we conducted interviews with stakeholders who were familiar with the health information exchange system to inform the development of a measure of benefits and challenges of the use of this system. Second, using this measure, we conducted a survey of current and former users of the health information exchange system using a modified Dillman method. Data were analyzed using structural equation modelling, implemented in partial least squares.

Results 105 current and former users completed the survey. The results showed information quality, ease of completing tasks and clinical process improvement as key benefits that reduced workload and improved patient care. Challenges related to system reliability, quality of reports, and service quality increased workload and decreased impact on care, though the effect of the challenges was smaller than that of the benefits.

Conclusion Even very limited health information exchange capabilities can improve outcomes for primary care users. Improving perceptions of benefits may be even more important than removing challenges to use, though it is likely that a threshold of quality must be achieved for this to be true.

INTRODUCTION

Health information technologies (HIT) offer tremendous potential to improve the quality and cost effectiveness of healthcare. However, projects to implement HIT around the world have met with mixed results.1–4 Central to all the explanations for HIT implementation success and failure is clinical adoption.5,6 Various models have been examined including TAM,7 UTAUT8–10 and an integrated model11 based on both TAM and Cenfetelli’s12 dual factors of adoption. In a related research stream, Lau and his colleagues13,14 and van der Meijden and his collegues15 have advanced a benefit evaluation framework that also addresses adoption issues.

This study examines the clinician’s perceptions of HIT (a common strategy in adoption research), but focuses on their perspectives regarding benefits arising from HIT adoption. In this way, this study bridges the two main themes in IS research on HIT identified by Agarwal et al.5: adoption and evaluation. It examines how a health information exchange technology was understood and valued by primary care practitioners. This research demonstrates both the potential benefits from a provider perspective and the limiting factors that must be addressed in ongoing system development.

The specific goal of the research was to understand the benefits and challenges of ‘PhysicianConnect’ (a pseudonym). PhysicianConnect is a technology designed to share data about patients between hospitals and primary care physicians. Sharing of data through IT remains a particular challenge in e-health. A recent survey16 showed that while primary care physicians have excellent adoption of electronic medical records (EMR) (i.e. internal systems), only about half routinely share information with other providers in an electronic format, even in countries where EMR adoption rates are 98%.

Two characteristics of PhysicianConnect are of particular importance to note. First, only three types of reports are sent through PhysicianConnect: results of in-hospital laboratory tests, in-hospital diagnostic imaging and discharge summaries for patients. Non-hospital-based laboratory tests or diagnostic images are not included nor are detailed patient reports for admitted patients. Second, the flow of data is one way; no data from the primary care EMR is shared with the hospital information system.

PhysicianConnect was developed in a bottom-up fashion. It was initiated by a physician and a hospital CIO, who saw an opportunity to connect their systems. It was designed using very simple technologies as a custom add-on to existing systems. It did not attempt to identify all of the possible data that a physician might want from a hospital information system, nor did it attempt to include other organizations from which primary care physicians received data, such as private laboratories. Regardless, it demonstrated the potential to connect systems, even in a rudimentary way, in order to avoid processing paper documents. After about two years, the hospital sought additional funding to expand the program to all physicians in its catchment area and to do some ‘recalibrating and adjusting’ of the system based on experience in working with it [interview with PhysicianConnect technologist]. Funding was provided by a government agency with the aim of expanding it beyond the initial hospital to two other hospital groups that existed within the region that had been developing a similar interface. The project ultimately expanded in four phases so that by the time of this research, the hospitals had been using it for at least 2 years, with individual physicians having access for a few months to 2 or more years.

METHODS

The study was conducted in two phases, using a mixed-methods design.17 During the qualitative phase, semi-structured interviews were conducted with two primary care physicians who had adopted the technology, two of their administrative staffs and one hospital IT manager who supported PhysicianConnect. A group interview was also conducted with three staff members at the government agency that funded and supported the project. Respondents were asked to explain when/how they had become PhysicianConnect adopters and how they had conducted various tasks related to the receipt of reports before and after the implementation and to identify the various benefits and challenges that they perceived related to the use of PhysicianConnect. Interviews were tape recorded and transcribed. Data were coded to identify salient benefits and challenges.

In the quantitative phase of the study, the benefits and challenges identified in phase one were used to develop a survey that was sent to all registered physician users of PhysicianConnect. The questionnaire included 12 items regarding benefits and 13 items focused on the challenges, all measured on a five-point scale. In total, 295 surveys were sent using an adapted Dillman method.18 Figure 1 shows the steps in the survey administration and the responses received at each stage.

Figure 1
Request permissionrequest permission icon
Figure 1

Based on the results of the interviews and a review of prior literature, a model for assessing the benefits of PhysicianConnect was constructed (Figure 2). The dual factor model represents the benefits (enablers) and challenges (inhibitors) as second-order latent constructs.19 Benefits are composed of three first-order factors: information quality, process improvement and ease of completing tasks. Challenges are composed of four first-order factors: reliability, report quality, number of reports and service quality. Dual-factor models are built on the premise that inhibitors influence behaviour differently than enablers and must, therefore, be examined separately. While increasing or decreasing an enabler is expected to promote or inhibit more use, removing an inhibitor does not necessarily promote more usage but simply does not discourage it.12,20 This model was assessed in the second phase of the study. Four hypotheses were tested, linking the benefits and challenges to each of the clinical outcomes.

Figure 2
Request permissionrequest permission icon
Figure 2

The model was tested using SmartPLS,21 following the Wetzels et al.22 approach for second-order factors. Box 1 provides a more detailed explanation of the partial least squares (PLS) method. The analysis proceeded in three stages: assessment of first-order measurement model, second-order measurement model and structural model. The seven first-order factors, modelled as reflective constructs, were assessed by examining the loadings and cross loadings, and some items were removed to improve measurement quality. The three reflective second-order factors were assessed based on their loadings and cross loadings, internal consistency reliability and discriminant validity. The structural model was assessed using a bootstrapping method with 500 samples. An additional model was run with the second-order factors modelled as formative. The structural model results are similar. The main difference is that only one first-order factor was a significant predictor of the second-order factors, an outcome of the high degree of multi-collinearity between the first order factors.

Box 1

Structural equation modelling (SEM) using PLS – an explanation of the method

PLS is a SEM technique, first developed by Lohmöller (1989). Broadly speaking, SEM tools allow researchers to assess complex relationships between constructs such as attitudes (including direct and indirect relationships) and to simultaneously assess the relationships between constructs and the questionnaire items that are designed to measure them. This holistic assessment of a complete model fits with contemporary views of theory and measurement, where theoretical concepts are deemed to derive part of their meaning from the way they are assessed and by the theoretical model in which they are embedded. In contrast to covariance-based SEM (CB-SEM), the PLS method uses the principals of ordinary least squares (OLS) regression and principal components factor analysis to assess the structural and measurement models.

PLS produces factor loadings (interpreted the same as loadings in factor analysis) to aid in assessment of the measurement model, and standardized regression coefficients to aid in assessment of the structural (path) model. Unlike CB-SEM which uses the assessment of model fit to judge the quality of the models, PLS uses R2 to assess the explained variance in constructs as well as the above-noted statistics.

Because PLS estimates the model iteratively in small portions (thus the term ‘partial’), the sample size requirements are lower than CB-SEM. The sample size requirement is determined as the greater of:

  • ten times the number of independent constructs in the most complex regression in the structural model. In our case, the most complex regression involves 2 independent constructs (benefits & challenges) that would demand at least 20 cases.

  • ten times the number of items in the construct with the greatest number of measures. In our case, that is process improvements, with 5 items, demanding 50 cases.

Most researchers suggest sample sizes greater than 100 are better, as the model estimates can be unstable at very low sample sizes.

In general, the stages of PLS interpretation proceed as follows:

  1. Assess internal consistency reliability of constructs. The ICR computed from PLS is similar to Cronbach’s alpha and can be similarly interpreted – ICR > 0.70 is the common benchmark.

  2. Assess convergent and discriminant validity at the item level using the loadings and cross loadings. In general, look for loadings in excess of 0.70 and cross loadings that are significantly lower than the loadings.

  3. Assess convergent validity and discriminant validity at the construct level by looking at the AVE and and inter-item correlations. An AVE > 0.50 implies the construct shares at least 50% variance with its items and this is the common threshold. In practice, the square root of the AVE is reported for ease of comparison with the inter-construct correlations. Thus, the square root of the AVE should be greater than 0.70. For discriminant validity, the square root of the AVE should be higher than the inter-construct correlations, indicating that the construct shares more variance with its measures than it does with other constructs in the model.

  4. Assess structural model by examining the significance of the regression coefficients and the substantiveness of the explained variance in dependent constructs. Significance is computed using bootstrapping. Bootstrapping is the preferred method for assessing significance as it does not require conformance to the distributional assumptions of traditional ordinary least squares (OLS) regression and thus produces more robust estimates.

Lohmöller J.-B. Latent Variable Path Modeling with Partial Least Squares, Heidelberg, Germany: Physica, 1989.

RESULTS

Interview findings

Participants in the interviews identified both positive and negative aspects of PhysicianConnect as well as numerous outcomes of using it. Appendix 1 summarizes the positive and negative attributes of the system identified along with the key outcomes experienced. For each item, a representative quote from the interviews is shown, along with an explanation of where the item fits within the research model.

Survey results

Of 295 surveys, 26 were returned as undeliverable leaving the potential sample at 269. Of the 118 respondents (44% response rate), three responses had questionable data quality (greater than 20% missingness) and were dropped, leaving 115 responses. Ninety-nine were current users of PhysicianConnect. Six respondents received information via PhysicianConnect but had discontinued their usage, while ten had never used it. No other data were collected from these ten respondents. Our final sample, then, was 105 current and former users. The majority of responses came from users of the three most commonly used EMRs. Ninety-nine surveys were completed by physicians, followed by one nurse, one nurse practitioner and four administrators. They represented each of the three hospital systems who were providing data. They had been using the system for between one month and five years, with an average usage of 19 months. A comparison of early versus late responders showed no significant differences on the variables of interest.23

Appendix 2 shows descriptive statistics for the survey items. Overall, the responses suggested positive outcomes from the use of PhysicianConnect. All of the means were above the scale midpoint. The most positive effects were for speed of obtaining patient results, reduction in filing errors and administrative staff workload. Beliefs about the benefits of PhysicianConnect were also generally positive. The most commonly cited benefit from using PhysicianConnect was ease of access to information. Improved information sharing, greater flexibility in where work can be done, staff ease of response to patient inquires, improved organization of information, improved ability to search for patient information and better follow-up with patients were also widely reported as benefits. Improved legibility of information was reported by some, but not all respondents. Three benefits were not as widely reported (average was less than 3): ­ease of creating referrals, updating the cumulative patient profile (CPP) and creating alerts.

The respondents report fewer challenges from using PhysicianConnect than they did benefits, and fewer than expected based on the phase one interviews, though most reported having experienced or observed each challenge. Duplicate reports were most often reported as a significant problem. Two other challenges were reported frequently: lack of support from the EMR vendor and not being able to tell whether the data are flowing. Three challenges were reported as not being significant concerns. The majority of respondents indicated having had no problems with date formats from PhysicianConnect not being compatible with the EMR. Nearly 50% of users reported that they had not experienced or observed an issue with getting too many reports from PhysicianConnect, nor had they been unable to update their EMR because of PhysicianConnect.

Survey – measurement model

First-order factors

For the three first-order factors of benefits, all but two loadings were above 0.7. The loading for ‘improved ability to search for patient information’ was 0.68, which is very close to the 0.7 threshold, so this item was maintained. The loading for ‘improved legibility compared to paper’ was 0.52. While the item seems to reflect information quality, it clearly did not fit with the other items. One possibility is that most respondents were already receiving reports by fax and so were not seeing PhysicianConnect as a replacement for handwriting but rather as a replacement of fax. Regardless, the item was the most variable of all of the benefits items, and since it did not load strongly on the construct, it was dropped.

For the four first-order factors of challenges, the measurement model was mixed. The service quality and reliability constructs both had loadings above 0.7 for each item. The number of reports construct had one very high (0.87) and one very low loading (0.23). Since removing the latter would leave a construct with a single item, and since both the number of reports and report quality issues related to the functionality of the system, the construct was merged with the report quality issues. The construct reflecting report quality had no loadings over 0.7. Examination of the items and reviews of the interview data suggested one possible explanation for this outcome. Several of the items in the report quality item reflected issues with a single EMR product: ‘inability to upgrade’, ‘incompatible date formats’ and ‘reports received in a way that does not enable cut and paste’. Since they were not experienced universally by respondents, they were dropped from the analysis.

The items for impact on workload are all loaded > 0.7. The loadings for the care outcomes construct were weaker, with one loading over 0.7, one loading (errors compared to reports that were scanned) of 0.56 and three items with loadings between 0.65 and 0.7. The higher loading items were retained consistent with prior decisions and the lower one was dropped.

With these modifications, the measurement model was reassessed. All internal consistency reliabilities were above 0.70 and each item loaded more strongly on its own construct than on the related constructs, indicating adequate discriminant validity. To proceed to the next phase of the assessment, the factor scores were saved for the first-order dimensions.

Second-order factors

The second-order factors were also modelled as reflective, since they were expected to be highly correlated. The three items reflecting benefits all had loadings (Table 1) greater than 0.7. Two of the three items reflecting challenges had loadings greater than 0.7 with reliability loading at 0.65. Since reliability is an established dimension of the benefits evaluation framework, it was retained despite the lower loading.

Table 1
|

The cross loadings for the items were also examined showing that each item loaded more strongly on its intended construct than on any of the others. To assess discriminant validity at the construct level, the square root of the average variance extracted (AVE) was compared with the inter-construct correlations. In all cases, the analysis showed that there was more variance in common between the constructs and their measures than between constructs, supporting discriminant validity. Each construct also had an internal consistency reliability above 0.70 (Table 2).

Table 2
|

Survey – structural model

As shown in Figure 3, all four hypotheses were supported. The benefits of using PhysicianConnect significantly and positively influenced the workload and care outcomes experienced by physicians. Stronger perceptions of information quality, process improvement and ease of completing tasks resulted in a perception that nursing, administrative and even physician workload were improved. They also influenced perceptions that care was improved by reducing errors and by improving the speed of referrals and receipt of patient results.

Figure 3
Request permissionrequest permission icon
Figure 3

Challenges with using PhysicianConnect also significantly influenced the outcomes, negatively as expected. Greater challenges with reliability, report quality and service quality promoted perceptions of worsening workload and worsening care.

The analysis also showed that benefits had a stronger positive influence on the outcomes than challenges had a negative one (p<0.001). Improving (perception of) benefits had a stronger impact than reducing (the perception of) challenges. This differential impact is consistent with the dual-factor conceptualization of enablers and inhibitors.

DISCUSSION

Findings

This study suggests that benefits were realized from the use of PhysicianConnect, particularly in relation to quality of care and productivity. Even a very limited form of connectivity (one-way communication of test results in a simple text format) resulted in perceived net benefits to clinical productivity and quality of care. This finding supports the view that integration of health records is a source of clinical value, and further demonstrates that even limited integration can provide some benefits.

Some what surprisingly, these benefits were realized in spite of challenges in both system and service qualities. Viewed from the perspective of traditional IS evaluation models,24 this should not be successful. Chatterjee et al.25 report that system and service qualities were essential to the realization of benefits in the context of mobile health technologies. They contend that if clinicians cannot depend on a system to function reliably or do not receive adequate support when there are problems, they will reject the system. A conversation with one of the staffs who had been managing the PhysicianConnect program echoed this view. Yet, despite acknowledged problems with PhysicianConnect, clinicians in this study were very positive towards it. They did not report a lot of problems with it (the average scores in the survey on the challenges related to system reliability and service quality are not as high as one might have expected given the acknowledged challenges from phase one). More importantly, the impact of the challenges construct on productivity and workload was lower than the impact of benefits.

It appears as though, in this case, the strong benefits related to information quality outweighed the limitations in system and service quality, and resulted in clinical acceptance of the system and ultimately the desired benefits. It appears as though the system and service were ‘good enough’ for adopters to realize the benefits associated with the access to information. This reflects quadrant II in Cenfetelli’s12 dual-factors model, and the results support his prediction that this quadrant can yield positive results (for adoption in his case, but benefits as observed here).

This finding is important because it shows that a new technology can be successful in a healthcare environment even if it falls short of the ideal on some quality aspects. Providing a workable system that gives clinicians tangible benefits in terms of health outcomes may be sufficient to encourage initial adoption, and thus may create the opportunity for continuing technology development to improve the system and extend its use. Thus, while the benefits evaluation framework that has been used in much of the literature24 presents system quality and service quality as linear predictors of use and satisfaction, our results suggest that they may represent a threshold factor. Whether this is unique to the PhysicianConnect context or whether this finding would generalize more broadly is unknown, and requires further investigation.

Limitations

A key limitation is the cross-sectional design with a single source of data (user perceptions). Still, examining perceptual data does provide some basis to estimate the potential benefits, and other studies26 have focused more on the quantification of benefits from HIT. As Goh et al.27 note, further research is needed on the complex processes of clinical adoption and acceptance that result in these benefits. This study extends prior literature to demonstrate how beliefs about enablers and inhibitors associated with the system influence perceptions of the benefits.

CONCLUSION

This study demonstrates the potential effectiveness of health information exchange, even in a limited fashion, and contributes to our understanding of benefits evaluation by linking the benefits evaluation model14,24,28 to the dual factors model of IT acceptance.11,12 In doing so, this study of PhysicianConnect shows the potential for successful HIT implementation and benefits realization, even in the face of system limitations.

APPENDIX 1: ASSESSMENTS OF PHYSICIAN CONNECT IN PHASE 1 INTERVIEWS

|

APPENDIX 2: MEASUREMENT ITEMS

PhysicianConnect impact (net benefits)

How does having the PhysicianConnect interface for receiving patient results in your practice affect each of the following outcomes?

|

PhysicianConnect benefits

To what extent have you personally observed or experienced each of the following potential benefits from having the PhysicianConnect interface for receiving patient results in your practice?

|

PhysicianConnect challenges

To what extent have you personally observed or experienced each of the following potential challenges from having the PhysicianConnect interface for receiving patient results in your practice?

|

Overview

Article metrics
Altmetricopen-url
Dimensionsopen-url