Connecting medical records: an evaluation of benefits and challenges for primary care practices

Background Implementation of systems to support health information sharing has lagged other areas of healthcare IT, yet offers a strong possibility for benefit. Clinical acceptance is a key limiting factor in health IT adoption. Objectives To assess the benefits and challenges experienced by clinicians using a custom-developed health information exchange system, and to show how perceptions of benefits and challenges influence perceptions of productivity and care-related outcomes. Methods We used a mixed methods design with two phases. First, we conducted interviews with stakeholders who were familiar with the health information exchange system to inform the development of a measure of benefits and challenges of the use of this system. Second, using this measure, we conducted a survey of current and former users of the health information exchange system using a modified Dillman method. Data were analyzed using structural equation modelling, implemented in partial least squares. Results 105 current and former users completed the survey. The results showed information quality, ease of completing tasks and clinical process improvement as key benefits that reduced workload and improved patient care. Challenges related to system reliability, quality of reports, and service quality increased workload and decreased impact on care, though the effect of the challenges was smaller than that of the benefits. Conclusion Even very limited health information exchange capabilities can improve outcomes for primary care users. Improving perceptions of benefits may be even more important than removing challenges to use, though it is likely that a threshold of quality must be achieved for this to be true.


IntrOduCtIOn
Health information technologies (HIT) offer tremendous potential to improve the quality and cost effectiveness of healthcare. However, projects to implement HIT around the world have met with mixed results. 1-4 Central to all the explanations for HIT implementation success and failure is clinical adoption. 5, 6 Various models have been examined including TAM, 7 UTAUT 8-10 and an integrated model 11 based on both TAM and Cenfetelli's 12 dual factors of adoption. In a related research stream, Lau and his colleagues 13, 14 and van der Meijden and his collegues 15 have advanced a benefit evaluation framework that also addresses adoption issues.
This study examines the clinician's perceptions of HIT (a common strategy in adoption research), but focuses on their perspectives regarding benefits arising from HIT adoption. In this way, this study bridges the two main themes in IS research on HIT identified by Agarwal et al. 5 : adoption and evaluation. It examines how a health information exchange technology was understood and valued by primary care practitioners. This research demonstrates both the potential benefits from a provider perspective and the limiting factors that must be addressed in ongoing system development.
The specific goal of the research was to understand the benefits and challenges of 'PhysicianConnect' (a pseudonym). PhysicianConnect is a technology designed to share data about patients between hospitals and primary care physicians. Sharing of data through IT remains a particular challenge in e-health. A recent survey 16 showed that while primary care physicians have excellent adoption of electronic medical records (EMR) (i.e. internal systems), only about half routinely share information with other providers in an electronic format, even in countries where EMR adoption rates are 98%.
Two characteristics of PhysicianConnect are of particular importance to note. First, only three types of reports are sent through PhysicianConnect: results of in-hospital laboratory tests, in-hospital diagnostic imaging and discharge summaries for patients. Non-hospital-based laboratory tests or diagnostic images are not included nor are detailed patient reports for admitted patients. Second, the flow of data is one way; no data from the primary care EMR is shared with the hospital information system.
PhysicianConnect was developed in a bottom-up fashion. It was initiated by a physician and a hospital CIO, who saw an opportunity to connect their systems. It was designed using very simple technologies as a custom add-on to existing systems. It did not attempt to identify all of the possible data that a physician might want from a hospital information system, nor did it attempt to include other organizations from which primary care physicians received data, such as private laboratories. Regardless, it demonstrated the potential to connect systems, even in a rudimentary way, in order to avoid processing paper documents. After about two years, the hospital sought additional funding to expand the program to all physicians in its catchment area and to do some 'recalibrating and adjusting' of the system based on experience in working with it [interview with PhysicianConnect technologist].
Funding was provided by a government agency with the aim of expanding it beyond the initial hospital to two other hospital groups that existed within the region that had been developing a similar interface. The project ultimately expanded in four phases so that by the time of this research, the hospitals had been using it for at least 2 years, with individual physicians having access for a few months to 2 or more years.

MethOds
The study was conducted in two phases, using a mixed-methods design. 17 During the qualitative phase, semi-structured interviews were conducted with two primary care physicians who had adopted the technology, two of their administrative staffs and one hospital IT manager who supported PhysicianConnect. A group interview was also conducted with three staff members at the government agency that funded and supported the project. Respondents were asked to explain when/how they had become PhysicianConnect adopters and how they had conducted various tasks related to the receipt of reports before and after the implementation and to identify the various benefits and challenges that they perceived related to the use of PhysicianConnect. Interviews were tape recorded and transcribed. Data were coded to identify salient benefits and challenges.
In the quantitative phase of the study, the benefits and challenges identified in phase one were used to develop a survey that was sent to all registered physician users of PhysicianConnect. The questionnaire included 12 items regarding benefits and 13 items focused on the challenges, all measured on a five-point scale. In total, 295 surveys were sent using an adapted Dillman method. 18 Figure 1 shows the steps in the survey administration and the responses received at each stage.
Based on the results of the interviews and a review of prior literature, a model for assessing the benefits of PhysicianConnect was constructed ( Figure 2). The dual factor model represents the benefits (enablers) and challenges (inhibitors) as second-order latent constructs. 19 Benefits are composed of three first-order factors: information quality, process improvement and ease of completing tasks. Challenges are composed of four first-order factors: reliability, report quality, number of reports and service quality. Dual-factor models are built on the premise that inhibitors influence behaviour differently than enablers and must, therefore, be examined separately. While increasing or decreasing an enabler is expected to promote or inhibit more use, removing an inhibitor does not necessarily promote more usage but simply does not discourage it. 12, 20 This model was assessed in the second phase of the study. Four hypotheses were tested, linking the benefits and challenges to each of the clinical outcomes.
The model was tested using SmartPLS, 21 following the Wetzels et al. 22 approach for second-order factors. Box 1 provides a more detailed explanation of the partial least squares (PLS) method. The analysis proceeded in three stages: assessment of first-order measurement model, second-order measurement model and structural model. The seven firstorder factors, modelled as reflective constructs, were assessed box 1 structural equation modelling (seM) using PLsan explanation of the method PLS is a SEM technique, first developed by Lohmöller (1989). Broadly speaking, SEM tools allow researchers to assess complex relationships between constructs such as attitudes (including direct and indirect relationships) and to simultaneously assess the relationships between constructs and the questionnaire items that are designed to measure them. This holistic assessment of a complete model fits with contemporary views of theory and measurement, where theoretical concepts are deemed to derive part of their meaning from the way they are assessed and by the theoretical model in which they are embedded. In contrast to covariance-based SEM (CB-SEM), the PLS method uses the principals of ordinary least squares (OLS) regression and principal components factor analysis to assess the structural and measurement models.
PLS produces factor loadings (interpreted the same as loadings in factor analysis) to aid in assessment of the measurement model, and standardized regression coefficients to aid in assessment of the structural (path) model. Unlike CB-SEM which uses the assessment of model fit to judge the quality of the models, PLS uses R 2 to assess the explained variance in constructs as well as the above-noted statistics.
Because PLS estimates the model iteratively in small portions (thus the term 'partial'), the sample size requirements are lower than CB-SEM. The sample size requirement is determined as the greater of: • ten times the number of independent constructs in the most complex regression in the structural model. In our case, the most complex regression involves 2 independent constructs (benefits & challenges) that would demand at least 20 cases. • ten times the number of items in the construct with the greatest number of measures. In our case, that is process improvements, with 5 items, demanding 50 cases.
Most researchers suggest sample sizes greater than 100 are better, as the model estimates can be unstable at very low sample sizes.
In general, the stages of PLS interpretation proceed as follows: 1) Assess internal consistency reliability of constructs. The ICR computed from PLS is similar to Cronbach's by examining the loadings and cross loadings, and some items were removed to improve measurement quality. The three reflective second-order factors were assessed based on their loadings and cross loadings, internal consistency reliability and discriminant validity. The structural model was assessed using a bootstrapping method with 500 samples. An additional model was run with the second-order factors modelled as formative. The structural model results are similar. The main difference is that only one first-order factor was a significant predictor of the second-order factors, an outcome of the high degree of multicollinearity between the first order factors.
alpha and can be similarly interpreted -ICR > 0.70 is the common benchmark.
2) Assess convergent and discriminant validity at the item level using the loadings and cross loadings. In general, look for loadings in excess of 0.70 and cross loadings that are significantly lower than the loadings.
3) Assess convergent validity and discriminant validity at the construct level by looking at the AVE and and interitem correlations. An AVE > 0.50 implies the construct shares at least 50% variance with its items and this is the common threshold. In practice, the square root of the AVE is reported for ease of comparison with the inter-construct correlations. Thus, the square root of the AVE should be greater than 0.70. For discriminant validity, the square root of the AVE should be higher than the inter-construct correlations, indicating that the construct shares more variance with its measures than it does with other constructs in the model.

Interview findings
Participants in the interviews identified both positive and negative aspects of PhysicianConnect as well as numerous outcomes of using it. Appendix 1 summarizes the positive and negative attributes of the system identified along with the key outcomes experienced. For each item, a representative quote from the interviews is shown, along with an explanation of where the item fits within the research model.

Survey results
Of 295 surveys, 26 were returned as undeliverable leaving the potential sample at 269. Of the 118 respondents (44% response rate), three responses had questionable data quality (greater than 20% missingness) and were dropped, leaving 115 responses. Ninety-nine were current users of PhysicianConnect. Six respondents received information via PhysicianConnect but had discontinued their usage, while ten had never used it. No other data were collected from these ten respondents. Our final sample, then, was 105 current and former users. The majority of responses came from users of the three most commonly used EMRs. Ninetynine surveys were completed by physicians, followed by one nurse, one nurse practitioner and four administrators. They represented each of the three hospital systems who were providing data. They had been using the system for Final reminder sent to nonrespondents after 5 weeks with paper survey option N=38 + =

Figure 1 Survey administration and sample
tell whether the data are flowing. Three challenges were reported as not being significant concerns. The majority of respondents indicated having had no problems with date formats from PhysicianConnect not being compatible with the EMR. Nearly 50% of users reported that they had not experienced or observed an issue with getting too many reports from PhysicianConnect, nor had they been unable to update their EMR because of PhysicianConnect.

First-order factors
For the three first-order factors of benefits, all but two loadings were above 0.7. The loading for 'improved ability to search for patient information' was 0.68, which is very close to the 0.7 threshold, so this item was maintained. The loading for 'improved legibility compared to paper' was 0.52. While the item seems to reflect information quality, it clearly did not fit with the other items. One possibility is that most respondents were already receiving reports by fax and so were not seeing PhysicianConnect as a replacement for handwriting but rather as a replacement of fax. Regardless, the item was the most variable of all of the benefits items, and since it did not load strongly on the construct, it was dropped.
For the four first-order factors of challenges, the measurement model was mixed. The service quality and reliability constructs both had loadings above 0.7 for each item. The number of reports construct had one very high (0.87) and one very low H1: The benefits of using PhysicianConnect will be positively related to impact on workload, such that higher benefits result in a sense of an improvement in workload.
H2: The benefits of using PhysicianConnect will be positively related to impact on care, such that higher benefits result in a sense of an improvement in care-related outcomes.
H3: The challenges of using PhysicianConnect will be negatively related to impact on workload, such that higher challenges result in a sense of a worsening in workload.
H4: The challenges of using PhysicianConnect will be negatively related to impact on care, such that higher challenges result in a sense of a worsening of care-related outcomes.
between one month and five years, with an average usage of 19 months. A comparison of early versus late responders showed no significant differences on the variables of interest. 23 Appendix 2 shows descriptive statistics for the survey items. Overall, the responses suggested positive outcomes from the use of PhysicianConnect. All of the means were above the scale midpoint. The most positive effects were for speed of obtaining patient results, reduction in filing errors and administrative staff workload. Beliefs about the benefits of PhysicianConnect were also generally positive. The most commonly cited benefit from using PhysicianConnect was ease of access to information. Improved information sharing, greater flexibility in where work can be done, staff ease of response to patient inquires, improved organization of information, improved ability to search for patient information and better follow-up with patients were also widely reported as benefits. Improved legibility of information was reported by some, but not all respondents. Three benefits were not as widely reported (average was less than 3): ease of creating referrals, updating the cumulative patient profile (CPP) and creating alerts.
The respondents report fewer challenges from using PhysicianConnect than they did benefits, and fewer than expected based on the phase one interviews, though most reported having experienced or observed each challenge. Duplicate reports were most often reported as a significant problem. Two other challenges were reported frequently: lack of support from the EMR vendor and not being able to  loading (0.23). Since removing the latter would leave a construct with a single item, and since both the number of reports and report quality issues related to the functionality of the system, the construct was merged with the report quality issues. The construct reflecting report quality had no loadings over 0.7. Examination of the items and reviews of the interview data suggested one possible explanation for this outcome. Several of the items in the report quality item reflected issues with a single EMR product: 'inability to upgrade', 'incompatible date formats' and 'reports received in a way that does not enable cut and paste'. Since they were not experienced universally by respondents, they were dropped from the analysis. The items for impact on workload are all loaded > 0.7. The loadings for the care outcomes construct were weaker, with one loading over 0.7, one loading (errors compared to reports that were scanned) of 0.56 and three items with loadings between 0.65 and 0.7. The higher loading items were retained consistent with prior decisions and the lower one was dropped.
With these modifications, the measurement model was reassessed. All internal consistency reliabilities were above 0.70 and each item loaded more strongly on its own construct than on the related constructs, indicating adequate discriminant validity. To proceed to the next phase of the assessment, the factor scores were saved for the first-order dimensions.

Second-order factors
The second-order factors were also modelled as reflective, since they were expected to be highly correlated. The three items reflecting benefits all had loadings (Table 1) greater than 0.7. Two of the three items reflecting challenges had loadings greater than 0.7 with reliability loading at 0.65. Since reliability is an established dimension of the benefits evaluation framework, it was retained despite the lower loading.
The cross loadings for the items were also examined showing that each item loaded more strongly on its intended construct than on any of the others. To assess discriminant validity at the construct level, the square root of the average variance extracted (AVE) was compared with the interconstruct correlations. In all cases, the analysis showed that there was more variance in common between the constructs and their measures than between constructs, supporting discriminant validity. Each construct also had an internal consistency reliability above 0.70 (Table 2).

Survey -structural model
As shown in Figure 3, all four hypotheses were supported. The benefits of using PhysicianConnect significantly and positively influenced the workload and care outcomes experienced by physicians. Stronger perceptions of information quality, process improvement and ease of completing tasks resulted in a perception that nursing, administrative and even physician workload were improved. They also influenced perceptions that care was improved by reducing errors and by improving the speed of referrals and receipt of patient results.
Challenges with using PhysicianConnect also significantly influenced the outcomes, negatively as expected. Greater challenges with reliability, report quality and service quality promoted perceptions of worsening workload and worsening care.
The analysis also showed that benefits had a stronger positive influence on the outcomes than challenges had a negative one (p<0.001). Improving (perception of) benefits had a  stronger impact than reducing (the perception of) challenges. This differential impact is consistent with the dual-factor conceptualization of enablers and inhibitors.

Findings
This study suggests that benefits were realized from the use of PhysicianConnect, particularly in relation to quality of care and productivity. Even a very limited form of connectivity (one-way communication of test results in a simple text format) resulted in perceived net benefits to clinical productivity and quality of care. This finding supports the view that integration of health records is a source of clinical value, and further demonstrates that even limited integration can provide some benefits.
Some what surprisingly, these benefits were realized in spite of challenges in both system and service qualities. Viewed from the perspective of traditional IS evaluation models, 24 this should not be successful. Chatterjee et al. 25 report that system and service qualities were essential to the realization of benefits in the context of mobile health technologies. They contend that if clinicians cannot depend on a system to function reliably or do not receive adequate support when there are problems, they will reject the system. A conversation with one of the staffs who had been managing the PhysicianConnect program echoed this view. Yet, despite acknowledged problems with PhysicianConnect, clinicians in this study were very positive towards it. They did not report a lot of problems with it (the average scores in the survey on the challenges related to system reliability and service quality are not as high as one might have expected given the acknowledged challenges from phase one). More importantly, the impact of the challenges construct on productivity and workload was lower than the impact of benefits.
It appears as though, in this case, the strong benefits related to information quality outweighed the limitations in system and service quality, and resulted in clinical acceptance of the system and ultimately the desired benefits. It appears as though the system and service were 'good enough' for adopters to realize the benefits associated with the access to information. This reflects quadrant II in Cenfetelli's 12 dualfactors model, and the results support his prediction that this quadrant can yield positive results (for adoption in his case, but benefits as observed here).
This finding is important because it shows that a new technology can be successful in a healthcare environment even if it falls short of the ideal on some quality aspects. Providing a workable system that gives clinicians tangible benefits in terms of health outcomes may be sufficient to encourage initial adoption, and thus may create the opportunity for continuing technology development to improve the system and extend its use. Thus, while the benefits evaluation framework that has been used in much of the literature 24 presents system quality and service quality as linear predictors of use and satisfaction, our results suggest that they may represent a threshold factor. Whether this is unique to the PhysicianConnect context or whether this finding would generalize more broadly is unknown, and requires further investigation.

Limitations
A key limitation is the cross-sectional design with a single source of data (user perceptions). Still, examining perceptual data does provide some basis to estimate the potential benefits, and other studies 26 have focused more on the quantification of benefits from HIT. As Goh et al. 27 note, further research is needed on the complex processes of clinical adoption and acceptance that result in these benefits. This study extends prior literature to demonstrate how beliefs about enablers and inhibitors associated with the system influence perceptions of the benefits.

COnCLusIOn
This study demonstrates the potential effectiveness of health information exchange, even in a limited fashion, and contributes to our understanding of benefits evaluation by linking the benefits evaluation model 14,24,28 to the dual factors model of IT acceptance. 11,12 In doing so, this study of PhysicianConnect shows the potential for successful HIT implementation and benefits realization, even in the face of system limitations. Staff ease of response to patient inquiries I love it. It definitely is… the turnaround time to have the results in the chart. A lot of patients come in and they're like 'oh I want to follow-up with such and such', so it's great (Admin 2) Improved information sharing among team This is important because 'only whoever has the report can see it. The idea of the electronic chart is that everyone can share it. That's one of the greatest benefits is that everybody can look at it at the same time. It's not this one piece of paper floating around'. (physician 1)

Construct: Information quality
Improved organization of information The reports all are very standardized so that's neat. It's boring but they all look very standardeverything is here. Sometimes with the hospital reports you get different things, nice headers etc. which is nice but with this you know everything is here and standardized. (physician 1)

Improved ability to search for patient information
We [are able to] do all kinds of searches. We do searches for whether people have had different types of screening (mammograms, colonoscopies). We do searches for diagnostic criteria, so for patients who have diabetes or other disease states. (physician 2)

Improved legibility of information compared to paper
And the other thing is the legibility of a doctor's handwriting. You know, in those charts, I look at them and think how can they read this? Seriously! It's just chicken scratch. But now everybody can read it, everybody knows if it's a 7 or a 0 for the meds and all this kind of stuff and I really think that that helps the next person to come along. (admin 1)

Ease of access to information
What I was doing differently before? Chasing down paper sometimes, so that's an advantage too … I mean, it's electronic so anybody can access the report even if it's signed off or not. But if it's sitting in my box here {referring to the inbox from the paper world} and I'm away for a day and a half, it's locked up. It's mine; people have to come in here to take a look at it (physician 1).

Construct: Ease of completing tasks
Ease of creating alerts by cutting and pasting So you can actually highlight this with your text and cut and paste and put it into an alert and then with the electronic chart the advantage would be that you can also put it into a recall. (physician 1)

Ease of creating referrals by cutting and pasting
Plus you have the ability to cut and paste within the report. So you can search it and you can…if I'm doing a letter to a consultant and there's a certain paragraph in the [computed tomography] (CT) report I can just copy and paste into my letter and it's wonderful. (physician 2) Ease of updating CPP The other part of my job is to update the CPP for bone density, mammograms and that type of thing so anybody can pick that up and see that a patient is due in two years or whatever. Instead of flipping through the chart [and deciphering hand written notes] the doctor now sends a task for me to update the CPP with the information.

Construct: Report quality issues
Extra space in reports that makes reading difficult Sometimes the way the reports come in is a bit awkward still. So, if it's a short report (here he picks up a piece of paper and indicates a report where the text occupies only about quarter of the page) and the rest is all blank, when it actually comes into the chart, this will be it and there will be a big blank space. So when I put my next entry into the EMR it will be down here (below the blank space). And I find myself putting the cursor in and deleting the extra space. (physician 2) Lack of support for images PhysicianConnect has no images, just text. Before a doctor could draw a simple image to represent part of a note.

Date formats from reports not compatible with EMR
And dates sometimes get confused, that's another one that happens to (e.g., DMY versus MDY).

Sometimes they file backwards. (Admin 2)
Not enough reports are available through PhysicianConnect I was asking our vendor this too…we should stick [lab tests from private labs] in there too. We should stick everything we possibly can through there.
Do not want all of the reports that currently come Initially, we received data on things happening during a hospital stay (e.g. bloodwork, ICU, etc.). This is problematic because of the risk of information overload and also for medical-legal reasons. We need to have it filtered so we only get what we previously received via fax.
(physician 1) Duplicate reports received There was a while there where we were getting multiple copies of reports and it had to do with any time the transcriptionist at the hospital edited something or changed the transcription, it would generate another copy of the report. So there was a while there where we were getting 3-4-5-6-7 copies of a discharge summary and it was just because every time there was a change or addendum made, we would get another copy. (physician 2) Unable to upgrade EMR due to version incompatibility We like it so much that we have in fact delayed our upgrade to the next version of [our EMR]. We've heard from other clinics that the new version of the EMR somehow broke the interface. … But lose the features of the upgrade that we can't take advantage of. That was a big downside.

Construct: System reliability
System reliability (i.e. stops sending reports) Even though, going into this, we were of the same mind; we can change what we want, but we went down a bit of a rabbit hole there and found out that it's a pretty temperamental interface. If you start changing things, for some reason, on the other side of the fence {the vendor side} things go south. We don't know why. (Hospital 1) Reports received in a way that does not allow cut-paste Now sometimes when the reports come in they're not cut and paste-able. And that's another weird negative. And I know every once in a while they go through a thing where 'we can't do this now' so I don't understand how that happens or works but…. (physician 2) Reports received for mis-identified or unknown patients Occasionally we get unidentified patients or unknown patients and I think it has to do something with maybe the way a name is spelled or … but it's happening less and less, that. (physician 2) No way to tell whether data are flowing I don't really have an understanding about when they are coming in, how often they are coming in. I just see them and when I stop seeing them I realize that maybe something is wrong. And that's when I say to the staff gee I haven't had any hospital reports in a while. (physician 2)

Construct: Service quality
Lack of support from hospitals about system Support is very frustrating when you are working with two different places and you're working with your EMR and with the hospital who's sending the reports, you don't know who to go to and how…that's a big one that's hard to deal with cause no one's really helpful and you're just expected to figure it out. (admin 2)

Lack of support from EMR vendor about system
And I don't think that the vendor has been quite as forthcoming as they could have been with what the problems are. I think [they] are a very popular company but they've grown really really quickly and they're probably in a little over their head right now trying to keep up. (physician 2)

Construct: Impact on care
Speed of obtaining patient results It is very cool, for instance, that if I see somebody in the morning, and I order a chest X-ray, that by the afternoon that chest X-ray is already on the interface and may be in my inbox and could well be in the chart. I mean, that's just never happened before…that quickly (physician 2). All items measured on a five-point scale where 1 = much worse with physicianConnect, 2 = worse with physicianConnect, 3 = no difference, 4 = better with physicianConnect and 5 = much better with physicianConnect a indicates items retained in final model b nursing and physician workload were discussed during phase 1 but not as having been significantly improved. they were included in the survey to provide a more fulsome assessment of the impact on workload in the practice.

physicianConnect benefits
To what extent have you personally observed or experienced each of the following potential benefits from having the PhysicianConnect interface for receiving patient results in your practice? All items measured on a 5-point scale where 1 = not at all, 3 = to some extent and 5 = to a very great extent a indicates items retained in final model