Introduction

Point-of-care decision support systems based on electronic guidelines have been suggested to successfully deliver the knowledge embedded in Evidence-Based guidelines (E-B guidelines) [1, 2]. A number of studies have already shown positive findings for some types of decision support systems such as drug-dosing systems and computer-based reminder systems for preventive care services [3, 4]. However, there is less evidence for more complex guideline-based implementation systems with real-time interaction during consultation and failure of these systems is not uncommon [5, 6].

An electronic clinical decision support system (EBMeDS) has been integrated in one of the Electronic Medical Records (SoSoeMe) of Belgian family physicians (Feb 2010).

Facilitators, barriers and issues of non-acceptance need to be understood in view of a successful implementation and to minimize unexpected adoption behavior. Expressed preferences for system attributes and functionality could be useful as a basis for future system re-engineering. Electronic systems that are not accepted by its users cannot be expected to contribute to improving quality of care.

Several models have been developed to explain users’ acceptance and use. These models originated from theoretical insights used in psychology, sociology and information systems. Examples are the Theory of Reasoned Action (TRA), the Technology Acceptance Model (TAM), the Theory of Planned Behavior (TPB), etc. To date, the Unified Theory of Acceptance and Use (UTAUT) is considered as the most sensitive model for explaining variance in technology acceptance [7]. The validated UTAUT model integrates all constructs from previous models. The model attempts to explain intention to use and has been found to explain 69 % of the variance in technology acceptance. Several studies demonstrated already the use of UTAUT for studying factors influencing health information technology adoption. [8, 9]

Objectives of the present study were the assessment of users’ perceptions towards the implemented EBMeDS system, the investigation of user-interactions with the system and possible relationships between perceptions and use.

Methods

This EBMeDS-system was developed by Duodecim in Finland and has been implemented in Belgium by one of the software vendors (SoSoeMe) in a translation of the English version to the Dutch language, with the support of the Belgian Center for Evidence-Based Medicine (CEBAM). The EBMeDS system receives structured patient data from electronic medical records and returns reminders, therapeutic suggestions and diagnosis-specific links to guidelines. Electronic forms and calculators (e.g. a calculator for glomerular filtration) are integrated in the system [10]. The EBMeDS system covers a full spectrum of clinical areas.

The EBMeDS system consists of a client component, an EBMeDS interface and the EBMeDS engine. The EBMeDS service is a platform-independent, service-oriented architecture (SOA) which can be integrated into any EMR system containing structured patient data.

EBMeDS scripts are written in the standard server-side JavaScript language (ECMA) which is compiled by the JavaScript interpreter of a web server application. EBMeDS works with any web server application supporting server-side JavaScript (e.g. IIS, Helma). The EBMeDS interface is defined in an XML Schema based on an object model for information exchange between the EMR and the EBMeDS service (HL7 Clinical Document Architecture R2). The interface is an XML-based request-response interface. The request is made by the client component of the EMR system when opening a patient’s file, entering a new diagnosis or prescribing a new drug. The request contains coded information (e.g. ICD10 or ICPC-2) extracted from the EMR and standardized in an XML-format. The EBMeDS module analyzes the data package and returns a decision support message that is transferred back to the EMR system in a respond message. The client component have to be designed and integrated by the local developers of the EMR software.

The EBMeDS engine consists of a service to normalize data and to create the standard variables and objects used in the EBMeDS scripts. A conversion filter converts incoming classification codes to neutral aliases and absolute finding values to values relative to reference values or SI standard values. The EBMeDS engine could call a function library, which performs standard calculations (e.g. BMI, GFR) needed by several EBMeDS scripts [10].

SoSoeMe is one of the computerised medical record systems in Belgium, fully coded in ICPC-2 and supported by a terminology system. Not only ICPC coding is supported but physicians can also use ICD-10, which could be mapped to their corresponding ICPC code. The SoSoeMe software enables physicians to record patient histories and contacts, to display test results, to access generated notes and reports and to support physicians in decision making for patient care. The SoSoeMe software is often chosen by physicians, keen to coded registration.

A mixed evaluation approach was performed consisting of a qualitative and a quantitative analysis. A qualitative analysis was used to identify factors that may account for acceptance and use of the EBMeDS system. A quantitative analysis of computer-recorded user interactions with the system was performed for an evaluation period of 3 months to assess the actual use of the system. Qualitative and quantitative analysis were linked to each other.

Quantitative analysis

To capture user interactions with the EBMeDS system during patient consultations at the office of the physician, data were obtained from a log file being part of the SoSoeMe software. Installation of the log file was automatically done while upgrading the software. Physicians had to manually send this log file to the researcher to be included in the study. Physicians knew they were participating in a study, by sending their log files they agreed to be included in analysis. Collection of data of actual use was done from the end of March 2011 until the beginning of June 2011. The analysis of log files was anonymous and no patient information was collected.

Five distinct events were registered in the log files, namely reminders shown automatically or on demand (1), number of seconds after which a reminder was closed (2) and number of requests for additional information on the evidence (3), the script (4) or the patient (5).

Qualitative analysis

Questionnaire development

The technology acceptance model of UTAUT was used as a structural model for the development of our questionnaire.

A literature review of barriers and facilitators to implementing electronic clinical practice guidelines was performed. These barriers and facilitators were classified under the different constructs of the UTAUT model: perceived ease of use, perceived usefulness, social influence (using the system because colleagues do) and the perception that organizational and technical support exist (facilitating conditions). General satisfaction, perceived influence and perceived use were added as additional constructs. The questionnaire was supplemented with open questions to allow free expression of ideas or perceptions.

Survey components differed between three different groups of SoSoeMe-users: users and ex-users of the EBMeDS system, physicians who did not use or never used the EBMeDS system.

The draft questionnaire was reviewed by two expert-users to establish content validity. This questionnaire was piloted in a group of 3 family physicians and was modified based on the feedback we received from these physicians.

Survey administration

The survey was administered online to all 334 Dutch-speaking family physicians using the EMR of SoSoeMe from March 2011 to June 2011, (only family physicians were invited). The study population included current users of the EBMeDS system, as well as ex-users and physicians who did never have experiences with the EBMeDS-system. Participants were recruited using an invitation by e-mail. The whole population sample received three reminders to fill in the survey. The survey was linked to a lottery of a tablet PC to increase the response rate. The survey had to be opened within the EMR of SoSoeMe. A unique individual identifier was automatically assigned to the survey. The same unique identifier was saved in the log files to link these data to the questionnaire.

Data analysis

Results of the literature review studying barriers and facilitators were classified under the different constructs of the (modified) UTAUT model. Responses to the items per construct (seven-point Likert scales as in the original model) were summed up and divided by the total number of items in a construct to yield mean composite scores. When one item of a construct was not answered, we imputed the mean individual score for the other completed items in the scale. When more than one items of a construct were not answered, the case was deleted from analysis. The reliability of the construct measurements was evaluated by using Cronbach’s alpha.

Descriptive statistics and graphical displays were conducted to describe the sample population and the data in the log files. Chi-square and Fisher exact tests, Mann–Whitney and Kruskal-Wallis were used as appropriate to compare data between different groups. Spearman correlation coefficients were determined between variables.

All statistics were performed using a two-sided test and a significance level of 0.05. Bonferroni adjustment was applied to correct for multiple comparisons. PASW Statistics 18 was used for statistical analyses.

Results

Characteristics of respondents

The response rate was 12 % with 39 out of 334 invited family physicians responding. 35 physicians were users of EBMeDS, two respondents had used EBMeDS in the past and two physicians had no experiences with EBMeDS. Main analysis was restricted to the category of current EBMeDS-users because of the limited responses in the other categories. 71 % of respondents were male and 26 % were female. Male respondents were roughly 10 years older than female respondents. When comparing the percentage of females in our group of respondents with the whole group of SoSoeMe users, relative more men than women participated in the survey. Characteristics of the respondents are shown in Table 1.

Table 1 Characteristics of the respondents

Perceived (dis)advantages of the EBMeDS system

The increased alertness created by the system was mentioned as most important advantage by 43 % of users. 37 % of respondents reported as most important advantage of the system the possibility to get patient-specific point-of-care information, as well about the evidence as about the individual patient.

Seventeen percent of users perceived the quantity of the reminders as most important disadvantage of the system. The same percentage of physicians reported the incorrectness of the reminders (17 %) as most important disadvantage. Wrong timing and on-screen positioning of the reminder were mentioned as most important disadvantage by another 17 % of users.

Most important reason to neglect reminders were mostly related to the quantity of the reminders (29 %) and lack of time (23 %).

Construct measures

The majority of EBMeDS users (66 %) had a positive attitude towards the system in general. 66 % of respondents would recommend the use of the system to others to some extent.

Mean perceived usefulness was 4,69 ± 1,35. 77 % of respondents believed the EBMeDS system could lead to better quality of care and 72 % found their knowledge increased while using the system. 53 % of users perceived EBMeDS as a useful information source. 35 % of physicians found they could perform their tasks faster while using EBMeDS. In general, respondents were positive towards the ease of use of the system (mean 5,04 ± 1,41). Only 12 % of physicians found that the system required too much ICPC coding.

A mean of 4,43 ± 1,13 was reached for the construct of facilitating conditions. Differences in opinion existed about whether or not sufficient information was provided when the system was launched. 68 % of users found that they did not get enough information while 32 % of users were positive. 43 % of respondents were satisfied with the technical support when problems occur while the same percentage of users (43 %) was not. Only 18 % of physicians found that technical problems frequently occurred. 58 % of physicians found that using the system fits well with the way they liked to work.

Eleven percent of respondents found that they changed their way of prescribing by using the system and 60 % found they became more careful with drug-drug interactions. 26 % of respondents found they changed their way of working. Mean intention to keep using the system was 5,91 ± 1,33. Mean social influence was 1,61 ± 1,03, participants did not use the system because their colleagues did.

Mean attitude towards reminders scored 4,57 ± 0,85. Differences in opinions existed for the perceptions concerning the flexibility (34 % positive versus 34 % negative). Only a minority of users was not satisfied with the relevance (24 %), the specificity (24 %) and the formulation (12 %) of the reminders. Lack of time was a barrier to read the reminders for 44 % of users.

Cronbach’s alpha, mean and standard deviation of the constructs are displayed in Table 2. Figures 1, 2 and 3 give the percentage of answers for the different items in the constructs.

Table 2 Cronbach’s alpha, mean and standard deviation of the constructs
Fig. 1
figure 1

Perceived Use and Perceived Influence. (IU1) I have the intention to keep using the system. (PU1) I always read the reminders in EBMeDS (PU2) I always read the guidelines in EBMeDS (PU3) I always read the messages for drug-drug interactions in EBMeDS. (PI1) I changed my way of working (PI2) I am more careful with drug-drug interactions (PI3) I changed my way of performing diagnostic tests (PI4) I changed my way of prescribing

Fig. 2
figure 2

Perceived Ease of Use, Perceived Usefulness or Performance Expectancy, Facilitating Conditions. (EU1) The system is easy to use (EU2) I became fast skillful in using the system (EU3) The system operates too slow (EU4) The system requires too much ICPC coding. (PE1) The use of the system could lead to better quality of care (PE2) Using the system enables me to accomplish my tasks more quickly (PE3) The information is useful in daily practice (PE4) My knowledge increases while using the system. (FC1) Sufficient information was provided when the system was launched (FC2) Technical support is available when needed (FC3) Using the system fits well with the way I like to work (FC4) Technical problems frequently occur (FC5) It is easy to use ICPC coding within my EMR

Fig. 3
figure 3

Social Influence, Attitude towards reminders, General Satisfaction. (SI) I use the system because my colleagues do. (AR1) The reminders are relevant (AR2) The reminders are too general (AR3) The reminders are flexible enough for all types of patients consulting me (AR4) I doubt about the quality of the information (AR5) The formulation of the reminders is clear (AR6) I have no problem with the foreign origin of most reminders. (GS1) I am satisfied with the EBMeDS system (GS2) I am satisfied with the reminders (GS3) I am satisfied with the guidelines (GS4) I am satisfied with the messages for drug-drug interactions (GS5) I am satisfied with the reliability of the system (GS6) I would recommend the use of the system to others

Actual use

The dataset contained 16 891 reminder events generated within the EMR of 17 family physicians over a period of three months. Seven percent of family physicians co-operated in both the qualitative and quantitative part of the study.

Table 3 Actual use

Thirty-one percent of reminders automatically closed after an installed time-out. 0,35 % of reminders were requested on demand, the other 99,62 % of reminders displayed automatically. The average reminder was manually closed after 23 % of the number of installed time-out seconds. Detailed guidelines (long) were requested for 0,47 % of reminders automatically shown versus 16,17 % of the reminders shown on demand. The script behind the reminders was requested for 8,4 % of reminders automatically shown versus 13,6 % of the reminders shown on demand. Data of actual use are displayed in Table 3.

Table 4 Correlations between the constructs

Correlations

Statistically significant correlations existed between the intention to keep using the EBMeDS system and general satisfaction (p < 0,001), perceived influence (p < 0,001), perceived usefulness (p < 0,001), and facilitating conditions (p < 0,001). The correlation with the attitude towards reminders in general was not significant. Perceived use was statistically significant correlated with the intention to keep using the system (p < 0,001), general satisfaction (p < 0,001) and perceived influence (p < 0,001). Correlations are shown in Table 4.

The scores in the constructs (or scales) were not related with sex, age, the number of years of experience in family medicine, the number of years using EMR and SoSoeMe, using ICPC coding or the time Spent searching and reading medical literature.

Perceived use was significantly correlated with the percentage of time of the installed time-out after which a reminder was closed (p = 0,002). The intention to keep using the system was correlated with none of the parameters of actual use.

Discussion

Intention to keep using the system was very high (5,91 ± 1,33). A study of Varonen et al. [11] analyzed the perceptions of users towards the same EBMeDS-system. A different methodology was used but results confirmed the positive conclusions of our study towards the EBMeDS-system. Most of the physicians who participated in our survey were positive towards the EBMeDS system.

The possibility to get patient-specific point-of-care information, as well about the evidence as about the individual patient was seen as one of the most important advantages. The increased alertness created by the system, as well for more unusual conditions as for preventive actions and systematic prescriptions was more than once mentioned as another important advantage. The EBMeDS system was perceived by its users as robust. Only 18 % of users found that technical problems frequently occurred, which is one of the most important reasons in the literature for not using the system. Formulation of the reminders was perceived as very clear (76 % of users were positive) and the attitude towards the content of the guidelines in general was positive [12, 13]. Although ICPC coding is needed for a correct use of the system, current users (71 %) perceived the system as easy to use which is an important factor in decreasing resistance for use [12]. Only 12 % of physicians found that the system required too much ICPC coding. All of the above have been reported in previous studies [14, 15] as important characteristics of successful computerized decision support.

Physicians generally did not like the EBMeDS system when they were very busy during consultations and had no time to read the reminders. One of the most important disadvantages as mentioned by its current users was the quantity of the reminders. Too many reminders popped up or are repeated according the respondents. The quantity of the reminders and doubts about the correctness of the reminders seemed to be the most important reasons to neglect reminders (43 %). This could be confirmed by the relatively high percentage of reminders in the log files (18,5 % ≤ 2 sec, 32,5 % ≤ 3 sec) which seemed to be clicked away as an automatic reaction to ‘escape’ out of the guidelines whenever they triggered. Electronic decision support systems rely on consistent data in the EMR. The quality of the reminders depends on the quality of coding. Inserting the wrong ICPC-code results in a selection of the wrong reminders. Furthermore, ICPC is not always specific enough resulting in the retrieval of more than one guidelines for some of the ICPC-codes. It is important to find a balance between the specificity and the sensitivity of reminder triggers. It could be considered to improve the specificity at the cost of the sensitivity to limit the quantity of the reminders.

Reminders automatically shown, might best be reserved for situations where the patient benefits the most [12, 16] to avoid that serious alerts are being ignored or to prevent reminder fatigue. We support the idea of one of the respondents to develop a forum to share ideas and to report incorrect or unnecessary reminders by way of improving the system.

A previous review on prescribing alerts found that between 49 % to 96 % of all prescribing alerts were being overridden by physicians [16, 17]. Although it has to be recognized that results of our questionnaire were self-reported measures of use and influence and that users are often poor estimators of their own behavior [14], it was interesting to find that 26 % of physicians reported they changed their way of working to some extent with the introduction of the EBMeDS system. A remarkably high percentage of physicians (60 %) found they became more careful with drug-drug interactions.

Most important limitations of this study were the sample size and the lack of information of the non-responders. Despite intensive efforts response rate was low (12 %). The results were limited to the perceptions of current EBMeDS users and did not include the valid input of physicians who stopped using the system of physicians who never used the system due to insufficient responses.

Perceptions of the family physicians in our sample did not necessarily represent the perceptions of the whole population of family physicians. Eighty-three percent of respondents of our questionnaire reported using ICPC coding while an unpublished survey learned us that a minority of Belgian family physicians code in their EMR. This makes us presume that family physicians who agreed to participate in the study may have been particularly interested in evidence-based decision support and could possibly be seen as ‘early adopters’. Especially the physicians who took part in both parts of the study (qualitative and quantitative) as these participants were also most inclined to keep using the system. Further qualitative research (such as in-depth interviews) is necessary to explore possible reasons for this low response rate and for validating the results.

The results of the study indicated that the intention to keep using the system was correlated with the perceptions that the EBMeDS system is useful (PE) and the perception that organizational and technical support exist (FC). Perceived use of the system was correlated with the intention to keep using the system but with none of the UTAUT scales. The results of the study did not find strong support for the UTAUT model in our population, probably because of the small sample and the resulting lack of power.

Although the results of the questionnaire might not represent the perceptions towards the system of all family physicians, the opinions of this limited group of respondents do represent real-world perspectives towards the system. These perceptions are very useful for the formulation of recommendations towards system improvements and further implementation initiatives.

The questionnaire clarified there was a need for better communication and support when the system was launched. Improvements in the availability of technical support in case of problems with the system and more focus on the integration of the system into the normal clinical workflow are important future points of interest.

Conclusion

The majority of respondents demonstrated a relatively high degree of acceptance. Although the majority of respondents was in general positive towards the ease of use of the system, usefulness and facilitating conditions, part of the statements gave rather mixed results and could be identified as important points of interest for future implementation initiatives and system improvements. Messages for drug-drug interactions were mostly read and had the highest perceived influence. The quantity of reminders, which was perceived as too many, seemed to be the most important reason for neglecting reminders. The quality of reminding obviously depended on the quality of coding. The constructs ‘Usefulness’ and ‘Facilitating conditions’ were statistically significant correlated with the intention to keep using the system. Perceived Use of the system was correlated with none of the UTAUT scales. Major drawback of the study was the low response rate which implies caution with generalizing the results.