Sample and data collection for use-case specific defined studies

We made many contacts but received few responses. We contacted 531 different organisations, and later individuals in EU countries (including eHR vendors) and received 56 valid responses. Of the health ministries we contacted, seven provided useful information and a further five responded but could not provide any helpful information. Only two site representatives declined to participate at this stage (Supplementary data file, Table S2).

eHR vendors

We also collected details of the national or international eHR vendors with a significant presence in one or more EU countries. We contacted 17 companies identified initially, as well as any reported by survey respondents. Nine of these eHR vendors had a presence in more than one country. Two of those contacted started to complete the TIRRE survey instrument but failed to complete the questionnaire. We also approached nine vendors listed by site representatives who completed the questionnaire but they once again expressed no interest in participating in the survey. They did suggest they might consider completing the survey in the future if and when we had something more definite to offer. Few vendors responded; however, when they did reply to the survey, their responses to the questions posed provided useful detail.

Telephone and online completion of the survey

Of the 531 organisations we made contact with, 45 respondents commenced but did not complete the TIRRE survey online (Supplementary data file, Table S3) and 26 made a partial response during telephone enquiries but were then either unavailable for their telephone interview or failed to go ahead and complete the online survey. The initial telephone interviews took 1.5 hours and with experience still took 50–75 minutes. The feedback from the pilot survey suggested that the process took too long and that there was very little incentive for the respondent for completing the survey. While this drawback of the survey could have possibly caused a bias for the responses collected, we consider this as a valuable learning to consider in similar database profiling activities conducted in the future.

Completion of the survey

The valid surveys were on an average returned with 76% of the questions completed and this was consistent across the three respondent groups. Looking at the survey by category, the Data source and Record system sections were the only ones that fell below the 75% level (many sections were returned with above 90% of the questions completed). The main reason for this was the variation in the skip logic for individual respondents in these sections of the questionnaire (Supplementary data file, Table S4). There was a little difference between the sites which we had identified as eligible to participate in the use-cases and those we had identified as not eligible (77% use-case sites versus 75% non-use-case sites).

Results micro-level data

The greater the number of coding systems in use, the harder it will be to achieve semantic interoperability; therefore, the micro-level data collection was primarily concerned with collecting information about the coding systems the repositories used. We found that the WHO International Classification of Disease (ICD)20 was the most common coding system used by 71% (n = 39) of respondents. ICD-10 (n = 32) was used by 82%; 13% (n = 5) used ICD-9; 23% (n = 9) used an ICD modification and 5% (n = 2) did not respond (Supplementary data file, Table S5).

The second most used coding system was the WHO International Classification of Primary Care (ICPC), this was used by 20% (n = 11) of respondents. Eighty-two percent (n = 9) of those using ICPC used ICPC-2 and 18% (n = 2) used ICPC-1, none reported using an ICPC modification (Supplementary data file, Table S6). The third most common coding system was the Systematised Nomenclature of Medicine (SNOMED),21 which was used by 13% (n = 7) of all the respondents; 44% (n = 4) of those using SNOMED used the Clinical Terms version; 33% (n = 3) used the Reference Terminology version and 22% (n = 2) did not respond (Supplementary data file, Table S7).

One of the least common coding systems used was the Read Coding system [version2 – 5-byte and the Clinical Terms Version 3 (CTv3)] and these were only used by the seven UK repositories. They represented 9% (n = 5) and 4% (n = 2) of all respondents, respectively; 87% (n = 50) did not respond (Supplementary data file, Table S8).

The survey highlighted that there was a great variety in the number of drugs dictionaries utilised by the repositories and this is one potential barrier to achieving semantic interoperability. Sixty percent (n = 33) of respondents said that they have a coding system for drugs (Primary care 83%, n = 24; Cancer 33%, n = 5; Genetic 36%, n = 4). Of these; 76% (n = 25) use the Anatomical Therapeutic Chemical classification system;22 9% (n = 3) use Multilex; 12% (n = 4) responded ‘other’ and 3% (n = 1) responded ‘no data’ (Supplementary data file, Table S9). We were interested to know whether it was possible to extract information about the administration of drugs and we asked respondents if it was possible to extract data about daily dose and administration route from their database. Only around one-third of the Primary care and Cancer registries could extract data of this nature, while none of the genetic databases held this information (Supplementary data file, Table S10).

The survey was designed to assess what systems the registries had in place to achieve interoperability and to ensure data quality. Thirty-four percent (n = 19) of respondents had no system at all; only 5% (n = 3) used Health Level-7 (HL-7), an international interoperability organisation who’s Reference Information Model underpins much interoperability in healthcare; 2% (n = 1) used the Clinical Data Interchange Standards Consortium (CDISC);23 none used the Biomedical Research Integrated Domain Group (BRIDG)24 and 52% (n = 29) used an ‘In-house or other’ system (Table 1). Nearly, all (93%, n = 52) of the respondents either had no system in place or used an in-house system or provided no data.

Data collection meso- and socio-cultural levels

Data extraction at this level was concerned with record level issues. The majority of respondents (82%; n = 45) have the ability to extract data in standardised formats such as Comma Separated Values, Excel and full text. All of the respondents have at least one appropriate format. The data collected have a wide application and this is reflected by the diverse nature of the information stored within these repositories which ranges from research to mortality records (Table 2).

The respondents reported that socio-cultural influences had a small but significant impact on the validity of their data. These factors included ethical, religious and legal factors (Table 3); these might delay or prevent participation in the TRANSFoRm studies.

Socio-cultural factors, which include legal and ethical constraints, as well as influences on diagnosis, and organisational components of the health system from which the data originates are often barriers to conducting research. In summary, 71% (n = 39) of respondents use ICD and 20% (n = 11) use ICPC; however, 86% (n = 48) do not use one of the three main systems for ensuring data quality; 29% opt instead for an in-house system. Very few sites are adopting national standards for interoperability in linking data. Whilst multiple drug dictionaries were used, 66% (n = 10) of cancer repositories did not use them. Extract formats for data were standardised and only 3% (n = 6) of respondents chose to use a non-standard format. Data were not forthcoming from eHR vendors (n = 40). Repositories had a broad range of applications for their data, the most important was research (49%, n = 51). The most common socio-cultural influence that could potentially affect the validity of their data was ethical (10%, n = 7) and social (10%, n = 7) factors although 49% (n = 36) reported no social issues at all.

Difference in response depending on eligibility

Data sources that were non-use-case eligible tended to produce much fewer positive responses than those that were eligible. Overall, the repositories identified as potentially being use-case eligible made 2098 positive responses to questionnaire items compared with 268 from non-use-case eligible data sources (OR: 4.59; 95% CI: 3.93–5.35; p < 0.008); for genetic databases, the respective figures were 380:44 (OR: 6.13; 95% CI: 4.25–8.85; p < 0.008) and for cancer registries, they were 553:44 (OR: 5.87; 95% CI: 4.13–8.34; p < 0.008); the full results are in Table 4.

Data repositories capable of participation in the survey

Of the 56 valid responses, there were 15 pairs eligible to complete one or other of the use-cases. The 56 valid responses were made up of 29 databases of routine primary care data, 12 genetic databases and 15 cancer registries. From the valid responses, we were able to identify the location of databases with the potential to participate in the research studies. We identified five locations for linking primary care databases with genetic databases and 10 for linking primary care databases with cancer registries. The 15 eligible sites were spread across 11 countries (Supplementary data file, Table S11).

Details of the eligible sites

The sites had a total of around 1.5 million potential patients eligible to participate in this research; over 30,000 in the genetics of diabetes use-case and over 1 million to participate in Barrett’s disease, oesophageal cancer and the prescription of 30 medicines used to treat dyspepsia use-case. The country of origin, the website for these sites, the main coding system used and the expected delay in ethical approval are shown in Tables 6. We sometimes found contradictions between the data sources which indicated that they could supply linked data and several of the participants were, on closer questioning, only linking on a pilot basis; we have shaded out in grey the sites which are not currently active. The outcome of this process is that we have identified one fully functional location able to run the diabetes use-case (Table 5) and five pairs of locations able to run Barrett’s disease use-case (Table 6). The one able to run the diabetes use-case is the Wellcome Type 2 Diabetes study group in Scotland. The five locations that can run the second use-case are as follows: Finland, Germany (Bremen), Norway, UK (General Practice Research Database), UK, Scotland (pilot).

Principal findings

The TIRRE survey has been completed by 56 data-repositories across Europe and six outside the EU. We have developed a usable instrument which can assess their potential to take part in linked data research. There were no equivalent International sites available to conduct this type of research. A challenge was to get databases to complete the questionnaire, when we did get a response, the completeness of information gathered was high and proved useful in identifying their potential to participate in linked research. Meso- and macro-level questions were important discriminators between use-case and non-use-case eligible data sources. There are currently no other survey instruments available to enable brokerage between databases potentially willing to participate in research. Micro levels informed about the data and its granularity.

Implications of the findings

The TIRRE survey is the first step towards assessing the potential of a database for linkage. It can identify data sources suitability in terms of data availability and readiness to participate in a study. Whilst the initial focus of TIRRE was on linking data sources (which were important and consistent), the meso- and macro-factors generally had higher OR of predicting use-case eligibility.

Different coding systems have varying levels of granularity. For example, at the time of this study, neither ICD-10 nor ICPC differentiated between types of diabetes according to the latest WHO classification. ICD-10 differentiates insulin-dependent and non-insulin-dependent, rather than the Type 1 (insulin for survival) and Type 2 diabetes used in the latest classifications. Although we acknowledge, this is now updated in later releases.

Comparison with the literature

It is possible to draw comparisons between the complexity of this task and the existing successful projects that involve linking data. However, the successful data repositories in the UK have all been based on a single vendor of GP eHR system. Clinical Practice Research Datalink previously only extracted data from a single vendor called In-Practice Systems, though they are expanding this to all UK vendors;25 Q-Research on the EMIS system26 and other UK research networks (The Health Improvement Network27 and ResearchOne28) and other networks following the same pattern. The only exception to this in the UK is the Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC);29 this network extracts data from all the different brand of medical record systems. It has published a cohort profile about patients in the RCGP RSC database with diabetes, one of the TRANSFoRm use-case areas30 Notwithstanding the RSCHP RSC success, the relatively simple task of linking data from this small number of brands of computer within the UK has proved challenging, both in terms of creating a summary care record31 and in developing a common data extraction system.32

Limitations of the method

Any initial screening process will need to be followed up by a detailed assessment of whether the dataset needed for a given study can be elicited from the data repositories. There was no real incentive for data repositories to supply us with the data required, as there was not a reciprocal offer of benefit. As a consequence, our results inevitably underestimate the number of sites where this type of research can be conducted. We propose that future projects should consider including incentives in their budget. An effective method to reduce the impact of this self-selection bias could be to approach databases with a partially completed survey (using information available in the public domain) in order to encourage participation. Furthermore, the collected data could be shared publicly as a metadata registry that would facilitate advertising data offered by organisations for prospective studies. We also recommend limiting surveys to 30–40 questions to improve the response rate.

Call for further research

We need to conduct test-retest studies to assess the reliability of the survey instrument. The reliability test could be carried out by repeating the data collection after a period of time. While this would help to validate the instrument, it will also potentially remove any bias introduced by the specific person responding to the survey. We should conduct simulated and real studies with data extractions to test its validity. However, conducting real studies may be affected by the availability of funding. Alternatively, we can promote reuse of the instrument in other projects with the research area.