Reliability of administrative data to identify sexually transmitted infections for population health: a systematic review

Introduction

Surveillance is the cornerstone of public health practice as well as research.1 Public health surveillance involves the systematic collection, analysis, interpretation and dissemination of health-related data to inform population health policies, programmes and interventions. The most prevalent type of data used by public health authorities to identify population health trends, examine healthcare access and assess population level outcomes is administrative data.

The term ‘administrative data’ in healthcare refers to data generated during routine healthcare delivery processes,2 which includes but is not limited to outpatient encounters, hospital admissions and pharmacy dispensing events. These data are used for a range of administrative purposes in healthcare, such as billing payers for healthcare services and measuring the efficiency of healthcare delivery. Administrative data include patient demographic information (eg, age, race), insurance plan enrolment (eg, Medicare, Medicaid, Blue Cross/Blue Shield, etc), hospital discharge information (eg, reason for admission, discharge disposition), procedures delivered during an outpatient encounter and pharmaceutical claims (eg, medication dispensed, route of administration). Administrative data are typically structured using information coding standards to enable interpretation and analysis. As a result, they represent accessible, available data suited for secondary purposes such as population health research.

Accessibility and availability are two important dimensions of data quality,3 yet they are only part of the quality equation with respect to population health research.4 To be of high quality, administrative data must also be accurate.5 Otherwise health policies, programmes and interventions that derive from observational research will be developed on false premises and may therefore fail to prevent disease, prolong life or promote health.

In the context of public health surveillance at the national level, scientists use diagnostic codes from administrative data in the form of International Classification of Diseases (ICD) codes to identify disease-specific cohorts for assessment of population level trends and outcomes. While other coding systems, such as the Logical Observation Identifiers Names and Codes, are used to electronically report positive disease cases to local and state health authorities,6 7 national surveillance systems lack these data as they are often removed when cases are reported to federal information systems. Moreover, scientists at federal health agencies are unable to measure positivity, adherence to testing guidelines and other population indicators when data reported to state health authorities come from just those individuals with a disease. Robust public health surveillance and research requires accessible, available and accurate data for population numerators and denominators. Therefore, national public health scientists leverage large, population health data sets that consist primarily of administrative data.

Existing literature on the accuracy of ICD codes in administrative data is mixed. Whereas a recent Canadian study found that ICD codes had a low positive predictive value (PPV) of 16% with respect to the identification of pertussis cases,8 an earlier study in Canada found that this data possessed high sensitivity (96.2%) and specificity (99.6%) with respect to the identification of HIV infection.9 The ability of ICD codes from administrative data to reliably identify true cases of disease therefore appears to vary, suggesting that validation of administrative data is important to conduct for all diseases of public health importance.

Sexually transmitted infections (STIs) are an important public health challenge and a focus of the Healthy People 2020 goals.10 Undiagnosed and untreated STIs are associated with adverse outcomes such as infertility, pelvic inflammatory disease, chronic pelvic pain, HIV acquisition, neurologic disease and adverse pregnancy outcomes. Several STI health services are recommended by the WHO to protect the reproductive and sexual health of men and women, yet referral to and performance of these services requires accurate identification of STI incidence, prevalence and outcomes. Chlamydia, gonorrhoea and syphilis are the most prevalent and curable STIs reportable under public health laws in the USA11 and more than 1 million STIs are acquired every day worldwide.12 Accuracy of administrative data for identifying cases of chlamydia, gonorrhoea and syphilis is unknown.

Methods

To examine the existing literature on the validity of using administrative data to identify STI cases, we conducted a systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis. We performed a comprehensive search of the literature using the MEDLINE and Scopus databases for peer-reviewed articles published before February 2018. We limited the scope of our review to chlamydia, gonorrhoea and syphilis, as these are the STIs that constitute the highest reportable morbidity for local health departments.

Articles were identified using search terms consisting of search strings as well as Medical Subject Heading (MeSH) terms associated with the key words. We used STI-related search terms including ‘syphilis’, ‘gonorrhoea’, ‘chlamydia’, ‘pelvic inflammatory disease’, ‘sexually transmitted disease’ and those related to administrative codes such as ‘international classification of diseases’, ‘administrative codes’, ‘ICD’, ‘diagnostic codes’, ‘diagnosis, classification’, ‘international classification of diseases’, ‘ICD code’, together with terms intended to identify validation studies: ‘validation studies’, ‘reliability’ and ‘validity’. Pelvic inflammatory disease (PID) was included as previous literature indicates that 33%–50% of all PID cases are due to either Chlamydia trachomatis or Neisseria gonorrhoeae.13 14

Only empirical publications appearing in peer-reviewed English language journals were included. As such, we excluded articles classified as letters to the editor, policy briefs, perspectives, commentaries, summaries of future research plans, as well as grey literature. Additionally, articles without abstracts were also excluded.

Article identification and selection followed the process outlined in figure 1. We used Covidence (Melbourne, Victoria, Australia), a web-based tool developed for systematic reviews by Cochrane, to facilitate article selection and review.

Request permission

In the first step, articles were reviewed with a focus on title and abstract. We eliminated articles that did not focus on STIs or did not validate STI administrative data as a component of the research. This included validation studies focused on conditions or diseases other than STIs. Articles were included if they focused on any combination or all of the STIs of interest, or on PID, and were validation studies focused on diagnostic testing or administrative codes.

Screened articles were then subjected to a second review which focused on including only validation studies examining the use of either ICD, Ninth Revision, Clinical Modification (ICD-9-CM) or ICD, 1oth Revision, Clinical Modification (ICD-10-CM) codes to identify diagnosis of chlamydia, gonorrhoea, syphilis or PID. Other forms of administrative data, such as pharmacy claims, to identify cases were not considered because the target STIs of interest are often treated with common antibiotics such as azithromycin which is also used to treat a number of other bacterial infections. Moreover, studies that focused on the validation of specific laboratory assessments such as PCR assays for diagnosing chlamydia were also excluded.

Two reviewers (SR and YH) reviewed each article to determine inclusion with conflicts in judgement reconciled by consensus. Finally, in order to be exhaustive, we used a snowball technique whereby we reviewed all references found in the bibliographies of included articles.

The final set of articles included for review met the following criteria: (1) the study included syphilis, chlamydia or gonorrhoea; (2) assessed and listed ICD-9-CM or ICD-10-CM codes and (3) measured accuracy, sensitivity, specificity, PPV and/or negative predictive value for ICD codes related to any of the three STIs. With the selected articles, references from each article were examined to identify other relevant articles.

Results

Our search strategy identified 1754 unique articles. Of these, only five (0.29%) studies met initial inclusion criteria for full-text review based on review of title and abstract. After full-text review, only two (40%) articles met the criteria for inclusion in the systematic review.15 16 The three articles that did not meet inclusion criteria focused on validating Healthcare Effectiveness Data and Information Set (HEDIS) measures for estimating screening rates for chlamydia among young women.17–19 HEDIS measures utilise administrative data, yet the identified articles were removed because the assessment focused on identifying whether individuals were tested for chlamydia rather than on positive diagnosis of disease. Finally, the two included articles resulted in 60 references identified using the snowball technique. No additional articles were identified in this step in the review process.

The two articles which met the inclusion criteria focused on the validation of administrative data to detect PID. Neither study examined syphilis nor did either study examine the accuracy of administrative data to diagnose chlamydia or gonorrhoea independent of PID. Furthermore, both studies examined only ICD-9-CM codes, because the data used were prior to October 2015. The table 1 presents the list of ICD-9-CM codes used in these articles to identify potential cases.

Ratelle and colleagues (2003) assessed women at a group practice in Massachusetts from 1995 to 1997.16 A cohort of 1051 patients with PID was identified based on ICD-9 codes (see table 1). From this cohort, chart reviews were conducted on de-identified medical records for a random sample of 296 patients focused on the 614.9 (unspecified inflammatory disease, female pelvic organs) code. Chart reviews were limited to the first encounter for each patient with PID with 72.5% of patients having one encounter for the ICD-9 code of interest. The study identified 39 cases which met US Centers for Disease Control and Prevention (CDC) case criteria for PID, resulting in a PPV of 18.1%. For each PID case, reviewers examined whether patients were tested for chlamydia or gonorrhoea. Most patients were tested for chlamydia (84.3%) and gonorrhoea (82.8%).When the PID diagnostic code was used in combination with a positive laboratory test for chlamydia or gonorrhoea, the PPV for PID was 56% (five out of nine cases) and 100% (four out of four cases), respectively. These determinations were based on laboratory testing results for chlamydia or gonorrhoea rather than use of ICD-9-CM codes for these diseases.

The second article by Satterwhite and colleagues (2011) assessed validity of a PID case-finding algorithm among women aged 15–44 years at healthcare organisations in Washington and Colorado from 2003 to 2007.15 Potential PID cases were identified from both healthcare organisations using ICD-9-CM codes (see table 1). In order to identify PID cases associated with chlamydia and gonorrhoea, only non-chronic cases were considered. A total of 2764 and 2685 potential PID cases were identified from Washington and Colorado, respectively. Chart reviews were conducted on 393 cases from Washington and 500 cases from Colorado. Relevant data were extracted from medical records during the chart review process including the ICD-9-CM codes. Data from Washington were used to develop the PID case-finding algorithm that used several variables from administrative data including ICD-9-CM codes to identify cases; this algorithm was then tested on the Colorado data. Using ICD-9-CM codes alone, the PPV of identifying PID in Washington was 78.8% while that in Colorado was 79.1%. When supplemented with other administrative data (eg, age at diagnosis, inpatient admission, etc) PPV increased to 87.9% in Washington and 84.5% in Colorado. Sensitivity was high at 96.4% in Washington and 90.3% in Colorado. In contrast, specificity was low at both Washington (45.9%) and Colorado (37%). The validity of chlamydia was not determined using either laboratory testing data extracted from the EHR or ICD codes.

Discussion

Our systematic review identified two population health studies that evaluated the validity of using diagnostic codes to identify PID. No studies could be identified that examined the validity of using ICD codes to identify positive diagnosis of chlamydia, gonorrhoea or syphilis. The reviewed studies reported a wide range of PPVs for PID using only diagnostic codes from 18.1% to 79.1%. The PPV range increased to 56%–100% when additional data were considered in addition to an ICD diagnosis. These findings suggest there is sparse evidence on the reliability of using administrative data to identify STI cases in the conduct of population health research.

Conclusion

Based on a review of the literature, there is scant evidence on the reliability of using administrative codes to identify cases of chlamydia, gonorrhoea or syphilis. In the available literature, we found high variability in the predictive value of using administrative codes. Given these findings, further studies are required to examine the predictive value of administrative codes for all three diseases in the general population as well as high risk populations, including pregnant women and men who have sex with men.