Accuracy of screening tools for Pap smears in general practice

Introduction

With the development of computer systems in Australian general practice, the potential has come for increased access to primary healthcare information on a much larger scale1 and the potential for evidence-based information to improve healthcare.2 This study attempts to explore the accuracy of Pap smear data available through a primary care database and accessible via an extraction tool.

It has been recognised that data extraction tools (DETs) have limitations relating to data entry and incomplete data.3 Current DETs do not work with all clinical software programs and the way in which the software architecture of clinical systems has been established does not easily lend itself to extraction of data.1 In this study, the screening rate obtained via a DET was compared to the rate obtained using a practice audit of patient files for a large multi-general practice (multi-GP) practice in regional New South Wales.

Although there has been a decline in the incidence of cervical cancer in Australia, there are still a substantial number of women in New South Wales who are either under-screened or who have never been screened,4 particularly rural women.5 One barrier to addressing this issue is the quality of available data relating to under-screened women and previous studies have shown variations in rates depending on the source of the data.6

This cross-sectional study attempted to explore the reliability of Pap smear data available through a primary care databases and accessible through a DET compared with that of data available from manual audit.

Methods

The focus of this report is a large multi-GP practice in a large regional centre in NSW, Australia. Two DETs were trialed but only one was compatible with the practice software. The current rate of Pap smears for all women aged 20–69 was determined via the compatible DET using multiple search terms and synonyms for Pap smear (Table 1).

A manual audit of files from a random sample of 100 patients within each of the four categories (listed in Table 2) output by the DET was undertaken to establish the accuracy of the tool in terms of recording Pap smears. This audit involved inspection of the pathology requests and results and review of specialists’ letters and other scanned documents. Outcome measures included the identification of relative rates of Pap smears and deficiencies in recording within the practice (number of women in the practice with hysterectomy, Pap smear from an outside source and Pap smear from another pathology provider). Sensitivity and specificity were calculated for the four groups as a whole. Rates were compared by Chi-square tests and analysed using OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version 2.3.1 (Atlanta, USA). A two tailed p-value of <0.05 was considered significant.

RESULTS

Screening rates for eligible female patients are presented in Table 2. The DET indicated that more than half of all eligible female patients (aged 20–69, n = 2625) had never had a Pap smear, and that only 348 (13.3%) had had a Pap smear within the last two years, the current guidelines for cervical cancer monitoring.

A random sample of 100 patients from each of the four groups listed in Table 2 was audited. Overall, the DET correctly identified 161/163 patients (98.8% specificity) as never having had a Pap smear. The DET only detected the most recent Pap smear in 161/237 patients (67.9% sensitivity). The DET picked up an earlier smear in 66/237 (27.8%) of patients.

It was found that 7% of those patients who were recorded as never having had a Pap smear by the DET had actually had a Pap smear (Table 3). The DET either missed the Pap smear altogether or there was a phone/specialist record of the Pap smear having been conducted elsewhere. Of those patients who had not had a Pap smear in ≥ 4 years, 20% had actually had a Pap smear. The DET failed to pick up the most recent Pap smear in 18% of patients. In those patients who had had a Pap smear in the last 2–4 years, this failure to pick up the most recent Pap smear increased to 42%.

If the proportion of patients indicated in the random sample (42% of patients being wrongly recorded as having had a Pap smear between two and four years ago, when in fact, they should be included in the Pap smear within the last two years category) is applied to the original population of 302 patients having had a Pap smear within the last 2–4 years, this suggests that 127 additional patients should be included in the within the last two years category. This brings the updated Pap smear rate within two years in Practice A to 18.1%, which is significantly higher than the rate indicated by the DET (X²=23.24, p<0.001).

Those patients who had Pap smears that were coded differently (Table 1) between time intervals (from different pathology providers or from within the same provider but using a different code) were less likely to have had their most recent Pap smear picked up by the DET (p <0.001). No trend could be observed in the DET’s preferences for picking up or failing to detect smears from specific pathology providers.

DISCUSSION

The average biennial cervical cancer screening rate in NSW for the 2009–2010 reporting period is 56.5%,7 with reportedly lower participation rates in rural and remote communities.8 The local health district is reported to have a significantly lower biennial screening rate [54.0 with 95% CI (53.6–54.4)] than the state average.

It was anticipated that the use of a DET would allow for rapid assessment of the cervical cancer screening rates in the practices and accurately reflect the participation rate in cervical screening. Such information would be useful in the effort to increase the rate of women following the current recommendation to undergo screening biennially. However, the DET failed to report accurate results, significantly underestimating the true level of cervical screening (X²=23.24, p=0.001).

One problem associated with using DETs may be the inability to archive information that is no longer relevant, for example woman who have left the practice. Essentially, data need to be accurate and complete, in addition to being entered in the correct field within the clinical software9 and in a format useable by the DET.

Of those patients identified by the DET as not having had a Pap smear, seven of 100 random patients had actually had a Pap smear. Three patients had phone/specialist records of Pap smears conducted elsewhere, which may account for the failure of the DET to pick up these results. Although not directly a problem of the DET, this does limit its usefulness in clinical practice, highlighting the importance of complete/accurate clinical data. The audit failed to identify a reason for why the other four patients were not identified.

The tool extracts data from two sources. There is a manual entry field called ‘Last Pap Date’, in addition to atomized pathology data that are extracted. The atomized pathology data are limited to test names that have been entered into the program. Pap smear results do not auto-populate in many clinical software programs and manual entry is time consuming and can be open to error.10 If these data are not entered manually, Pap smear rates determined via DETs will be underestimated.10

This appears to be a more significant problem with Pap smear data than other pathology tests, and accuracy of general practice records in terms of Pap smears has been previously questioned. Laurence et al.11 found that the cervical screening rate determined using only immediately available electronic medical records (EMRs) indicated a low screening rate in participating practices (45%). However, telephone follow-up and adjustments to the denominator indicated that the rate was 86%. In the current study, the DET identified that 13.3% of women in the practice had had a Pap smear within the previous two years; however, available Practice Incentives Program data stated that 63.5% of whole patient equivalents at this practice had claimed the Medicare item number for a Pap smear during the same reporting period. This is 50% points greater than the rate reported by the DET, highlighting the current limitation of the tool.

Schattner et al.12 suggest that there is a need to improve eHealth data transmission to increase accessibility of clinical data by DETs for Pap smear results, in addition to improving the functionality of DETs themselves.

Increasingly, DETs are being used for research and audit of general practice. A role for DETs has been suggested for gathering such diverse information from electronic patient databases as patient demographics, disease and risk factor profiles, immunisation rates and cancer surveillance.13 However, examples of their use often make no comment about the accuracy of the tools in terms of extracting clinical data despite the recognised limitations of using DETs.13^–16

CONCLUSION

The results from this study highlight the deficiencies in current electronic recording of Pap smears within general practice. DETs should be used with caution as they may underestimate the rate of Pap smears from EMRs.