Validation of US CDC National Death Index mortality data, focusing on differences in race and ethnicity
•,,,,.
...
Abstract
Objectives The US Center for Disease Control and Prevention’s National Death Index (NDI) is a gold standard for mortality data, yet matching patients to the database depends on accurate and available key identifiers. Our objective was to evaluate NDI data for future healthcare research studies with mortality outcomes.
Methods We used a Kaiser Permanente Mid-Atlantic States’ Virtual Data Warehouse (KPMAS-VDW) sourced from the Social Security Administration and electronic health records on members enrolled between 1 January 2005 to 31 December 2017. We submitted data to NDI on 1 036 449 members. We compared results from the NDI best match algorithm to the KPMAS-VDW for vital status and death date. We compared probabilistic scores by sex and race and ethnicity.
Results NDI returned 372 865 (36%) unique possible matches, 663 061 (64%) records not matched to the NDI database and 522 (<1%) rejected records. The NDI algorithm resulted in 38 862 records, presumed dead, with a lower percentage of women, and Asian/Pacific Islander and Hispanic people than presumed alive. There were 27 306 presumed dead members whose death dates matched exactly between the NDI results and VDW, but 1539 did not have an exact match. There were 10 017 additional deaths from NDI results that were not present in the VDW death data.
Conclusions NDI data can substantially improve the overall capture of deaths. However, further quality control measures were needed to ensure the accuracy of the NDI best match algorithm.
What is already known on this topic
While prior literature describes quality control methods, and enhancements to the National Death Index (NDI) death registry algorithm for identifying true deaths, our study highlighted how race and ethnicity and sex were associated with the ability to match with this death registry for over one million people.
What this study adds
An evaluation of the NDI matching algorithm at a large integrated healthcare system highlighted the algorithm strengths and pitfalls. The NDI can provide significantly more mortality data than other US death registry sources, but the matching algorithm missed some deaths determined with other sources. Women, Hispanic and Asian/Pacific Islander populations were more frequent in poorly matched records compared to those well matched to NDI.
How this study might affect research, practice or policy
Large studies with mortality outcomes and incomplete follow-up should use multiple mortality databases and will benefit from a similar quality control of data from death registry sources.
Introduction
The National Death Index (NDI), managed by the National Center for Health Statistics (Hyattsville, Maryland, USA) of the Center for Disease Control and Prevention (CDC; Atlanta, Georgia, USA), is a database of death certificate data from the state vital statistics offices in the USA and territories. NDI is frequently used to ascertain deaths and cause of death in studies where participants may be lost to follow-up.1 Kaiser Permanente Mid-Atlantic States (KPMAS) is a large multisite healthcare system, which derives death data from databases sourced from the medical record, select state registries, and quarterly updates from the Social Security Administration (SSA) Death Master File. However, more complete information on vital status for members without a valid social security number (SSN) and cause of death on all deceased members was needed for medical research studies.
NDI provides a detailed user’s guide to submission and analysis of returned results.1 However, quality control and testing matching algorithm validity is the responsibility of the submitter. NDI returns a probabilistic score (PS) based on a matching algorithm that evaluates the likelihood of a match over nine demographic variables and their components.2 Researchers have reported different methods for identifying and verifying a true match to the NDI dataset given varying available data, by comparing it to their large institutional databases.3–6 Most report that submitting SSN is the best identifier with high sensitivity and specificity.4 Where SSN is not available or partially matched, researchers have developed algorithms to enhance NDI results or use alternative matching results.
Race and ethnicity may be differentially linked with the availability of SSN, affecting matching with the NDI database, potentially biasing mortality estimates.7 Hispanic people’s records appear to have higher missing SSN and other NDI fields compared with white and black people, possibly leading to inappropriate inferences about survival differences by ethnicity.7
Our objective was to produce a comprehensive decedents dataset for KPMAS that could be used for research queries on vital status and cause of death for over 1 million members. We evaluated the NDI PS matching algorithm and developed quality control steps for our racially diverse population.
Methods
Study population selection
The primary objective of our NDI submission was to find current vital status (by 31 December 2017) and the cause of death of a cohort of KPMAS members that were lost to follow-up, disenrolled or died without record in our electronic health record (EHR) system. We submitted members’ data naïve to death status on over one million members enrolled at KPMAS who had a date of last contact (identified by an outpatient encounter or blood pressure measurement) between 1 January 2005 and 31 December 2017. We calculated follow-up time from the year of last contact to 2018, equivalent to the number of years searched per person by NDI.
Patient data were obtained from the Virtual Data Warehouse (VDW) which is a database derived from the KPMAS EHR. We submitted fields required by NDI including first, middle and last names, SSN, birth month, day and year, sex, and state of residence and marital status. We did not submit race, state of birth or father’s surname (for women).
To account for significant missing self-reported race and ethnicity data, we used the Bayesian Improved Surname and Geocoding algorithm8 9 probabilities available in a Kaiser Permanente data repository. We combined the Asian and Pacific Islander (A/PI) populations into one group and combined the American Indian and Alaskan Native (AI/AN) populations with the multiracial population into an other group due to small sample sizes. We tested the association of missing SSN with race and ethnicity using a χ2 test and by adjusting for sex in a logistic regression analysis.
Evaluation of returned results and additional quality control
NDI returned record level data and summary statistics on people for our submission that matched records in their database. For the matched records, NDI provided information on which fields matched and an overall PS based on weighted matching. KPMAS members with scores above the cut-off were presumed dead because they matched well with someone with a death certificate in the NDI database. KPMAS members with scores below the cut-off were not considered a good match and so presumed alive. However, there is the possibility that some were actually a true match and should not be presumed alive, especially if the PS was borderline and a rerun with additional variables could increase the score. For people with multiple matching records, the PS provided a way to determine the best match. We evaluated each field for percent matched.
Quality assurance
NDI stratified groups of matching fields into classes. In brief, Class 1 requires matching at least eight digits of the 9-digit SSN, and the fields: first name, middle initial, last name, sex, state of birth, birth month, and birth year. Class 2 requires matching at least 7 digits of the SSN and one or more of the fields may not match. For Classes 3 and 4, SSN is unknown but for Class 3, eight or more items match (first name, middle initial, last name, father’s surname (for women), birthday, birth month, birth year, sex, race, marital status or state of birth) and for Class 4, fewer than eight of the items match. For Class 5, the SSN exists but does not match and all are considered false matches. The recommended cut-off values for the PS are 44.5, 37.5 and 32.5 for Classes 2, 3 and 4, respectively. The NDI User guide1 stated that evaluations of the PS algorithm revealed biases in the classification of NDI match status for women and non-white persons, due to changing surnames for women and ‘lower reporting of SSNs and incorrect spelling or recording of ethnic names’, particularly for Class 4 matches.1 We compared percent differences in all above and below cut-off PSs for sex and for imputed race and ethnicity using a χ2 test. For Class 4 within above and within below cut-off group, we used linear regression analyses to test the associations of PS with imputed race and ethnicity and sex, age at submission to NDI, state of residence and matched birthday, and matched birth year (everyone in Class 4 matched on birth month, so this variable was not included). For comparison, we did the same analyses with self-reported race instead of imputed race.
To determine additional deaths identified by NDI and potential mismatches, we stratified by above and below PS cut-off records and compared NDI results to the KPMAS VDW on vital status, and differences or no difference in death dates. We calculated validation metrics of sensitivity and specificity for the VDW compared with NDI as the gold standard.10 We used SAS V.9.4 for analyses, where statistical tests were two-sided with a significance level of 0.05.
Results
We submitted 1 036 449 people to NDI. Follow-up time from the date of last contact was 1–13 years with a median of 6 years and average of 6.3 years. However, since we had 69% missing self-reported race and ethnicity (online supplemental table 1), we imputed race and ethnicity resulting in populations that were 9% A/PI, 31% black, 13% Hispanic, 2% AI/AN/multiracial, 45% white and 0.3% not reported due to a missing address (table 1).
Table 1
|
Characteristics of the population submitted to NDI (n=1 036 449) and the population with scores above the cut-off (presumed dead) (n=38 862) and scores below the cut-off (presumed alive) (n=334 003)
Of the 1 033 477 people, with imputed race information, we found 143 452 (13.8%) were missing SSN. Percent missing SSN significantly differed among race and ethnicity (χ2, p<0.001), and a race–sex-adjusted logistic regression analysis showed all race and ethnicity categories were significant compared with the white population as a reference group, with Other having the largest OR (OR=14.3 (11.9, 17.1)) followed by the Hispanic population (OR=2.34 (2.30, 2.39)) and by the A/PI population (OR=1.56 (1.53, 1.60)). An analysis of self-reported data showed similar results, except that the Other group had a smaller OR (online supplemental tables 2 and 3).
We obtained summary statistics on 372 865 people that matched records in the NDI (figure 1). We found 663 061 people did not match the NDI database and 522 people were rejected for inclusion of special characters in the fields and one duplicate record was removed. Therefore, we classified 663 583 as alive according to NDI. However, there were 1188 deaths among these patients according to the VDW sources by 31 December 2017.
KPMAS submission matches the National Death Index and the process of inclusion. KPMAS, Kaiser Permanente Mid-Atlantic States.
Above PS cut-off (presumed dead)
Of the 372 865 people that matched records in NDI, we found 38 862 unique people had PSs above the class cut-off threshold (tables 2 and 3, figure 1).
Table 2
|
Exact match on submitted variables for records with scores above the cut-off (presumed dead)
Table 3
|
NDI matches (n=372 865) stratified by class and probabilistic score, with death dates compared with the VDW
There were no records in Class 1 because state of birth was not submitted. Class 3 had very few matches because father’s surname and race were not submitted and very few people submitted had marital status information. Table 2 shows that there was a high percentage of exact matches between the data submitted and the NDI database for each field within each class. Only middle initial and state of residence had less than 95% matching for all classes. A high percentage of above PS cut-off records (89%) were matched exactly by their 9-digit SSN.
Below PS cut-off (presumed alive)
Of the 372 865 people that matched records in NDI, we found 334 003 had PSs that were below the threshold for class, therefore considered poor matches and the KPMAS member was presumed alive (table 3). Most, (82.7%) were in Class 5 (people who had an SSN that did not match any SSN in NDI).
Comparison between NDI and the KPMAS VDW on death date
Comparing death dates more closely, we analysed the number of NDI records above the PS cut-off, where the death dates matched exactly with the VDW, were missing in the VDW or did not match the VDW (table 3, figure 1).
Of the 38 862 above PS cut-off records, there were 1539 (4.0%) records where NDI and VDW matched in deceased vital status but had non-matching death dates. The death dates differed with a median of 2 days, IQR (1–9 days), maximum of 10 212 days. Of these, there were 1421 that also matched exactly on first and last names, and birth month, and birth year and sex and 1021 that matched on all variables.
For the below PS cut-off records (table 3), we analysed possible matching on 1858 death dates between the VDW and NDI, due to the possibility that some of these NDI records were true matches and therefore could be linked to identifiable deaths and cause of death could be obtained. We found 870 records where death dates matched exactly to our VDW. Of these, 300 also matched exactly on first and last names, and birth month, and birth year and sex; there were only 17 that mismatched on day of birth but 139 had mismatches on state of residence.
Sensitivity and specificity
The submission to NDI provided an additional 10 017 records presumed dead compared with using our VDW alone at the time of submission (tables 3 and 4). Table 4 shows a comparison of deaths found with the NDI best score algorithm to the deaths found in the VDW for all records submitted. We had 74.2% sensitivity and 99.7% specificity on death matches, with NDI as the gold standard.
Table 4
|
NDI matches compared with VDW deaths by 31 December 2017 for complete submission
PSs stratified by sex and race and ethnicity
The above cut-off scores had a higher percentage of men (55.7%) compared with below cut-off scores (52.0%), p<0.0001. Imputed race and ethnicity were significantly different between above and below cut-off scores, p<0.0001: above scores had lower percentages of A/PI and Hispanic people and had higher percentages of white and black people (table 1). Similar results were seen for self-reported race (online supplemental table 1).
Class 4 matches are independent of SSN and dependent on name matching. In multivariate linear regression analyses of Class 4 scores for above and below the PS cut-off, we found sex and imputed race and ethnicity were significantly associated with PS in each group. In the above cut-off-score group, women had slightly higher scores compared with men (online supplemental table 4). Compared with the reference white population, the imputed A/PI populations group had 4.8 times higher PS, while scores for Hispanic people were not significantly different; similar results were seen for self-reported race (online supplemental table 4). In the below cut-off-score group, women had lower PS compared with men. The imputed Hispanic and black populations had higher PS while the A/PI group and the other group had lower PS compared with the reference white population, both statistically significant; similar results were seen for self-reported race, except that the black and other group did not have statistically different PS scores compared with the white population (online supplemental table 4).
Discussion
Data from the NDI can substantially improve the overall capture of deaths over other nationally available sources. We obtained information on an additional 10 017 deaths compared with the KPMAS VDW (sourced from the SSA Death Master File, some state registries and the local KPMAS EHR). However, there were deaths in the KPMAS VDW that were classified as presumed alive by the NDI algorithm. We observed variation in PSs with sex and race and ethnicity. The above cut-off scores had lower percentages of women, A/PI and Hispanic populations compared with the scores below the cut-off. For records not matched on SSN (in Class 4), NDI scores also varied by sex and by race and ethnicity.
The primary source of VDW data is the SSA death master file (SSA-DMF) which, while economic and refreshed quarterly, has been reported to have limitations in the capture of deaths, particularly for younger people and by state.11 12 The SSA-DMF became more limited when on 1 November 2011, the SSA removed 4.2 million protected state death records and continues to add 1 million fewer annually, as a result of changes in the Social Security Act (section 205(r)).12 13 Authors comparing NDI and the SSA Master File emphasise the importance of using multiple sources to ascertain death status for mortality studies and noted the more complete capture of death by NDI even prior to 2011.6 12 14
Deaths that appear in our local database but not in NDI could be explained by missing or mismatched fields in the submission which could result in a non-match or lower than expected PS. Geisinger notes that the NDI algorithm did not capture some known deaths in the World Trade Center Health Registry, with higher discrepancies among those missing SSN and non-white populations.15 Sayer16 showed that the NDI algorithm is sensitive to mismatch on exact first names when SSN is missing. In multiple matches to the database, the top scoring match may not be the true match. When we also examined ‘presumed alive’ matches, we found there were over 300 matched people who had exact matching on death date, first name, last name and birth month and year. State of residence appeared to be the most frequently mismatched field for these people. Additional cause of death on these probable matches could be obtained by resubmitting data to NDI.
The A/PI and Hispanic populations were smaller compared with the white and black populations at our institution and had a higher percent missing SSN. Arias et al (2016)17 studied misclassification of race and ethnicity on death certificates used in the National Longitudinal Mortality Study and reported there was accurate race and ethnicity reporting for white and black populations during the 1999–2011 period, but there was 40% misclassification for the AI/AN population and 3% misclassification for A/PI and for Hispanic populations. When we focused on the Class 4 population which were missing SSN, we also found differences in the scores by sex and by race and ethnicity, particularly for the A/PI and Hispanic populations compared with white population. In the above-cut-off Class 4 group, the A/PI had much higher PSs compared with the white population, but the reasons for the higher scores are unclear, possibly due to less variability or missingness for other matching variables. While matches for below the cut-off may be discarded, slight changes in information available on race, sex and state of residence may be influential in accepting a match for borderline cases.
There were some limitations to this study. Several variables that could have improved matching to NDI were not used or were unavailable or were very limited at the time of submission including: self-reported race and ethnicity, father’s surname, state of birth and marital status. There may have also been deaths missed due to a reporting lag by the states to NDI. Further, a review of death certificates would have been more definitive for the questionable matches. The primary strengths of this study were the large sample size, a diverse population and the ability to identify key identifiers for high scoring matches to NDI. We also demonstrated that imputed (and self-reported) race and ethnicity is correlated with missing SSN and that both race and ethnicity and sex (directly or indirectly through surname) impact matching to NDI.
Conclusion
In conclusion, NDI complements other sources of death data and provides increased information on vital status and cause of death. Other researchers using NDI data may benefit from a comparison group of known deaths either from SSA or a manual validation of internal data. At our institution, for records with a deceased vital status that had scores above the cut-off, over 90% matched on at least 7 digits of the SSN. However, it is important to investigate quality control measures by NDI class and matching variables, particularly for people missing SSN or with specific ethnic backgrounds. This validation step is essential to ensure the accuracy of the NDI best match algorithm and to obtain the maximum return on data submitted to NDI.
