Article Text
PDF
Abstract
Objectives Electronic medical record (EMR) tools can identify specific populations among hospitalised patients, allowing targeted interventions to improve care quality and safety. We created an EMR alert using readily available data elements to identify hospitalised people with HIV (PWH) to facilitate a quality improvement study intended to address two quality/safety concerns (connecting hospitalised PWH to outpatient HIV care and reducing medication errors). Here, we describe the design and implementation of the alert and analyse its accuracy of identifying PWH.
Methods The EMR alert was designed to trigger for at least one of four criteria: (1) an HIV ICD-10-CM code in a problem list, (2) HIV antiretroviral medication(s) on medication lists, (3) an HIV-1 RNA assay ordered or (4) a positive HIV-antibody result. We used manual chart reviews and an EMR database search to determine the sensitivity and positive predictive value (PPV) of the overall alert and its individual criteria.
Results Over a 24-month period, the alert functioned as intended, notifying an intervention team and a data abstraction team about admissions of PWH. Manual review of 1634 hospitalisations identified 18 PWH hospitalisations, all captured by the alert (sensitivity 100%, 95% CI 82.4% to 100.0%). Over the 24 months, the alert triggered for 1191 hospitalisations. Of these, 1004 were PWH hospitalisations, PPV=84.3% (95% CI 82.2% to 86.4%). Using fewer criteria (eg, using only ICD-10-CM codes) identified fewer PWH but increased PPV.
Conclusion An EMR alert effectively identified hospitalised PWH for a quality improvement intervention. Similar alerts might be adapted as tools to facilitate interventions for other chronic diseases.
- Electronic Health Records
- Common Data Elements
- PatientIdentification Systems
- Infectious Diseases Medicine
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- Electronic Health Records
- Common Data Elements
- PatientIdentification Systems
- Infectious Diseases Medicine
WHAT IS ALREADY KNOWN ON THIS TOPIC
Electronic medical record tools may be able to identify special hospitalised patient populations in real time for quality and safety interventions, research or other purposes.
WHAT THIS STUDY ADDS
We incorporated diagnosis (ICD-10-CM) codes, laboratory results and medication lists into an electronic medical record inbox message alert that accurately identified hospitalised persons with HIV for a quality improvement study.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
When designing an alert, balancing its sensitivity and specificity is a function of the criteria used, with ICD-10-CM codes having the highest utility for identifying persons with HIV. Iterative testing of individual criteria is important to improving the accuracy of an electronic medical record alert.
Introduction
In the USA, people with HIV (PWH) are hospitalised at a rate 2–3 times the general population.1–4 In the past decade, over 90% of these hospitalisations have been for conditions (non-AIDS-defining conditions) not typically associated with HIV.3 Two leading quality and safety concerns among hospitalised PWH are low rates of engagement in outpatient HIV care and high rates of inpatient antiretroviral medication prescription errors.5–10 In 2017, we initiated a trial to evaluate the ability of a hospital HIV Support Team (HST) to address both issues for patients admitted to our large, academic, urban hospital. The team consisted of a nurse and an HIV-specialist pharmacist who met hospitalised PWH at the bedside.
To facilitate this work, we developed a novel electronic medical record (EMR) alert to identify PWH among all adult hospital admissions. The alert was based on readily available discrete data elements in the EMR and was designed to notify two groups of people by EMR message: (1) our HST (‘intervention team’) in real time and (2) a data abstraction team to collect data from charts captured by the alert. In this report, we describe the design of our EMR alert, explore challenges in its implementation and analyse the sensitivity and positive predictive value (PPV) of the alert’s criteria for identifying admitted PWH. We plan to report results of the HST’s effects on HIV care engagement and medication errors in future manuscripts.
Methods
The trial evaluating the HST’s effects was designed as a phased, cluster-randomised (‘step-wedge’) trial with each of six randomly determined clusters of nine hospital admitting services becoming successively included in the intervention group over contiguous 4-month periods. PWH hospitalised on services not yet included in the intervention group served as controls. We designed the EMR alert to identify all PWH on both intervention and control services. In this manuscript, we describe the EMR alert’s function in identifying PWH on all services included in the randomised trial. Further description of the methods and results of the trial itself will be the focus of future manuscripts.
Our hospital uses Epic corporation’s Hyperspace software as its EMR in both inpatient and outpatient settings. Our alert, programmed by an Epic physician builder within our health system (TG-B), screened records of adult patients admitted to inpatient status (excluding observation hospitalisations) to determine if they met criteria of PWH. We excluded observation hospitalisations (<48 hours) because these might be too short for the HST to be effective. The alert’s output consisted of a message with patient name and medical record number delivered to the EMR’s ‘In-Basket’ system.
The alert was designed to trigger for any one of four criteria, chosen to identify PWH using discreet EMR data elements: (1) an HIV International Classification of Diseases 10th Revision (ICD-10-CM) code (B20, Z21, O98.711–O98.73) in any current or prior outpatient or inpatient problem lists, (2) any antiretroviral therapy (ART) medication(s) which can be used (but are not necessarily specific) for HIV (identified from pharmaceutical subclasses maintained and updated externally by the EMR vendor, online supplemental S1) on the patient’s current inpatient or historical outpatient medication lists, (3) an HIV-1 RNA level assay ordered (regardless of result) during the hospitalisation or (4) a positive HIV-antibody (Ab) result during the hospitalisation or at any prior point in time. The third criterion was intended to capture both instances of the clinical team performing virological monitoring of individuals with diagnosed HIV, (in which case the result could be either detectable or undetectable) and instances of diagnosing acute HIV infection during the hospitalisation (in which case the result would be detectable). We did not look at HIV-1 RNA level testing prior to the hospitalisation because we felt this may introduce a high number of false-positives due to prior attempts at diagnosing acute HIV, and because we felt these individuals would be well captured by the fourth criterion.
Supplemental material
We initially considered a fifth criterion, a laboratory order for a CD4 cell count during the index admission. In 2 weeks of predeployment testing (40 hospitalisations alerted), this criterion triggered for 3 hospitalisations of HIV-uninfected persons, all of whom had CD4 cell counts ordered to assess immunodeficiency in the setting of cancer chemotherapy. The CD4 criterion did not identify any PWH who were not identified by one or more other criteria; thus, it was eliminated as an alert trigger for subsequent hospitalisations.
In May 2017, we deployed the alert and began the randomised trial. We reviewed all alert instances in the first month of deployment (approximately 60 charts). We identified a single instance where the alert was activated by the HIV-1 RNA criterion but failed to recognise that the patient also had a positive antibody. In this case, we identified and fixed a coding error that resulted in an unintended upper age limit for the antibody criterion. We then considered our specifications for the HIV alert criteria finalised. The alert build and post go-live support required 74 total hours of physician builder time.
The alert separately notified a data abstraction team tasked with confirming the patient’s HIV status (through reviewing chart notes and/or lab results) and the intervention team (HST members), who used the alerts to know which patients to see at the bedside. Rather than manually define individual data abstraction team members and intervention team members, the alert sent messages to separate recipient pools for each of these teams. Individuals could be added and removed from each pool as needed for team member turnover.
To analyse sensitivity, two nurses (EH and KH) conducted manual chart reviews of hospitalisations selected without regard to the EMR alert. A manual review of all adult patients admitted during the 2-year intervention period (approximately 100 000 hospitalisations) was beyond our capacity, so we collected a random sample of charts over a 4-month interval during the midpoint of the intervention period, from admitting services that averaged more than 10 hospitalisations among PWH per year. We aimed to review 1500–2000 charts, approximately 3%–4% of annual hospital volume. The protocol for each review began with reading the admission history and physical note and the most recent progress note looking for HIV (or AIDS) described as an active or historical diagnosis. If there was no indication of HIV (or AIDS) in the clinical notes, the reviewers then examined laboratory, medication and problem list chart sections and finally screened outpatient visits for any visits at the hospital-affiliated HIV clinic.
We also determined the proportion of hospitalisations identified by the full alert that was identified by each alert criterion alone and in two-way combinations. Assuming the full alert would approach 100% sensitivity, these results would then approximate sensitivity estimates for the individual criteria.
For the analysis of PPV, we started with the abstractor team’s manual reviews of each alerted hospitalisation indicating whether the patient was, indeed, living with HIV. We then performed a secondary review of all 199 hospitalisations the abstractors initially classified as false-positives (not having HIV infection despite the alert triggering). The secondary review involved detailed examination of current and prior discharge summaries, inpatient progress notes, outpatient office visits, medication lists and a search of outside records (available through EMR links) including antibody measurements and viral loads. Finally, we performed a retrospective EMR query of the charts that were identified by the alert to analyse which criteria (ICD-10-CM, ART, HIV-1 RNA, antibody or a combination) activated the alert.
Through this post-deployment analysis, we observed that the antibody criterion had never alerted, confounding our expectations. On a targeted review of a sample of 10 charts, 4 of which were known to have a positive antibody result within our EMR, we determined that the alert, as coded, was not successfully capturing antibody tests. Our 2017 audits did not include charts in which the only positive criterion was the antibody, and thus we failed to identify this issue prior to or during the intervention period. The error appears to have originated from failure of a function within the EMR to translate textual results of antibody tests into discrete normal/abnormal data.
To evaluate the potential impact of the antibody criterion failure, we determined from our retrospective EMR query that only 95 of 110 028 (0.09%) adult hospital admissions during the study period (29 unique patients) were associated with a positive HIV antibody and no other criteria. Thus, we considered the percentage of charts missed by the failure of this criterion to be negligible. We limited our analysis of individual criteria to the information available for the three functioning criteria: ICD code, HIV-1 RNA or prescription of any ART. We performed analyses using Stata V.16.1 software (StataCorp)11 with an α value of 0.05 for significance and CI calculations of sensitivity and PPV.
Results
Between May 2017 and May 2019, the EMR alert met criteria for 1191 hospitalisations among 849 unique patients. The majority of identified patients were male (63.0%), and black (72.4%) with a median age of 53.2 (IQR 41.6–60.6) years (table 1). During the 24-month intervention period, 671 (79.0%) patients were hospitalised once, 107 (12.6%) were hospitalised twice, 32 (3.8%) three times and 39 (4.6%) four or more times.
Demographic characteristics of patients identified by EMR alert
Our random sample to assess sensitivity comprised 1634 hospitalisations (approximately 3% of typical annual hospital volume). Among these hospitalisations, we identified 18 PWH admitted to inpatient status (with a total of 18 hospitalisations over the review period). The three-criteria-based alert identified all 18, yielding a sensitivity of 100% (95% CI 82.4% to 100%).
Using all three criteria, the alert was activated in 1191 instances, of which 1004 were true-positives, PPV=84.3% (95% CI 82.2% to 86.4%) (table 2). Using only two criteria (ICD code and ART) identified 988 (98.4%) of the 1004 true-positives, with a higher PPV of 94.2%. Using only ICD codes identified 947 (94.3%) of all true-positives, and further increased PPV to 99.1%. Results for other individual criteria and combinations are shown in table 2. The HIV-1 RNA criterion had the most false-positives, 141 out of 1078 (13.1%) alerts, due to this assay being used to evaluate for acute HIV infection and resulting negative. The 59 false-positive (6.1% of 960 total) ART alerts were for instances of individuals taking antiretrovirals for HIV pre-exposure prophylaxis (n=46), HIV post-exposure prophylaxis (n=4), treatment of Hepatitis B (n=3, three medications are approved by the U.S. Food and Drug Administration for both viral infections), or a mixture of indications (n=6) including experimental colorectal cancer treatment11 or research protocols for HIV pre-exposure prophylaxis. The nine false-positive ICD code instances were errors in entering HIV into the problem list, typically for individuals undergoing testing for HIV or receiving pre-exposure prophylaxis. The ICD-10-CM codes in these cases were B20 ‘HIV (HIV) disease’ (n=8) and Z21 ‘Asymptomatic HIV infection’ (n=1). Cases we evaluated in postdeployment analysis (approximately 2 years after the trial concluded) had already been corrected in the EMR by removal of HIV from the problem list.
Rates of true and false-positive detection of HIV by EMR alert and individual criteria
Discussion
This study has several important findings. First, using a combination of readily available discrete data (ICD-coded problem lists, medication prescriptions and disease-specific laboratory test orders), the EMR alert achieved both a high sensitivity and PPV, correctly identifying most admitted PWH. Second, our results demonstrate the impact of the number and choice of criteria on the balance between PPV and sensitivity of an EMR alert. Finally, our experience offers several practical lessons from successes and limitations of the implementation of our alert.
We demonstrated a high sensitivity and PPV of EMR alerts to accurately determine persons with HIV in order to facilitate a prospective evaluation of a quality improvement intervention. The same alert (potentially with a repaired antibody criterion) could be used to facilitate additional quality improvement interventions for hospitalised PWH. Indeed, there is a growing body of evidence that EMR data can be used to improve diagnosis, linkage and engagement in care for PWH.12–15 This analysis adds to that literature by determining sensitivity and PPV of an EMR alert and its individual elements. Because the EMR data elements should be similar if not identical in versions of the same EMR software used at different health systems, this alert might be easily reproduced and used by others for quality interventions among PWH. Finally, we propose that similar alerts might be used for quality and safety interventions in other chronic diseases. We are currently evaluating the performance of an alert using ICD code, medication and laboratory testing data to identify persons with chronic Hepatitis C virus (HCV) infection at the time of hospital admission. Such an alert could be used to target efforts to engage these individuals in outpatient care and to initiate curative HCV therapy. We suspect alerts based on similar discrete data elements might also be applied to non-infectious chronic diseases such as diabetes, inflammatory bowel disease and rheumatological disorders in order to deploy multidisciplinary teams for quality interventions including medication reviews, patient education, targeted case management and/or outpatient linkage to follow-up.
Our findings also highlight the importance of criteria selection. Specifically, as a single criterion, ICD coded problem lists captured the most patients with the highest PPV compared with either HIV-1 RNA or ART. For individuals with previously diagnosed HIV and prior contact within our health system, it is not surprising that ICD codes entered into problem lists by providers would be highly accurate. Conversely, the CD4 criteria introduced false-positives without adding sensitivity beyond the other criteria, so it was eliminated during early revisions of the alert. The HIV-1 RNA criterion also introduced a notable number of false-positives; however, we considered it important to identifying newly diagnosed PWH who were in-need of initial linkage to care. Our results demonstrate that adding criteria identifies more patients but increases false-positives. The right balance between sensitivity and PPV may vary for different diseases and intended purposes of identifying patients.
Related to creation and implementation of the alert, several design aspects of the alert are notable. The use of EMR medication classes for the ART criteria minimised upkeep, as these classes are updated independently by the software manufacturer, obviating the need to manually update the alert when new medications were brought to market. A disadvantage of their use is the lag between marketing approval for new drugs and updating the classes. However, based on our observation with one recently approved antiretroviral (fostemsavir), we suspect this duration is typically a matter of days or weeks. The use of EMR recipient pools facilitated conducting an interventional trial by having separate data abstractor and intervention team pools and simplified EMR programming when study personnel transitioned.
Our experience with the alert also offered several valuable lessons related to the iterative nature of quality improvement. The original alert included an HIV-antibody criterion, designed to capture newly diagnosed PWH or PWH who otherwise may not ever have engaged in outpatient HIV care such that any provider added an HIV diagnosis to their problem list or prescribed ART. Early iterations of the alert were repeatedly tested, with manual chart reviews of identified patients. Despite this, it was not until later review of a larger sample of charts that we learnt that the HIV-antibody criterion was not functioning as intended, demonstrating the importance of frequent testing and possibly of targeted exploration of each individual criterion and various criteria in combination. Overall, however, even missing one criterion, the alert in this case remained effective at identifying our target patient population.
The main limitation of the assessment of the sensitivity of the alert was the sample size, as this required a manual chart review. While over 1600 charts were reviewed, amounting to approximately 3% of annual inpatient volume, only 18 patients were PWH that could contribute to a sensitivity calculation. Our study was performed at a large, academic, urban medical centre with a high prevalence of HIV and may not generalise to other care systems.
Conclusion
In summary, EMR alerts have significant potential as tools to identify PWH when hospitalised. The use of such alerts can facilitate the deployment of multidisciplinary inpatient teams for medication review, education, targeted case management and outpatient linkage to follow-up. Our approach using readily available discrete data elements could potentially be applied to other chronic illnesses to facilitate quality and safety interventions. The selection of criteria, however, plays an important role in the functioning of such an alert, and dictates its sensitivity and PPV, which should factor heavily into design. Lastly, as with any quality improvement project, iterative revision and regular monitoring of the intervention itself are clearly important.
Ethics statements
Patient consent for publication
Ethics approval
The Johns Hopkins School of Medicine Institutional Review Board determined the development of the EMR alert (as part of the study of the HST) to be not human subjects/quality improvement research (IRB00100665).
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors TG-B built the electronic medical record alert. EH and KH reviewed charts. WE-N completed the data analysis, conducted a secondary review of charts, and prepared the initial manuscript draft. SB conceived the project, supervised the findings, and is responsible for the overall content as guarantor of this work.
Funding This work was supported by a grant from the Agency for Healthcare Research and Quality, R01HS024079, PI, SB. WE-N is supported by the National Institutes of Health T32 AI007291.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.