Original research•

Adaptability of prognostic prediction models for patients with acute coronary syndrome during the COVID-19 pandemic

Abstract

Background The detrimental repercussions of the COVID-19 pandemic on the quality of care and clinical outcomes for patients with acute coronary syndrome (ACS) necessitate a rigorous re-evaluation of prognostic prediction models in the context of the pandemic environment. This study aimed to elucidate the adaptability of prediction models for 30-day mortality in patients with ACS during the pandemic periods.

Methods A total of 2041 consecutive patients with ACS were included from 32 institutions between December 2020 and April 2023. The dataset comprised patients who were admitted for ACS and underwent coronary angiography for the diagnosis during hospitalisation. The prediction accuracy of the Global Registry of Acute Coronary Events (GRACE) and a machine learning model, KOTOMI, was evaluated for 30-day mortality in patients with ST-elevation acute myocardial infarction (STEMI) and non-ST-elevation acute coronary syndrome (NSTE-ACS).

Results The area under the receiver operating characteristics curve (AUROC) was 0.85 (95% CI 0.81 to 0.89) in the GRACE and 0.87 (95% CI 0.82 to 0.91) in the KOTOMI for STEMI. The difference of 0.020 (95% CI −0.098–0.13) was not significant. For NSTE-ACS, the respective AUROCs were 0.82 (95% CI 0.73 to 0.91) in the GRACE and 0.83 (95% CI 0.74 to 0.91) in the KOTOMI, also demonstrating insignificant difference of 0.010 (95% CI −0.023 to 0.25). The prediction accuracy of both models had consistency in patients with STEMI and insignificant variation in patients with NSTE-ACS between the pandemic periods.

Conclusions The prediction models maintained high accuracy for 30-day mortality of patients with ACS even in the pandemic periods, despite marginal variation observed.

What is already known on this topic

  • Several prognostic prediction methods, such as GRACE, and a machine learning model, KOTOMI, have been developed to estimate the mortality rates of patients with acute coronary syndrome (ACS).

  • The exacerbated circumstances induced by the COVID-19 pandemic have critically compromised the healthcare system to provide optimal medical care for patients suffering from ACS, necessitating a rigorous re-evaluation of prognostic prediction models in the context of the pandemic environment.

What this study adds

  • The prognostic prediction model, GRACE, and a machine learning model, KOTOMI, maintained a high prediction accuracy for short-term mortality of patients with STEMI and NSTE-ACS during the COVID-19 pandemic periods.

How this study might affect research, practice or policy

  • The prognostic prediction models can contribute to the improvement of the quality of care and the therapeutic strategy for patients with ACS regardless of overwhelmed situation due to an emerging infectious disease.

Introduction

The exacerbated circumstances induced by the COVID-19 pandemic have critically compromised the healthcare system to provide optimal medical care for patients suffering from acute coronary syndrome (ACS). Delayed medical contact, diminished requisite hospital admissions and a propensity for less invasive management were reported in the course of care for patients with ACS during the pandemic.1 2 The restriction due to the pandemic is concomitantly associated with increased cardiac damage for patients with ST-elevation acute myocardial infarction (STEMI) treated with percutaneous coronary intervention (PCI),3 and the infection of the virus itself is regarded as an adverse prognostic factor for patients with myocardial infarction.4 Additionally, the overburden imposed during the pandemic has impaired the quality of emergency medical care for savable lives and deteriorated the mortality rate among patients with out-of-hospital cardiac arrest, the predominant aetiology of which is cardiogenic.5–7

Several prognostic prediction methods have been developed to estimate the mortality rates of patients with ACS. The Global Registry of Acute Coronary Events (GRACE) risk score retains durable performance in contemporary medical practice for ACS, although it was developed before PCI was established as a primary therapy for acute myocardial infarction (AMI).8 9 A literature shows that the GRACE risk score has the capability to accurately predict in-hospital mortality in patients with STEMI concomitant with COVID-19.10 Furthermore, the GRACE risk score demonstrated versatility to predict in-hospital mortality, major ischaemic events and the necessity for advanced ventilatory support and intensive care unit admission even for patients without ACS hospitalised due to COVID-19.11 A machine learning–based prediction model, KOTOMI, has shown the prediction capability for in-hospital mortality in patients with STEMI with enhanced precision compared with conventional methods.12 However, there exists a knowledge gap regarding the durability and adaptability of these predictive models in accurately determining short-term mortality for patients with ACS, irrespective of the COVID-19 status, during the pandemic.

The negative impact of COVID-19 pandemic on quality of care and clinical outcomes for patients with ACS necessitate a re-evaluation of the prognostic prediction models within the pandemic’s context as it will facilitate preparation and measures for the outbreak of the next emerging infectious disease. In this study, we aimed to demonstrate the adaptability of the prediction models for 30-day mortality of patients with ACS under conditions exacerbated by the pandemic.

Methods

Setting and design

The AMI-Kyoto Multi-Center Risk Study is a multicentre observational study, executed in collaboration with 32 partnering hospitals in Japan (online supplemental table 1). The study collected demographic, clinical, laboratory, procedural, angiographic and outcome-related data of patients diagnosed with ACS.12 13 Data were collected from patients who were admitted for ACS and underwent coronary angiography (CAG) for the diagnosis during their hospitalisation. A total of 2041 consecutive patients with ACS who were admitted between 24 December 2020 and 21 April 2023 were incorporated into the study. The 30-day mortality was assessed for patients with STEMI and non-ST-elevation acute coronary syndrome (NSTE-ACS). Monthly COVID-19 case data were extracted using the open data provided by the Ministry of Health, Labour and Welfare, Japan.14

Following the procurement of verbal informed consent, which did not necessitate a formal agreement document, all data were transmitted to the centre located at the Department of Cardiology in Kyoto Prefectural University of Medicine for the analysis.

Statistical analysis

The designated prediction models were implemented in Python V.3.10.4. Thirty-day mortality was predicted with binary classification of cases as either survival or deceased. Logistic regression model was employed to provide the predictive probability of the GRACE for 30-day mortality based on the coefficient and intercept.15 The KOTOMI, which was developed by a machine learning using random forest classifier due to its higher discrimination performance for in-hospital mortality compared with extreme gradient boosting classifier and logistic regression, was implemented according to our preceding report.12 All patients with ACS were categorised into either STEMI or NSTE-ACS patient groups. The predictive accuracy of the GRACE and the KOTOMI was ascertained in each ACS group with the area under the receiver operating characteristics curve (AUROC) and the area under the precision recall curve (AUPRC). Net reclassification improvement (NRI) was also calculated. The calibration_curve function was used for the model calibration.

Comprehensive descriptive statistics were conducted in R V.4.2.0.16 Categorical values represented as numbers (%) and numerical values as median (IQR). The χ2 test or Fisher’s exact test was applied to categorical values, and Mann-Whitney U test was employed for continuous values with non-parametric distribution, aiming to delineate the characteristics between survival and deceased groups. A p value <0.01 was interpreted as indicative of statistical significance.

Patient and public involvement

Neither patients nor members of the public were directly involved in the design, conduct or reporting of this research.

Results

Patient characteristics

Total ACS cases (n=2041) were divided into STEMI (n=1232) and NSTE-ACS (n=809), and further stratified to survival and deceased groups. In the STEMI patient subset, median age was 73 years, and male constituted 71.7%. Compared with survival patients (90.2 %, n=1112), the deceased patients with STEMI (9.7 %, n=120) were older, had a higher proportion of severe Killip class, had lower systolic blood pressure (BPs) and haemoglobin (Hb). Additionally, they exhibited elevated white cell count (WCC), blood sugar (BS) levels, creatinine (Cr), C-reactive protein (CRP), maximum creatine phosphokinase (CPK), CK-MB and GRACE risk score (table 1). In the NSTE-ACS subset, median age was 75 years, and male constituted 72.0%. Compared with survival patients (96.0 %, n=777), the deceased patients with NSTE-ACS (3.9 %, n=32) were older, had a higher proportion of severe Killip class, had lower BPs and Hb, and had higher BS, Cr, CRP, maximum CPK, maximum CK-MB and GRACE risk score (table 2).

Table 1
|
Characteristics of patients with STEMI
Table 2
|
Characteristics of patients with NSTE-ACS

Evaluation of predictive values during the COVID-19 pandemic

The receiver operating characteristics (ROC) curve and precision recall (PR) curve were depicted to assess the prediction accuracy of the models for 30-day mortality in patients with STEMI and NSTE-ACS (figure 1). The AUROC was 0.85 (95% CI 0.81 to 0.89) for the GRACE and 0.87 (95% CI 0.82 to 0.91) for the KOTOMI in cases of STEMI, reflecting no significant disparity between the models (the difference in AUROC was 0.020 (95% CI −0.098 to 0.13)). The AUROC was 0.82 (0.73–0.91) for the GRACE and 0.83 (0.74–0.91) for the KOTOMI in cases of NSTE-ACS, revealing no significant disparity between the models (the difference in AUROC was 0.010 (95% CI −0.023 to 0.25)). The AUPRC was 0.36 for the GRACE and 0.40 for the KOTOMI in cases of STEMI, and 0.22 for the GRACE and 0.20 for the KOTOMI in cases of NSTE-ACS. Calibration plot was depicted to assess the consistency between the actual mortality rate and the predictive probability yielded by the prediction models (figure 2). The mortality rate (fraction of positives) was increased along with the average prediction probability in both the GRACE and the KOTOMI for STEMI and NSTE-ACS. Nonetheless, for patient groups with high average prediction probability, the probability was inclined to be understated compared with the actual mortality rate. Overall, both prediction models exhibited high prediction accuracy for patients with ACS including STEMI and NSTE-ACS during the COVID-19 pandemic.

Figure 1
Request permissionrequest permission icon
Figure 1

Model accuracy for 30-day mortality of patients with ACS. Left panel shows receiver operating characteristics curve, and right panel shows precision recall curve. ACS, acute coronary syndrome; NSTE-ACS, non-ST-elevation acute coronary syndrome; STEMI, ST-elevation acute myocardial infarction.

Figure 2
Request permissionrequest permission icon
Figure 2

Model calibration for patients with STEMI and NSTE-ACS. Samples were divided into four bins according to probability. NSTE-ACS, non-ST-elevation-acute coronary syndrome; STEMI, ST-elevation acute myocardial infarction.

Variation of model accuracy across pandemic phases

Subsequently, we analysed the deviations in the prediction accuracy correlating with the COVID-19 cases. Aligning with the monthly COVID-19 case counts in Japan, the pandemic timeline was divided into two distinct periods, with December 2021 serving as the boundary (online supplemental figure 1). The AUROC of both the GRACE and the KOTOMI was compared between the first and second period for patients with STEMI and NSTE-ACS, respectively (figure 3). For STEMI, the AUROC was 0.87 (95% CI 0.81 to 0.93) in the first period, 0.83 (95% CI 0.76 to 0.89) in the second, yielding an insignificant disparity of 0.041 (95% CI −0.13 to 0.21) in the GRACE model; concurrently, the KOTOMI model exhibited an AUROC of 0.88 (95% CI 0.82 to 0.93) in the first period and 0.85 (95% CI 0.79 to 0.91) in the second, with a non-significant disparity of 0.028 (95% CI −0.13 to 0.19). For NSTE-ACS, the AUROC was 0.85 (95% CI 0.74 to 0.95) in the first period, 0.78 (95% CI 0.62 to 0.93) in the second, with an insignificant disparity of 0.072 (95% CI −0.29 to 0.44) in the GRACE model; 0.88 (95% CI 0.78 to 0.97) in the first period, 0.73 (95% CI 0.57 to 0.90) in the second, with an insignificant disparity of 0.14 (95% CI −0.22 to 0.51) in the KOTOMI model. NRIs of the KOTOMI and the GRACE were not significant in any pandemic phases for each ACS group (online supplemental table 2). Collectively, the prediction accuracy of both models had consistency in patients with STEMI, and non-significant variation between the distinct phases of the pandemic periods in patients with NSTE-ACS.

Figure 3
Request permissionrequest permission icon
Figure 3

Variation of model accuracy between the pandemic periods. The pandemic period was divided into two periods, with December 2021 as the boundary, and the models’ accuracy was evaluated. Error bar indicates 95% CI. AUROC, area under the receiver operating characteristics curve; NSTE-ACS, non-ST-elevation-acute coronary syndrome; STEMI, ST-elevation acute myocardial infarction.

Discussion

The present study elucidated that the prediction models, the GRACE and the KOTOMI, have durable prediction accuracy for 30-day mortality of patients with STEMI and NSTE-ACS during the COVID-19 pandemic, and the variation was not significant between the pandemic periods. The model accuracy of the KOTOMI marginally surpassed that of the GRACE for STEMI as well as NSTE-ACS, although the disparity of the prediction accuracy was not significant. These findings indicate that the accurate prediction of 30-day mortality of patients with ACS is feasible during the overwhelmed situation under such COVID-19 pandemic, despite its negative impact on medical service and clinical outcomes for patients with ACS.

The GRACE and the KOTOMI model were formulated based on different combination of prediction factors. The GRACE is inclusive of eight predictors, including age, Killip class, BPs, HR, Cr, ST-segment deviation, cardiac arrest during presentation and positive initial cardiac enzyme findings. Conversely, the KOTOMI integrates 10 predictors, including age, Killip class, BPs, HR, WCC, Hb, BS, Cr, CRP and max CPK. Both models were superior to the conventional TIMI (Thrombolysis in Myocardial Infarction) risk index regarding the prediction of in-hospital mortality of patients with STEMI.12 The GRACE can be used to predict in-hospital mortality as well as 6-month prognosis in patients with STEMI and NSTE-ACS.15 17 It has enhanced discriminatory power for in-hospital mortality of NSTE-ACS compared with the traditional TIMI risk score.18 19 Conversely, the KOTOMI was developed to predict in-hospital mortality of patients with STEMI. Machine learning–based prediction model for 1-year adverse events among patients following ACS have been previously developed and validated using data collected before the COVID-19 outbreak.20 Indeed, the evaluation of short-term prognosis is also necessary, given the poor 30-day mortality rate—approximately 10% for STEMI and 4% for NSTE-ACS, as shown by our results. In this context, the present study demonstrated the adaptability and robustness of both prediction models for 30-day mortality of patients with STEMI and NSTE-ACS even in the COVID-19 pandemic.

The types of biomarkers and their times of measurement are different between the GRACE and the KOTOMI. In contemporary clinical practice for patients with ACS, cardiac troponin has emerged as a pivotal biomarker for early diagnosis due to its high sensitivity.21–23 Cardiac enzymes including CPK or CK-MB, as well as troponin, are useful prognostic markers for ACS, reflecting cardiac damage and infarct size.24 25 The GRACE uses positive initial cardiac enzyme including troponin, whereas the KOTOMI employs maximum CPK for prediction of mortality of patients with ACS. The GRACE can be used at admission for patients with ACS, while the KOTOMI demonstrates the strength after peak out of CPK levels following the diagnosis by CAG. The present study observed that the KOTOMI marginally surpasses the GRACE without significant difference for 30-day mortality of patients with STEMI and NSTE-ACS. However, the types of biomarkers and their times of measurement need to be taken into consideration.

There are several limitations in the study. First, the validation study in other countries is imperative because the pandemic impact on medical system differed between countries. Second, the validation was performed for prediction of 30-day mortality of patients with ACS. Therefore, further study is needed to assess the models’ adaptability for longer term outcomes during the pandemic. In addition, it will be warranted to establish a prediction model for antithrombotic risk following ACS, adaptable to situations overwhelmed by emerging infectious disease.

Conclusions

The GRACE and the KOTOMI model maintained high prediction accuracy for 30-day mortality of patients with STEMI and NSTE-ACS during the COVID-19 pandemic. These prediction models contribute to the improvement of the quality of care and the therapeutic strategy for patients with ACS regardless of overwhelmed situation due to an emerging infectious disease.

Article metrics
Altmetricopen-url
Dimensionsopen-url