key: cord-0920446-btkde3l9
authors: Muñoz, Oscar M; Ruiz-Talero, Paula; Hernández-Florez, Catalina; Lombo-Moreno, Carlos Ernesto; Casallas-Rivera, Martha Alejandra; Mayorga-Hernández, Carol Anne
title: Validation of the PESI Scale to Predict in-Hospital Mortality in Patients with Pulmonary Thromboembolism Secondary to SARS CoV − 2 Infection
date: 2022-05-20
journal: Clin Appl Thromb Hemost
DOI: 10.1177/10760296221102940
sha: b2531e570ccc6420a41cf61a6d5aca7ab9109c13
doc_id: 920446
cord_uid: btkde3l9

OBJECTIVE: To evaluate the discriminative ability and the calibration of the Pulmonary Embolism Severity Index (PESI) to predict in-hospital mortality in patients with Pulmonary Embolism (PE) secondary to COVID 19 in two hospitals in Bogotá. METHODS: External validation study of a prediction model based on a retrospective cohort of patients with PE secondary to COVID-19 treated at Hospital Universitario San Ignacio and Hospital universitario La Samaritana, between March 2020 and August 2021. Calibration of the scale was evaluated using the Hosmer-Lemeshow test and a calibration belt diagram. Discrimination ability was evaluated using a ROC curve. RESULTS: 272 patients were included (median age 61.5 years, male 58.8%). PE was diagnosed in 45.6% of the patients at the time of admission. Of the remaining 54.4%, 95.9% received thromboprophylaxis until the time of diagnosis.17.6% of the patients died. Regarding calibration, the scale systematically underestimates risk in all classes of PESI. For class I, the ratio of observed/expected events was 4.4 vs 0.8%, class II 4.8 vs 1.8%, class III 15.2 vs 4.2%, class IV 14.3 vs 5.9% and class V 46.7 vs 5.8%. The calibration test rejected the adequate calibration hypothesis (p < 0.001). The discriminatory ability was adequate (AUC = 0.7128, 95% CI 0.63-0.79). CONCLUSIONS: The PESI scale in patients with PE secondary to COVID 19 underestimates the risk of in-hospital mortality, while maintaining adequate discrimination. It is suggested not to use the PESI scale until it is recalibrated in this context.

As of January 2022, 715 inhabitants per million died because of coronavirus disease worldwide and 2595 per million died in Colombia. 1 Although mortality has been mainly associated with the development of Acute Respiratory Distress Syndrome (ARDS), 2 there are also multiple different pathophysiological alterations in patients with COVID-19. 3 Patients with COVID-19 can have pulmonary embolism (PE) secondary to an increase in the systemic inflammatory response generated by an imbalance between procoagulant and anticoagulant factors. 4 Between 15% and 30% of patients with COVID-19. [5] [6] [7] develop PE. Also, PE is associated with increased mortality, up to 45%. 8, 9 International PE clinical practice guidelines recommend the use of the PESI (Pulmonary Embolism Severity Index) to stratify mortality risk. 10 PESI scale has been validated to predict 30-day mortality in patients with PE, [11] [12] [13] which allows to guide patient management according to mortality risk. However, at the moment there is no consensus about using PESI in patients with PE secondary to COVID-19. 14, 15 There are multiple concerns about PESI underestimating mortality risk in this context and this hypothesis has not been evaluated yet.

The purpose of this study is to validate the PESI scale estimation of in-hospital mortality risk in patients with PE secondary to COVID-19, to evaluate PESI scale ability to discriminate between high and low risk patients and to evaluate the relationship between observed and predicted in-hospital mortality by the PESI scale, in two hospitals in Bogotá, Colombia.

This study is a validation of a prediction model based on a retrospective cohort. The databases of the institutional anticoagulation registry of Hospital Universitario San Ignacio and the COVID-19 registry of Hospital Universitario La Samaritana were considered, which include the patients managed between March 6, 2020, and August 31, 2021. The inclusion criteria were hospitalized patients older than 18 years with a confirmed diagnosis of PE and COVID-19. Patients with missing information on medical records and patients referred to another health facility were excluded. This determination was made to secure information about mortality outcome. The institutional research ethics committee of both institutions approved the study.

Sociodemographic data was systematically collected during patient care using standard recording practices in the database described. Variables related to comorbidities, laboratories, thromboprophylaxis, use of in-hospital anticoagulation, days of hospitalization, hospitalization in general ward or Intensive Care Unit (ICU), oxygen supply system, in-hospital death and hemorrhagic complications were obtained from the mentioned databases and complemented with revision of medical records when needed. Standard thromboprophylaxis was defined as a Low Molecular Weight Heparin (LMWH) in standard dosis, generally enoxaparin 40 mg/day subcutaneous (SC). Adjusted thromboprophylaxis was defined as any different dose, according to individualized decisions of the treating physician based on thrombotic risk, weight or kidney failure.

Regarding the PESI scale, 16 the version validated in Colombia was used. 13 Data necessary to calculate PESI scale (Age, sex, cancer, heart failure, chronic obstructive pulmonary disease, heart rate >110 bpm, blood pressure, systolic < 100 mm Hg, respiratory rate > 30 rpm, Temperature < 36 centigrade degrees, altered mental status and arterial oxygen saturation (SaO 2 ) < 90%) was collected when PE diagnosis was made.

The diagnosis of PE was operatively defined by detection of thrombi in pulmonary arteries in a computed tomography pulmonary angiogram (CTPA), contrast-enhanced chest computed tomography or ventilation/perfusion lung scan. COVID 19 infection was confirmed with RT PCR or SARS-CoV 2 antigen positivity. Finally, major bleeding was defined as fatal bleeding, bleeding in a critical area or organ or bleeding with a decrease in hemoglobin ≥ 2 g/dl, according to the guidelines of the International Society of Thrombosis and Haemostasis. 17 

For description of qualitative variables, absolute and relative frequencies were used. Central tendency and dispersion measures were calculated for quantitative variables. Mean and standard deviation were described for variables with normal distribution and median and interquartile range for variables with non-normal distribution. Normal distribution was Subsequently, the PESI scale was validated by analyzing its calibration and discrimination capacity. Calibration was evaluated comparing the number of observed and predicted in-hospital mortality events within each of the five risk classes proposed by PESI scale. Expected proportion of In-hospital mortality was obtained from the original study of Aujesky (16) , in the derivation sample, for each PESI class. The expected events were calculated by multiplying the expected proportions by the number of patients in each class.

Proper calibration hypothesis was evaluated according to the Hosmer-Lemeshow statistical test. 18 A graphical analysis was performed through a calibration belt diagram. This methodology allowed visual inspection to determine whether the observed frequencies differ significantly from the expected probabilities. Also, this methodology allowed to localize the direction of the miscalibration deviation, 19 80% and 95% confidence interval levels were used. Finally, a calibration test was obtained to assess if the deviation from the bisector (45-degree line of perfect fit) was significant. 20 The discriminatory ability (ability of the scoring system to differentiate between patients with different outcomes) was evaluated with a Receiver operating Characteristic (ROC) curve. An area under the curve (AUC) greater than 0.7 was considered appropriate.

Statistical analysis was performed with the statistical program STATA (Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC). Table 1 summarizes the demographic and clinical characteristics of the 272 patients with PE secondary to COVID-19. Median age was 61.5 years, with a higher prevalence of males. Hypertension was the main associated comorbidity. It is noteworthy in the laboratory reports that 78.9% of the patients had D-dimer levels greater than 1000 ng/ml. In 45.6%, PE diagnosis was made at the time of admission; within the 54.4% remaining, 95.9% had received thromboprophylaxis at the time of diagnosis. Low molecular weight heparin (LMWH) was the most frequent anticoagulant.

Regarding the outcomes of interest17.6% died, 34.9% required ICU,28.7% required mechanical ventilation and 16.9% required high-flow oxygen supply systems (nonrebreather mask or high-flow nasal cannula), 7.7% did not require O 2 supply during hospitalization. Finally, 8.8% presented hemorrhagic complications, half of them characterized as major bleeding and the other half as non-major bleeding.

The evaluation of the calibration of the PESI scale showed that the observed in-hospital mortality was higher than the expected mortality (17.7 vs 5.8%). This result was consistent within each class of the PESI scale (Hosmer-Lemeshow 2.02, p = 0.364) ( Table 2 ). The calibration belt diagram demonstrates an inadequate calibration of the model for all risk ranges with confidence intervals of 80% and 95%. Similarly, the calibration test rejected the hypothesis of adequate calibration (p < 0.001) (Figure 1 ). PESI scale discriminatory ability was acceptable with an area under the ROC curve of 0.71 (95% CI 0.63, 0.79) (Figure 2 ), similar to results reported by the original scale.

This is the first study that formally evaluates the performance of the PESI scale to predict in-hospital mortality in patients with PE secondary to COVID 19. Our results show that the calibration of the scale is inadequate since it systematically underestimates the probability of death at all PESI classes. The ratio of observed events/expected events is close to 3:1. However, PESI maintains its acceptable ability to discriminate patients with lower or higher risk.

Studies have evaluated the usefulness of the PESI scale in the context of PE secondary to COVID-19. Miró et al 21 showed a higher mortality in patients with PE secondary to COVID-19 compared to PE not secondary to COVID-19 (12.8 vs 5.3%, p < 0.001). Their results suggest an underestimation of in-hospital mortality with different scores on the sPESI scale (simplified PESI) in patients with PE secondary to COVID-19 compared to PE not caused by COVID-19 (sPESI 0 p = 0.003, sPESI 1 p = 0.028, sPESI, 2 p = 0.012, sPESI>2 p = 0.055). Additionally, Xu et al. 22 showed that in patients with PE secondary to COVID-19, the PESI scale apparently adequately discriminated in-hospital mortality when comparing high-intermediate risk patients (PESI class III, IV and V) compared to low-risk patients (PESI class I and II) (27 vs 6%, p = 0.007). However, these studies did not formally assess the discrimination and calibration of the PESI scale in this context. Kagan et al 23 studied a modified sPESI scale to determine the need for ICU admission in patients with COVID-19. It used the variables: age ≤ 65 years, history of cancer, cardiopulmonary diseases, heart rate < 110 bpm, systolic blood pressure ≥ 100 mm Hg and O 2 saturation ≥ 90%. They report an area under the ROC curve of 0.95, sensitivity 84.6% and specificity 94.6% for a sPESI score > 2. However, their study did not include patients with PE. Additionally, sPESI recommendations have been made without having studied its discriminative capacity or calibration. [21] [22] [23] [24] Therefore, this is the first study to evaluate the discriminative capacity and calibration of the PESI scale in patients with PE secondary to COVID-19.

Our results show an adequate discrimination ability (area under ROC curve 0.7377), similar to AUC reported in the original PESI validation study (0.79 CI95% 0.65-0.93) 16 and in the Colombian validation study (AUC 0 0.78 CI95% 0.6-0.96). 13 The main strength of the PESI scale is its ability to identify patients with low mortality risk (class I and II). 10, 25 However, our results show an increased mortality rate in these classes (4.4 and 4.8%, respectively). Therefore, in the context of PE secondary to COVID-19, it is possible that PESI classes I and II need in-hospital surveillance, rather than early hospital discharge and outpatient follow-up as current guidelines recommend. 10, 25 Regarding PESI classes III to V, the in-hospital mortality observed in these patients is greater than 15%. Hence, it would be advisable to hospitalize these patients in an Intermediate Care Unit or ICU.

The sociodemographic, clinical and laboratory characteristics of these patients are similar to other reported studies of patients with COVID-19 and PE. A meta-analysis 5 that included more than 7178 patients with COVID-19 and PE showed a median age of 60.4 years with a prevalence of men between 58% and 80%. They reported a prevalence of PE in hospitalized patients between 1.6 and 62.5% and in the ICU between 4.2% and 75%. Regarding comorbidities, the main associated comorbidity was hypertension (47%-85%), followed by diabetes (4%-26%) and cancer (3%-20%), similar to our study. This data is also reported in other studies. 7, 26 When comparing laboratory reports, other studies have also shown elevated levels of lactate dehydrogenase (LDH), CRP, and D-dimer in patients with PE. Compared to patients with COVID-19 without PE, there are statistically significant differences in D-dimer and LDH levels. 7, 27 Expected proportion of In-hospital mortality was obtained from the original study of Aujesky (16) , It is concerning that nearly half of the patients who presented PE secondary to COVID-19 were receiving thromboprophylaxis at the time of PE diagnosis. Another meta-analysis showed a similar result, they reported 38% of patients hospitalized in the ICU presented thromboembolic events under thromboprophylaxis treatment. 28 One hypothesis is the possible resistance to heparin treatment in patients with severe COVID-19 manifestation. 29 It is also notable the high proportion of bleeding complications in our study (8.8%). Other studies report an incidence of clinically significant bleeding or major bleeding between 6 and 14.7%. Additionally, they report an association between major bleeding and the administration of therapeutic anticoagulation compared to prophylactic, ECMO treatment and dialysis treatment. 30, 31 Among the strengths of this study, it stands out that we used different methodologies to evaluate the calibration of the scale obtaining consistent results. Additionally, the external validity of our results is strong since we included patients from two different university hospitals.

One of the main limitations of this study is its retrospective nature which could lead to misclassification bias. The underreporting of comorbidities (given the self-report by patients) or the underreporting of altered consciousness state in medical records are possible biases in this study. Additionally, the relatively small number of events limits the accuracy of our estimates, mainly in low-risk classes. Therefore, prospective validation studies with a larger number of patients are required to confirm our results and to recalibrate the PESI scale in this context.

In conclusion, our study suggests that PESI scale in patients with PE secondary to COVID-19 underestimates the risk of in-hospital mortality while maintaining adequate discrimination. Therefore, PESI scale utilization in this context is not suggested until it has been recalibrated. When used, we suggest careful interpretation, bearing in mind in-hospital mortality underestimation in patients with PE secondary to COVID-19.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval of this study was obtained from "Investigation and ethical institutional committee from Hospital Universitario San Ignacio (Approval number 238/2021)".

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Carlos Ernesto Lombo-Moreno https://orcid.org/0000-0002-6392-7554 

Our world in data

Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study

Epidemiology, pathogenesis, clinical presentations, diagnosis and treatment of COVID-19: a review of current evidence

Pulmonary embolism in patients with coronavirus disease-2019 (COVID-19) pneumonia: a narrative review

Incidence of acute pulmonary embolism in COVID-19 patients: systematic review and metaanalysis

Arterial and venous thromboembolism in COVID-19: a study-level meta-analysis

Pulmonary embolism in hospitalized patients with COVID-19: a multicenter study

The impact of deep vein thrombosis in critically ill patients: a meta-analysis of major clinical outcomes

Incidence and mortality of pulmonary embolism in COVID-19: a systematic review and meta-analysis

ESC guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European respiratory society (ERS)

Prospective validation of the Pulmonary Embolism Severity Index. A clinical prognostic model for pulmonary embolism

Prognostic models in acute pulmonary embolism: a systematic review and meta-analysis

Validación de la escala pronóstica (Pulmonary Embolism Severity Index-PESI-) en pacientes con diagnóstico de tromboembolia pulmonar

Consenso colombiano para la prevención, el diagnóstico y el tratamiento de condiciones trombóticas en adultos con COVID-19: aplicando el Marco de la Evidencia a la Decisión (EtD) GRADE

Recomendaciones sobre el tratamiento antitrombótico durante la pandemia COVID-19. Posicionamiento del Grupo de Trabajo de Trombosis Cardiovascular de la Sociedad Española de Cardiología

Derivation and validation of a prognostic model for pulmonary embolism

Anticoagulation the S on C of. Definition of clinically relevant non-major bleeding in studies of anticoagulants in atrial fibrillation and venous thromboembolic disease in non-surgical patients: communication from the SSC of the ISTH

Applied Logistic Regression

A new test and graphical tool to assess the goodness of fit of logistic regression models

A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes

Pulmonary embolism severity and in-hospital mortality : an international comparative study between COVID-19 and non-COVID patients

Pulmonary embolism in patients hospitalized with COVID-19 (from a New York health system)

Can a modified-simplified pulmonary embolism severity index (m-sPESI) be used to predict the need for intensive care in hospitalized COVID-19 patients?

Differences between surviving and non-surviving venous thromboembolism COVID-19 patients: a systematic review

American Society of hematology 2020 guidelines for management of venous thromboembolism: treatment of deep vein thrombosis and pulmonary embolism

Acute pulmonary embolism associated with COVID-19 pneumonia detected with pulmonary CT angiography

Acute pulmonary embolism in patients with COVID-19 at CT angiography and relationship to d-dimer levels

Venous thromboembolism in critically ill COVID-19 patients receiving prophylactic or therapeutic anticoagulation: a systematic review and metaanalysis

Heparin resistance in COVID-19 patients in the intensive care unit

Anticoagulation and bleeding risk in patients with COVID-19

Bleeding risk by intensity of anticoagulation in critically ill patients with COVID-19: a retrospective cohort study