key: cord-0901058-2zxibya7
authors: Aletreby, Waleed Tharwat; Mumtaz, Shahzad Ahmad; Shahzad, Saima Akhtar; Ahmed, Intekhab; Alodat, Mohammed Ali; Gharba, Mohamed; Farea, Zohdi Ahmed; Mady, Ahmed Fouad; Mahmood, Waqas; Mhawish, Huda; Abdulmowla, Majd Munia; Nasser, Rehab Mohammed
title: External Validation of 4C ISARIC Mortality Score in Critically ill COVID-19 Patients from Saudi Arabia
date: 2022-01-12
journal: Saudi J Med Med Sci
DOI: 10.4103/sjmms.sjmms_480_21
sha: 7f1b19e96952861d71594e5e071a3a123c8abfee
doc_id: 901058
cord_uid: 2zxibya7

BACKGROUND: ISARIC mortality score is a risk stratification tool that helps predict the in-hospital mortality of COVID-19 patients. However, this tool was developed and validated in a British population, and thus, the external validation of this tool in local populations is important. OBJECTIVES: External validation of the ISARIC mortality score in COVID-19 patients from a large Saudi Arabian intensive care unit (ICU). METHODS: This is a retrospective study that included all adult patients with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, from March 2020 to June 2021. Patients who were pregnant or had pulmonary tuberculosis/human immunodeficiency virus were excluded along with patients with missing variables. Data were collected to calculate the ISARIC mortality score and then fitting receiver operator characteristic curve against patients’ outcome. RESULTS: A total of 1493 critically ill COVID-19 patients were included. The mortality was 38%, the area under the curve of the score was 0.81 (95% confidence interval [CI]: 0.79–0.83, P < 0.001) and the cutoff value correctly classified 72.7% of the cohort. The cutoff value of >9 had sensitivity of 70.5% (95% CI: 66.6–74.3); specificity, 73.97% (95% CI: 71–76.8); positive predictive value, 62.4% (95% CI: 59.5–65.2) and negative predictive value, 80.2% (95% CI: 78.2–82.4). CONCLUSION: The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion.

requiring intensive care unit (ICU) admission and mechanical ventilation. [6] Life-threatening COVID-19 presentations are commonly characterized by acute respiratory distress syndrome, multiorgan failure, hyperinflammatory responses and thromboembolic manifestations, among others. [7] Therefore, it is comprehendible that the mortality rate in severe and critically ill COVID-19 patients is high, [8] particularly when complemented with poor prognostic factors such as comorbidities, elevated inflammatory biomarkers and old age. [3, 7] The variability in presentation necessitated the development of risk stratification tools that would allow early identification of COVID-19 patients at higher risk of mortality, using readily available objective criteria. [8] [9] [10] Accordingly, Knight et al. [8] utilized The International Severe Acute Respiratory and emerging Infections Consortium (ISARIC) World Health Organization Clinical Characterization Protocol to develop and validate such a tool, the ISARIC 4C mortality score (hereafter referred to as ISARIC), which likely outperforms other risk stratification models. [11] The ISARIC mortality score utilizes variables that are readily available on hospitalization, thereby avoiding reliance on parameters such as radiological imaging or those that only become available after ICU admission. [12, 13] The model has been demonstrated to have a high discriminatory ability for in-hospital mortality and categorizes patients into four categories of severity with a uniformly increasing mortality risk [Supplementary Table 1 ].

However, the tool's development and validation were primarily done for the population in the United Kingdom. For generalizability, there is a need to validate the score within different populations. To the best of our knowledge, no study from Saudi Arabia has previously validated this tool, and thus, the current study was performed to validate the performance of the ISARIC score within the setting of a Saudi Arabian ICU.

This is a retrospective study that included all adults with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, between January 1, 2020, and June 30, 2021; the first case was recruited on March 20, 2020. The study was approved by the hospital's institutional review board.

King Saud Medical City is a 1200-bed government hospital, with an ICU capacity that has been increased from 127 to 300 beds since the beginning of the COVID-19 pandemic. The ICU is closed, operated round the clock by intensivists and has a nurse:patient ratio of 1:1. For the pandemic, almost half of the beds were converted to single rooms, and all beds are fully equipped with capabilities of invasive and noninvasive monitoring and ventilation. The admission criteria of COVID-19-positive patients to the ICU were broad, as the center serves as a COVID-19 referral center of the region. COVID-19 was confirmed by nasopharyngeal reverse transcription-polymerase chain reaction (RT-PCR), and patients were admitted if they had any of the following: supplemental oxygen of any kind to maintain peripheral oxygen saturation (SpO 2 ) ≥92%, mechanically ventilated, any level of consciousness disturbance or respiratory rate ≥28 per minute. The criteria were the same during both waves of COVID-19 and in line with the COVID-19 management recommendations of the Saudi Ministry of Health. [14] Inclusion and exclusion criteria All adult (≥18 years) patients with a nasopharyngeal RT-PCR-confirmed COVID-19 admitted to the ICU of our hospital were included in this study. Nasopharyngeal RT-PCR was repeated for all patients who previously tested positive either before ICU discharge or after 2 weeks of ICU admission, whichever occurred first.

Pregnant women as well as those known to have pulmonary tuberculosis (PTB) and human immunodeficiency virus (HIV) were excluded from this study, as common COVID-19 medications such tocilizumab or antiviral drugs may not be offered to these patients. After the completion of data collection, patients with missing variables that preclude the calculation of ISARIC score were also excluded.

For calculating the ISARIC score [8] [Supplementary Table 2 ], the following variables were collected from the electronic database of our ICU and/or manually from the patients' medical records: age; gender; number of comorbidities; respiratory rate, SpO 2 on room air, and Glasgow coma scale (GCS) at hospital admission; first available blood urea level (mmol/L); and C-reactive protein (CRP) (mg/L). In addition, ethnicity, smoking status, and nature of comorbidities were also noted. The collection of data was performed by a group of data collectors supervised by authors: IA, ZS, MA and WA. The electronic records of the hospital were used by the data collectors to gather demographic data, COVID-19 test and laboratory results along with all relevant dates such as admission and discharge, while data of the condition upon hospitalization (such as GCS and SpO 2 ) were gathered manually from the patients' files, and the aforementioned authors verified the same.

The primary outcome of the study was the performance of ISARIC score in our settings by evaluating its discriminatory ability of survivors and deceased in the all-cause hospital mortality outcome. Secondary outcomes included comparison between subjects with score above and below the optimal criterion derived from the analysis with regard to mortality rate, ICU length of stay (LOS) and survival analysis.

Statistical tests were performed using the STATA ® statistical software (Release 16; StataCorp LLC, College Station, TX, USA). For descriptive purposes, the demographic and clinical characteristics of survivors and nonsurvivors are presented separately. Continuous variables are summarized as mean ± standard deviation and discrete variables as frequency and percentages. Continuous variables were compared by Student's t-test or Wilcoxon rank-sum test, as appropriate, while discrete variables were compared by Chi-square test or Fisher's exact test, as appropriate. Comparative results are presented with corresponding 95% confidence interval (CI) and P value.

The primary outcome was evaluated by fitting receiver operator characteristic (ROC) curve using known outcomes as reference variable and ISARIC score as classification variable, through 1000 bootstrap with replacements. The area under the curve (AUC) and P value are presented. Furthermore, the analysis identified the optimal criterion associated with Youden's index, which was used as a cutoff value for the ICU survival analysis, presented as Kaplan-Meier curve and log-rank P value, in addition to sensitivity, specificity, positive predictive values (PPV) and negative predictive value (NPV), and a 2 × 2 contingency table. All tests were two tailed, and P < 0.05 was considered statistically significant.

During the study period, there were 1749 ICU admissions of confirmed COVID-19 [ Figure 1 ]. Of these, 32, 12, 5 and 2 patients were excluded because they were aged <18 years, were pregnant, had PTB and were HIV positive, respectively. Another 205 (11.7%) patients were excluded due to missing CRP values precluding score calculation. Accordingly, the study finally included 1493 patients. Table 1 depicts the demographic and clinical characteristics of the cohort and comparison between survivors and nonsurvivors. The majority of the patients were Asians (42%) followed by Saudis (28.2%) and Middle Eastern (24.4%). There were 567 (38%) hospital deaths, with a mean age of 53.2 ± 14.1 years and the majority (76.3%) being males.

Survivors had significantly different values than nonsurvivors in all variables used in the calculation of ISARIC score, in the direction of a lower score, with the exception of gender distribution, and significantly lower prevalence of all recorded comorbidities, except for chronic kidney disease, ischemic heart disease and smoking. Only Saudis and Asians had significantly higher survival than mortality rates. The average ICU LOS of the cohort was 10.5 ± 9 days, being significantly shorter for survivors compared to nonsurvivors (8.1 ± 7.2 vs. 14.3 ± 10.2, 95% CI of difference: −7 to −5.3, P < 0.001). The cohort had a mean ISARIC score of 8.6 ± 5.3; survivors had significantly lower score than nonsurvivors (6.3 ± 4.3 vs. 12.4 ± 4.3, 95% CI of difference: −6.5 to −5.6; P < 0.001). Supplementary Figure 1 shows the numbers of patients, mortality rate and median (interquartile range) ISARIC score for different modes of supplemental oxygen.

The fitted ROC curve of ISARIC score against known outcomes had AUC of 0.81 (95% CI: 0.79-0.83, P < 0.001) [ Figure 2 ]. The associated criterion with Youden's index was a score of >9; accordingly, survival based on that cutoff value estimated by the Kaplan-Meier curve showed a median survival of 22 days with ISARIC scores ≤9 and 16 days with scores >9, and the associated log-rank test was highly significant [ Figure 3 ]. The sensitivity and specificity with scores >9 were 70.5% (95% CI: 66.6-74.3) and 73.8% (95% CI: 71-76.8), respectively, the PPV and NPV was 62.4% (95% CI: 59.5-65.2) and 80.4% (95% CI: 78.2-82.4), respectively, and the cutoff value correctly classified 72.7% of the cohort [Supplementary Table 3 ].

Dividing the enrolled patients into two groups based on ISARIC score of ≤9 and >9 showed significantly higher mortality in the >9 group compared to ≤9 (62.4% vs. 19.6%, 95% CI: 38%-47.4%; P < 0.001). Similarly, the group with a score of ≤9 had ICU LOS of 9.1 ± 8.2 days, which was significantly shorter than that of the group with score >9 (LOS: 12.3 ± 9.7 days) (95% CI: 2.3-4.1, P < 0.001) [Supplementary Table 4 ].

In the population of our study, the ISARIC score was found to have excellent [15] predictive ability for mortality risk, with an AUC of 0.81 (95% CI: 0.79-0.83, P < 0.001). This study excluded cases with missing data (11.2%) precluding calculation of ISARIC score, although modern imputation methods (such as multiple imputation) are likely to produce valid estimates; however, only when critical assumptions have been met, [16] which may have not been true for our data, they still carry the potential to cause bias. [17] Enrolled patients in our analysis were of multiple ethnicities, representing the general population of Saudi Arabia, and thus providing a good external validation cohort for the ISARIC score.

The AUC in our study was slightly higher than that reported by the original development and validation study (AUC = 0.786; 95% CI: 0.781-0.79). Although this is uncommon in external validation studies, higher AUCs were similarly reported by van Dam et al. [9] (AUC = 0.84; 95% CI: 0.79-0.88) and Wellbelove et al. [18] (AUC = 0.83; 95% CI: 0.71-0.95). Other studies have reported AUC values similar to the validation study [19] or lower. [10] This variation among studies -although minimal -utilizing the same prediction model may be a reflection of the variations in the studied populations, with regard to their demographic characteristics, clinical severity and sample size. For example, in our study, various ethnic groups were included, the age range was wide (from 18 to 104 years) and there was an unequal gender distribution (males: 76.3%). Furthermore, our cohort included only patients admitted to the ICU, which means that they presented a more critical picture of the disease, while the original study enrolled all hospital admissions. Regardless of those variations, all studies in the literature, including ours, found , that is, a directly proportional relationship between mortality risk and increase in score. This reflects that the model performs optimally, especially when considering that the higher mortality rates were higher within all groups in our study compared with the original study. [8] In our analysis, the optimal cutoff value associated with Youden's index was a score >9, with this value correctly classifying 72.7% of the cohort.

The diagnostic parameters (sensitivity, specificity, PPV and NPV) of the cutoff of >9 in our model were considerably lower compared to the same cutoff value in the British study. [8] This could possibly be due to inclusion of only critically ill COVID-19 patients, which makes predicting outcome more difficult than mild cases, given the larger number of complications [7] that could be the indirect cause of death. This is also reflected in the fact that our study had a higher mortality rate compared with the study by Knight et al. [8] (38% vs. 32.2%, respectively). One of the most important diagnostic parameters is NPV, which indicates the probability of survival in patients with scores ≤9. In our model, NPV was 80.2% (95% CI: 78.2-82.4), which provides a reasonable risk probability to guide clinical decision making; this discrimination was supported by our finding of a significantly higher survival of patients with scores ≤9, shorter ICU LOS, and lower mortality.

This study has several limitations. First, the inherent limitations of the retrospective study design are applicable. Second, this was a single-center study, reflecting the management in only our ICU, and thus, our results may not be extrapolated to other centers in Saudi Arabia or non-ICU patients. Third, not using imputation methods on missing data may have reduced our sample size and impacted our results. Fourth, despite working with a quite diverse population, our cohort lacked several ethnicities, and thus, our results may not be applicable in other regions of the world. Finally, the generalizability of our results may be limited to only critically ill patients, as non-ICU admissions were not included. Further multicenter studies with a larger sample size and including those with varied severities are required to validate the score in the larger Saudi population and possibly explore predictors of mortality in COVID-19 patients.

The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion.

T he study was approved by the institutional review board of King Saud Medical City, Riyadh (reference no.: H1R1-24-Feb21-04, dated: April 8, 2021). The requirement of patient consent was waived in view of the retrospective study design. The study followed the general principles outlined by the Declaration of Helsinki, 2013.

The datasets generated during and/or analyzed during the current study are not publicly available, but are available from the corresponding author on reasonable request.

This article was peer-reviewed by two independent and anonymous reviewers.

Nil.

There are no conflicts of interest.

Dynamics of SARSCoV2 outbreak in the Kingdom of Saudi Arabia: A predictive model

A novel coronavirus from patients with pneumonia in China

Isaric 4c mortality score as a predictor of in-hospital mortality in COVID-19 patients admitted in ayub teaching hospital during first wave of the pandemic

World Health Organization. WHO Director-General's opening remarks at the media briefing on COVID -19-11

World Health Organization. WHO coronavirus (COVID-19) dashboard

Tocilizumab in the treatment of rapidly evolving COVID-19 pneumonia and multifaceted critical illness: A retrospective case series

Clinical characteristics and predictors of 28-day mortality in 352 critically ill patients with COVID-19: A retrospective study

Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO Clinical characterisation protocol: Development and validation of the 4C mortality score

Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: A retrospective study

Predicting in-hospital mortality in COVID-19 older patients with specifically developed scores

Prediction models for diagnosis and prognosis of COVID-19: Systematic review and critical appraisal

Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images

Influenza-associated pneumonia as reference to assess seriousness of coronavirus disease (COVID-19)

Saudi MoH Protocol for Patients Suspected of/Confirmed with COVID-19

Receiver operating characteristic curve in diagnostic test assessment

Advanced statistics: Missing data in clinical research -Part 2: Multiple imputation

Imputations: Benefits, Risks and a Method for Missing Data

Comparing the 4C mortality score for COVID-19 to established scores (CURB65, CRB65, qSOFA, NEWS) for respiratory infection patients

Characteristics and outcomes of hospital admissions for COVID-19 and influenza in the Toronto area

Sensitivity=70.5% (95% CI: 66.6-74.3), Specificity=73.97% (95% CI

ISARIC -International Severe Acute Respiratory and emerging Infections Consortium; PPV -Positive predictive value