key: cord-0981828-cjbu88s3
authors: Monterde, David; Carot-Sans, Gerard; Cainzos-Achirica, Miguel; Abilleira, Sònia; Coca, Marc; Vela, Emili; Clèries, Montse; Valero-Bover, Damià; Comin-Colet, Josep; García-Eroles, Luis; Pérez-Sust, Pol; Arrufat, Miquel; Lejardi, Yolanda; Piera-Jiménez, Jordi
title: Performance of Three Measures of Comorbidity in Predicting Critical COVID-19: A Retrospective Analysis of 4607 Hospitalized Patients
date: 2021-11-23
journal: Risk Manag Healthc Policy
DOI: 10.2147/rmhp.s326132
sha: 9e917b698047c154109abe1e0e2df9199d860089
doc_id: 981828
cord_uid: cjbu88s3

BACKGROUND: Comorbidity burden has been identified as a relevant predictor of critical illness in patients hospitalized with coronavirus disease 2019 (COVID-19). However, comorbidity burden is often represented by a simple count of few conditions that may not fully capture patients’ complexity. PURPOSE: To evaluate the performance of a comprehensive index of the comorbidity burden (Queralt DxS), which includes all chronic conditions present on admission, as an adjustment variable in models for predicting critical illness in hospitalized COVID-19 patients and compare it with two broadly used measures of comorbidity. MATERIALS AND METHODS: We analyzed data from all COVID-19 hospitalizations reported in eight public hospitals in Catalonia (North-East Spain) between June 15 and December 8 2020. The primary outcome was a composite of critical illness that included the need for invasive mechanical ventilation, transfer to ICU, or in-hospital death. Predictors including age, sex, and comorbidities present on admission measured using three indices: the Charlson index, the Elixhauser index, and the Queralt DxS index for comorbidities on admission. The performance of different fitted models was compared using various indicators, including the area under the receiver operating characteristics curve (AUROCC). RESULTS: Our analysis included 4607 hospitalized COVID-19 patients. Of them, 1315 experienced critical illness. Comorbidities significantly contributed to predicting the outcome in all summary indices used. AUC (95% CI) for prediction of critical illness was 0.641 (0.624–0.660) for the Charlson index, 0.665 (0.645–0.681) for the Elixhauser index, and 0.787 (0.773–0.801) for the Queralt DxS index. Other metrics of model performance also showed Queralt DxS being consistently superior to the other indices. CONCLUSION: In our analysis, the ability of comorbidity indices to predict critical illness in hospitalized COVID-19 patients increased with their exhaustivity. The comprehensive Queralt DxS index may improve the accuracy of predictive models for resource allocation and clinical decision-making in the hospital setting.

During 2020, the rapid spread of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and the severity of coronavirus disease 2019 led to the collapse of many healthcare systems worldwide, particularly hospital and intensive care unit (ICU) resources. Recently approved vaccines against SARS-CoV-2 infection are expected to ease hospital resources pressure. 1 Nevertheless, in many settings, COVID-19 continues to cause high hospital demand, which requires adequate healthcare provision planning, particularly in scenarios of a shortage of resources or overburdening of hospitals.

Early in the pandemic, underlying comorbidities were pointed out as significant prognostic factors for the development of severe illness. Initial analyses of a large series of COVID-19 patients revealed that hospitalized patients with chronic conditions like diabetes, hypertension, chronic obstructive pulmonary disease, or cardiovascular diseases were more likely to develop severe COVID-19, with no consensus regarding the contribution of each comorbidity to explaining differences in COVID-19 outcomes. [2] [3] [4] [5] Based on this evidence, the patient's comorbidity burden on admission has been included in many models for predicting hospital outcomes in COVID-19 patients, sometimes combined with post-admission information (eg, vital signs, laboratory and imaging results). 6 However, the comorbidity burden is typically represented by a simple count of few chronic conditions (ie, up to 12) that does not fully capture patients' complexity. [7] [8] [9] [10] [11] [12] [13] [14] Another approach to representing the patient's comorbidity burden is using a comorbidity index that combines information about several comorbidities into a single score. 17, 18 This approach yields more complex models with a more challenging interpretation; however, it allows adjusting for multiple factors, likely improving model accuracy. 18 Known examples of comorbidity indices include the Charlson Comorbidity Index 15 and the Elixhauser index for hospitalized patients. 16 In 2020, we developed a comprehensive risk index tool for hospitalized patients, the Queralt Index, which includes a measure of pre-existing comorbidities (Queralt DxS). 19 The Queralt DxS combines and weighs more than 2100 relevant acute and chronic diagnostic codes. Queralt DxS has shown optimal performance for risk adjustment when measuring comorbidity burden in hospitalized patients. 19 Understanding the risk factors, such as comorbidities associated with hospital outcomes in COVID-19 patients, is crucial for healthcare planning during future waves of the infection. However, most predicting models published to date rely on simple counts of comorbidities, and no studies have explored the appropriate measure for adjusting for multimorbidity in the COVID-19 setting. In this study, we assessed the performance of Queralt DxS as a comorbidity measure in models for predicting outcomes in hospitalized COVID-19 patients and compared it with other widely used comorbidity measures: the Charlson and Elixhauser indices.

In this retrospective analysis, we included data from patients hospitalized due to COVID-19 in any of the eight hospitals of the Catalan Institute of Health (ICS), the leading health care provider in Catalonia (North-East Spain; 7.5 million inhabitants). The ICS provides universal health care to nearly 70% of the Catalan population, and accounts for approximately 30% of all hospitalizations reported in Catalonia. ICS hospitals systematically collect and store data on diagnoses and resource utilization into a centralized database. We screened the ICS database for all admissions due to COVID-19 occurred within the investigated period, which was restricted to June 15-December 8, 2020, to prevent biases associated with the unprecedented hospital overburdening experienced in Spain during the first wave of the COVID-19 outbreak. 20, 21 The database was locked on March 30, 2021.

Hospital admissions due to COVID-19 were identified according to the following codes of the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) system: B97.29, B97.21, B34.2, J12.81, J12.89, and the recently added code for labconfirmed COVID-19 U07.1. We included all hospital admissions of patients who either died in the hospital or were discharged home. Records from patients transferred from or to other hospitals were excluded from the analysis to ensure the completeness of clinical data during the full index hospitalization.

All data were handled according to the General Data Protection Regulation 2016/679 on data protection and privacy for all individuals within the European Union and the local regulatory framework regarding data protection. The independent ethics committee of the Bellvitge Biomedical Research Institute (IDIBELL) approved the study protocol (Ref. PR195/21) and waived the need for informed consent, as the data were generated as part of routine clinical care and fully de-identified for analytic purposes.

The study outcome ("critical illness" in patients hospitalized with COVID-19) was a composite that included the need for invasive mechanical ventilation, transfer to the intensive care unit (ICU), or in-hospital death. The need for invasive mechanical ventilation was identified by the presence of any of the following procedure codes recorded in the hospital database for billing purposes: 5A09357, 

5A09457, 5A09557, 5A1935Z, 5A1945Z, 5A1955Z, 09HN7BZ, 09HN8BZ, 0BH13EZ, 0BH17EZ, 0BH18EZ, and 0CHY7BZ.

Potential predictors considered for the analysis included age, sex, and comorbidities present on admission. The latter were measured using three multi-comorbidity indices: the Charlson Comorbidity Index, the Elixhauser index, and Queralt DxS. In another model, we also used the 27 comorbidities included in the Elixhauser index, each of them separately. The Charlson and Elixhauser indices were estimated using the ICD-10 coding system proposed by Quan et al. 22 For the Charlson index, diagnostic weights were assigned based on the original formulation by Charlson et al. 15 For the Elixhauser index, we used the weights proposed by Moore et al. 23 Queralt DxS belongs to a family of three indices for predicting clinical outcomes in hospitalized patients that provide a numerical value from the weighted sum of diagnoses (primary, secondary present on admission, and complications) from a list of 2119 diagnostic code groups. 19 In this analysis, we used an updated version of the index (version 6.0), computed using healthcare data recorded in the routine care setting between 2018 and 2019.

Data on demographic and clinical variables and outcomes were described as frequency and percentage, mean and standard deviation (SD), and/or median and interquartile range (IQR, defined as the 25th and 75th percentiles), as appropriate. The Charlson and Elixhauser indices were computed using the Comorbidity library by Gasparini; 24 Queralt DxS was generated using R functions and instructions provided in the Supplementary File. For descriptive purposes only, patients were also stratified at four risk levels (ie, low, moderate, high, and very high) based on the 50th, 80th, and 95th percentiles of Queralt DxS, as previously described. 19 Since the primary objective was to compare the performance of each index of comorbidity when added to a given multivariate risk prediction model, we built five logistic regression models to predict the composite outcome of critical illness: a baseline model with age (introduced into the model as a continuous variable) and sex, three models including the baseline and each of the comorbidity measures (ie, Charlson index, Elixhauser index, and Queralt DxS), and one including the 27 diagnoses included in the Elixhauser index separately. The performance of each model was evaluated using five statistical measures of model performance: the deviance, the Akaike information criterion (AIC), 25 the Bayesian Information Criterion (BIC), 26 the area under the receiver operating characteristic curve (AUROCC), and the area under the precisionrecall curve (AUPRC). 27 The 95% confidence intervals (CI) of AUROCC were estimated using 1000 bootstrap samples and confirmed with the DeLong criteria. 28 All analyses were performed using the R statistical package (version 4.0.3). 29

Between June 15 and December 8, 2020, 4607 patients were admitted to the ICS hospitals with COVID-19. Table 1 summarizes the demographic characteristics of the study population according to the incidence of critical illness (ie, need for invasive mechanical ventilation, transfer to ICU, or death) on admission and/or during hospitalization. The corresponding comorbidity burden, measured using the three analyzed indices, is shown in Table 2 .

Patients who experienced critical illness on/after admission were significantly older and had higher scores of comorbidity burden irrespective of the index used. Men experienced critical illness more frequently.

According to the distribution of patients across the five risk levels of Queralt DxS, 2145 (46.6%) patients were at low risk, 1509 (32.8%) at moderate risk, 730 (15.8%) at high risk, and 223 (4.8%) at very high risk. Figure 1 shows the distribution of patients with and without critical illness, according to age, sex, and Queralt risk group. Table 3 shows the overall performance of each model, computed using various measures of the magnitude of the effect. The addition of a multimorbidity measure to the baseline model increased the AUROCC and AUPRC values of the model, which were consistent with all other metrics of model performance. In all, the best performance was consistently observed for the model that included age, sex, and Queralt DxS (Table 3 ). Figure 2 shows the ROC and precision-recall curves of the five models explored.

In the three models including a comorbidity index, this factor significantly contributed to predicting critical illness, irrespective of the comorbidity index used in the model (Figure 3 ). In the model including Queralt DxS as a comorbidity measure, age groups lost statistical significance. 

Our results using data from the healthcare databases of Catalonia confirm the relevance of comorbidities on admission as a strong predictor of outcomes in patients hospitalized with COVID-19, which is consistent with prior reports. Accordingly, the addition of any of the summary measures of comorbidity burden used in our analysis improved the performance of predictive models compared with age and sex only. Furthermore, for the first time to our knowledge, we showed how different measures of the comorbidity burden can change the accuracy of models for predicting critical illness in hospitalized COVID-19 patients.

In the context of comorbidity burden being a strong predictor of outcomes in COVID-19 patients, we observed that the most comprehensive multi-comorbidity index evaluated, Queralt DxS, showed the strongest contribution to explaining critical illness. Unlike the other two indices investigated (ie, the Charlson and Elixhauser indices), which estimate comorbidity burden from a discrete list of diagnoses, Queralt DxS considers all conditions present on admission among a list of 2119 diagnostic groups. Hence, our findings align with the concerns highlighted by Ording and Sørensen, who warned about residual confounding potentially introduced when underestimating the comorbidity burden by using numerical indices based on rather restrictive definitions of comorbidity. 18 Remarkably, when introducing an exhaustive summary measure of comorbidities, such as Queralt DxS, to the model, age was no longer a significant predictor. This suggests that the risk of critical illness in older patients is driven by the accumulation of chronic conditions over time rather than by age per se.

In addition to comorbidities, various authors have identified other relevant predictors of hospital outcomes in patients with COVID-19, such as inflammatory biomarkers, need for oxygen therapy, and diagnostic images analyzed using advanced machine learning approaches. [30] [31] [32] [33] [34] [35] While these models are helpful for informing clinical decisions based on assessments performed during the hospital stay in individual patients, models that use retrospective information available from electronic health records can provide hospital outcome estimates at the time of hospital admission, thus aiding not only clinicians but also managers and policymakers in hospital resource planning. Our results suggest that all models, irrespective of whether they include data from in-hospital assessments, may benefit from a summary, highly comprehensive measure of the comorbidity burden on admission, such as Queralt DxS. We have made this index and the related code freely available online for research purposes as an open-source tool to facilitate evaluation by other groups either in its current form in other populations or potentially as part of clinical risk estimation tools combining comorbidities and clinical data on admission. 

Indices that summarize a high number of variables into a single numerical index, such as Queralt DxS, also have the advantage of allowing more parsimonious models. This is particularly important for machine learning approaches, which may lose performance when increasing the number of variables included in the model. Hence, despite losing sight on the individual effect of each diagnosis, summarizing the information from thousands of pre- existing diagnoses into a single index that accounts for the relative weights of each one and their prognostic relevance for hospital outcomes may enrich other models while minimizing the risk of overfitting.

One of the limitations of Queralt DxS is the lack of validation in settings other than Catalonia; however, we hope that making the software publicly available for research purposes will facilitate external validation moving forward. Moreover, it is worth mentioning that the weights used to estimate the relative contribution of each diagnostic group to health risk were calculated using data collected between 2018 and 2019, before the COVID-19 pandemic. Hence, although the source population of the eight relevant hospitals has remained unchanged since then, the sample of patients admitted with COVID-19 included in this study, and their clinical and sociodemographic profiles do differ from those used when developing the Queralt indices, including Queralt DxS. Therefore, this analysis may be interpreted as a pseudo-validation of Queralt DxS in a Catalan subpopulation hospitalized with COVID-19. Also, as often occurs in retrospective analyses, our dataset was limited to the data recorded in electronic records during routine care. Nonetheless, the universal coverage of our healthcare system, and the cross-linking of healthcare data from the primary care and specialized settings allowed us to access various sources and very comprehensive health data from our study population and consider all possible diagnoses present on admission, as well as basic demographic data, such as age and sex. Conversely, we were not able to include some preadmission variables such as blood groups, which have been associated with COVID-19 severity 36 and might modulate the weight of comorbidities when estimating the risk of critical illness, or the SARS-CoV-2 variant, which may influence clinical outcomes of COVID-19 patients. Nevertheless, even without those pieces of information, the performance of Queralt DxS was very robust.

Finally, it is worth mentioning that we did not adjust for treatments administered or the type of care received. COVID-19 is an evolving pandemic, and the development and implementation of pathogenesis-directed therapies 37 and care pathways 38 may strongly influence the clinical outcome, regardless of the baseline risk factors of hospitalized patients. Due to the relatively narrow investigated period and early stage of the pandemic (ie, June-December 2020), we expect the type of care to be homogeneous among study patients. However, future analyses are warranted to investigate how newly developed therapies, vaccination, and improved care pathways influence the weight of the comorbidity burden on predicting critical illness.

Our findings show that the burden of pre-existing comorbidities significantly improves the prediction of the risk of critical illness in patients hospitalized with COVID-19, particularly when measured exhaustively using a tool such as Queralt DxS. This comorbidity index, which is freely available for research purposes, may improve the accuracy of risk models aimed at supporting clinical decision-making and hospital resource planning in hospitalized patients, including those with COVID-19. 

The study protocol was approved by the independent ethics committee of the Bellvitge Biomedical Research Institute (IDIBELL) (Ref. PR195/21), which waived the need for informed consent, as the data were generated as part of routine clinical care and fully de-identified for analytic purposes. or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

David Monterde declares that he is the developer of the Queralt System. This tool is available online for research purposes at no cost. The authors report no other conflicts of interest in this work, which did not receive specific funding.

Current status of COVID-19 vaccine development: focusing on antigen design and clinical trials on later stages

Risk factors associated with mortality among patients with COVID-19 in intensive care units in Lombardy, Italy

Risk factors for severe illness in hospitalized Covid-19 patients at a regional hospital

Global, regional, and national estimates of the population at increased risk of severe COVID-19 due to underlying health conditions in 2020: a modelling study

Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

Prognostic factors for COVID-19 pneumonia progression to severe symptoms based on earlier clinical features: a retrospective analysis

Development and validation of an early warning score (EWAS) for predicting clinical deterioration in patients with coronavirus disease 2019

Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score

Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection

Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19

Evaluation of novel coronavirus disease (COVID-19) using quantitative lung CT and clinical data: prediction of short-term outcome

Acute physiology and chronic health evaluation II score as a predictor of hospital mortality in patients of Coronavirus Disease

Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score

A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

Comorbidity measures for use with administrative data

Measures of multimorbidity and morbidity burden for use in primary care and community settings: a systematic review and guide

Concepts of comorbidities, multiple morbidities, complications, and their clinical epidemiologic analogs

Performance of comprehensive risk adjustment for the prediction of in-hospital events using administrative healthcare data: the queralt indices

Duty to plan: health care, crisis standards of care, and novel Coronavirus SARS-CoV-2. NAM Perspectives

Saturation of critical care capacity and mortality in patients with the novel coronavirus (COVID-19) in Italy

Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data

Identifying increased risk of readmission and in-hospital mortality using hospital administrative data: the AHRQ Elixhauser Comorbidity Index

A new look at the statistical model identification

Estimating the dimension of a model

The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach

R: A Language and Environment for Statistical Computing

AI-assisted CT imaging analysis for COVID-19 screening: building and deploying a medical AI system

Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy

Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network

CT image visual quantitative evaluation and clinical classification of coronavirus disease (COVID-19)

Severity assessment of Coronavirus Disease 2019 (COVID-19) using quantitative features from chest CT images

End-to-end automatic differentiation of the coronavirus disease 2019 (COVID-19) from viral pneumonia based on chest CT

Genomewide association study of severe Covid-19 with respiratory failure

Pathogenesis-directed therapy of 2019 novel coronavirus disease

The exponential phase of the covid-19 pandemic in central Italy: an integrated care pathway

The authors would like to thank Toni Fuentes for his support in building the dataset used in this analysis.

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation,

Risk Management and Healthcare Policy is an international, peerreviewed, open access journal focusing on all aspects of public health, policy, and preventative measures to promote good health and improve morbidity and mortality in the population. The journal welcomes submitted papers covering original research, basic science, clinical & epidemiological studies, reviews and evaluations, guidelines, expert opinion and commentary, case reports and extended reports. The manuscript management system is completely online and includes a very quick and fair peer-review system, which is all easy to use. Visit http://www.dovepress.com/testimonials.php to read real quotes from published authors.