key: cord-0973469-ayxandv6
authors: Evans, D. S.; Kim, K. M.; Jiang, X.; Jacobson, J.; Browner, W.; Cummings, S. R.
title: Prediction of In-hospital Mortality among Adults with COVID-19 Infection
date: 2021-01-25
journal: nan
DOI: 10.1101/2021.01.22.21249953
sha: 1edd6607349494fd75df4f20ec39a073153fbc00
doc_id: 973469
cord_uid: ayxandv6

IMPORTANCE: Accurate and rapid prediction of the probability of dying from COVID-19 infection might help triage patients for hospitalization, intensive care, or limited treatment. OBJECTIVE: To develop a simple tool to estimate the probability of dying from acute COVID-19 illness. DESIGN, SETTING AND PARTICIPANTS: This cohort study included 13,190 adult patients admitted to one of the 11 hospitals in the New York City Health + Hospitals system for COVID-19 infection between March 1 and June 30, 2020. EXPOSURES: Demographic characteristics, vital signs, and laboratory tests readily available at the time of hospital admission. MAIN OUTCOME: Death from any cause during hospitalization. RESULTS: Patients had a mean age (interquartile range) of 58 (45-72) years; 5421 (41%) were women, 5258 Latinx (40%), 3805 Black (29%), 1168 White (9%), and 2959 Other (22%). During hospitalization, 2,875 (22%) died. Using separate test and validation samples, machine learning (Gradient Boosted Decision Trees) identified eight variables, oxygen saturation, respiratory rate, systolic and diastolic blood pressures, pulse rate, blood urea nitrogen level, age and creatinine, that predicted mortality, with an area under the ROC curve (AUC) of 94%. A score based on these variables classified 5,677 (46%) as low risk (a score of 0) who had 0.8% (95% confidence interval, 0.5, 1.0%) risk of dying, and 674 (5.4%) as high-risk (score [≥] 12 points) who had a 97.6% (96.5, 98.8%) risk of dying; the remainder had intermediate risks. A risk calculator is available online at https://danielevanslab.shinyapps.io/Covid_mortality/). CONCLUSIONS AND RELEVANCE: In a diverse population of hospitalized patients with COVID-19 infection, a clinical prediction model using a few readily available assessments may precisely estimate in-hospital mortality and can rapidly assist decisions to prioritize admissions and intensive care.

COVID-19 infection. Accurate prediction of the probability of death using rapidly available data might help prioritize patients for hospitalization, intensive care and intubation, or to receive limited treatments.

Previous studies of predictors of dying from COVID-19 infection have had a small number of deaths; 1-7 used comorbid conditions, diagnoses ,and severity indices from electronic medical records that may not be known at the time of admission; 4, [8] [9] [10] [11] [12] [13] or included specialized laboratory tests-such as levels of C-reactive protein, troponin, and D-dimers-that are not readily available for urgent triage of patients for hospital admission or intensive care. 14 Some studies were done in ethnically homogenous populations such as Wuhan, China 1 or Italy, 15 in specific populations such as nursing home residents, 16 or among patients already being treated in an intensive care unit (ICU) 6, 9 Although some studies applied machine learning methods to develop predictive models, 1, 5, 17, 18 none have been based on measurements available at the time of triage in a large diverse population, nor have they been translated into calculators that can be used in clinical settings.

To address these needs, we studied the large and diverse population of patients admitted to New York City Health + Hospitals (NYC H+H) public hospital system. We used machine learning to select strong predictors of mortality, developed and validated a multivariable model and score to estimate the risk of dying, and translated the model into an online calculator to estimate the risk of in-hospital mortality.

We used data extracted from the electronic medical records of all patients at least 18 years old who were admitted to any of the 11 hospitals of the New York City Health + Hospitals (NYC H+H) system with a diagnosis of Covid 19 infection verified with a positive polymerase chain reaction (PCR) test. NYC H+H is the largest public health system in the United States, providing health services to more than one million New Yorkers across the city's five boroughs. These hospitals account for approximately one-fifth of all general hospital discharges and more than one-third of emergency department and hospital-based clinic visits in New York City.

We abstracted demographic characteristics (sex, age, race and ethnicity), weight, body mass index, vital signs, oxygen saturation (SpO2) from peripheral monitors, and routine clinical laboratory tests (serum chemistry panel, complete blood counts) and D-dimer levels ( Table 1) from electronic medical records. When there was more than one value, we selected the first.

Missing values were not imputed. Non-transformed values were used. Sex and race/ethnicity were coded as categorical variables (Table 1) ; all others were treated as continuous variables.

(Results did not change when continuous features were centered to their mean and scaled to a standard deviation of one.) The outcome was death from any cause during hospitalization; length of hospitalization was also noted.

Baseline characteristics are presented N (%) or mean (SD). Comparisons between continuous variables were tested using Student's t test or the Mann-Whitney U test if the distribution was skewed. Categorical variables were analyzed with the Chi-squared test.

Our overall goal was to develop a clinical prediction model to estimate the probability of dying in the hospital. We used Extreme Gradient Boosted Decision Trees (XGBoost), followed by the identification of cut-points of the selected variables using classification and regression trees (CART), then developed a score to predict mortality. Train and test data partitions were created using an 80% and 20% random split stratified by death status to ensure an even proportion of mortality in the train and test datasets. Gradient Boosted Decision Trees implemented in the XGBoost R package v 1.2.0.1 with R v 4.0.2. was used to generate an ensemble of multiple decision trees to minimize errors in the classification of mortality in patients. The XGBoost model was developed in the train partition, using four boosting rounds, a maximum depth of three for each decision tree, a learning rate of 0.3, a binary:logistic learning objective with error rate used as the evaluation metric with a minimum child weight of 75. Feature importance was evaluated using the information gain metric of a split on a variable. XGBoost model performance was evaluated in the test partition using accuracy and area under the curve (AUC) from a receiver operating characteristic (ROC) curve. Selected features and model performance did not change with 10-fold cross-validation. All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 25, 2021. ;

To develop a clinical prediction score, we used Classification and Regression Tree (CART) analyses in the original 80% training set to identify optimum cut-points for each variable selected by XGBoost. There was no cut-point for creatinine and it had low importance in the XGBoost model, therefore it was not included in the final calculation of clinical risk score. There were 1,501 patients missing a BUN value. To determine whether a missing value was associated with risk of mortality, we created an indicator variable for a missing BUN value. We entered the selected variables and cut-points into a logistic regression model to estimate the multivariable odds ratios. To assign risk scores, the odds ratio for each of these categorical variables were divided by 2.6 (the lowest odds ratio), rounded, and then summed for each patient to calculate a risk score. After excluding 703 patients with missing values for one or more variables, the proportions of patients who died were calculated for each 1-point interval in risk score; the highest-risk categories, which had similar scores and small numbers of patients, were combined. Because the predicted mortality by risk score categories were very similar in the training and test sets, these sets were combined to estimate the probabilities and 95% confidence intervals for the entire population. An online calculator reports the probability of inhospital mortality based on the risk score (danielevanslab.shinyapps.io/Covid_mortality). To report the probability of dying, all variables must have non-missing values except for the blood urea nitrogen (BUN) test which includes a term for missing. There were statistically significant differences between those who died and those who survived for almost all variables ( Table 1 ). The XGBoost algorithm identified eight variables ( Figure 1 ) All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 25, 2021. Of the variables that the XGBoost model selected, SpO2 was the strongest predictor; respiratory rate and blood pressure were also major contributors; body temperature was not.

Although race and ethnicity were associated with mortality in univariable analyses (Table 1) , they were not selected in the predictive model.

CART analysis identified cut-points for each of the XGBoost-selected variables. A multivariable logistic model showed that the selected cut-points were all significant predictors of mortality ( Table 2 ). The risk score based on the odds ratios for these variables ranged from 0 to 22 points and had an AUC of 0.934 for predicting mortality (Figure 3 ). There were 5,677(45.5%) patients with a score of 0, and 674 (5.4%) with a score ≥ 12 points (Table 4 ). In-hospital mortality increased continuously with higher risk scores, ranging from ranged from 0.8% (95% confidence interval, 0.5 -1.0 %) for those with a score of 0 to 97.6% (96.5 -98.8%) for patients with a score ≥ 12 points (Table 3 ).

The mean times between admission and death was 18 days (IOR 6-27 days) for those with a risk score of 0, compared with 9 days (IQR 3-11 days) for those with a risk score of 12 or greater.

We translated the models into an online calculator to report the probability of mortality and the corresponding 95% confidence interval: danielevanslab.shinyapps.io/COVID_mortality/. Not surprisingly, physiologic variables, such as SpO2, respiratory rate, and blood pressure were important predictors, indicating that the pulmonary and systemic effects of the infection are its most important prognostic features. As expected, mortality also increased with age and with higher BUN levels. 5, 19, 20 Notably, after considering other variables, race and ethnicity were not All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 25, 2021. ; significant predictors of mortality, as has been seen in other studies. 11, 21, 22 While previous studies have identified other laboratory values, such as red cell distribution width and D-dimer levels, as significant predictors, they did not contribute to this algorithm. 3, 14 BUN was the only laboratory value in the algorithm and a missing value did not influence the score. This suggests that clinicians do not need to order or wait for other test results to estimate a patient's probability of dying.

These data were collected before effective treatments, such as corticosteroids, were used This analysis has several strengths. The algorithm was derived from a very diverse population of patients in New York City using data from 11 hospitals. The study population and number of deaths were large enough to produce estimates of mortality with narrow confidence intervals and high AUC values; it is unlikely that adding additional variables to the model would substantially improve its already high accuracy. The algorithm produced consistent results by 10-fold cross-validation and by testing its performance in a random 20% of the data.

Multivariable regression analysis of the variables selected by machine learning confirmed that they were strong and independent predictors of mortality. An easy-to-use version of the model is also available online for use in acute care settings (danielevanslab.shinyapps.io/Covid_mortality/). All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The analysis also has limitations. The model represents the natural history of COVID-19 infection before hospital care improved-and mortality rates declined-so it could not be calibrated to predict mortality with current standards of care. By design, the data did not include measurements, such as sophisticated markers of inflammation and coagulation, or indices of comorbidity and severity of illness, that predict mortality but that may not be readily available in the initial assessment of a patient.

Mortality from COVID-19 infection can be predicted accurately from a few clinical observations and laboratory tests that are readily available in acute care settings. When resources are scarce, estimates of the probability of dying might aid decisions about prioritizing patients to receive intensive care or other scarce resources. The prediction model is available online for use in clinical settings.

All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 25, 2021. ; (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 25, 2021. ; 

Supplemental Figure Legends .

Supplemental Figure 1 . Cover is the sum of the second order gradient of training data classified to the leaf. Gain is the information gain metric of a split. Value is the margin value that the leaf contributes to prediction.

Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients

Comorbidity and clinical factors associated with COVID-19 critical illness and mortality at a large public hospital in New York City in the early phase of the pandemic

Association between biomarkers and COVID-19 severity and mortality: a nationwide Danish cohort study

Laboratory Findings Associated With Severe Illness and Mortality Among Hospitalized Individuals With Coronavirus Disease 2019 in Eastern Massachusetts

Machine Learning for Mortality Analysis in Patients with COVID-19

Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease

Predictors of In-Hospital Mortality in Older Patients With COVID-19: The COVIDAge Study

Comparison of inhospital mortality risk prediction models from COVID-19

Factors Associated With Death in Critically Ill Patients With Coronavirus Disease 2019 in the US

Risk Factors for Hospitalization, Mechanical Ventilation, or Death Among 10 131 US Veterans With SARS-CoV-2 Infection

Association of Race and Ethnicity With Comorbidities and Survival Among Patients With COVID-19 at an Urban Medical Center

Risk Factors Associated With Mortality Among Patients With COVID-19 in Intensive Care Units in Lombardy, Italy

Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation

Can we predict the severe course of COVID-19 -a systematic review and meta-analysis of indicators of clinical outcome? Predictors of In-Hospital Mortality in Older Patients With COVID-19: The COVIDAge Study. medRxiv

Clinical risk score to predict in-hospital mortality in COVID-19 patients: a retrospective cohort study

Risk Factors Associated With All-Cause 30-Day Mortality in Nursing Home Residents With COVID-19

Clinical features of COVID-19 mortality: development and validation of a clinical prediction model

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation

Predicting Hospital Utilization and Inpatient Mortality of Patients Tested for COVID-19. medRxiv

Predictive modeling of morbidity and mortality in COVID-19 hospitalized patients and its clinical implications. medRxiv

Association of Race With Mortality Among Patients Hospitalized With Coronavirus Disease 2019 (COVID-19) at 92 US Hospitals

Assessment of Racial/Ethnic Disparities in Hospitalization and Mortality in Patients With COVID-19 in New York City

Association Between Administration of Systemic Corticosteroids and Mortality Among Critically Ill Patients With COVID-19: A Metaanalysis

Variation in US Hospital Mortality Rates for Patients Admitted With COVID-19 During the First 6 Months of the Pandemic