key: cord-0810482-iugbasy7
authors: Mann, Charlotte Z.; Abshire, Chelsea; Yost, Monica; Kaatz, Scott; Swaminathan, Lakshmi; Flanders, Scott A.; Prescott, Hallie C.; Gagnon-Bartsch, Johann A.
title: Derivation and external validation of a simple risk score to predict in-hospital mortality in patients hospitalized for COVID-19: A multicenter retrospective cohort study
date: 2021-10-08
journal: Medicine (Baltimore)
DOI: 10.1097/md.0000000000027422
sha: dfe48ec237d0d70134497c2a7d93d7dbb3d4b124
doc_id: 810482
cord_uid: iugbasy7

As severe acute respiratory syndrome coronavirus 2 continues to spread, easy-to-use risk models that predict hospital mortality can assist in clinical decision making and triage. We aimed to develop a risk score model for in-hospital mortality in patients hospitalized with 2019 novel coronavirus (COVID-19) that was robust across hospitals and used clinical factors that are readily available and measured standardly across hospitals. In this retrospective observational study, we developed a risk score model using data collected by trained abstractors for patients in 20 diverse hospitals across the state of Michigan (Mi-COVID19) who were discharged between March 5, 2020 and August 14, 2020. Patients who tested positive for severe acute respiratory syndrome coronavirus 2 during hospitalization or were discharged with an ICD-10 code for COVID-19 (U07.1) were included. We employed an iterative forward selection approach to consider the inclusion of 145 potential risk factors available at hospital presentation. Model performance was externally validated with patients from 19 hospitals in the Mi-COVID19 registry not used in model development. We shared the model in an easy-to-use online application that allows the user to predict in-hospital mortality risk for a patient if they have any subset of the variables in the final model. Two thousand one hundred and ninety-three patients in the Mi-COVID19 registry met our inclusion criteria. The derivation and validation sets ultimately included 1690 and 398 patients, respectively, with mortality rates of 19.6% and 18.6%, respectively. The average age of participants in the study after exclusions was 64 years old, and the participants were 48% female, 49% Black, and 87% non-Hispanic. Our final model includes the patient's age, first recorded respiratory rate, first recorded pulse oximetry, highest creatinine level on day of presentation, and hospital's COVID-19 mortality rate. No other factors showed sufficient incremental model improvement to warrant inclusion. The area under the receiver operating characteristics curve for the derivation and validation sets were .796 (95% confidence interval, .767–.826) and .829 (95% confidence interval, .782–.876) respectively. We conclude that the risk of in-hospital mortality in COVID-19 patients can be reliably estimated using a few factors, which are standardly measured and available to physicians very early in a hospital encounter.

2019 novel coronavirus (COVID-19) was declared a pandemic by the World Health Organization in March 2020 and continues to devastate much of the world. As of July 26, 2021, there have been approximately 194 million reported COVID-19 cases worldwide and 4.2 million reported deaths attributed to COVID-19, with a mortality rate per reported infection of 2.1%. [1] During the pandemic, hospitals have needed to constantly adapt how they operate as COVID-19 cases rise and wane. Empirical tools to assess individual patients' risk for mortality could be lifesaving in hospitals, where decisions must be made as to how to allocate resources. To fill this need, since March 2020, a number of risk score models for predicting adverse outcomes of COVID-19 have been published. [2] [3] [4] [5] However, many of these models were developed in patients within a single hospital or hospital system, lacked validation, or incorporated variables that are not routinely available or not measured consistently across hospitals or required subjective evaluation. Concerns of generalizability and ease of implementation remain.

In this study, we aimed to develop a model to predict the risk of in-hospital mortality among patients hospitalized for COVID-19, utilizing variables that are readily available when a patient is hospitalized. We used a systematic variable selection approach to determine a small number of highly predictive factors from over 100 variables in a high-quality observational dataset abstracted from patient electronic health records in the state of Michigan.

For this retrospective observational study, we used data abstracted from patients in hospitals across the state of Michigan (Mi-COVID19 registry) to fit a model predictive of risk of inhospital death amongst patients hospitalized for COVID-19. The Mi-COVID19 data registry is a joint initiative between 10 collaborative quality initiatives sponsored by an insurance provider, Blue Cross Blue Shield of Michigan and Blue Care Network to create a multihospital data registry. [6] Forty noncritical access, non-federal Michigan hospitals voluntarily participated in abstracting data from patients, following a coordinated abstraction protocol, beginning on April 2, 2020.

Data were abstracted from medical records by trained abstractors. The data included a pseudo-random sample of COVID-19 positive cases discharged between March 5, 2020 and August 14, 2020 (with the majority of discharges before April 24, 2020). If a hospital had the ability to abstract full patient data for all eligible COVID-19 positive patients that entered the hospital, they did so. However, if a hospital was unable to abstract all cases, each day the eligible cases were ordered by the timestamp (minute) of discharge and were included starting with the smallest minute value of discharge, until the hospital reached their abstraction capacity.

For this study, we only included patients who tested positive for severe acute respiratory syndrome coronavirus 2 within the hospital in which they were enrolled or were discharged with an ICD-10 code for COVID-19 (U07.1). We split the data to include 20 hospitals in the derivation set and 20 hospitals in the validation set. Our derivation set included the 20 hospitals with the largest number of patients with abstracted data and we reserved the data on patients in the 20 hospitals with the smallest sample sizes for the validation set. We split the data by hospitals, rather than taking a random sample of patients, to emulate external validation and ensure that our estimates of model discrimination were not overly optimistic. We used a complete case analysis, so we ultimately excluded 1 hospital in the validation set since it had no observations with complete cases for the variables included in our final model. It is worth noting that the sample size available in the Mi-COVID19 data registry did not necessarily correlate with the size of the hospital, so there are small and large hospitals in both the derivation and validation sets.

The outcome of interest was in-hospital mortality. We considered as possible risk factors 145 variables abstracted in Mi-COVID19 which were measured on the first or second days of hospitalization. These risk factors included a patient's demographics and health behaviors (e.g., smoking), medical history (comorbidities and previous medications and treatments), exposure risk factors (e.g., being a service or healthcare worker), symptoms and primary complaint at hospital arrival, triage and first day vital signs, first or second hospitalization day lab values, and first or second hospitalization day chest x-ray findings. See Table S1 , Supplemental Digital Content, http://links.lww.com/MD2/A525, for a full list of factors considered in model derivation.

We predicted in-hospital mortality using a logistic regression model. Our approach to variable selection ensured that our model was robust across the hospitals participating in Mi-COVID19 and used factors that are commonly measured, while maintaining high discrimination. In order to control for variability between hospitals, we set our base model to be each hospital's COVID-19 mortality rate, which can be thought of as the patient's pre-test probability for mortality at a given hospital. We included the hospital's COVID-19 mortality rate as an adjustment to ensure that hospital-level differences in mortality would not be falsely attributed to patient characteristics or mask important patient characteristics. From this base model, we used forward and backward selection to choose which variables to include in the model.

In the forward and backward selection, we assessed the model with 3 quality metrics: mean squared error (MSE), R-squared, and an adjusted area under the receiver operating characteristics curve (AUC). The adjusted AUC (denoted AUC (w) ) assessed the discrimination of the model in a way that was unaffected by the value of the mortality rate of each hospital (see eMethods, Supplemental Digital Content, for details on the calculation, http://links.lww.com/MD2/A538). We used this metric so that the AUC was not falsely inflated by including hospital mortality rate in the model. The value of AUC (w) can be interpreted as an estimate of the probability that, for 2 randomly selected patients with opposite outcomes from the same hospital, the model estimates a higher risk of mortality for the patient who is deceased. The AUC (w) calculation is a weighted average of the individual hospital AUCs, with weights proportional to the sample size. If AUC (w) is calculated for only 1 hospital, it is equivalent to the standard AUC.

For each step of forward selection, we added every potential risk factor individually to the model determined from the previous step. We then assessed how much the model's MSE, Mann et al. Medicine (2021) 100:40 Medicine AUC (w) , and R-squared improved when including a variable as compared to the model excluding that variable. We assessed improvement in the quality metrics (MSE, AUC (w) , and Rsquared) both for the full derivation dataset (all 20 hospitals) and within each hospital individually. We considered adding variables to the base model only if there was consistent improvement in MSE, AUC (w) , and R-squared across the individual hospitals as well as all hospitals combined and if the improvement across all derivation hospitals was sufficiently large to warrant inclusion. Amongst factors that had consistent quantitative support for being added to the model, we considered whether these factors were likely to be routinely available and standardly reported across hospitals, as inferred by our clinical experience. If the answer was "yes," we added the variable to the model. We repeated this process until there were no remaining clinically meaningful and standardly measured variables that also dependably improved MSE, AUC (w) , and R-squared in a meaningful way across hospitals in the derivation set when added to the model. Following this forward selection protocol, we additionally performed 1 step of backward selection to ensure that all variables, given all others in the model, were predictive of inhospital mortality. We removed each variable individually and again assessed the change in MSE, AUC (w) , and R-squared from the initial model to this model with 1 variable removed. If the MSE consistently increased and the AUC (w) and R-squared consistently decreased across hospitals when the variable was removed from the model, then we kept that variable in our final model.

In both forward and backward selection, predictions for each individual hospital were made using a model fit on all other hospitals in the derivation set. In this way, all models were assessed using leave-one-out predictions, which reduced the concern for overfitting.

We assessed the performance of our final model on the external validation set of 19 hospitals described previously. We additionally assessed the performance of the model across subgroups of patients (Black vs White, male vs female, and 75 years of age or older vs younger than 75) in terms of AUC for the full dataset.

We aimed to develop a mortality risk assessment model that was as accessible as possible; therefore, we shared the model using an online app developed in R (see Figure S1 , Supplemental Digital Content, http://links.lww.com/MD2/A521). One can input the values for the patient characteristics included in the final model into the app and the predicted risk of mortality is outputted. We set up the app such that one does not need to have or guess the values of every variable used in our final risk score model in order to estimate a patient's risk of mortality. Instead, the app will refit the model using variables that the user does have access to (see eMethods, Supplemental Digital Content, for details, http://links. lww.com/MD2/A538). The app can be accessed at https:// micovidriskcalc.org/.

The study was deemed exempt by the University of Michigan Institutional Review Board on April 2, 2020, for the reason that MI-COVID19 is a Quality Assurance/Quality Improvement 

The Mi-Covid19 dataset included 2193 patients who met our inclusion criteria. In the final model validation, we excluded 79 (4.5%) individuals in the derivation set and 26 (6.1%) individuals in the validation set who were missing data for 1 of the predictive factors in the final model. Therefore, our final derivation and validation sets included 1690 and 398 patients, amongst the 20 and 19 hospitals, respectively. The demographic and clinical characteristics of the patients in the derivation set are described in Table 1 . See Table S2 , Supplemental Digital Content, http://links.lww.com/MD2/A526 and Table S3 , Supplemental Digital Content, http://links.lww.com/MD2/A527, for hospital characteristics in the Mi-COVID19 registry and characteristics of the patients in the validation set. The overall in-hospital mortality rates in the derivation and validations sets were 19.6% and 18.6%, respectively. However, there was variability of mortality rates between individual hospitals. The mortality rates in individual derivation set hospitals ranged from 7.4% to 54.3% (see Figure S2 , Supplemental Digital Content, http:// links.lww.com/MD2/A522 and Table S2 , Supplemental Digital Content, http://links.lww.com/MD2/A526, for all hospital COVID-19 mortality rates).

Through our forward and backward variable selection process, we arrived at a final risk score model that included patient age, first recorded pulse oximetry, first recorded respiratory rate, and highest creatinine level on the first day of admission, along with the overall mortality rate at the patient's hospital. The first variable that met the forward selection criteria was patient age, then pulse oximetry, respiratory rate, and heart rate in the second step of forward selection, and then finally creatinine in the third step. Then, based on the backward selection criteria, we removed heart rate. In a final step of forward selection, no further factors met our criteria. See eAppendix A, Supplemental Digital Content and Figure S3 , Supplemental Digital Content, http://links.lww. com/MD2/A523, for details.

During the forward selection process, the manner in which a patient arrived at the Emergency Department (e.g., by ambulance or by foot) seemed to improve the model (improving AUC (w) by 2%-13% in the first steps of forward selection), however this data was not reliably available in different hospitals. The interleukin-6 and creatine phosphokinase labs additionally appeared to improve the model in terms of MSE (improving MSE by .003, the most out of other variables in the second and third steps of forward selection), however these lab values were available for only 156 (7%) and 562 (26%) patients of the patients in the MiCOVID-19 data, respectively, and are unlikely to be commonly measured. Finally, the patient's emergency department triage score (a number from 1-5 indicating acuity with 1 as highest acuity and 5 as lowest acuity) and the presence Thus, we predict risk of in-hospital mortality using a logistic regression model with 5 covariates. We applied a spline basis transformation to age, with knots at 35, 50, 65, and 80 years, to allow for a non-linear relationship with risk, and a log transformation to creatinine level. Pulse oximetry on admission was collected as "70% or less," "71%-80%," "81%-90%," and "91%-100%." Respiratory rate was collected as "Abnormal (20-24)," "Abnormal (25-30)," "Abnormal (greater than 30)," and "Normal (less than 20)." Table 2 reports the odds ratios corresponding to the coefficients of this model, fit on the derivation set (see Figure S4 , Supplemental Digital Content, http://links.lww.com/MD2/A524, for the age odds ratios). In a logistic regression model, the odds (i.e., the probability of mortality divided by the probability of survival) is estimated by eb 0 X where b is a vector of coefficients and X is the matrix of covariates. We estimate that the odds of mortality increases as age, respiratory rate, and creatinine increases, and as pulse oximetry decreases. See Table 3 for examples of the predicted risk of mortality for 12 patients with different characteristics.

The AUC (w) for the model on the derivation set was .796 (95% confidence interval, .767-.826). The model had similar discrimination on the validation set, with an AUC (w) of .829 (95% confidence interval, .782-.876). The individual hospital AUC (w) values for the validation set vary around .83 and show good discrimination (see Fig. 1 ). We also found that the model shows similar discrimination for Black and White patients as well as male and female patients, although the discrimination is not as strong for patients 75 years or older (see eAppendix B, Supplemental Digital Content and Table S4 , Supplemental Digital Content, http://links.lww.com/MD2/A529). We observe good model calibration when using the individual hospital mortality rates (see Fig. 2 ).

Using a systematic variable selection approach, we built a simple model that estimates the risk of in-hospital mortality for patients hospitalized for COVID-19 with comparable discrimination to more complex models. We performed external validation of the model in 19 separate hospitals for our validation set. It is notable that only age and a few vital signs and labs on hospital presentation and the hospital specific COVID-19 mortality rate are able to predict the risk of in-hospital mortality with high discrimination. Further, our model can be used even when not all variables are available to the user. Age, creatinine, respiratory rate, and pulse oximetry have been found to be associated with mortality in COVID-19 patients in other studies with different populations, attesting to their validity as outcome predictors. [2] [3] [4] [5] 7] Furthermore, the model was developed and validated on data from 39 diverse hospitals from different hospital systems, with variable size, urbanicity, profit status, and academic affiliation. Because of the diversity of hospitals included in Mi-COVID19, we expect that our risk score will generalize well to other hospitals, including those outside the state of Michigan.

We diverge from previous models by including individual hospital's mortality rate for patients with COVID-19 in our model. The inclusion of this variable controls for unmeasured differences between hospitals, as well as the patient populations. We expect including this indicator for each hospital helps control for differing social determinants of health. We do find that after conditioning on the hospital, a patient's race is not predictive of in-hospital mortality. If a user is uncertain about the current Table 2 Odds ratios associated with coefficients of final risk score model.

Odds ratio 95% CI P We applied a spline transformation on age, however the spline regression coefficients are not interpretable. Therefore, in this table we display the estimated odds ratios for different ages as compared to 50 years as a reference age. The coefficients for the spline are significant with p-values <.001. See Figure S4 , Supplemental Digital Content, http://links.lww.com/MD2/A524, for an illustration of the odds ratios and confidence intervals for age with 50 years as a reference age for the full range of ages in the data. this is essentially incorporating a pre-test probability into the model, which can be updated over time as the mortality rate at an individual hospital changes. Our study should be interpreted in the context of some limitations. First, this is a retrospective observational study on patients admitted to the hospital, so it is unclear how the model may generalize to predict eventual outcomes for patients with COVID-19 before hospital admittance. Second, the cutoffs used in the factor variables such as pulse oximetry and respiratory rate were decided before data abstraction occurred, and it is possible that other or additional cutpoints may yield better prediction of hospital mortality. In particular, a cut point of <95% SpO2 may be beneficial as a marker of early disease not yet requiring supplemental oxygen and should be the subject of future investigation. Likewise, breakpoints <80% may not be helpful since there is poor correlation with arterial oxygen at this level. Third, many of the hospitalizations in our dataset were in Southeastern Michigan during the spring 2020 COVID-19 surge, when many hospitals in this region were experiencing very high patient volumes and treatment differed from current best practices. For example, in March and April 2020, dexamethasone and remdesivir were used only rarely, while hydroxychloroquine use was common. Thus, because our model was developed and validated using data from the Spring 2020 surge, it may overestimate the in-hospital mortality of patients treated in non-surge settings and with current best practices. Importantly, however, our model includes the hospital's mortality rate for COVID-19 as a predictor, such that the model automatically recalibrates over time. Furthermore, while in-hospital mortality has changed over time in relation to patient volume and the introduction of new therapies, we expect that age, respiratory rate, pulse oximetry, and creatinine will remain important predictors of in-hospital mortality for COVID-19, as these variables are consistently identified for inclusion in riskprediction models. However, future studies that evaluate the model discrimination and calibration for patients hospitalized after the summer of 2020 will need to confirm that the model performance does not degrade over time.

In sum, we developed a parsimonious risk-prediction model for in-hospital mortality in patients from COVID-19. The use of data from a statewide registry, systematic approach to variable inclusion, and external validation should improve applicability in diverse hospital settings.

Johns Hopkins Coronavirus Resource Center. COVID-19 Map

A clinical risk score to identify patients with COVID-19 at high risk of critical care admission or death: an observational cohort study

Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score

Development and validation of a survival calculator for hospitalized patients with COVID-19

Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19

Initiative j Michigan Hospital Medicine Safety Consortium

Factors associated with disease severity and mortality among patients with COVID-19: a systematic review and meta-analysis

We thank all BCBSM Collaborative Quality Initiatives that partnered together on data collection and all hospitals that volunteered to be part of Mi-COVID19.

Conceptualization: Hallie C. Prescott, Johann A. Gagnon-Bartsch.