key: cord-0757405-yi0x6m5x
authors: Castro, V. M.; McCoy, T. H.; Perlis, R. H.
title: Consistency of performance of adverse outcome prediction models for hospitalized COVID-19 patients
date: 2021-03-01
journal: medRxiv : the preprint server for health sciences
DOI: 10.1101/2021.02.25.21252410
sha: a04576669951ee89ff2954f5c12e3b4368930fa7
doc_id: 757405
cord_uid: yi0x6m5x

Background: We previously reported and validated a risk prediction tool based on COVID-19 hospitalizations prior to June 2020. Here, we report performance of that model on subsequent data from 6 hospitals and among individual patient subgroups. Method: We included individuals age 18 or older hospitalized at one of 2 academic medical centers and 4 community hospitals from 6/7/2020 through 1/22/2021 with positive PCR test for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) within 5 days of admission. Coefficients from our previously reported least absolute shrinkage and selection operator (Lasso) risk models were applied to estimate probability of a mortality, as well as a composite severe illness outcome, including admission to the ICU, mechanical ventilation or mortality. Results: Overall model performance for mortality included AUC of 0.83 (95% CI:0.80-0.87) for mortality, with a PPV 0.55 and NPV of 0.95 when using a cutoff corresponding to the highest 20% of predicted risk derived in the training set. For all adverse outcomes, AUC was 0.79 (95% CI:0.75-0.81) and PPV 0.48 and NPV 0.98 in the top 20% risk group. Model discrimination was generally similar between genders and race/ethnicity groups, but markedly poorer for younger age groups. Conclusion: Although the population of individuals hospitalized for COVID-19 has shifted and outcomes have improved overall, prediction models derived earlier in the pandemic may maintain utility.

The challenge of managing limited resources during COVID-19 has sparked efforts to stratify risk among hospitalized patients 1 . Few risk models have been validated or investigated for potential bias, 2 even though COVID-19 inpatient populations, treatments, and outcomes have changed over time. We previously reported and validated a risk prediction tool based on COVID-19 hospitalizations prior to June 2020 3 . Here, we report performance of that model on subsequent data from 6 hospitals and among individual patient subgroups.

We included individuals age 18 or older hospitalized at one of 2 academic medical centers and 4 community hospitals from 6/7/2020 through 1/22/2021 with positive PCR test for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) within 5 days of admission, excluding those with an outcome on the day of hospitalization.

Features of hospital course were extracted from the Mass General Brigham Data Registry 4 and the Enterprise Data Warehouse, including laboratory values and high/low flags. Charlson comorbidity index was calculated using coded ICD-10 diagnostic codes 5 . The study protocol was approved by the Mass General Brigham Human Research Committee, waiving informed consent as detailed by 45 CFR 46.116. STROBE reporting guidelines for cohort studies were applied. All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Patients were followed from admission to hospital discharge or death, with follow-up censored at discharge. Primary outcomes of interest were 1) a composite severe illness outcome, including admission to the ICU, mechanical ventilation or mortality;

and 2) mortality. Coefficients from our previously reported least absolute shrinkage and selection operator (Lasso) risk models were applied to estimate probability of each outcome, without recalibration. We applied median imputation of missing data.

We characterized model performance with standard metrics of discrimination and calibration. All analyses utilized R v4.

Features of the new cohort are summarized in Table 1 and compared to those of the previously-reported cohort in which the predictive model was trained. For the 2,892 individuals, mean age was 63.0 (SD=19.1); they were 50.5% female, 23.3% Hispanic, and 11.9% Black. Mean length of hospital stay was 6. (Table 2) , model discrimination was generally similar between genders and race/ethnicity groups, but markedly poorer for younger age groups. All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 

Applying a previously-validated model to 2,892 new COVID-19 admissions in the same 6 hospitals, we find that model performance decreased modestly from the initial validation study which reported AUC's of 0.81 for severe illness and 0.85 for mortality 3 . Discrimination is generally similar across subgroups, with the notable exception of younger age groups, where performance is poorer.

Our results highlight the observation that the population of individuals hospitalized for COVID-19 has shifted and outcomes have improved overall, but suggest that prediction models derived earlier in the pandemic may maintain utility. The extent to which this finding of robustness over time in a rapidly evolving disease generalizes to other predictive models requires investigation. Our results also illustrate the importance of investigating risk stratification models across patient subgroups, as a step toward ensuring that particular groups of patients are not adversely impacted by application of such tools particularly in settings of potential resource constraints.

Models that explicitly address the heterogeneity in severe COVID-19 across age groups are candidates for further study.

This study was supported by the National Institute of Mental Health (R01MH116270; Dr. Perlis). The sponsors did not contribute to design and conduct of the study; collection, management, analysis, and interpretation of the data; All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted March 1, 2021. ; https://doi.org/10.1101/2021.02.25.21252410 doi: medRxiv preprint preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Dr. Perlis had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. He is an associate editor for JAMA Network Open.

Dr. Perlis has received consulting fees from Burrage Capital, Belle Artificial Intelligence, Genomind, RID Ventures, and Takeda. He holds equity in Outermost Therapeutics and Psy Therapeutics. The other authors report no disclosures. All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.

(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Table 1 . Sociodemographic and clinical feature comparison between the initial model training COVID admissions and the subsequent admissions used to evaluate the model. The training dataset consisted of the initial surge of COVID cases in Eastern Massachusetts whereas the replication cohort includes the summer nadir and second wave in Fall 2020. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 

Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score

Collider bias undermines our understanding of COVID-19 disease risk and severity

Laboratory Findings Associated With Severe Illness and Mortality Among Hospitalized Individuals With Coronavirus Disease 2019 in Eastern Massachusetts

Calculating the benefits of a Research Patient Data Repository