key: cord-0850543-kt8jim1d
authors: Halalau, Alexandra; Imam, Zaid; Karabon, Patrick; Mankuzhy, Nikhil; Shaheen, Aciel; Tu, John; Carpenter, Christopher
title: External validation of a clinical risk score to predict hospital admission and in-hospital mortality in COVID-19 patients
date: 2020-10-09
journal: Annals of medicine
DOI: 10.1080/07853890.2020.1828616
sha: bd17bb92d7095faa06a748a736bd004b8a9c0139
doc_id: 850543
cord_uid: kt8jim1d

BACKGROUND: Identification of patients with novel coronavirus disease 2019 (COVID-19) requiring hospital admission or at high-risk of in-hospital mortality is essential to guide patient triage and to provide timely treatment for higher risk hospitalized patients. METHODS: A retrospective multi-centre (8 hospital) cohort at Beaumont Health, Michigan, USA, reporting on COVID-19 patients diagnosed between 1 March and 1 April 2020 was used for score validation. The COVID-19 Risk of Complications Score was automatically computed by the EHR. Multivariate logistic regression models were built to predict hospital admission and in-hospital mortality using individual variables constituting the score. Validation was performed using both discrimination and calibration. RESULTS: Compared to Green scores, Yellow Scores (OR: 5.72) and Red Scores (OR: 19.1) had significantly higher odds of admission (both p < .0001). Similarly, Yellow Scores (OR: 4.73) and Red Scores (OR: 13.3) had significantly higher odds of in-hospital mortality than Green Scores (both p < .0001). The cross-validated C-Statistics for the external validation cohort showed good discrimination for both hospital admission (C = 0.79 (95% CI: 0.77–0.81)) and in-hospital mortality (C = 0.75 (95% CI: 0.71–0.78)). CONCLUSIONS: KEY POINTS: Can an electronic health record generated risk score predict the risk of hospital admission and in-hospital mortality in patients diagnosed with coronavirus disease 2019 (COVID-19)? In both validation cohorts of 2,025 and 1,290 COVID-19, the cross-validated C-Statistics showed good discrimination for both hospital admission (C = 0.79 (95% CI: 0.77–0.81)) and in-hospital mortality (C = 0.75 (95% CI: 0.71–0.78)), respectively. The COVID-19 Risk of Complications Score may help predict the need for hospital admission if a patient contracts SARS-CoV-2 infection and in-hospital mortality for a hospitalized patient with COVID-19.

Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) is a positive-sense RNA virus belonging to the Coronaviridae family, first reported in a cluster of patients with viral pneumonia in Wuhan, China [1, 2] . Rapid spread ensued and novel coronavirus infections were declared as a pandemic on 11 March 2020, resulting in global aggressive social distancing measures to limit viral transmission [3] . As of 19 May 2020, there are 4,897,492 confirmed cases of COVID-19 with 323,285 deaths in 188 countries [4] .

SARS-CoV-2 primarily spreads via respiratory droplets and direct contact [5] [6] [7] . Medical procedures that induce aerosol production, such as nebulizer treatments or intubation, are reported to increase the risk of transmission [1, 6, 7] . A wide clinical spectrum of severity is reported, and worse clinical outcomes are observed with older patients and patients with comorbidities such as hypertension, diabetes, and chronic obstructive lung disease (COPD) [8, 9] . Severe cases can result in shock, acute respiratory distress syndrome (ARDS), acute kidney, cardiac, liver, gastrointestinal, neurological injury, coagulopathy and death [10, 11] .

Despite COVID-19 infection severity in higher risk populations, most drugs have proven no significant efficacy in large-scale studies [12] [13] [14] , except remdesivir, currently considered the most promising antiviral agent [12, 15] . Hospitalized patients with advanced COVID-19 and lung involvement who received remdesivir had a 31% faster recovery than similar patients who received placebo in the Adaptive COVID-19 Trial sponsored by the National Institute of Allergy and Infectious Disease [15] . However, given the lack of widely available effective therapies, COVID-19 continues to be a global health threat with a massive burden on health care systems. Beyond social distancing and personal protective equipment use, intensive care unit (ICU) capacity expansion and treatments to reduce ICU demand are potential strategies to mitigate the pandemic's impact [16] .

Developing and validating clinically applicable prognostic tools to identify high-risk patients is necessary to guide resource allocation efforts. Recently proposed prediction models for COVID-19, derived primarily from populations in China, Italy, and international registries, suffer from high risk of bias due to small sample sizes, model overfitting, and lack of external validation, and are not yet recommended for clinical practice [17, 18] .

We aimed at validating a risk assessment tool for patients with COVID-19, stratifying patients based on their hospitalization and in-hospital mortality risk.

Beaumont Health is the largest health system in Southeast Michigan, USA providing healthcare services to about one third of patients in the Detroit Metropolitan Area [19] . A retrospective cohort was created from patients with positive SARS-CoV-2 testing on nasopharyngeal swabs per WHO definitions [20] between 1 March 2020 and 1 April 2020 presenting to any of Beaumont Health's eight emergency departments (EDs). COVID-19 confirmed patients who remained hospitalised beyond 12 May 2020 were excluded given the absence of final outcome data in this group. Additionally, ambulatory (clinic) setting testing was not available during the study timeframe, and hence, was not evaluated. Data on the cohort were abstracted using automated reports generated through ToadDataPoint multi-platform database query tool from Beaumont's electronic health record (EHR) (EPIC System, Verona, WI, USA). The risk score was automatically calculated and reported in Epic, the most commonly utilised EHR platform in the United States [21, 22] (see Supplementary Appendix). This study was approved as an exempt retrospective chart review by the Beaumont Health Institutional Review Board.

We defined admitted patients as patients with confirmed SARS-CoV-2 infection who required hospital admission to any of the eight Beaumont hospitals. We defined outpatients as patients who were sent home from their initial ED encounter during which a COVID-19 diagnosis was established. To validate the utility of the risk score in triaging patients on their initial visits to the healthcare system, we excluded outpatients who presented to the ED on subsequent encounters and were admitted to the hospital.

Two outcome variables were measured, both using a yes/no binary scale: hospital admission and in-hospital mortality. Hospital admission on the first encounter to the ED was evaluated for the entire cohort, while mortality was evaluated only for inpatients who were discharged prior to 12 May 2020. Mortality was evaluated only for the duration of the COVID-19 hospitalization and out of hospital mortality was not evaluated.

The risk score components are: (i) age divided into four categories, <60 years old, 60-69 years old, 70-79 years old and ! 80 years old; (ii) male sex; (iii) the presence of coronary artery disease; (iv) the presence of congenital heart disease (v) the presence of congestive heart failure; (vi) the presence of end-stage renal disease (ESRD); (vii) the presence of end-stage liver disease (ESLD); (viii) the presence of chronic pulmonary disease (such as pulmonary fibrosis/chronic obstructive pulmonary disease/bronchial asthma); (ix) the presence of diabetes; (x) the presence of hypertension; (xi) the presence of obesity; (xii) nursing home residence; (xii) pregnancy status; and (xiii) immunocompromised status defined by one of: (a) diagnosis of human immunodeficiency virus (HIV) infection, (b) actively receiving chemotherapy, (c) receiving immunosuppressive agents, (d) carrying a diagnosis of iatrogenic immunosuppression. The items included in the score are automatically retrieved by the EHR from different areas of the patient's chart, including problem lists and local hospital registries for chronic conditions, computed, and entered into the patient's record. The maximum score is 15 and each of the 12 items reported receives 1 point if present, apart from age where a patient <60 years old receives 0 points, 60-69 years old receives 1 point, 70-79 years old receives 2 points, and >80 years old receives 3 points (Supplementary Appendix). The score is subsequently divided into three risk categories: (i) green (score 0-2), (ii) yellow (score 3-5), (iii) red (score 6-15), and once validated, it is meant to be available for providers to view and aid in triage decisions in both outpatient and inpatient settings.

The risk score was developed by Dr. David Daniel at Confluence Health. The elements of the risk score were derived from the guidance published by the Centre for Disease Control (CDC) in the United States on conditions that increase the risk of severe illness from COVID-19 for all patients [23] . The risk score was not created from a validation cohort, was only based on expert opinion of the data available. Given the lack of a prior validation cohort from which these variables were given weights, we sought to evaluate the initially selected components of the risk score. No modifications in weight assignment (defined by number of points assigned to each risk factor) were made given the absence of a secondary cohort to validate these modifications.

We included all available patients in the final sample size to minimise selection bias, only excluding patients who did not have an outcome at the time of the analysis and those who were initially discharged from the ED and then returned for re-evaluation. Ascertainment bias was limited via automated reports data collection.

We did not calculate a sample size as we included in our cohort all the patients that met the inclusion criteria as of 1 April 2020.

Descriptive statistics were generated for all variables, which were stratified by both admission and mortality with T-Test to compare COVID-19 Risk of Complications Score and Chi-Square tests to compare all other variables. All numbers were rounded to two decimal places. Multivariate logistic regression models were fit for admission on all inpatient and outpatient encounters and in-hospital mortality for inpatient encounters only. All variables incorporated in the Epic COVID-19 Risk of Complications Score were included in the regression for admission. Due to issues with complete separation of some covariates in the external validation dataset, all variables except pregnancy, congenital heart disease, and ESLD were included in the regression for in-hospital mortality. Multivariate logistic regression results are presented in terms of Adjusted Odds Ratios (AOR) with corresponding 95% confidence intervals and p-values.

An analysis using an external validation cohort was performed. Discrimination was evaluated using a Cross-Validated C-Statistic, along with its corresponding 95% Confidence Intervals and Receiver Operating Characteristic (ROC) curve. C-Statistics !0.7 were considered good and !0.8 were considered strong values [24] . Calibration was assessed using the decile method, a method where patients were divided into ten deciles based on their predicted probability for the outcome as predicted from the regressions. Decile calibration plots with superimposed Local Regression-based (LOESS) calibration curves [25] were generated.

In addition to the external validation analysis, the discrimination of the Green/Yellow/Red categorizations also was evaluated using C-Statistics. The C-Statistics for the Green/Yellow/Red categorizations were compared for differences from the full external validation models with a Wald Test.

Any p-values <.05 were considered as statistically significant associations. All analysis was done in SAS 9.4 (SAS Institute Inc., Cary, NC, USA).

There were 2126 encounters with data extracted from Epic (1305 inpatient encounters and 821 outpatient encounters). We excluded 86 outpatient encounters who subsequently returned to ED at a later date, 14 inpatient encounters where the patient was still admitted as of 12 May 2020 and their ultimate disposition was still unknown, and one inpatient encounter where Epic was unable to procure the necessary information to calculate the COVID-19 Risk of Complications Score.

The final sample includes 2025 encounters, divided in 1290 hospital admission encounters and 735 outpatient encounters who were never admitted to one of the eight hospitals. Each of these encounters represents a unique patient. Descriptive statistics are shown in Table 1 .

Of all encounters in the external validation dataset, 63.70% (95% CI: 61.60%, 65.80%) were hospital admissions. Of the hospital admissions, 17.29% (95% CI: 15.23%, 19.35%) experienced in-hospital mortality. The majority of patients were <60 years old (55.11%) and there was a nearly even split between males (50.72%) and females (49.28%). Half of all patients had hypertension (50.47%) and were obese (56.84%). Around one quarter of the patients had diabetes (27.65%) and chronic pulmonary disease (25.63%), while one-tenth had CAD (10.67%).

The average length of stay for the hospital admission encounters was 8.25 days. Of those whose discharge information is known, the majority were discharged home (58.07%).

In the multivariate model to predict admission, older age, male gender, congestive heart failure, end-stage renal disease, chronic pulmonary disease, diabetes mellitus, hypertension, obesity, and nursing home residence were independently associated with admission (all AOR > 1 and p < .05). While immunocompromised, congenital heart disease, coronary artery disease, endstage liver disease, and pregnancy had lower odds of admission, there were no significant differences found (all AOR < 1 and p ! .05) ( Table 2) .

For prediction of in-hospital mortality in the multivariate model, older age, end-stage renal disease, chronic pulmonary disease, and nursing home residence were significantly associated with in-hospital mortality (all AOR > 1 and p < .05). Other variables that had greater odds, but were not significantly associated with in-hospital mortality, included male gender, immunocompromised status, congestive heart failure, coronary artery disease, diabetes, and obesity (all AOR > 1 and p ! .05). Hypertension had lower odds of in-hospital mortality, but did not meet statistical significance (AOR ¼ 0.70; p ¼ .0607) ( Table 2) .

When reducing the risk score algorithm to categories, the categories were highly predictive of both admission and in-hospital mortality. Compared to Green scores, Yellow Scores (OR: 5.72) and Red Scores (OR: 19.1) had significantly higher odds of admission (both p < .0001). Similarly, Yellow Scores (OR: 4.73) and Red Scores (OR: 13.3) had significantly higher odds of in-hospital mortality than Green Scores (both p < .0001) (Tables 3 and 4) .

The cross-validated C-Statistics for the external validation cohort showed good discrimination for both Admission (C ¼ 0.79 (95% CI: 0.77, 0.81)) and in-hospital mortality (C ¼ 0.75 (95% CI: 0.71, 0.78)). Figure 1 shows the ROC curves for both models.

Calibration for Admission and In-Hospital Mortality are depicted in Figure 2 . For admission, the model significantly overestimated the predicted probability of admission for encounters in the lowest decile of predicted probability of admission (< 22%); however, there was no significant overestimation or underestimation for any encounters in any of the other nine deciles. For in-hospital mortality, there was no evidence of significant overestimation or underestimation of the external validation model.

Upon examination of the stoplight categories (Green/ Yellow/Red), the categorization demonstrates less than good discrimination, in terms of cross-validated C-Statistics, for both Admission (C ¼ 0.59 (95% CI: 0.56, 0.61)) and in-hospital mortality (C ¼ 0.52 (95% CI: 0.48, 0.56)). Not categorizing the algorithm to Green/ Yellow/Red categories led to significantly better discrimination for both admission (C-Statistic increase of 0.20) and in-hospital mortality (C-Statistic increase of 0.23) (both p < .0001). Figure 3 compares the ROC between the models that reduce the scoring system to Green/Yellow/Red categories and the full external validation models for admission and in-hospital mortality, respectively. 

In this study, we utilized a large multicentric retrospective cohort to validate the COVID-19 Risk of Complications Score for predicting hospital admission and in-hospital mortality of patients diagnosed with COVID-19 when presenting to the emergency department (ED). In general, there was very good calibration of the models predicting admission and in-hospital mortality. For admission, the model significantly overestimated the predicted probability of admission for encounters in the lowest decile of predicted probability of admission (<22%); however, there was no significant overestimation or underestimation for any encounters in any of the other nine deciles. For inhospital mortality, there was no evidence of significant overestimation or underestimation. The risk score demonstrated satisfactory discriminatory ability for both outcomes as demonstrated by the AUCs of 0.79 and 0.75, respectively. Categorizing patients into stoplight categories (Green/Yellow/Red) proposed by tool developers, in contrast to using the score in a linear fashion, proved unsatisfactory discriminatory value for hospital admission and mortality, demonstrated by AUCs of 0.59 and 0.52 respectively. Contributing factors to the latter observation include a discrepancy between optimal cut-offs for outcomes in our cohort (score of 2 and 4 for hospital admission and mortality, respectively) and the proposed category cut-offs (0-2 green, 3-5 yellow, 6-15 red). Additionally, different risk score constituents had variable predictive abilities for outcomes and hence using uniform weights to score these constituents may affect the overall predictive ability of the model. The variables constituting the tool have been reported as risk factors for severe COVID-19 illness or mortality, are constituents of other well validated prognostic tools such as the Charlson Comorbidity Index (CCI), or are mortality predictors for other respiratory illnesses [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] . Older age in particular heralds worse outcomes in COVID-19 patients in an incremental [26, 30, 32, 34, 35] . On multivariable analysis of our cohort, we found that in addition to older age, end-stage renal disease, chronic pulmonary disease, and nursing home residence were independently predictive of both hospital admission and in-hospital mortality. Additionally, male gender, congestive heart failure, diabetes mellitus, hypertension, and obesity were predictive of admission. However, different risk score constituents had variable predictive abilities for outcomes and hence using uniform weights to score these constituents may affect the overall predictive ability of the model. An example of that effect is evident contrasting risk posed by being male (AOR: The COVID-19 Risk of Complications Score can be easily distributed and readily accessible to a large portion of United States (US) healthcare providers, due to the availability of risk score constituents in the EHR, the automatic computation of the score, and the wide prevalence of Epic EHR in US healthcare systems [21, 22] . These factors offer an advantage compared to recently published prediction tools [18] that involve web-based calculators requiring physicians to manually input factors and allow for better generalizability in the United States. Following validation with other external cohorts and after further optimisation of cutoffs, this tool could have significant implications in triaging patients in the outpatient setting for ED referral and in the ED for hospital admission. Patients with higher mortality risk could then be triaged to centres with more available intensive care unit (ICU) beds and advanced oxygenation modalities such as extracorporeal membrane oxygenation (ECMO) in anticipation of worse outcomes. These patients may benefit from earlier or more aggressive medical or procedural interventions such as specific pharmacologic therapy or early prone positioning.

This risk score could be instrumental to identifying higher risk COVID-19 patients that may benefit from close follow-up after discharge. The role of close follow-up in reducing readmission rates in patients with heart failure and cirrhosis is well described [36] [37] [38] . Additionally, a longer time to follow-up after hospital discharge is associated with worse outcomes in patients with community acquired pneumonia (CAP) [39] . Tele-visits or provision of monitoring modalities such as ambulatory oximetry might be beneficial to reduce readmissions and improve outcomes. Further analysis should focus on investigating the impact of the risk score usage in decreasing the insurance/ healthcare expenditure for COVID-19 patients.

Our study has several limitations. Limitations relating to the tool include the absence of an initial validation cohort for its constituents resulting in uniform scoring weights of different risk factors, and bias created by missing variables reported in other studies as multivariate predictors of outcomes such as imaging findings, levels of C-reactive protein (CRP), lactate dehydrogenase, D-Dimer, and absolute lymphocyte counts [17, 18, 40] . Limitations relating to our cohort include its retrospective nature, not evaluating mortality in outpatients if it happened outside of our health system, limited outcome data of patients transferred to other facilities, and not analysing time-based outcomes. Additionally, our cohort is limited to the available data in our health system. Statistical limitations included inability to analyse pregnancy, congenital heart disease and end-stage liver disease in the model due to complete separation of these variables. Attempting a penalized regression did not ameliorate these separation effects.

In conclusion, the COVID-19 Risk of Complications Score is a promising, easily distributable, and EHR integrated tool for prediction of hospital admission and in-hospital mortality of COVID-19 patients. However, further refinement of the risk score is required prior to widespread reliance on its use. Future steps should include: (a) validation with other cohorts to assess optimal category cut-offs, (b) evaluation for additional stoplight categories, (c) consideration of different weights of risk score constituents based on their predictive ability, and (d) validation in prospective cohorts with longer follow-up times and with time-based data from symptom onset to outcomes to evaluate timebased outcomes (i.e. time to mortality or hospital admission).

The risk score was created by Dr. David Daniel from Confluence Health, Physician builder/Informaticist, EPIC Emeritus, Clinical Professor of Medicine, UWMC, further modified by the Mayo Clinic and later integrated in EPIC.

A.H. was involved with the development and implementation of the study design and methods and revised the manuscript. All other authors were involved with manuscript preparation, multiple draft revisions, conception of tables and have reviewed and approved the manuscript for submission.

A.H., P.K., and Z.I. had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Book V R 2018 j Red Book Online j AAP Point-of-Care-Solutions

Epidemiology, virology, and clinical features of severe acute respiratory syndrome -coronavirus-2 (SARS-CoV-2

COVID-19) events as they happen

COVID-19 Map -Johns Hopkins Coronavirus Resource Center

Persistence of coronaviruses on inanimate surfaces and their inactivation with biocidal agents

Aerosol and Surface Stability of SARS-CoV-2 as Compared with SARS-CoV-1

Airborne transmission of SARS-CoV-2: the world should face the reality

Comorbidity and its impact on 1590 patients with Covid-19 in China: A Nationwide Analysis

Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

Characteristics of and important lessons from the Coronavirus Disease 2019 (COVID-19) outbreak in China: summary of a report of 72-314 cases from the Chinese Center for Disease Control and Prevention

Treatment options for COVID-19: The reality and challenges

A Trial of Lopinavir-Ritonavir in Adults Hospitalized with Severe Covid-19

Observational study of hydroxychloroquine in hospitalized patients with Covid-19

NIH Clinical Trial Shows Remdesivir Accelerates Recovery from Advanced COVID-19 j NIH: National Institute of Allergy and Infectious Diseases

Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period

Prediction models for diagnosis and prognosis of covid-19 infection: Systematic review and critical appraisal

An interpretable mortality prediction model for COVID-19 patients

About Us j Beaumont Health

Clinical management of severe acute respiratory infection when COVID-19 is suspected

clinical-management-of-severe-acuterespiratory-infection-when-novel-coronavirus-. (ncov)-infection-is-suspected

All of US News' top 20 hospitals use Epic

Information for Healthcare Professionals: COVID-19 and Underlying Conditions j CDC

Applied logistic regression

Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

Report of the WHO-China Joint Mission on Coronavirus Disease 2019 (COVID-19)

A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

Prediction of in-hospital death from community-acquired pneumonia by varying crb-age groups

Comorbidity as the dominant predictor of mortality after hip fracture surgeries

Host susceptibility to severe COVID-19 and establishment of a host risk score: Findings of 487 cases outside Wuhan

Risk factors and trends associated with mortality among adults with hip fracture in Singapore

Estimation of risk factors for COVID-19 mortality -preliminary results

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

Building a COVID-19 Vulnerability Index. medRxiv

Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19. medRxiv

Relationship between early physician follow-up and 30-day readmission among medicare beneficiaries hospitalized for heart failure

The association between outpatient follow-up visits and all-cause non-elective 30-day readmissions: a retrospective observational cohort study

Outpatient telephonic transitional care after hospital discharge improves survival in cirrhotic patients

Long-term prognosis in community-acquired pneumonia

Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China

The authors report no conflicts of interest. The inpatient cohort reported in this manuscript has been evaluated by our research group for predictors of mortality separately in another study that is currently submitted for publication. The outpatient cohort is part of a larger cohort that was analysed in a separate study currently submitted for publication. Neither of those studies evaluated the outcomes reported in this study.

Zaid Imam http://orcid.org/0000-0002-3030-6188