key: cord-0946254-xu9pb9ul
authors: Guillamet, C. V.; Guillamet, R. V.; Kramer, A. A.; Maurer, P. M.; Menke, G. A.; Hill, C. L.; Knaus, W. A.
title: TOWARD A COVID-19 SCORE-RISK ASSESSMENTS AND REGISTRY
date: 2020-04-20
journal: nan
DOI: 10.1101/2020.04.15.20066860
sha: 324a00d600f3314ab4470c8eb6d29b04ffeca4ac
doc_id: 946254
cord_uid: xu9pb9ul

ABSTRACT Importance Critical care resources like ventilators, used to manage the current COVID-19 pandemic, are potentially inadequate. Established triage standards and guidelines may not contain the most appropriate severity assessment and outcome prediction models. Objectives Develop a draft pandemic specific triage assessment score for the current COVID-19 pandemic. Design a website where initial Toward a COVID-19 Scores (TACS) can be quickly calculated and used to compare various treatment strategies. Create a TACS Registry where data and outcomes for suspected and confirmed COVID-19 patients can be recorded. Use the TACS Registry to develop an influenza epidemic specific database and score for use in future respiratory based epidemics. Design, Setting, Participants Retrospective analysis of 3,301 ICU admissions with respiratory failure admitted to 41 U.S. Intensive Care Units from 2015-19. Independent external validation on 1,175 similar ICU Admissions using identical entry criteria from Barnes Jewish Hospital (BJH), Washington University from 2016-2019. Main Outcomes TACS was created with 16 readily available predictive variables for risk assessment of hospital mortality 24 hours after ICU Admission and the need for prolonged assisted mechanical ventilation (PAMV) ( >>96 hours) at 24- and 48-hours post ICU admission. Results TACS achieved an Area Under the Curve (AUC) for hospital mortality after 24 hours of 0.80 in the development dataset; 0.81 in the internal validation dataset. At a probability of 50% hospital mortality, positive predictive value (PPV) was 0.55, negative predictive value (NPV) 0.89; sensitivity 22%, specificity 97%. For PAMV after 24 hours, the AUC was 0.84 in the development dataset, 0.81 in the validation dataset. For PAMV after 48 hours, the AUC was 0.82 in the development dataset, 0.78 in the validation dataset. In the external validation the AUC for TACS was 0.76 +/- 0.024. We launched a website that is scaled for mobile device use ( https://covid19score.azurewebsites.net/) that provides open access to a user-friendly TACS Calculator for all predictions. We also designed a voluntary TACS Registry for collection of data and outcomes on ICU Admissions with COVID-19. Conclusions and Relevance Toward a COVID-19 score is a starting point for an epidemic specific triage assessment that could be used to evaluate various approaches to treatment. The TACS Registry provides the ability to establish a respiratory specific outcomes database that can be used to create a triage approach for future such pandemics.

Critical care resources like ventilators, used to manage the COVID-19 pandemic, are reported inadequate worldwide 1 . In the US established triage standards, such as Ventilator Allocation Guidelines developed by New York State 2 , by Johns Hopkins 3 , and Pennsylvania 4 , have been recently endorsed 5 .These represent substantial efforts to design an ethical allocation system for a range of disaster scenarios.

One component of these may not be optimal in the current pandemic: sole reliance on the Sequential Organ Failure Assessment (SOFA) score for severity measurement 6 . SOFA was developed primarily for use in hospitalized sepsis. It has one respiratory measurement: the PaO2/FIO2 ratio. The other five measures document deterioration in other organ systems. Published data consistently report many COVID-19 patients did not develop multi organ failure during the Intensive Care Unit stay [7] [8] [9] . US and European colleagues treating COVID-19 patients confirm that many deaths are from primary respiratory, not multiple system, failure 10 . SOFA scores for triage in a COVID-19 pandemic may then not discriminate who would benefit from intubation and only be useful late in a patient's course, after they have received multiple days or weeks of mechanical ventilation. More so, since COVID-19 patients will presumably have lower SOFA score at the time of intubation compared to septic patients they will be allocated a ventilator even though current articles report a significantly higher mortality 7, 11 .

Our objective was to begin a process whereby a primarily respiratory based severity assessment measure that also incorporated other known patient characteristics associated with . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . https://doi.org/10.1101/2020.04. 15.20066860 doi: medRxiv preprint this COVID-19 epidemic and patient outcomes could be established. More importantly, we sought to establish a COVID-19 Registry so a database that could be used to develop a new score for future pandemics.

We designed and validated an initial TACS based on 3,301 adult patients with a clinical diagnosis of respiratory distress admitted to 41 ICUs in the US from 2015-19.

We performed an additional external validation of the TACS in 1,175 ICU Admissions to Barnes Jewish Hospital (BJH), Washington University from 2016-2019.

The statistical models were initially developed and internally validated on admissions from 1/1/2015 through 9/30/2015 in Medical Decision Network's , Charlottesville, Va. (MDN) Phoenix ICU database. Phoenix provides data critical to research efforts such as the development of TACS as well as providing insight and understanding to clinicians and administrators making front line decisions that impact patient care. The dataset provided by MDN for the development of TACS consists of 3,301 de-identified admissions admitted to 41 adult ICUs. To qualify, an ICU had to send electronically captured vital signs to MDN. These ICUs were situated in the Southeast, Northeast, Mid-Atlantic and Pacific West regions. Patients had to be 18 years and above and have one of the following admitting diagnoses: asthma, COPD, pneumonia (bacterial, viral, or parasitic), pulmonary edema, respiratory arrest, restrictive lung disease (fibrosis, sarcoidosis), sleep apnea, or hemorrhage/hemoptysis. Variables considered as predictors were organized as demographics, comorbidities and , pulmonary physiology parameters. Vital signs and labs were formatted dependent on methodology taken from the Acute Physiology, Age, and Chronic Health Evaluation (APACHE) system [12] [13] [14] . The vital signs data were aggregated into minimum and maximum value within the first 24 hours after admission (e.g. max_hr24) and within 24 and 48 hours (e.g. max_temp48). The same was carried out for albumin and ph. Based on clinical knowledge the following vitals and labs were entered into the multivariable models: Maximum heart rate Maximum respiratory rate Minimum mean arterial pressure Maximum temperature Minimum albumin Minimum pH Missing values for albumin were imputed as the median value of 2.80, while missing pH values were imputed with the median value of 7.39. None of the min/max vital signs had missing values. Along with vital signs and the two lab values, a patient's age, gender = female, square root of the length of time between hospital admission and ICU admission, (This was done to reduce the . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 20, 2020. . https://doi.org/10.1101/2020.04. 15.20066860 doi: medRxiv preprint effect of extreme ( long stay) patients), .minimum PaO2:FiO2 ratio during the first 24 hours after admission, maximum Glasgow Coma Score during the first 24 hours after admission, the presence or absence of the following comorbidities: COPD, congestive heart failure, chronic alcoholism, body mass index > 30, and chronic tobacco use. When arterial blood gases where not available, PaO2/FiO2 ratio was imputed by the APACHE methodology, = 80/.0.21. Another variable combined seven comorbidities indicative of immunosuppression was derived by a patient having one or more of the following comorbidities based on the APACHE methodology: AIDS, cirrhosis, leukemia, lymphoma, metastatic cancer, hepatic failure, and diabetes. Finally, interaction terms were created for age with COPD, congestive heart failure, body mass index, and male gender. All of these variables were chosen based on clinical expertise, the emerging literature describing COVID-19 cases, and descriptions provided by front-line practitioners and no addition or removal of variables was performed based on statistical significance. To account for the effects of when a patient was placed on mechanical ventilation, a variable called "initMV" was computed as the duration between ICU admission and hospital admission. If greater than 1.00 days, this value was set equal to zero for the 24-hour model (see below); for the 48-hour models (see below) initMV was truncated at 2.00 days.

The MDN data set was divided randomly 2:1 into development and validation data sets. A logistic regression procedure was carried on the development data set. Variables remained in the model regardless of statistical significance, as they were deemed clinically important. Model accuracy was determined by the area under the receiver operating characteristic curve 15 and the ratio of observed to predicted outcomes. The coefficients from the development data set model were then fed into the validation data set, and the same statistics on this group of patients were obtained. Finally, a cut-point probability of 0.50 was used to calculate the positive predictive value, negative predictive value, sensitivity and specificity of the model. There were three outcomes for which the above procedures were carried out: hospital mortality given labs and vital signs at 24 hours, PAMV given labs and vital signs at 24 hours, and PAMV given labs and vital signs at 48 hours. For the outcome mortality at 24 hours, only a patient's first ICU admission was included (to avoid counting one patient twice). When developing the models with PAMV as the outcome, all ICU admissions were included, except patients not in the ICU after 24 and 48 hours respectively were excluded.

For external validation, the 24h TACS mortality model was replicated in the Barnes Jewish Hospital dataset. The patients were selected with the same entry criteria as our developmental and validation datasets (see statistical appendix).

University -BJH datasets, respectively. All values given are the mean or percentage, depending on the variable's distribution. The MDN data set had patients who were older, had less immunocompromised comorbidities, longer previous length of stay, ICU length of stay, and a higher percentage of patients with prolonged acute mechanical ventilation ("PAMV"; duration on ventilation > 96 hours) (14) .

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. Table 3 displays the performance for predicting hospital mortality within the first 24 hours of ICU admission and prolonged ventilation within the first 24 and 48 hours. TACS achieved an Area Under the Curve (AUC) for predicting hospital mortality after 24 hours of ICU treatment of 0.80 in the development dataset; 0.81 in the internal validation dataset. At a probability of 50% for hospital mortality, the positive predictive value (PPV) is 0.55, negative predictive value (NPV) 0.89; sensitivity 22%, specificity 97%. We also performed an external validation of the TACS 24-hour mortality model on 1,1175 ICU patients to Washington University/BJH treated between 2016-19. Mortality prediction at 24 hours the TACS AUC was 0.76 +/-0.024.

We have developed an initial model of a respiratory oriented Toward a COVID 19 Score designed to be useful in possible triage decisions and to compare outcomes from various treatment approaches in the current pandemic. It was designed specifically with this pandemic in mind. We used the early reports from China and discussions with front line practitioners treating COVID-19 patients as the basis for selecting predictor variables. While it focuses on respiratory physiology parameters, it also has seven possible choices for immunosuppression among other pre-existing conditions, as these are well-documented and very important risk factors. The final model is broad enough to include all possible current and future COVID trajectories but also narrow enough to focus on death from respiratory causes such as Adult Respiratory Distress Syndrome (ARDS). While previous attempts at developing an ARDS mortality prediction models were inferior to more general multi organ failure scores such as APACHE, those models combined a very heterogenous ARDS population 16 The model relied on age, bilirubin, hematocrit and net volume balance over 24hrs. Pneumonia severity prediction tools including respiratory parameters have maintained their predictive performance (curb-65, psi). Since current reports describe COVID-19 pneumonia as predominantly a single organ failure we decide to focus on respiratory parameters.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . https://doi.org/10.1101/2020.04. 15.20066860 doi: medRxiv preprint A precursory study by Luo et al described that direct ARDS had fewer organ failures and different mortality predictors than indirect ARDS (e.g. non-pneumonia sepsis) despite similar mortality rates. The lung injury scores correlated with mortality in direct ARDS but not in indirect ARDS 17 . In our study we also focused on lung physiology parameters along with variables that have been described in COVID reports.

In rationing situations, the main use for a risk stratification score would be to discriminate survivors and non-survivors among the critically ill so patients more likely to survive will be allocated a ventilator. A secondary use would be serial measurements to assess the trajectory of intubated patients and predict patients who will require prolonged mechanical ventilation. A third use would be to compare outcomes from various treatment strategies. We initially focused on these objectives. While earlier reports from China showed that higher SOFA scores were associated with higher mortality, all hospitalized patients were included 18 . More than half of the patients were admitted to the general wards therefore comparing non-critically ill and critically ill patients. This comparison will not be relevant in crisis situations when the selection criteria will be applied solely to critically ill patients. This is why in our model, we included patients who had an ICU stay. Other studies have used SOFA in critically ill patients but with very small samples sizes, only 9 outcomes thus questioning the validity of the analysis 19 . When analyzing 52 critically ill patients, Yang et al found the initial 24hr SOFA score was 4 in survivors compared to 6 in non-survivors 19 . The traditional cutoff for SOFA is < 6 which correlates with a mortality of 10% 20 . The findings by Yang et al reinforce the concept that early SOFA will not discriminate between survivors and non-survivors as most patients present to the ICU in single organ failure. In the study by Yang et al, the in-hospital mortality was 61.5% although the SOFA score in non-survivors predicted a mortality below 10%.

The final TACS 24 hour mortality model displayed good discrimination within the independent external Washington University dataset. We anticipate, however, that when the model is applied to a COVID-19 ICU cohort, the observed hospital death rates will be much higher than the 13% -14 % found in our derivation, validation, and external datasets 9 . Under these circumstances, the model's PPV at higher probabilities will be strengthened. In a pandemic, patients with the highest likelihood of survival should have priority access to therapy. It is our expectation that TACS should be able to identify early the patients more likely to benefit from life support when resources are limited.

The major limitation of TACS is that it has not been applied to patients with a confirmed COVID 19 diagnosis. This is a major question. But, waiting for a substantial database of COVID-19 cases with confirmed outcomes in this country, meant there will have been many deaths and many decisions made with only a potentially adequate severity/outcome estimate, SOFA, as part of the decision making. We have therefore chosen to release it now with the intent of being able to acquire new COVID 19 cases as quickly as possible and improving upon this base. We will commit to providing new data on TACS's performance with COVID-19 patients as soon as possible. We also acknowledge that we will need to create outcome predictions for later in . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . https://doi.org/10.1101/2020.04. 15.20066860 doi: medRxiv preprint the ICU stay. These will be a major focus of the TACS Registry along with development of a database in order to create a new revised score for use in the next such pandemic.

We have launched a website with mobile access ( https://covid19score.azurewebsites.net/) that give open access to a user-friendly TACS Calculator for all predictions. Because TACS was developed on non-COVID cases, we believe it can be applied to other causes of respiratory failure as appropriate. A voluntary registry option reporting subsequent patient outcomes will enable us to further validate and make improvements in the score and our understanding of COVID-19. The TACS registry requests registration and the recording of patient outcomes ( hospital mortality, hospital LOS, and duration of mechanical ventilation). It also enables data to be recorded over the course of the hospital stay. Collection of a pandemic specific database will allow use to create a new respiratory based resource for use in future such pandemics.

The Toward A COVID-19 score is designed to be used within an overall rationing approach when critical care resources are scarce (2) (3) (4) (5) . It provides an earlier and conceptually superior prediction of outcomes compared to SOFA. It also can be used to compare outcomes from various approaches to treatment of COVID-19 ICU admissions. Finally, we have established a COVID-19 Registry to collect and store data and outcomes on patients treated during this pandemic. TACS Registry data will hopefully be available to develop a new score for use in the next such pandemic (21).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted April 20, 2020. . An SMR > 1.0 indicates more outcomes ( mortalities or prolonged ventilations ) than expected based on patient characteristics. An SMR < 1.0 indicates fewer observed outcomes then estimated. p-value is the probability under the Chi-square distribution that the test statistic Σ(observed (0,1) -predicted)^2 / Σ(observed (0, 1) * predicted ) exceeds 3.842. AU-ROC = area under the Receiver Operating Characteristics curve. AUC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUC of 0.0; one whose predictions are 100% correct has an AUC of 1.0. A value of 0.5 means a prediction equal to a coin flip or chance.

Positive Predictive Value is probability that subjects with a positive prediction truly have the outcome ( hospital mortality and/or prolonged ventilation).

Negative Predictive Value is the probability that subjects with a negative prediction truly don't have the outcome (hospital mortality and/or prolonged ventilation).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 20, 2020. Observed vs. Predicted Prolonged Ventilation at 24 hr.

Observed Predicted . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted April 20, 2020. 

Facing Covid-19 in Italy -Ethics, Logistics, and Therapeutics on the Epidemic's Front Line

New York State Department Health. Ventilator Allocation Guidelines

Too Many Patients…A Framework to Guide Statewide Allocation of Scarce Mechanical Ventilation During Disasters

A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic

The Toughest Triage -Allocating Ventilators in a Pandemic

Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit

Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease

Association of Cardiac Injury With Mortality in Hospitalized Patients With COVID-19 in Wuhan, China. JAMA Cardiol

Covid-19 in Critically Ill Patients in the Seattle Region -Case Series

John E. Arbo, MD, Associate Director of the Albert Einstein

Contacted 3/25/2020. Douglas B. White MD, Vice Chair and Professor of Critical Care Medicine

Characteristics and Outcomes of 21 Critically Ill Patients With COVID-19 in Washington State

APACHE II: a severity of disease classification system

The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults

Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients

The meaning and use of the area under a receiver operating characteristic (ROC) curve

Short-Term Mortality Prediction for Acute Lung Injury Patients: External Validation of the ARDSNet Prediction Model

Clinical Predictors of Hospital Mortality Differ Between Direct and Indirect ARDS

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. The Lancet

Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. The Lancet Respiratory Medicine

Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Working group on "sepsis-related problems" of the European Society of Intensive Care Medicine