key: cord-0921341-8zq5mdg9
authors: Fisman, D.; Greer, A. L.; Tuite, A.
title: Derivation and Validation of Clinical Prediction Rule for COVID-19 Mortality in Ontario, Canada
date: 2020-06-23
journal: nan
DOI: 10.1101/2020.06.21.20136929
sha: 382920ece2259d175814c088412bb862b4964817
doc_id: 921341
cord_uid: 8zq5mdg9

Background: SARS-CoV-2 is currently causing a high mortality global pandemic. However, the clinical spectrum of disease caused by this virus is broad, ranging from asymptomatic infection to cytokine storm with organ failure and death. Risk stratification of individuals with COVID-19 would be desirable for management, prioritization for trial enrollment, and risk stratification. We sought to develop a prediction rule for mortality due to COVID-19 in individuals with diagnosed infection in Ontario, Canada. Methods: Data from the Ontario provincial iPHIS system were extracted for the period from January 23 to May 15, 2020. Both logistic regression-based prediction rules, and a rule derived using a Cox proportional hazards model, were developed in half the study and validated in remaining patients. Sensitivity analyses were performed with varying approaches to missing data. Results: 21,922 COVID-19 cases were reported. Individuals assigned to the derivation and validation sets were broadly similar. Age and comorbidities (notably diabetes, renal disease and immune compromise) were strong predictors of mortality. Four point-based prediction rules were derived (base case, smoking excluded as a predictor, long-term care excluded as a predictor, and Cox model based). All rules displayed excellent discrimination (AUC for all rules > 0.92 ) and calibration (both by graphical inspection and P > 0.50 by Hosmer-Lemeshow test) in the derivation set. All rules performed well in the validation set and were robust to random replacement of missing variables, and to the assumption that missing variables indicated absence of the comorbidity or characteristic in question. Conclusions: We were able to use a public health case-management data system to derive and internally validate four accurate, well-calibrated and robust clinical prediction rules for COVID-19 mortality in Ontario, Canada. While these rules need external validation, they may be a useful tool for clinical management, risk stratification, and clinical trials.

Since the COVID-19 pandemic was declared by the World Health Organization on March 12, 2020 (1), the spread of SARS-CoV-2 has taken a fearsome toll on global mortality. As of June 11, 2020, over 400,000 deaths worldwide have been attributed to SARS-CoV-2, with many more excess deaths likely related either to infection with the virus or disruption of health systems by epidemics (2) . While most infections with SARS-CoV-2 are mild or even asymptomatic, approximately 20% of recognized infections are sufficiently severe to require hospitalization (3, 4) . Among those hospitalized, 10-20% have an intensive care requirement, usually related to respiratory failure (3-5), though multiorgan system failure (6), clotting abnormalities (7) and angioneogenesis (8) with resultant bleeding are increasingly recognized as severe complications of COVID-19.

Numerous studies have identified clinical factors associated with requirements for intensive care and death among those with COVID-19 infection (9-11). Published prediction models to date have evaluated case-level factors that might predict care diagnosis, more severe disease requiring hospitalization, and poor outcomes (critical illness or death) (9). A recent review identified 16 prediction models focused on prognosis; 14 were based on the COVID-19 epidemic in China and the other two used aggregated public data from a variety of sources (9). The generalizability of these rules to the North American context is unclear. Furthermore, few of these efforts included conversion of prediction models into parsimonious, simple, score-based tools that can be used easily for risk stratification in clinical settings. In the context of COVID-19, a such rules might have important implications for risk-stratification of patients, streamlining decisions around hospital care vs. self-isolation (12), and prioritizing 6 individuals for enrollment in clinical trials of emerging therapies (e.g., convalescent plasma or antiviral drugs), as has been the case with similar tools developed for community acquired pneumonia (13).

Ontario, Canada, had identified over 30,000 virologically confirmed cases of in the province as of June 11, 2020 (14). Each confirmed case is the subject of epidemiological investigation by local public health authorities, who enter epidemiological, clinical and outcome data into the Province's Integrated Public Health Information System (iPHIS). Our objective was to make use of iPHIS data to develop and validate parsimonious, sensitive and specific prediction rules for infection-related death in individuals with COVID-19 in Ontario.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint 7

Ontario is Canada's most populous province, with a current population of 14.7 million (15) . The Province identified imported COVID-19 cases from China, and Iran, in January and February 2020 (16); local epidemic spread of SARS-CoV-2 has been evident since late February 2020 (17). Each of Ontario's 34 public health units is responsible for local case investigation and uploading of case information into the iPHIS data system, which is used for surveillance and case management of notifiable diseases in the Province (18). Ontario's case definition for a confirmed case of requires a positive laboratory test using a validated nucleic acid amplification test, including real-time PCR and nucleic acid sequencing (19).

Information on patient characteristics -including age group (by 10-year intervals), sex, medical comorbidities, long-term care residence, healthcare and emergency service work, case symptoms, dates of symptom onset, testing and reporting, hospitalization and intensive care admission, and mortality was collected for cases. Approximately 80% of all deaths during the Ontario COVID-19 epidemic have occurred in long term care facilities (20), and there has been little transfer of long-term care residents to intensive care units (17).

We randomly assorted cases into derivation and validation sets. Characteristics of the two sets are presented in Table 1 . Univariable logistic regression was used to 8 identify factors associated with mortality in the derivation group. Continuous variables were dichotomized to facilitate score generation and ease of application in clinical settings. When a factor was found to be protective, the covariate evaluated was absence of the factor, so that resultant odds ratios were > 1.

Risk factors significant at P < 0.2, or which were thought a priori to confer important increases in risk (age and sex) were included in model building using a forward stepwise selection algorithm, with covariates selected for P < 0.05, and retained in the model for P < 0.15. We did not include interaction terms in efforts to keep a final prediction rule as simple as possible. The final regression model was transformed to a point-based rule, with each regression coefficient divided by half of the smallest coefficient and rounded to the nearest integer to obtain weighted values. where S is the individual's score, C is the prediction rule's coefficient in a logit model using score as a predictor of death, I is the intercept from the same model.

The discriminatory ability of the prediction rule in the derivation group was quantified through the area under the receiver-operating characteristic curve (ROC AUC), with 95% confidence intervals estimated through 1000 bootstrap replicates.

Calibration was assessed visually and using the Hosmer-Lemeshow test for goodness of fit, which evaluates expected and observed probabilities in population deciles (21).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. .

Some analysts have expressed concern that failure to account for right censoring in could lead to bias in COVID-19 clinical prediction rules (9). As such we created a second prediction rule using Cox-proportional hazards analysis, by identifying factors associated with increased hazard of death using the same selection algorithm as applied to the logistic model described above. Log transformed hazard ratios were converted to point scores using the approach described above.

Discriminative ability of the rule was evaluated using Harrell's C-statistic after constructing a Cox proportional hazards model with the score as the sole covariate in both the derivation and validation sets. ROC analysis, and score calibration, were performed by using the Cox-model-derived score as a predictor in a logistic model.

Smoking status emerged as a protective effect in our base case prediction model; this is likely to be controversial with some users. Furthermore, it might be argued that the known high mortality associated with COVID-19 in long term care settings favors creation of a rule for non-long-term care residents. As such, we made additional rules which excluded smoking status, and which excluded long-term care residents, using the approach described above.

In the base case, models were built using only observations from individuals with complete data; we tested the robustness of our models by evaluating the discriminative ability and calibration of rules in datasets in which missing fields were replaced at random, and in datasets where an attribute was assumed not present if a field was left blank (e.g., if an individual had no record of presence or absence cardiac disease, they were assumed not to have cardiac disease). All analyses were performed . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. .

using Stata version 14.0 (Stata Corporation, College Station, TX). The study was approved by the research ethics board of the University of Toronto.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. .

Of 21,922 COVID-19 cases reported between January 23 and May 15, 2020, 57% were female, and 43% were aged > 59 years. The median time from symptom onset to case reporting was 5 days (IQR 4 to 10 days). Fourteen percent of cases were residents of long-term care facilities; 17% were healthcare workers. Thirteen percent of cases were hospitalized; 2% had record of intubation and/or mechanical ventilation, and case-fatality was 8%. Individuals assigned to the derivation and validation sets were broadly similar, but were significantly more likely to be smokers, less likely to have a history of chronic liver disease, and less likely to die ( Table 1) .

In univariable analyses, death was associated with a broad array of demographic characteristics and comorbid conditions. No association was seen between risk of death and mean neighborhood income or asthma which were not included in subsequent model building ( Table 2) . As age was provided as ordinal, 10-year age groupings (0 to 9, 10 to 19, 20 to 29, etc.) the age coefficient in models represents increased risk per increase in (age/10). Using a forward selection algorithm, we identified 7 independent predictors of death in the derivation group: age, long-term care residence, a history of renal disease, diabetes, chronic obstructive pulmonary disease, and immune compromise, and non-smoking. ( Table 3 ).

The point-based prediction rule was well-calibrated between quantiles of observed and expected risk (Hosmer-Lemeshow χ 2 =1.58; p=0. 0.90) in the derivation group and discriminated extremely well between those who did and did not develop die (ROC AUC in the derivation group=0.95; 95% CI, 0.91-0.96). The median score (interquartile . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. . 12 range) was 13 (6) for survivors and 25 (6) for those who died (p<0.001 by the Wilcoxon rank-sum test). The rule displayed good calibration to outcomes in the validation set (Hosmer-Lemeshow χ 2 =9.16; p=0.16), as well as excellent discrimination (AUC 0.92, 95% CI 0.89 to 0.94) (Figure 1 and Figure 2) .

Three alternate rules (based on a Cox proportional hazards model, a logistic model excluding smoking status, and a model with long-term care residents excluded) were created. These models had excellent discrimination. We statistical found evidence for poor calibration of the model that excluded long-term care residents in the validation set (P = 0.04). The Harrell's C-statistic for a Cox model including age, male sex, diabetes, COPD, and immune compromise was 0.97 in the derivation set, and 0.96 in the validation set. Other fit statistics, and c-statistics for AUC, as well as values of the model intercept and smallest logit model coefficient (for calculation of death probability) are presented in Table 3 and presented graphically in the Supplement.

We re-evaluated all four prediction rules in datasets in which missing variables were assumed to not be present, and in which missing variables were replaced randomly.

Discriminative ability remained good for both randomly replaced datasets (ROC curve AUC 0.84-0.90 for missing observations replaced with zeroes; AUC 0.79-0.83 for missing observations replaced randomly). The large number of observations in datasets with all missings replaced (N = 21,922) resulted in statistically significant differences between observed and expected mortality probabilities (P < 0.001 for all . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. .

analyses by Hosmer-Lemeshow test), but visual inspection suggested that calibration of rules remained very good (Supplement).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. .

Accurate prediction of mortality from COVID-19 has a number of potential applications, including rational decision making for hospital admission, prioritization of high-risk individuals for inclusion in trials of novel therapeutic agents, and to identify high risk individuals for policy purposes (e.g., to inform decisions around risks and benefits of remote work). We demonstrate here that COVID-19 mortality in identified cases can be predicted with remarkable accuracy based on the limited, is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . 15 bias (e.g., individuals predisposed to very mild COVID-19 infection as a result of young age or good general health might be over-represented among those tested for COVID-19 due to smoking-related health concerns like cough). Regardless, a non-causal association with risk may still be useful for clinical prediction; if this association reflects peculiarities of Ontario's approach to COVID-19 testing we expect that it may not be generalizable to other jurisdictions that test more widely.

Similarly, the strong effect of long-term care residence on mortality is unsurprising, given the high fraction of long-term care deaths seen during the Canadian COVID-19 epidemic to date (20). As such we created alternate rules that exclude smoking and long-term care residence; these rules can be used in place of our base case rule, as they have similar discriminative ability. Lastly, to avoid biases that might be introduced by right-censoring (i.e., lack of mortality in individuals in the study cohort as a result of insufficient follow up time) we derived an additional rule using survival methods, which also performed well. There was substantial overlap between all four prediction rules in included covariates: notably, age, diabetes, and immune compromise were included in all four rules we derived, and renal or chronic obstructive pulmonary disease were included in 3 of four rules.

Our analysis had many limitations; the use of a public health record system not explicitly designed as a research tool means that we lack laboratory and radiological results that have been useful in other prediction models (23, 29). Furthermore, missing data was a significant limitation of our dataset, although our models appeared robust even with random replacement of predictors and outcomes that should bias associations towards the null. In that sense, the ability to derive simple, accurate and parsimonious rules, which perform well in split-halves validation, despite limitations . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint 16 in our dataset, may suggest generalizability of application outside Ontario. We hope that other groups will evaluate our rules in other settings.

In summary, we developed and internally validated a prediction rule for COVID-19 mortality using a large and detailed public health line list in the Canadian province of Ontario. The rule was well calibrated and discriminated well and was robust in sensitivity analyses to assess the impact of missing information on predictor variables.

If externally validated, this rule might facilitate decision making during future epidemic waves.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. .

Guan 2020.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint 24 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. 

Mortality in Ontario, Canada

As noted in the text, we used a Cox proportional hazards model to derive point scores for a prediction score. The use of Cox models was intended to avoid bias that might be introduced as a result of incomplete follow up and right censoring. Point scores were derived based on log-hazard ratios. The scores, and values for I and C necessary to predict mortality probability are presented in Table 3 . The score itself was then evaluated as a single covariate in a logit model predicting mortality. Model calibration in derivation (top panel) and validation (bottom panel) sets are presented in Figure S1 below. Figure S2 presents the ROC curves for the derivation (top panel) and validation . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. 

We anticipated that a prediction rule incorporating non-smoking as a risk factor for mortality would be controversial. As such, we created an alternate rule with nonsmoking excluded. Again, rule-based scores, and values for I and C necessary to predict mortality probability are presented in Table 3 . As above model calibration in derivation (top panel) and validation (bottom panel) sets are presented in Figure S3 below. Figure S4 presents the ROC curves for the derivation (top panel) and validation (bottom panel) sets.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint Figure S3 . Observed (circles) and predicted (curve) probability of death in logit modelderived alternate prediction rule that excludes smoking status. Circle size is proportionate to number of deaths for each score. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. 

The extremely high mortality in the long-term care setting might make a prediction rule unhelpful. As such, we created an alternate rule with long-term care residents excluded. Again, rule-based scores, and values for I and C necessary to predict mortality probability are presented in Table 3 . As above model calibration in derivation (top panel) and validation (bottom panel) sets are presented in Figure S5 below. Figure S6 presents the ROC curves for the derivation (top panel) and validation (bottom panel) sets.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. 

Observed Probability of Death . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint Figure S5 . Observed (circles) and predicted (curve) probability of death in logit modelderived alternate prediction rule that excludes long term care residents. Circle size is proportionate to number of deaths for each score.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. 

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. .

The iPHIS dataset was limited by substantial data missingness. To assess the robustness of our rules, we replaced missing variables in two ways: first, by assuming that missingness (for covariates or death) indicated that they were not present or did not occur (i.e., replaced missing values as zero); and second, by assuming that variables were missing completely at random, and replacing missing values randomly based on their frequency of observation among non-missings. The latter approach had the effect of substantially increasing the number of deaths available in the dataset. Notwithstanding the likely introduction of misclassification via both of these approaches, model calibration (based on visual inspection) and discrimination remained very good. Calibration (left sided panels) and discrimination (right sided panels) are presented graphically for all four prediction rules with missing values replaced as zeroes ( Figure S7 ) and missing values replaced at random ( Figure S8 ).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted June 23, 2020. . 

Observed Probability of Death . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. . https://doi.org/10.1101/2020.06.21.20136929 doi: medRxiv preprint numbers of deaths at each score level. Right panel figures represent ROC curves for the same prediction rules with missing variables replaced as zeroes.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted June 23, 2020. 

World Health Organization. WHO announces COVID-19 outbreak a pandemic

Coronavirus resource tracker. Available via the Internet at

Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan