key: cord-0330021-beb7vx8d authors: Ebell, M.; Hamadani, R.; Kieber-Emmons, A. title: Simple risk scores to predict hospitalization or death in outpatients with COVID-19 date: 2022-01-14 journal: nan DOI: 10.1101/2022.01.14.22269295 sha: 496d96553ab5eb0da9880920d061f4615470030e doc_id: 330021 cord_uid: beb7vx8d Importance Outpatient physicians need guidance to support their clinical decisions regarding management of patients with COVID-19, in particular whether to hospitalize a patient and if managed as an outpatient, how closely to follow them. Objective To develop and prospectively validate a clinical prediction rule to predict the likelihood of hospitalization for outpatients with COVID-19 that does not require laboratory testing or imaging. Design Derivation and temporal validation of a clinical prediction rule, and prospective validation of two externally derived clinical prediction rules. Setting Primary and Express care clinics in a Pennsylvania health system. Participants Patients 12 years and older presenting to outpatient clinics who had a positive polymerase chain reaction test for COVID-19. Main outcomes and measures Classification accuracy (percentage in each risk group hospitalized) and area under the receiver operating characteristic curve (AUC). Results Overall, 7.4% of outpatients in the early derivation cohort (5843 patients presenting before 3/1/21) and 5.5% in the late validation cohort (3806 patients presenting 3/1/21 or later) were ultimately hospitalized. We developed and temporally validated three risk scores that all included age, dyspnea, and the presence of comorbidities, adding respiratory rate for the second score and oxygen saturation for the third. All had very good overall accuracy (AUC 0.77 to 0.78) and classified over half of patients in the validation cohort as very low risk with a 1.7% or lower likelihood of hospitalization. Two externally derived risk scores identified more low risk patients, but with a higher overall risk of hospitalization (2.8%). Conclusions and relevance Simple risk scores applicable to outpatient and telehealth settings can identify patients with very low (1.6% to 1.7%), low (5.2% to 5.9%), moderate (14.7% to 15.6%), and high risk (32.0% to 34.2%) of hospitalization. The Lehigh Outpatient COVID Hospitalization (LOCH) risk score is available online as a free app: https://ebell-projects.shinyapps.io/LehighRiskScore/. Most patients with COVID-19 are initially evaluated in the outpatient setting, and a decision must be made whether to manage them as outpatients or to hospitalize them. For those managed initially in the outpatient setting, a decision must also be made whether they require close follow-up or whether they are at a low risk of deterioration and can be told to simply follow-up as needed. A risk score designed for outpatients could help support clinician decisionmaking around hospitalization and follow-up. However, while many COVID-19 risk scores have been proposed, almost all have been developed and validated to predict mortality in hospitalized patients. 1 These include the 4C risk score, 2 the ABCS risk score, 3 and the COVID-GRAM risk score. 4 Almost all of these risk scores require laboratory tests and in some cases imaging, which is often unavailable in the outpatient or telehealth settings. An exception is the COVID-NoLab risk score for inpatient mortality, which requires only oxygen saturation, age, and the respiratory rate. 5 Additionally, the OutCoV score was derived and internally validated in Switzerland to predict the likelihood of hospitalization among outpatients and does not require laboratory testing. 6 It consists of 5 easily ascertained variables: age, fever, dyspnea, hypertension, and chronic respiratory disease. However, it has not been externally validated. Lehigh Valley Hospital, Inc. (LVH) is a not-for-profit academic community hospital, which is a legal entity of Lehigh Valley Health Network (LVHN). LVHN has grown to become the largest health care provider in the Pennsylvania Lehigh Valley region, with nine not-for-profit hospital campuses, nearly 2,000 inpatient beds, more than 275 physician practice locations, 25 health centers and 20 ExpressCARE locations, and includes the region's only Children's Hospital. The LVH hub is located in the Lehigh Valley region of eastern Pennsylvania, approximately 90 miles from New York City, 60 miles from Philadelphia, and 20 miles from the New Jersey border, and encompasses Allentown, the third largest city in Pennsylvania. According to the 2020 Census, Allentown is home to 125,845 people. Of this population, 52% identify as Latino, 14.7% as Black/African American, and 2.9% as Asian. 7 We assembled a dataset with patients who had been diagnosed with COVID-19 in any LVHN outpatient primary care or ExpressCARE clinic. The objectives of this study were to prospectively validate the OutCoV and COVID-NoLab risk scores for the prediction of hospitalization or death among outpatients, and to develop and temporally validate novel risk scores that do not require laboratory tests or imaging to predict the likelihood of hospitalization or death in outpatients. The electronic health record was used to identify all outpatients diagnosed with COVID-19 by polymerase chain reaction test between March 13, 2020 and September 30, 2021. Data were obtained from query of LVHN's Epic electronic medical record and included all primary care and ExpressCARE outpatient visits during the time window that included a COVID-19 PCR order from that visit, and a subsequent positive COVID-19 PCR test result. Patients with missing data for age, respiratory rate, or oxygen saturation were excluded, as were patients less than 12 years of age. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint Analytic plan Univariate analysis was performed using a chi-square test for dichotomous variables, Student's t-test for normally distributed continuous variables, and the Kruskal-Wallis test for nonnormally distributed variables to identify individual patient characteristics associated with hospitalization or death. Potential cutpoints for continuous variables were selected by inspection of histograms and contingency tables. All comorbidities significantly associated with hospitalization were combined into a single "any comorbidity" variable. Patients presenting prior to March 1, 2021 were used to derive the risk score (early cohort), and patients presenting from March 1, 2021 onwards were used to validate it (late cohort). Logistic regression was performed with hospitalization or death as the dependent variable, and all patient characteristics significantly associated with hospitalization or death in univariate analysis as the independent variables. We evaluated multiple models using different cutoffs for oxygen saturation, age and respiratory rate in the derivation group to identify the models with the highest area under the receiver operating characteristic curve. The final models were converted to simple point scores by dividing each beta-coefficient by the smallest beta-coefficient and rounding the resulting number. Low, moderate and high-risk groups were identified by inspection of a table showing the likelihood of hospitalization associated with each risk score. The primary goal was to identify as large as possible a low-risk group with a less than 3% likelihood of hospitalization. The resulting risk score was then validated in the late cohort of outpatients. Analysis was performed using Stata v. 17 (StataCorp, College Station, Texas). The study was approved by the University of Georgia Human Subjects Office (PROJECT00004060) and LVHN staff received approval to use the University of Georgia determination. Data were available for 13,418 outpatients diagnosed with COVID-19. After excluding patients under age 12 and patients with missing data, the final dataset included 9649 outpatients with COVID-19. The early cohort had 5843 patients while the late cohort had 3806 patients. A total of 641 patients were hospitalized and of that number, 55 died. Of the hospitalized patients, 89 of 641 (13.9%) were hospitalized on the same day as their outpatient visit. No non-hospitalized patients died following a diagnosis of COVID-19. The overall likelihood of hospitalization or death was lower in the late cohort than in the early cohort (5.5% vs 7.4%). The characteristics of patients who were hospitalized or died and of those who were not are summarized in Table 1 . Increasing age, increasing respiratory rate, lower oxygen saturation, a complaint of dyspnea, and all comorbidities were associated with an increased likelihood of hospitalization. Three logistic regression models were developed based on the assumption that in telehealth settings oxygen saturation, respiratory rate, or both may not be available. Model A used age, dyspnea, and the presence of a comorbidity; Model B added the respiratory rate, and Model C added oxygen saturation. The models are shown in Table 2 , along with corresponding points assigned to each variable based on the beta-coefficient. The three models are of similar overall All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint accuracy based on the area under the receiver operating characteristic curve, with a range from 0.772 to 0.785. The performance of each risk score in the early (derivation) and late (validation) cohorts is summarized in Table 3 . Each model classified over half of patients in the very low risk group, with a 2.5% or lower risk of hospitalization or death in the early derivation cohort, and less than or equal to 1.7% in the later validation group. To determine the accuracy in a partially vaccinated population, we evaluated the accuracy of Risk Score A in a very late cohort of patients identified after June 1, 2021. We found hospitalization rates of 12/1199 (1.0%), 23/519 (4.4%), 32/259 (12.4%) and 15/49 (30.6%) in the very low, low, moderate and high-risk groups respectively. The classification accuracy of the OutCoV and COVID-NoLab risk scores in the late cohort is also shown in Table 3 . These risk scores identified more patients in the low-risk groups, but also had higher rates of hospitalization in that group, 3.8% to 4.0% in the early cohort and 2.8% in the late cohort. We developed 3 simple risk scores for hospitalization in outpatients with COVID-19, and prospectively validated them in the same population. Each risk score included age, dyspnea and the presence of a comorbidity, with model B adding the respiratory rate and Model C also adding oxygen saturation. The three models all had good accuracy based on the AUC (0.772 -0.785). In the validation group, the risk scores identified a very low risk group that comprised 53% to 57% of the entire population with a likelihood of hospitalization of 1.7% or less. These patients could potentially be managed initially as outpatients with guidance to contact their primary care physician in the event of worsening symptom. The low risk had a 5.2% to 5.9% likelihood of hospitalization, which is similar to that for the population as a whole. About 1 in 6 patients fell into either the moderate risk (14.7% to 15.6% hospitalized) or high-risk groups (32.0% to 34.2% hospitalized). If managed as outpatients, these patients should be given an oxygen saturation monitor with daily check-in from a nurse or other clinician to assess their status. We also evaluated the accuracy of two previously developed clinical prediction rules. The COVID-NoLab score was originally designed for inpatient risk prediction, but very few patients in our sample had either oxygen saturation < 93% or respiratory rate > 30 breaths/minute. It identified very few high-risk patients, and the low-risk group had a higher likelihood of hospitalization than in any of our 3 novel risk scores (2.8% vs. 1.6% to 1.7%). The OutCoV risk score placed a larger number of patients in the low-risk group (77% vs 53% to 57% with the novel risk scores), but a higher percentage of low-risk patients were hospitalized (2.8% vs 1.6% to 1.7%). Depending on the tolerance for risk, this may be an acceptable trade-off. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint Our study had several notable strengths. We had access to a large, real-world population of outpatients being managed in primary and urgent care settings. The risk scores developed in the earliest 60% of patients validated very well in the later cohort of patients, with similar classification accuracy. The actual hospitalizations were consistently slightly lower in the validation cohort due to the overall lower likelihood of hospitalization as the pandemic evolved. The risk scores are simple, and even the simplest had very good overall accuracy and was able to identify clinically useful lower and higher risk groups. In addition, it was found to be accurate in a late cohort of patients who presented in summer and fall of 2021. While telehealth patients were not included in our sample, all of our clinical variables (especially for Risk Score A) can be easily obtained in that setting. The study also had limitations. Data were obtained retrospectively from the electronic health record, and we were unable to directly verify diagnoses. Additionally, visits to the LVHN network for COVID-19 concerns were available both through telehealth (video or phone encounters) as well as outpatient and ExpressCARE visits during the initial months of the pandemic and then also during surge windows. Thus, this sample will not include some individuals in the health system who chose to do telehealth visits for their symptoms and COVID-19 testing needs. Selection bias regarding patient characteristics of those individuals who did come to an in-person visit could have affected the risk scores. However, it is more likely that those who sought in-person outpatient or ExpressCARE visits were actually sicker, and hence the risk score population may trend towards those with a higher risk of hospitalization. An additional limitation is that the variants of focus for this work included the ancestral, Beta, Alpha and Delta strains, and was completed prior to the emergence of the Omicron variant. Further refinement and validation of the prediction rule, with Omicron's likely decreased hospitalization rates, including in other populations, will be an important next step of this work. It will also be important to evaluate the risk score in vaccinated populations, possibly incorporating lack of vaccination as an additional risk factor. We have developed and temporally validated 3 simple risk scores for outpatients with COVID-19 that identify a large proportion of outpatients who have a ≤1.7 risk of hospitalization. They also identify one in six patients in the moderate and high-risk groups who have a substantial likelihood of eventual hospitalization and warrant very close follow-up or referral to the emergency department for further evaluation. Risk score A, which we call the Lehigh Outpatient COVID Hospitalization (LOCH) risk score is available online as a simple interactive app: https://ebell-projects.shinyapps.io/LehighRiskScore/. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.14.22269295 doi: medRxiv preprint Table 3 . Classification accuracy of 3 novel risk scores and 2 externally derived risk scores in the early derivation and late validation cohorts. External validation of prognostic scores for COVID-19: a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: Development and validation of the 4C Mortality Score. The BMJ A biomarker-based age, biomarkers, clinical history, sex (ABCS)-mortality risk score for patients with coronavirus disease 2019 Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients with COVID-19 Development and Validation of the COVID-NoLab and COVID-SimpleLab Risk Scores for Prognosis in 6 US Health Systems Development and validation of the OUTCoV score to predict the risk of hospitalisation among patients with SARS-CoV-2 infection in ambulatory settings: a prospective cohort study