key: cord-0686883-db7d1g68
authors: Nyman, Mark A.; Jose, Thulasee; Croghan, Ivana T.; Parkulo, Mark A.; Burger, Charles D.; Schroeder, Darrell R.; Hurt, Ryan T.; O’Horo, John C.
title: Utilization of an Electronic Health Record Integrated Risk Score to Predict Hospitalization Among COVID-19 Patients
date: 2022-01-22
journal: J Prim Care Community Health
DOI: 10.1177/21501319211069748
sha: 37c2eede5a08abe3513baee84eb443d47a627037
doc_id: 686883
cord_uid: db7d1g68

OBJECTIVE: To evaluate the performance of an Electronic Health Record (EHR) integrated risk score for COVID-19 positive outpatients to predict 30-day risk of hospitalization. PATIENTS AND METHODS: A retrospective observational study of 67 470 patients with COVID-19 confirmed by polymerase chain reaction (PCR) test between March 12, 2020 and February 8, 2021. Risk scores were calculated based on data in the chart at the time of the incident infection. RESULTS: The Mayo Clinic COVID-19 risk score consisted of 13 components included age, sex, chronic lung disease, congenital heart disease, congestive heart failure, coronary artery disease, diabetes mellitus, end stage liver disease, end stage renal disease, hypertension, immune compromised, nursing home resident, and pregnant. Univariate analysis showed all components, except pregnancy, have significant (P < .001) association with admission. The Mayo Clinic COVID-19 risk score showed a Receiver Operating Characteristic Area Under Curve (AUC) of 0.837 for the prediction of admission for this large cohort of COVID-19 positive patients. CONCLUSION: The Mayo Clinic COVID-19 risk score is a simple score that is easily integrated into the EHR with excellent predictive performance for severe COVID-19. It can be leveraged to stratify risk for severe COVID-19 at initial contact, when considering therapeutics or in the allocation of vaccine supply.

The COVID-19 pandemic continues to challenge clinical practice. In 2020, COVID-19 was the third leading cause of death in the United States. 1 In general, most cases of COVID-19 are mild, but up to 14% are severe and up to 5% are critical. 2 With up to 20% of patients needing hospitalization, algorithms have been proposed to direct the care efficiently and effectively for patients newly diagnosed with COVID-19, with mild cases managed at home and severe cases managed in hospital. 3 Those managed at home who are at high risk of developing severe COVID-19 disease have been offered remote patient monitoring to assess vital signs and symptoms to allow early detection of severe disease in need of hospital care. [4] [5] [6] [7] [8] Given limited resources, it is important to accurately identify those patients at highest risk for severe disease who warrant remote monitoring while at home. An easily accessible risk score in the electronic health record (EHR) calculated using available discrete data in the chart to estimate the likelihood of hospital admission may assist clinicians to allocate remote patient monitoring resources appropriately to provide optimal care for patients. As such, we describe the performance of an EHR integrated risk score to predict the 30-day risk of hospitalization among a large cohort of COVID-19 positive patients.

This retrospective cross-sectional research study of a practice-improvement initiative was reviewed and deemed exempt from human subjects' research by the Mayo Clinic Institutional Review Board (IRB) and approved by the Institutional COVID-19 Taskforce.

Mayo Clinic, a large multistate U.S. health system with locations in Arizona, Florida, Wisconsin, and Minnesota, sees over 1.5 million annual unique patient visits. The health system uses the same single instance EHR (Epic© Systems Corporation [www.epic.com]) allowing all clinical sites to be connected across the enterprise health system.

The present study was a retrospective observational study of 67 470 COVID-19 positive registry patients from all Mayo Clinic sites. This registry included patients based on positive COVID-19 diagnosis made at a Mayo Clinic site or imported from the health information exchange to confirm the diagnosis. The positive COVID-19 diagnosis was confirmed by PCR positive for SARS-CoV-2 between March 12, 2020 and February 8, 2021. Patients who did not give permission via the Minnesota Research Authorization were excluded from this cohort.

The MCC19-RS was an adaptation of the scoring system developed by Dr. David Daniel from Confluence Health. In developing the original scoring system, Dr. Daniel used available data with regards to risk factors for severe disease with COVID-19 via articles posted on the Centers for Disease Control and Prevention website for Morbidity and Mortality Weekly Reports on COVID-19 (https:// www.cdc.gov/coronavirus/2019-ncov/publications.html). Since the score was not developed using a derivation cohort, we adjusted the original score by eliminating weighting and gave each component equal weight in the score except for age. This was done because of uncertainty surrounding the contribution of each risk factor (age aside) in combined risk. The score was initially seen as a way to rapidly display pertinent comorbidities and provide information on known risk factors to bedside providers. Subsequently, the score was validated externally by Halalau et al 9 in a small cohort of patients presenting to one of their 8 Emergency Rooms. The present study builds on this initial validation by including a larger cohort of patients with known COVID-19 diagnosis regardless of presenting location.

MCC19-RS was implemented within the EHR system in May 2020. MCC19-RS utilized an individualized score-based point system under 13 broad domains including demographics and clinical conditions with points assigned to each domain. The demographic criteria evaluated included age and sex. One point was given for male sex and 1to 3 points for age (ie, 0 points for less than 60 years old; 1 point for 60-69 years old; 2 points for 70-79 years old; and 3 points for 80 years or older). Registries were used to assess the presence of a clinical condition and were accorded 1 point including hypertension; congestive heart failure; congenital heart disease; coronary artery disease; chronic lung disease or asthma; diabetes mellitus; immune compromised (Human Immunodeficiency Virus [HIV] diagnosis; or currently receiving chemotherapy or immunosuppressive drugs); nursing home resident; chronic dialysis; chronic liver disease; and current pregnancy. An individual patient's score equals the sum of the assigned points (0-15 points). For the current study we used the demographics and clinical conditions at the time of COVID-19 diagnosis to calculate the score.

The Charlson Comorbidity Index 10 predicts the ten-year mortality for a patient who may have a range of comorbid conditions by calculating a score for each patient as an accepted measure of comorbid burden. 11 We calculated a Charlson Comorbidity Index Score for each patient based on EHR documentation including appropriate codes indicating comorbid conditions, as has been published elsewhere. We used this as a comparator for the MCC19-RS to evaluate if the performance was better than an accepted standard for assessing comorbidity burden and risk.

Demographics and clinical characteristics of patients were stratified based upon hospitalization status ( Table 1 ). The significance of association between each of the variables and hospitalization status was determined by either using two-sample t-test for continuous variables such as age or by using chi-squared test for all the categorical variables. Fisher's exact test was used to evaluate the statistical significance of association in cases where the number of classes in categorical variables was less than three. Calibration curves were constructed via logistic regression models of the risk score against the dependent variable of hospitalization risk. Area under the curve confidence intervals were determined with a bias corrected bootstrap. Statistical analysis was done with R Studio Statistical Software.

This study is based on data from 67 470 COVID-19 patients. Only index infection episodes were used. Of these, 4626 (6.9%) patients were subsequently admitted within 30 days of testing positive for COVID-19 and 62 844 (93.1%) were treated entirely as outpatients; in addition, 405 (0.6%) expired within 30 days of diagnosis. Table 1 shows the individual factors used in the MCC19-RS and the tendency for them to be associated more proportionally in the admitted cohort. Of the 67 470 patients analyzed, 53 625 (79.5%) were under the age of 60 with only 1685 (3.1%) requiring admission. However, of those 60 and older (13 845; 20.5%) 2941 (21.4%) required admission, with about 1/3 in each successive decade. Admission rates for patients aged 60 to 69, 70 to 79, and 80+ were respectively 12.8%, 25.6%, and 43.2%. COVID patients who were admitted tended to be male (56%) and on average 66 years of age (52-77, lower and upper quartile respectively). Regards race, American Indian, Pacific Islander, and Asian were represented more proportionally in the admitted cohort versus the not-admitted cohort. All clinical condition components were at least 4 times more proportionally represented in the admitted than the non-admitted cohorts, except for congenital heart disease and end stage liver disease. Pregnancy was also represented more proportionally with the non-admitted cohort. Mortality was significantly more common in the admitted cohort (7.2%) compared to the non-admitted cohort (0.1%).

Although the MCC19-RS has a maximum possible score of 15, the highest observed score was 15, with a median score of 1 (interquartile range 0-2). Charlson scores had a range of 0 to 23, with a median score of 0.

Performance metrics of the MCC19-RS are outlined in Figure 1 . The ROC curve of the MCC19-RS having AUC of 0.837 (95% Confidence interval [CI] 0.830-0.843) in comparison to the Charlson Comorbidity Index score AUC of 0.740 (95% CI 0.733-0.748). The calibration curve, Figure  2 , demonstrates that the MCC19-RS predicts well in tose with the lowest predicted risk (< 5%).

In a sensitivity analysis comparing variations across regions, the AUC in the Midwest was 0.838, (95% CI 0.829-0.846), Arizona was 0.787 (95% CI 0.774-0.800), and Florida was 0.845 (95% CI 0.829-0.861).

Using a large cohort of patients positive for COVID-19 in any setting, we have further validated the strength of the MCC19-RS. This risk score demonstrated excellent discrimination power with an AUC of 0.837 and reasonable calibration. The performance of the score was consistent across regions, suggesting good reproducibility. We propose using a color coding of the score with green associated with scores of 0 to 2, yellow with scores of 3 to 5, and red with scores of 6 and above. The linear score, as noted, holds good discriminatory value, and should be maintained in any display. Table 2 shows that patients with scores in the 3 to 5 range have an average risk of admission in the 17% to 35% range, while scores of 2 and under have a 10% average risk of admission or lower and scores of 6 or greater have an average risk of admission of 50%.

Overall, this study identified age, sex, and comorbidities to be strong predictors of 30-day hospital admission among COVID-19 patients in agreement with other published reports. [12] [13] [14] Several scores have been proposed to assess the risk in COVID-19 positive patients progressing to more severe disease in the outpatient and inpatient settings, but they are limited by bias. 15 Of the available outpatient-based scores assessing risk to progress to hospitalization (Table 3) , [16] [17] [18] [19] [20] all require symptoms or objective signs at the time of diagnosis except for SARS2 and MCC19-RS. The primary purpose of developing the MCC19-RS was to allow the clinical practice to utilize real-time EHR data to classify patients into the appropriate risk category and to assess risk for need of admission based on the risk score.

The strength of the MCC19-RS lies in the lack of a need to assess symptoms or objective signs thereby enabling the clinical practice to independently stratify risk either at the individual positive patient level or for assessing larger cohorts of patients. Accurate risk-predicting tools for various decision points are thought to be necessary to assist managing patients in the current COVID-19 pandemic to allocate limited resources. 21 An EHR data integrated scoring system may assist the clinician who has to decide which patients are at high-risk for severe COVID-19 and that may benefit from remote patient monitoring. Mayo Clinic has also used MCC19-RS to assist in the identification of a higher risk cohort of patients to offer COVID-19 immunization when available.

Regarding the build within our EHR, we did create cutpoints for mild (0-2 points), moderate (3-5 points) , and high risk (6-15 points) to allow for color coding of the risk score when displayed in chart view or list views within the system. The score was added to inpatient and outpatient lists, as well as care management lists. We included it in multiple visualizations within the chart including "story board," "snapshot," and "sidebar" views. Figure 3 shows the score with coloring in a patient list view with "hover to discover" regarding the components for that patient. As noted, the score value is critical, but the color visualizations can help to quickly establish a general risk for the patient. In general, a clinical scoring system should not override overall clinical judgment as a clinician's overall clinical assessment of the individual patient does take into consideration multiple inputs to approximate the risk for admission. The MCC19-RS may act as an important input to assist the clinician in assessing the risk of admission of the COVID-19 positive patient.

As with all retrospective studies, there are opportunities for improvement. For example, it is likely that some patients from the outpatient setting who were not admitted to the hospital may have had co-morbid conditions that were not discretely documented in the EHR. Since most of the components for MCC19-RS were chronic conditions, the majority of the conditions would have been accounted for discretely. Additionally, pregnancy appeared to not suggest admission, but the number of patients who were pregnant were few and did not appear to cause the score to perform poorly. Finally, the study cohort is not representative of the general population of patients in the United States, both in terms of factors such as race and ethnicity, and because the patients specifically sought treatment as a multi-campus health system.

The MCC19-RS is a functional score easily integrated into the EHR with excellent predictive performance for severe COVID-19 that can easily inform clinical practice and cohort management. Calculating the score does not require current symptoms, objective signs, or lab values so it can be easily used to stratify risks for COVID-19 severity at initial contact with the health systems, while considering therapeutics or within the context of distributing of COVID-19 vaccine.

Provisional mortality data -United States

Characteristics of and important lessons from the Coronavirus Disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Figure 3. Visualization of MCC19-RS in the Electronic Medical Record list view

Algorithm for COVID-19 Triage and Referral: Patient Triage and Referral for Resource-Limited Settings During Community Transmission. WHO Regional Office for the Western Pacific

Rapid implementation of a COVID-19 remote patient monitoring program

Remote monitoring of COVID-19 positive high-risk patients in domestic isolation: a feasibility study

Home monitoring for COVID-19

Monitoring of COVID-19 patients by telemedicine with telemonitoring

Front-line innovation: rapid implementation of a nurse-driven protocol for care of outpatients with COVID-19

External validation of a clinical risk score to predict hospital admission and in-hospital mortality in COVID-19 patients

New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality

The Charlson comorbidity index is adapted to predict costs of chronic disease in primary care patients

Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease

Global, regional, and national estimates of the population at increased risk of severe COVID-19 due to underlying health conditions in 2020: a modelling study

Coronavirus disease 2019 hospitalizations attributable to cardiometabolic conditions in the United States: a comparative risk assessment analysis

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

Fast track triage for COVID-19 based on a population study: the soda score

Development and validation of the OUTCoV score to predict the risk of hospitalisation among patients with SARS-CoV-2 infection in ambulatory settings: a prospective cohort study

SARS2 simplified scores to estimate risk of hospitalization and death among patients with COVID-19

Early prognostication of COVID-19 to guide hospitalisation versus outpatient monitoring using a point-of-test risk prediction score

Development of a predictive prognostic rule for early assessment of COVID-19 patients in primary care settings

Calculated decisions: COVID-19 calculators during extreme resource-limited situations

We acknowledge the contributions of Jackie Vaughn, Cory J. Kudrna, and Jason R. Buckmeier toward the development and maintenance of the COVID-19 data architecture in the EHR.

All authors assert that all procedures contributing to this work comply with the ethical standards of the Mayo Clinic.

Ivana T. Croghan https://orcid.org/0000-0003-3464-3525

All data supporting the study findings are contained within this manuscript.

All the authors participated in the study concept and design, analysis and interpretation of data, drafting and revising the paper, and have seen and approved the final version of the manuscript.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported in part by Mayo Clinic Department of Medicine, Division General Internal Medicine.

In accordance with the Declaration of Helsinki, this study (ID 20-003278) was reviewed and found to be exempt by the Mayo Clinic Institutional Review Board (IRB). Mayo Clinic IRB approved informed consent waiver.