key: cord-0938761-2b9ehyvy authors: Sen-Crowe, Brendon; McKenney, Mark; Elkbuli, Adel title: Consistency and reliability of COVID-19 projection models as a means to save lives date: 2020-07-12 journal: Am J Emerg Med DOI: 10.1016/j.ajem.2020.07.020 sha: 57fb77d2e93730a15a4815d9bf7ac1372b481f02 doc_id: 938761 cord_uid: 2b9ehyvy nan In January 2020, the COVID-19 pandemic took the United States (US) by surprise. Projections were needed to estimate the magnitude and plan responses at national and local levels. In turn, the projections influenced decisions regarding social distancing, resource distribution, and lockdowns like the mandatory closing of businesses and schools. 1 Many predictive models have been used during the pandemic such as the Imperial College of London (ICL) and Institute of Health Metrics and Evaluation's (IHME). In addition to information regarding testing and mortality, the IHME contains projections about ICU admissions, a metric needed for resource planning. However, some concerns regarding the validity of this model have been raised. 2 One of the guidelines for reopening that was communicated by The Whitehouse is 40% of ICU beds must be available in case of a surge of COVID-19 cases. 3 However, one limitation of ICU and hospitalization metrics is the information is based on data collected from a sampling of hospitals in each state. 4 Thus, ICU bed occupancy, may be underestimated. For example, if a section of the state or regional has low utilization while another portion is overwhelmed, the sampling may miss this misallocation and the effects could be devastating. Furthermore, the results from some other predictive models are not reproducible due to a lack of transparency and exact methodology utilized. Reproducible results are an inherent expectation in the scientific community. In the case of ICL, a report was released communicating their underlying methodology and model choices, as well as their sensitivity analysis for their projections. 5 Reports that communicate methodologies could result in improvements in the projection model and enhance the overall accuracy. Developers of projection models should provide specifications to enable other researchers to reproduce results and assess for accuracy. Validation of these models is paramount to their utilization. From May 1 to July 3, 2020, there was a decrease in hospitalization rates accompanied by a decrease in the overall casefatality ratio (CFR) in 4 of 5 examined states ( Figure 1 ). However, in the case of New York, there was an overall decrease in hospitalization rates, but an overall increase in the CFR (Figure 1 ). The latter opposing trends seem counterintuitive, and models need to look at further Journal Pre-proof J o u r n a l P r e -p r o o f confounding variables leading to better understand unexpected relationships to make better predictions in the future. Another concern is the use of these projection models to make conclusions. Texas has been reporting increasing cases since mid June. 6 The increase in cases may be due to more contact between the community and additional spread of COVID-19. On the other hand, it is possible that the increase is due to increased testing. 3 However, there was an increase in positive COVID-19 tests started roughly 2 weeks after the state began to reopen (Figure 2 ). In light of these results, it is likely that the surge of cases is not due to additional testing. Ideally, the models will help with these types of questions and conclusions. It is likely that the pandemic will continue to disrupt life in the US for many of the of COVID-19 cases in California are due to patients in the 18-34 age group, a population that has only recently comprised a significant portion of COVID-19 cases. 9 The mandated use of masks accompanied by contact tracing and social distancing measures will be vital to minimize the risk of transmission in the communities experiencing outbreaks and to prevent further spread to other communities throughout the nation. 10, 11 Decreasing the margin of error of prediction models for COVID-19 projections may be beneficial for informing public health policy not only at the national level, but extending worldwide. The evaluation of the reliability of models should focus on the transparency, reproducibility and validity of these models. Despite There was a maximum of 25.4% hospitalization rate recorded on May 1, 2020 and steadily declined to 22.7% by July 3 rd , 2020. However, despite the decrease in hospitalization rate, there was an overall increase in the CFR. Florida: The hospitalization rate of FL increased from 16.7% on May 1, 2020 and reached a maximum of 21.8% on May 8 th , 2020. However, there was a consistent decrease in hospitalization rate from May 13 th , to July 3 rd , 2020. The CFR displayed a similar trend in which a maximum was reached on May 20 th , 2020 (6.5%) and steadily declined until July 3 rd , 2020 (6.3%). Arizona: A steady decline in the hospitalization rate was observed. A maximum of 24.9% hospitalization rate was recorded on May 1 st , 2020 and consistently declined to 5.5% by July 3 rd , 2020. However, the CFR displayed somewhat of an opposite trend, in which there was an initial increase from May 1 st , 2020 (4.1%) to May 20 th , 2020 (5.0%), followed by a decline to 1.9% CFR by July 3 rd , 2020. *Hospitalization data not currently available for Texas and California. Institute for Health Metrics and Evaluation Caution Warranted: Using the Institute for Health Metrics and Evaluation Model for Predicting the Course of the COVID-19 Pandemic Opening Up America Again. The Whitehouse Module Data Dashboard -Patient Impact and Hospital Capacity Pathway Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries. Imperial College London Total by State. The COVID Tracking Project United States Census Bureau Public Health Measures and the Reproduction Number of SARS-CoV-2 Cases and Deaths Associated with COVID-19 by Age Group in California. California Department of Public Health Social distancing during the COVID-19 pandemic: Staying home save lives COVID-19 laboratory testing issues and capacities as we transition to surveillance testing and contact tracing