key: cord-0744590-7549rpxi
authors: Nadella, Pranay; Swaminathan, Akshay; Subramanian, S. V.
title: Forecasting efforts from prior epidemics and COVID-19 predictions
date: 2020-07-17
journal: Eur J Epidemiol
DOI: 10.1007/s10654-020-00661-0
sha: 929c5aa05b1cf8e47d92f3fa4dd50355ac89898c
doc_id: 744590
cord_uid: 7549rpxi

Since the onset of the COVID-19 pandemic, countless disease prediction models have emerged, shaping the focus of news media, policymakers, and broader society. We reviewed the accuracy of forecasts made during prior twenty-first century epidemics, namely SARS, H1N1, and Ebola. We found that while disease prediction models were relatively nascent as a research focus during SARS and H1N1, for Ebola, numerous such forecasts were published. We found that forecasts of deaths for Ebola were often far from the eventual reality, with a strong tendency to over predict. Given the societal prominence of these models, it is crucial that their uncertainty be communicated. Otherwise, we will be unaware if we are being falsely lulled into complacency or unjustifiably shocked into action. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s10654-020-00661-0) contains supplementary material, which is available to authorized users.

and 50,000 global deaths for H5N1 and mad cow disease respectively [4, 5] . However, these were drastic overpredictions, as only 455 and 177 deaths ensued [6, 7] . To systematically evaluate the success of prior forecasting models, we reviewed predictions from three twenty-first century epidemics: the 2002-2004 Severe acute respiratory syndrome (SARS) outbreak, the 2009 H1N1 influenza pandemic, and the 2014 Ebola virus disease outbreak. We found that during the SARS and H1N1 outbreaks, only a few studies attempted to predict future cases and were ultimately unsuccessful. During the Ebola epidemic, the number of forecasting studies increased dramatically, and most overestimated-quite substantially-the true number of cases and deaths.

We identified studies that forecasted cases or deaths for each epidemic by employing a broad PubMed search strategy with the terms "estimate", "model", "forecast", "predict", "transmission", and "intervention." We only included studies that made predictions while the outbreak was occurring. Because not all forecasts may have been published in peer-reviewed literature, we applied this search strategy to news articles from major media outlets as well. For our analysis of the Ebola epidemic, we utilized the references from a prior Ebola review [8] . Of the reviewed studies, only three predicted deaths. For the remaining studies that predicted cases, we extrapolated deaths by multiplying cases with the studies' estimates of case fatality rates (CFR). For studies that did not estimate CFR, we applied the average Ebola Electronic supplementary material The online version of this article (https ://doi.org/10.1007/s1065 4-020-00661 -0) contains supplementary material, which is available to authorized users. CFR of 50%, as reported by the World Health Organization [9] . Finally, we numerically compared the studies' predicted deaths to eventual true deaths to assess prediction accuracy.

For SARS, only one prediction was found, which vastly overpredicted the number of cases in Canada by June 2003 (Predicted (P): 4432, Actual (A): 188) [10, 11] . During H1N1, there were only two studies that predicted future case counts and both significantly underpredicted the number of cases in the U.S. by June 2009 (P: 2000-2500, A: 100,000) [12, 13] .

There were 17 studies (reporting 35 predictions) that forecasted the Ebola epidemic. Of the 35 predictions, 71% (n = 25) overpredicted and 29% (n = 10) underpredicted the number of deaths. These mispredictions varied from projecting 89% (n = 2256) less than the actual deaths in Guinea to 9495% (n = 456,690) more than the actual deaths in Liberia [14] . Only 37% (n = 13) of the predictions were within the range of 50% greater or less than the actual number of deaths (Fig. 1) . Additionally, several predictions were made assuming best case (all interventions implemented) or worst case (no interventions implemented) scenarios. Of the 12 predictions that assumed a worst-case scenario, 92% (n = 11) overpredicted and 8% (n = 1) underpredicted. Of the 7 predictions that assumed a best-case scenario, 57% (n = 4) still overpredicted (Supplementary File 1) .

As of June 19th, 2020, there were over 50 studies that predicted the course of COVID-19. It is reassuring to have scientists rise to the challenge to help leaders make informed decisions. However, a review of forecasts from Ebola suggests that the majority of predictions were far from the eventual reality. In fact, COVID-19 predictions too have ranged from massive underestimates-a worst case scenario of 50,562 cases in Italy by May 31st (there were over 230,000 cases) [15, 16] -to proven overestimates-190,000 cases in Wuhan, China by April (there were 50,339 cases by the middle of May) [17, 18] . A model developed by the Institute for Health Metrics and Evaluation has gained prominence and has been widely utilized for state and federal policymaking. However, even this model has been often inaccurate at local and national levels, tending to provide overly narrow confidence intervals [19, 20] . Thus, we must consider the historically poor performance of disease prediction models when engaging with predictions for COVID-19.

Imperfect data, unverifiable assumptions, and the unpredictability of human behavior make forecasting epidemics an inherently uncertain task. For disease models to appropriately inform policy, we must acknowledge not only the uncertainty of prediction estimates (via confidence intervals), but also the uncertainty inherent to the exercise of prediction itself.

One approach for improving predictions is to incorporate a broader set of disciplinary perspectives. Often, disease forecasts are made on the basis of individual expertise in virology, infectious disease epidemiology, or demography. However, the psychology of how behavior changes, the economics of unemployment that ensues, and the policy options with which nations can respond also influence a pandemic's course but are typically left unconsidered. Models that integrate various forms of epidemic information will bring much needed nuance and humility to the challenge of prediction.

Furthermore, we recommend standardized reporting guidelines for forecasting studies, much like STROBE for observational epidemiological studies and CONSORT for randomized controlled trials. Forecasting studies should Fig. 1 Frequency of predictions based on accuracy compared to actual numbers of deaths discuss the "Current Forecasting Effort in Context" to summarize other predictions for the same outbreak as well as relevant predictions from prior outbreaks. These models should also report how their data was collected, detail the assumptions made and how realistic they are, and incorporate key epidemiological factors like age structure into the model [21] . Lastly, researchers should indicate how their forecast builds upon the existing landscape of predictions. As research and the media pivot focus towards the second surge of COVID-19, it is critical to quickly improve reporting standards so that future models are more honestly appraised.

Niels Bohr once said, "it is difficult to predict, especially the future". Only once COVID-19 is behind us will we know whether prediction models did better than their counterparts from the Ebola epidemic. Until then, it is critical that researchers communicate the contexts and uncertainties of their predictions to best inform policy and the public.

Coronavirus screening may miss two-thirds of infected travelers entering U.S. Harvard Gaz

Impact of Non-Pharmaceutical Interventions (NPIs) to Reduce COVID-19 Mortality and Healthcare Demand. London

India must prepare for a tsunami of coronavirus cases

sturc ke#:~:text=%22The %20con seque nces%20in%20 ter ms%20of,betwe en%20fiv e%20and %20150 %20mil lion%22

Estimating the human health risk from possible BSE infection of the British sheep flock

Cumulative number of confirmed human cases for avian influenza A(H5N1) reported to WHO

Variant Creutzfeldt−Jakob disease Current Data

Forecasting the 2014 West African Ebola Outbreak

Ebola virus disease

A simple approximate mathematical model to predict the number of severe acute respiratory syndrome cases and deaths

Cumulative Number of Reported Probable Cases Of SARS. World Health Organization

Predicting Flu With the Aid of (George) Washington. The New York Times

CDC Telebriefing on Investigation of Human Cases of H1N1 Flu

Estimating the Reproduction Number of Ebola Virus (EBOV) During the 2014 Outbreak in West Africa

Advanced forecasting of SARS-CoV-2 related deaths in Italy

Italy Records 75 New Coronavirus Deaths, 355 New Cases. Reuters. 2020

When will the battle against novel coronavirus end in Wuhan: a SEIR modeling analysis

China's Wuhan kicks off mass testing campaign for new coronavirus

Widely cited health institute keeps missing the mark on Maine death projections

Caution warranted: using the institute for health metrics and evaluation model for predicting the course of the COVID-19 pandemic

Forecasting for COVID-19 has failed