key: cord-0926343-qbvgnr4r authors: Ioannidis, John P.A.; Cripps, Sally; Tanner, Martin A. title: Forecasting for COVID-19 has failed date: 2020-08-25 journal: Int J Forecast DOI: 10.1016/j.ijforecast.2020.08.004 sha: 1e3d9b368efbf038af7290c7052d86af9f95fcfe doc_id: 926343 cord_uid: qbvgnr4r Epidemic forecasting has a dubious track-record, and its failures became more prominent with COVID-19. Poor data input, wrong modeling assumptions, high sensitivity of estimates, lack of incorporation of epidemiological features, poor past evidence on effects of available interventions, lack of transparency, errors, lack of determinacy, looking at only one or a few dimensions of the problem at hand, lack of expertise in crucial disciplines, groupthink and bandwagon effects and selective reporting are some of the causes of these failures. Nevertheless, epidemic forecasting is unlikely to be abandoned. Some (but not all) of these problems can be fixed. Careful modeling of predictive distributions rather than focusing on point estimates, considering multiple dimensions of impact, and continuously reappraising models based on their validated performance may help. If extreme values are considered, extremes should be considered for the consequences of multiple dimensions of impact so as to continuously calibrate predictive insights and decision-making. When major decisions (e.g. draconian lockdowns) are based on forecasts, the harms (in terms of health, economy, and society at large) and the asymmetry of risks need to be approached in a holistic fashion, considering the totality of the evidence. Modeling resurgence after reopening also failed (Table 2 ). E.g. a Massachusetts General Hospital model 10 predicted over 23,000 deaths within a month of Georgia reopening -actual deaths were 896. Table 3 lists some main reasons underlying this forecasting failure. Unsurprisingly, models failed when they used more speculation and theoretical assumptions and tried to predict long-term outcomes, e.g. using early SIR-based models to predict what would happen in the entire season. However, even forecasting built directly on data alone fared badly, 11, 12 failing not only in ICU bed predictions ( Figure 1 ) but even in next day death predictions when issues of long-term chaotic behavior do not come into play (Figures 2 and 3 ). Even for short-term forecasting when the epidemic wave waned, models presented confusingly diverse predictions with huge uncertainty (Figure 4 ). Failure in epidemic forecasting is an old problem. In fact, it is surprising that epidemic forecasting has retained much credibility among decision-makers, given its dubious track record. Modeling for swine flu predicted 3,100-65,000 deaths in the UK. 13 Eventually 457 deaths occurred. 14 Models on foot-and-mouth disease by top scientists in top journals 15, 16 were subsequently questioned 17 by other scientists challenging why up to 10 million animals had to be slaughtered. Predictions for bovine spongiform encephalopathy expected up to 150,000 deaths in the UK. 18 However, the lower bound predicted as low as 50 deaths, 18 Let's be clear: even if millions of deaths did not happen this season, they may happen with the next wave, next season, or some new virus in the future. A doomsday forecast may come handy to protect civilization, when and if calamity hits. However, even then, we have little evidence that aggressive measures focusing only on few dimensions of impact actually reduce death toll and do more good than harm. We need models which incorporate multicriteria objective functions. Isolating infectious impact, from all other health, economic and social impacts is dangerously narrow-minded. More importantly, with epidemics becoming easier to detect, opportunities for declaring global emergencies will escalate. Erroneous models can become powerful, recurrent disruptors of life on this planet. Civilization is threatened from epidemic incidentalomas. Cirillo and Taleb thoughtfully argue 19 that when it comes to contagious risk, we should take doomsday predictions seriously: major epidemics follow a fat-tail pattern and extreme value theory becomes relevant. Examining 72 major epidemics recorded through history, they demonstrate a fat-tailed mortality impact. However, they analyze only the 72 most-noticed outbreaks, a sample with astounding selection bias. For example, according to their dataset, the first epidemic originating from sub-Saharan Africa did not occur until 1920 AD, namely 20, 21 One of them, OC43 seems to have been introduced in humans as recently as 1890, probably causing a "bad influenza year" with over a million deaths. 22 Based on what we know now, SARS-CoV-2 may be closer to OC43 than SARS-CoV-1. This does not mean it is not serious: its initial human introduction can be highly lethal, unless we protect those at risk. A heavy tail distribution ceases to be as heavy as Taleb imagines when the middle of the distribution becomes much larger. One may also argue that pandemics, as opposed to epidemics without worldwide distribution, are more likely to be heavy-tailed. However, the vast majority of the 72 contagious events listed by Taleb were not pandemics, but localized epidemics with circumscribed geographic activity. Overall, when a new epidemic is detected, it is even difficult to pinpoint which distribution of which known events it should be mapped against. Blindly acting based on extreme value theory alone would be sensible if we lived in the times of the Antonine plague or even in 1890, with no science to identify the pathogen, elucidate its true prevalence, estimate accurately its lethality, and carry out good epidemiology to identify which people and settings are at risk. Until we accrue this information, immediate better-safethan-sorry responses are legitimate, trusting extreme forecasts as possible (not necessarily likely) scenarios. However, caveats of these forecasts should not be ignored 1, 23 and new evidence on the ground truth needs continuous reassessment. Upon acquiring solid evidence about the J o u r n a l P r e -p r o o f Journal Pre-proof epidemiological features of new outbreaks, implausible, exaggerated forecasts 24 should be abandoned. Otherwise, they may cause more harm than the virus itself. The insightful recent essay of Taleb 25 offers additional opportunities for fruitful discussion. Taleb 25 ruminates on the point of making point predictions. Serious modelers (whether frequentist or Bayesian) would never rely on point estimates to summarize skewed distributions. Even an early popular presentation 26 from 1954 has a figure (see page 33) with striking resemblance to Taleb's Figure 1. 25 In a Bayesian framework, we rely on the full posterior predictive distribution, not single points. 27 Moreover, Taleb's choice of a three-parameter Pareto distribution is peculiar. It is unclear this model provides a measurably better fit to his (hopelessly biased) pandemic data 19 than, say, a two parameter Gamma distribution fitted to log counts. Regardless, either skewed distribution would then have to be modified to allow for the use of all available sources of information in a logically consistent fully probabilistic model, e.g. via a Bayesian hierarchical model (which can certainly be formulated to accommodate fat tails if needed). In this regard, we note that examining the NY daily death count data studied in ref. 12, these data are found to be characterized as stochastic rather than chaotic. 28 Taleb seems to fit an unorthodox model, and then abandons all effort to predict anything. He simply assumes doomsday has come, much like a panic-driven Roman would have done in the Antonine plague, lacking statistical, biological, and epidemiological insights. J o u r n a l P r e -p r o o f Journal Pre-proof Taleb 25 caricatures the position of a hotly debated mid-March op-ed by one of us, 29 alluring it "made statements to the effect that one should wait for "more evidence" before acting with respect to the pandemic", an obvious distortion of the op-ed. Anyone who reads the op-ed unbiasedly realizes that it says exactly the opposite. It starts with the clear, unquestionable premise that the pandemic is taking hold and is a serious threat. Immediate lockdown certainly makes sense when an estimated 50 million deaths are possible. This was stated emphatically in multiple occasions these days in interviews in multiple languages -for examples see refs. 30-32. Certainly, adverse consequences of short-term lockdown cannot match 50 million lives. However, better data can recalibrate estimates, re-assessing downstream the relative balance of benefits and harms of longer-term prolongation of lockdown. That re-appraised balance changed markedly over time. 9 Another gross distortion propagated in social media is that supposedly the op-ed 29 had predicted that only 10,000 deaths in the USA. The key message of the op-ed was that we lack reliable data, i.e. we don't know. The self-contradicting misinterpretation as "we don't know, but actually we do know that 10,000 deaths will happen" is impossible. The op-ed discussed two extreme scenarios to highlight the tremendous uncertainty absent reliable data: 10,000 deaths in the US and 40,000,000 deaths. We needed reliable data, quickly, to narrow this vast uncertainty. We did get data and did narrow uncertainty. Science did work eventually, even if forecasts, including those made by one of us (confessed and discussed in Box 1), failed. Taleb 25 offers several analogies to assert that all precautionary actions are justified in pandemics, deriding "waiting for the accident before putting the seat belt on, or evidence of fire before buying insurance". 25 The analogies assume that the cost of precautionary actions are small J o u r n a l P r e -p r o o f Journal Pre-proof in comparison to the cost of the pandemic, and that the consequences of the action have little impact on it. However, precautionary actions can backfire badly when they are misinformed. In March, modelers were forecasting collapsed health systems, e.g. 140,000 beds would be needed in New York, when only a small fraction were available. Precautionary actions damaged the health system, increased COVID-19 deaths, 33 and exacerbated other health problems (Table 4 ). Seat belts cost next to nothing to produce in cars and have unquestionable benefits. Despite some risk compensation and some excess injury with improper use, eventually seat belts prevent ~50% of serious injuries and deaths. 34 Measures for pandemic prevention equivalent to seat belts in terms of benefit-harm profile are simple interventions like hand washing, respiratory etiquette and mask use in appropriate settings: large proven benefit, no/little harm/cost. 35,36 Even before the COVID-19 pandemic, we had randomized trials showing 38% reduced odds of influenza infection with hand washing and (non-statistically significant, but possible) 47% reduced odds with proper mask wearing. 35 Despite lack of trials, it is sensible and minimally disruptive to avoid mass gatherings and decrease unnecessary travel. Prolonged draconian lockdown is not equivalent to seat belts. It resembles forbidding all commute. Similarly fire insurance offers a misleading analogy. Fire insurance makes sense only at reasonable price. Draconian prolonged lockdown may be equivalent to paying fire insurance at a price higher than the value of the house. Taleb refers to the Netherlands where maximum values for flooding, not the mean, are considered. 25 Anti-flooding engineering has substantial cost, but a favorable decision-analysis profile after considering multiple types of impact. Lockdown measures were decided based on examining only one type of impact, COVID-19. Moreover, the observed flooding maximum to-J o u r n a l P r e -p r o o f Journal Pre-proof date does not preclude even higher future values. Netherlands aims to avoid devastation from floods occurring once every 10,000 years in densely populated areas. 37 A more serious flooding event (e.g. one that occurs every 20,000 years) may still submerge the Netherlands next week. However, prolonged total lockdown is not equivalent to building higher sea walls. It is more like abandoning the country -asking the Dutch to immigrate, because their land is quite unsafe. Other natural phenomena also exist where high maximum risks are difficult to pinpoint and where new maxima may be reached. E.g., following Taleb's argumentation, one should forbid living near active volcanoes. Living at the Santorini caldera is not exciting, but foolish: that dreadful island should be summarily evacuated. Same applies to California: earthquake devastation may strike any moment. Prolonged lockdown zealots might barely accept a compromise: whenever substantial seismic activity occurs, California should be temporarily evacuated until all seismic activity ceases. Furthermore, fat-tailed uncertainty and approaches based on extreme value theory may be useful before a potentially high-risk phenomenon starts and during its early stages. However, as more data accumulate and the high-risk phenomenon can be understood more precisely with plenty of data, the laws of large numbers may apply and stochastic rather than chaotic approaches may become more relevant and useful than continuing to assume unlikely extremes. Further responses to Taleb 25 appear in Table 5 . The short answer is: using science and more reliable data. We can choose measures with favorable benefit-risk ratio, when we consider together multiple types of impact, not only on COVID-19, but on health as a whole, as well as society and economy. J o u r n a l P r e -p r o o f Currently we know that approximately half of the COVID-19 deaths in Europe and the USA affected nursing home residents. 38,39 Another sizeable proportion were nosocomial infections. 40 If we protect these locations with draconian hygiene measures and intensive testing, we may avert 70% of the fatalities without large-scale societal disruption and without adverse consequences on health. Other high-risk settings, e.g. prisons, homeless shelters, meatprocessing plants also need aggressive protection. For the rest of the population, we have strong evidence on a very steep age gradient with ~1000-fold differences in death risk for people >80 years old versus children. 41 We have also detailed insights on how different background diseases modify COVID-19 risk for death or other serious outcome. 42 We can use hygiene and some least disruptive distancing measures to protect people. We can use intensive testing (i.e. again, use science) to detect resurgence of epidemic activity and extinguish it early -the countries that faced most successfully the first wave, e.g. Singapore and Taiwan, did exactly that highly successfully. We can use data to track how the epidemic and its impact evolves. Data can help inform more granular models and titrate decisions considering distributions of risk ( Figure 5 ). 42 Poorly performing models and models that perform well for only one dimension of impact can cause harm. It is not just an issue of academic debate, it is an issue of potentially devastating, wrong decisions. 36 Taleb 25 seems self-contradicting: does he espouse abandoning all models (since they are so wrong) or using models but always assuming the worst? However, there is no single worst scenario, but a centile of the distribution: should we prepare for an event that has 0.1%, 0.001%, or 0.000000000001% chance of happening? Paying what price in harms? Abandoning all epidemic modeling appears too unrealistic. Besides identifying the problems of epidemic modeling, Table 3 also offers suggestions on addressing some of them. To summarize here some necessary (although not always sufficient) targets for amendments: • Invest more on collecting, cleaning, and curating real, unbiased data, not just theoretical speculations unfortunately flirting with this slippery, defensive path. 45 Total lockdown is a bundle of dozens of measures. Some may be very beneficial, but some others may be harmful. Hiding uncertainty can cause major harm downstream and leaves us unprepared for the future. For papers that fuel policy decisions with major consequences, transparent availability of data, code, and named peer-review comments is also a minimum requirement. The possibility of calibrating model predictions for looking at extremes rather than just means is sensible, especially in early days of pandemics, when much is unknown about the virus and its epidemiological footprint. However, when calibration/communication on extremes is adopted, one should also consider similar calibration for the potential harms of adopted measures. For example, tuberculosis has killed 1 billion people in the last 200 years, it still kills 1.5 million people (mostly young and middle age ones) annually, and prolonged lockdown may Use of extreme case predictions for COVID-19 deaths should be co-examined with extreme case predictions for deaths and impact from many other lockdown-induced harms. Models should provide the big picture of multiple dimensions. Similar to COVID-19, as more reliable data accrue, predictions on these other dimensions should also be corrected accordingly. Eventually, it is probably impossible (and even undesirable) to ostracize epidemic forecasting, despite its failures. Arguing that forecasting for COVID-19 has failed should not be misconstrued to mean that science has failed. Developing models in real time for a novel virus, with poor quality data, is a formidable task and the groups who attempted this and made public their predictions and data in a transparent manner should be commended. We readily admit that it is far easier to criticize a model than to build one. It would be horrifically retrograde if this debate ushers in a return to an era where predictions, on which huge decisions are made, are kept under lock and key (e.g. by the government -as is the case in Australia). We wish to end on a more positive note, namely where we feel forecasting has been helpful. Perhaps the biggest contribution of these models is that they serve as a springboard for discussions and debates. Dissecting variation in performance of various models (e.g. casting a sharp eye to circumstances where a particular model excelled) can be highly informative and a systematic approach to the development and evaluation of such models is needed. 12 This demands a coherent approach to collecting, cleaning and curating data, as well as a transparent approach to evaluating the suitability of models with regard to predictions and forecast uncertainty. What we have learned from the COVID-19 pandemic can be passed to future generations that hopefully should be better prepared to deal with a new, different pandemic, learning from our failures. There is no doubt that, again, an explosive literature of models and forecasting will emerge again as soon as a new pandemic is suspected. However, we can learn from our current mistakes to be more cautious with interpreting, using, and optimizing these models. Being more cautious does not mean not to act decisively, but it requires looking at the totality of the data; considering multiple types of impact; having scientists from very different disciplines involved; replacing speculations, theories and assumptions with real, empirical data as quickly as possible; and modifying and aligning decisions to the evolving best evidence. In the current pandemic, we largely failed to protect people and settings at risk. We could have done much better in this regard. It is difficult to correct mistakes that have already led to people dying, but we can avoid making the same mistakes in future pandemics from different pathogens. We can avoid making the same mistakes even for COVID-19 going forward, since this specific pandemic has not ended as we write. In fact, its exact eventual impact is still unknown. For example, the leader of the US task force, Dr. Anthony Fauci, recently warned of reaching 100,000 COVID-19 US cases per day. 48 Maybe this prediction is already an underestimate, because with over 50,000 cases diagnosed per day in early July 2020, the true number of infections may be many times larger. There is currently wide agreement that the number of infections in many parts of the United States is more than 10 times higher than the reported rates. 49 "According to the Penn Wharton Budget Model (PWBM), reopening states will result in an additional 233,000 deaths from the virus -even if states don't reopen at all and with social distancing rules in place. This means that if the states were to reopen, 350,000 people in total would die from coronavirus by the end of June, the study found." Yahoo, May 3, 2020 (https://www.yahoo.com/now/reopeningstates-will-cause-233000-more-people-to-die-from-coronavirusaccording-to-wharton-model-120049573.html) Based on JHU dashboard death count, number of additional deaths as of June 30 was 5,700 instead of 233,000, i.e. total deaths was 122.7 thousand instead of 350 thousand. It is unclear also whether any of the 5,700 deaths were due to reopening rather than error in the original model calibration of the number of deaths without reopening. "Dr. Ashish Jha, the director of the Harvard Global Health Institute, told CNN's Wolf Blitzer that the current data shows Within less than 4 weeks of this quote the number of J o u r n a l P r e -p r o o f that somewhere between 800 to 1,000 Americans are dying from the virus daily, and even if that does not increase, the US is poised to cross 200,000 deaths sometime in September. "I think that is catastrophic. I think that is not something we have to be fated to live with," Jha told CNN. "We can change the course. We can change course today." "We're really the only major country in the world that opened back up without really getting our cases as down low as we really needed to," Jha told CNN." Business Insider, June 10, 2020 (https://www.businessinsider.com/harvard-expert-predictscoronavirus-deaths-in-us-by-september-2020-6) daily deaths was much less than the 800-1000 quote (516 daily average for the week ending July 4). Then it increased again. The number of actual total deaths as of September will be added here when available. Lack of consensus as to what is the 'ground truth" even for seemingly hard-core data such as daily the number of deaths. They may vary because of reporting delays, changing definitions, data errors, and more reasons. Different models were trained on different and possibly highly inconsistent versions of the data. As above: investment should be made in the collection, cleaning and curation of data. Wrong assumptions in the modeling Many models assume homogeneity, i.e. all people having equal chances of mixing with each other and infecting each other. This is an untenable assumption and in reality, tremendous heterogeneity of exposures and mixing is likely to be the norm. Unless this heterogeneity is recognized, estimated of the proportion of people eventually infected before reaching herd immunity can be markedly inflated Need to build probabilistic models that allow for more realistic assumptions; quantify uncertainty and continuously readjust models based on accruing evidence High sensitivity of estimates For models that use exponentiated variables, small errors may result in major deviations from reality Inherently impossible to fix; can only acknowledge that uncertainty in calculations may be much larger than it seems Lack of incorporation of epidemiological features Almost all COVID-19 mortality models focused on number of deaths, without considering age structure and comorbidities. This can give very misleading inferences about the burden of disease in terms of quality-adjusted life-years lost, which is far more important than simple death count. For example, the Spanish flu killed young people with average age of 28 and its burden in terms of number of quality-adjusted person-years lost was about 1000fold higher than the COVID-19 (at least as of June 8, 2020). Incorporate best epidemiological estimates on age structure and comorbidities in the modeling; focus on quality-adjusted lifeyears rather than deaths The core evidence to support "flatten-the-curve" efforts was based on observational data from the 1918 Spanish flu pandemic on 43 US cites. These data are >100-years old, of questionable quality, unadjusted for confounders, based on ecological reasoning, and pertaining to an entirely different While some interventions in the broader package of lockdown measures are likely to have beneficial effects, assuming huge benefits is incongruent with the past (weak) evidence and should be avoided. Large benefits may be feasible from J o u r n a l P r e -p r o o f (influenza) pathogen that had ~100-fold higher infection fatality rate than SARS-CoV-2. Even thus, the impact on reduction on total deaths was of borderline significance and very small (10-20% relative risk reduction); conversely many models have assumed 25-fold reduction in deaths (e.g. from 510,000 deaths to 20,000 deaths in the Imperial College model) with adopted measures precise, focused measures (e.g. early, intensive testing with through contact tracing for the early detected cases, so as not to allow the epidemic wave to escalate [e.g. Taiwan or Singapore]; or draconian hygiene measures and thorough testing in nursing homes) rather than from blind lockdown of whole populations. Lack of transparency Many models used by policy makers were not disclosed as to their methods; most models were never formally peer-reviewed and the vast majority have not appeared in the peer-reviewed literature even many months after they shaped major policy actions While formal peer-review and publication may take more time unavoidably, full transparency about the methods, and sharing of the code and data that inform these models is indispensable. Even with peer-review, many papers may still be glaringly wrong, even in the best journals. Errors Complex code can be error-prone and errors can happen even by experienced modelers; using oldfashioned software or languages can make things worse; lack of sharing code and data (or sharing them late) does not allow detecting and correcting errors Promote data and code sharing; use up-todate and well-vetted tools and processes that minimize the potential for error through auditing loops in the software and code Lack of determinacy Many models are stochastic and need to have a large number of iterations run, perhaps also with appropriate burn-in periods; superficial use may lead to different estimates Promote data and code sharing to allow checking the use of stochastic processes and their stability Looking at only one or a few dimensions of the problem at hand Almost all models that had a prominent role in decision-making focused on COVID-19 outcomes, often just a single outcome or a few outcomes (e.g. deaths, or hospital needs). Models prime for decision-making need to take into account the impact on multiple fronts (e.g. other aspects of health care, other diseases, dimensions of the economy, etc.) Interdisciplinarity is desperately needed; since it is unlikely that single scientists or even teams can cover all this space, it is important for modelers from diverse ways of life to sit on the same table. Major pandemics happen rarely and what is needed are models which fuse information from a variety of sources. Information from data, from experts in the field, from past pandemics, need to fused in a logically consistent fashion if we wish to get any sensible predictions. Lack of expertise in crucial disciplines The credentials of modelers are sometimes undisclosed; when they have been disclosed, these teams are led by scientists who may have strengths in some quantitative fields, but these fields may be remote from infectious diseases and clinical epidemiology; modelers may operate in subject matter vacuum Make sure that the modelers' team is diversified and solidly grounded in terms of subject matter expertise Groupthink and bandwagon effects Models can be tuned to get desirable results and predictions, e.g. by changing the input of what are deemed to be plausible values for key variables. This is especially true for models that depend on theory and speculation, but even data-driven forecasting can do the same, depending on how the modeling is performed. In the presence of strong groupthink and bandwagon effects, modelers may consciously fit their predictions to what is the dominant thinking and expectations -or they may be forced to do so. Maintain an open-minded approach; unfortunately models are very difficult, if not impossible, to pre-register, so subjectivity is largely unavoidable and should be taken into account in deciding how much forecasting predictions can be trusted J o u r n a l P r e -p r o o f Forecasts may be more likely to be published or disseminated, if they are more extreme Very difficult to diminish, especially in charged environments; needs to be taken into account in appraising the credibility of extreme forecasts J o u r n a l P r e -p r o o f Inform the public that we are doing our best, but it is likely that hospitals will be overwhelmed by COVID-19 Honest communication with the general public Patients with major problems like heart attacks did not come to the hospital to be treated, 5 while these are diseases that are effectively treatable only in the hospital; an unknown, but probably large share of excess deaths in the COVID-19 weeks were due to these causes rather than COVID-19 itself 54 Re-orient all hospital operations to focus on COVID-19 Be prepared for the COVID-19 wave, strengthen the response to crisis Most hospitals saw no major COVID-19 wave and also saw a massive reduction in overall operations with major financial cost, leading to furloughs and lay-off of personnel; this makes hospitals less prepared for any major crisis in the future J o u r n a l P r e -p r o o f Table 5 . Taleb's main statements and our responses Forecasting single variables in fat tailed domains is in violation of both common sense and probability theory. Serious statistical modelers (whether frequentist or Bayesian) would never rely on point estimates to summarize a skewed distribution. Using data as part of a decision process is not a violation of common sense, irrespective of the distribution of the random variable. Possibly using only data and ignoring what is previously known (or expert opinion or physical models) may be unwise in small data problems. We advocate a Bayesian approach, incorporating different sources of information into a logically consistent fully probabilistic model. We agree that higher order moments (or even the first moment in the case of the Cauchy distribution) do not exist for certain distributions. This does not preclude making probabilistic statements such as P(a90% of the potential deaths. >90% of the population could possibly continue with non-disruptive measures, since they account for only <10% of the total potential deaths. Wrong but useful -what Covid-19 epidemiologic models can and cannot tell us Will Have 100 Million Cases Of COVID-19 In Four Weeks, Doubling Every Four Days Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months Reduced rate of hospital admissions for ACS during covid-19 outbreak in northern Italy Collateral damage: the impact on outcomes from cancer surgery of the COVID-19 pandemic Years of life lost due to the psychosocial consequences of COVID19 mitigation strategies based on Swiss data Should governments continue lockdown to slow the spread of covid-19? The totality of the evidence Learning as We Go: An Examination of the Statistical Accuracy of COVID19 Daily Death Count Predictions A case study in model failure? COVID-19 daily deaths and ICU bed utilisation predictions The 2009 influenza pandemic review Transmission intensity and impact of control policies on the foot and mouth epidemic in Great Britain The foot-and-mouth epidemic in Great Britain: Pattern of spread and impact of interventions Use and abuse of mathematical models: an illustration from the 2001 foot and mouth disease epidemic in the United Kingdom Estimating the human health risk from possible BSE infection of the british sheep flock Tail risk of contagious diseases Clinical impact of human coronaviruses 229E and OC43 infection in diverse adult populations An outbreak of human coronavirus OC43 infection and serological cross-reactivity with SARS coronavirus Complete genomic sequence of human coronavirus OC43: molecular clock analysis suggests a relatively recent zoonotic coronavirus transmission event Five ways to ensure that models serve society: a manifesto Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe The illusory effects of non-pharmaceutical interventions on COVID-19 in Europe US could see 100,000 new COVID-19 cases per day