key: cord-0718292-qb5ekr19 authors: Bryant, P.; Elofsson, A. title: The limits of estimating COVID-19 intervention effects using Bayesian models date: 2020-08-16 journal: nan DOI: 10.1101/2020.08.14.20175240 sha: 126fee9a5942ace179a76dd47a70fd6cb626f69f doc_id: 718292 cord_uid: qb5ekr19 To limit the rapid spread of COVID-19, most governments have introduced different non-pharmaceutical interventions, which might have severe costs for society. Therefore, it is crucial to evaluate the most cost-effective interventions, using, for instance, Bayesian modelling. Such modelling efforts have deemed lockdown to account for 81% of the reduction in R0, contributing to government policies. Here, we show that these conclusions are unsupported and that policies therefore should not be based on these studies. A recent study 2 uses MCMC simulations 6, 7 to infer impacts on the basic reproductive number (R 0 ) through governmental non-pharmaceutical interventions (NPIs). From the estimated R 0 , the number of daily cases is predicted, which then through an infection-to-death distribution, is used to infer the number of daily deaths. The model fits the number of daily deaths to the observed number of deaths 8 using a Bayesian posterior distribution. The paper reports that current interventions have been sufficient to drive R 0 below 1 (which is necessary for the number of infected people to decrease) for all 11 European considered countries. They "show that major non-pharmaceutical interventions and lockdown, in particular, have had a large effect on reducing transmission" . The authors conclude that lockdown accounts for 81% [75% -87%] of the reduction in R 0 , i.e. that all other interventions only have minimal effects on the decline of R 0 . The peculiar aspect of the claim that lockdown accounts for 81% of the reduction in R 0 , is that Sweden did not implement any lockdown, but still see a similar decrease in R 0 as the other countries, even though the other NPIs were reported to have no substantial effect on R 0 . To solve this problem, as compared with the authors' earlier work 9 , which showed a significantly higher R 0 for Sweden, they invoke a country-specific last intervention parameter, which is only implemented for Sweden 10 (see equation i ). The "last intervention" parameter is multiplied with R 0 , and can therefore be seen as a parameter adjusting the model for Sweden independently. As can be seen, when analysing the posterior distributions of the intervention parameters, the "last intervention" parameter for Sweden results in 73.5 % of Sweden's reduction in R 0 ( Figure 1 ). The last intervention impact on R 0 is not reported or discussed in the Nature publication, possibly misleading decision-makers on the importance of lockdowns. where I m, 1-7 states if country m has introduced the following interventions, on the given day, (1) closing of schools and universities, (2) Self-isolating if ill, (3) banning of public events, (4) the first intervention introduced, (5) lockdown, (6) social distancing encouraged and (7) last intervention. The additional country-specific parameters added for the lockdown (here referred to as "extra lockdown") and last intervention are sampled from the following distributions: . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted August 16, 2020. . https://doi.org/10.1101/2020.08.14.20175240 doi: medRxiv preprint lockdown ~ normal(0,gamma) last intervention~ normal(0,gamma) gamma ~ normal(0,.2) To analyze the importance of the last intervention option only introduced for Sweden, we remove it (see equation ii ). The resulting reduction in R 0 for Sweden is identical within statistical margins, but the fit to the daily deaths is worse (Figure 2) . Interestingly, the banning of Public events (Sweden's last introduced NPI) now becomes more important (median reduction of 50.1%), being almost on par with that of the lockdown (median reduction of 64.9%, Figure 3 ). is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted August 16, 2020. . Finally, suppose the "extra lockdown" introduced for all countries except for Sweden and the "last intervention" options only added for Sweden are removed (equation iii ). In that case, the banning of public events now becomes the NPI with the most importance (reduction of 78%). Now also the effect of the closing of schools and universities, and social distancing rise (reductions of 5.4%, and 5.3% respectively), while the self-isolating if ill importance remains unchanged (median 0.7 %), and the lockdown importance greatly diminishes (0.2 %), is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted August 16, 2020. . The Bayesian model used by ICL to estimate the impact of each intervention appears to be sensitive to small perturbations. The reason the lockdown seems to have such high importance is likely due to it being the last intervention implemented, and not because it had the most substantial impact. We highlight this by showing that the importance of the "last intervention" diminishes when treating Sweden in the same way as all other countries. Further, the modification of the initial model 9 enables adjustments to the daily deaths from the model in each country. It thus creates the possibility of fitting a wide range of data to the observations but loses the ability to identify cross-country patterns. We believe that allowing country-specific effects of the NPIs makes it very difficult to estimate the relative impacts of NPIs, since there is no possibility of distinguishing between NPI effects and other country-specific effects, e.g. population density. In conclusion, it is peculiar that the model displays an almost identical change in R 0 in all countries, dropping sharply below one at the final NPI, independently on the nature of that NPI. In reality, all countries had different NPIs implemented at different time points, likely with varying strength and efficiency, and it is quite likely that NPIs such as enforcing social distancing at least had some effects, not seen in the models. Given the importance the initial report had on government policies and the fact that we show here that the conclusions made about the significance of the lockdown are not entirely correct, we do think that we should pinpoint this to readers and policymakers. Correct assumptions on the effects of NPIs are becoming even more urgent as many nations still are imposing different NPIs, and that these might go on for an extended period ( https://www.bbc.com/news/explainers-53640249 , last accessed 20200811). Updated rapid risk assessment from ECDC on coronavirus disease 2019 (COVID-19) pandemic: increased transmission in the EU/EEA and the UK. Eurosurveillance Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions Estimating the impact of mobility patterns on COVID-19 infection rates in 11 European countries Hierarchical Modeling and Analysis for Spatial Data Download today's data on the geographic distribution of COVID-19 cases worldwide Report 13 -Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries ImperialCollegeLondon/covid19model We acknowledge Claudio Bassot's contribution by sharing the Imperial College. Without this information, this study would not be possible. We are also grateful to various colleagues and friends that contributed to the discussion. Finally, we thank the authors of the Imperial College London report for making their data and model freely available.