key: cord-1012957-0e3c70q0
authors: Gnanvi, Janyce; Salako, Kolawolé Valère; Kotanmi, Brezesky; Kakaï, Romain Glèlè
title: On the reliability of predictions on Covid-19 dynamics: A systematic and critical review of modelling techniques
date: 2021-01-12
journal: Infect Dis Model
DOI: 10.1016/j.idm.2020.12.008
sha: ab1eefcf061f157ac133a9b42e703065d0871d72
doc_id: 1012957
cord_uid: 0e3c70q0

Since the emergence of the novel 2019 coronavirus pandemic in December 2019 (COVID-19), numerous modellers have used diverse techniques to assess the dynamics of transmission of the disease, predict its future course and determine the impact of different control measures. In this study, we conducted a global systematic literature review to summarize trends in the modelling techniques used for Covid-19 from January 1st(,) 2020 to October 30th(,) 2020. We further examined the reliability and correctness of predictions by comparing predicted and observed values for cumulative cases and deaths. From an initial 4311 peer-reviewed articles and preprints found with our defined keywords, 242 were fully analysed. Most studies were done on Asian (46.52%) and European (27.39%) countries. Most of them used compartmental models (namely SIR and SEIR) (46.1%) and statistical models (growth models and time series) (31.8%) while few used artificial intelligence (6.7%), Bayesian approach (4.7%), Network models (2.3%) and Agent-based models (1.3%). For the number of cumulative cases, the ratio of the predicted over the observed values and the ratio of the amplitude of confidence interval (CI) or credibility interval (CrI) of predictions and the central value were on average larger than 1 indicating cases of inaccurate and imprecise predictions, and large variation across predictions. There was no clear difference among models used for these two ratios. In 75% of predictions that provided CI or CrI, observed values fall within the 95% CI or CrI of the cumulative cases predicted. Only 3.7% of the studies predicted the cumulative number of deaths. For 70% of the predictions, the ratio of predicted over observed cumulative deaths was less or close to 1. Also, the Bayesian model made predictions closer to reality than classical statistical models, although these differences are only suggestive due to the small number of predictions within our dataset (9 in total). In addition, we found a significant negative correlation (rho = - 0.56, p = 0.021) between this ratio and the length (in days) of the period covered by the modelling, suggesting that the longer the period covered by the model the likely more accurate the estimates tend to be. Our findings suggest that while predictions made by the different models are useful to understand the pandemic course and guide policy-making, some were relatively accurate and precise while other not.

For each of the 242 papers selected (see supplementary material 1 for the full list), the data extracted were: the 122 country for which the modelling study was conducted, the published or unpublished status of the study, the time 123 period covered by the data (in number of days), the topics addressed in the study, the modelling techniques used, 124 and whether the modelling was data-driven or not. We also noted whether the study accounted for 125 asymptomatics, pre-symptomatics, both asymptomatics and pre-symptomatics, or none of these classes of 126 J o u r n a l P r e -p r o o f play in the transmission of the disease (He et al., 2020). When the study made predictions, we further recorded the date at which the predicted values of the number of cumulative cases will be observed, and the uncertainty 130 parameters around the predictions (95% Confidence Interval -CI or 95% Credibility Interval -CrI). The data 131 analyses considered three aspects. The first aspect was related to the geographical coverage (continents and 132 number of countries covered per continent) and topics addressed in the studies, whether the modelling was data-133 driven and include symptomatics, pre-symptomatics, or not; we used count and relative frequencies to describe 134 this trend. The second aspect was related to the modelling techniques and was also addressed using count and 135 relative frequencies after grouping modelling techniques in relatively similar groups. The third aspect was the 136 accuracy and precision of the predictions made by studies. Accuracy refers to how close a prediction is to the 137 true value but precision refers to how certain is the prediction (Stallings & Gillmore, 1971) . For this, we used 138 three parameters; the first is the ratio between the value predicted and the value actually observed on the day on 139 which the prediction was made. This ratio is a measure of accuracy of the prediction. A value close to 1 indicates 140 that the prediction was accurate. Values less or larger than one indicates underestimation or overestimation, 141 respectively. The second parameter which is a measure of precision was the ratio between the amplitude of the 142 uncertainty parameter (95% CI or 95% CrI) and the central value. For studies that used statistical methods, the 143 uncertainty parameter is the 95% CI. For studies which used Bayesian methods, the 95% CrI is the uncertainty 144 parameter. The uncertainty parameters indicate that given the observed data, the prediction has 95% probability 145 of falling within this range. This ratio is an estimate of the accuracy of the predictions. A value of 1 for this ratio 146 indicates that the amplitude is larger as the central value. Smaller values indicate more accurate prediction (i.e. countries out of the 44) of European countries with Italy (39 studies), France (25 studies) and Spain (22 studies) 160 being the countries where more studies were done. From our sampled studies, 40 focused on African countries 161 either at country level (9 in Nigeria, 7 in South Africa) or region level (i.e. west, east, north, south, or central), or 162 the whole continent level. Some studies did not focus on a specific country but on an entire continent or part of 

Confidence interval (CI) or credibility interval (CrI) are essential measurements of precision in parameter 299 estimations. As an indicator of precision, the ratio of the amplitude of the 95% CI or 95% CrI and the predicted 300 value was calculated to also assess reliability of the predicted values. Overall, very few studies have reported CI 301 or CrI. Only 5.79% of studies (14 out of 242) have reported CI or CrI for the predicted number of cumulative (95%) (Fig. 6a) . This ratio seems relatively lower for statistical models compared to compartmental models,

indicating relatively more precise predictions for statistical models (Fig. 6b) . However, this difference cannot be 305 confirmed statistically since the compartmental, and the statistical models were used for 4, and 13 predictions 306 respectively, that we judged not enough for a robust statistical significance test. More data would be needed to 307 better make this comparison. Including either asymptomatics, or pre-symptomatics or none of these classes in the 308 modelling does not seem to affect the precision of the predictions (Fig. 6c) . There was not enough information in 309 our dataset to compare this ratio between models parameterized on real data and models that were not 310 parameterized on real data (Fig. 6d) . This ratio decreases with the length of the period (in number of days) 311 covered by the data used for the estimation, although it was not significant (linear regression analysis: β = -312 0.001; p-value = 0.242, Fig. 6e ). 

The third parameter of reliability was based on the 95% CI or CrI provided for each prediction of the number of 322 cumulative cases. This parameter checked whether the true value (i.e. the value actually observed for the 323 prediction) is within the 95% CI or CrI provided for the prediction (Fig. 7) . Fig. 7 shows a graphical 324 representation of the cross-tabulation of the number of predictions that presented a 95% CI or CrI (20 in total) 325 and whether or not the value actually observed belongs to the 95% CI or CrI. This figure shows that 75% (15 out 326 20) of the values actually observed were within the 95% CI or CrI provided for the prediction. 65% of these (13 327 out 20) were predictions made based on statistical models (Fig. 7) . predicted over the actual number of cumulative deaths was lower or close to 1 (Fig. 8a) . One prediction largely 336 exceeded (more than 6 times) the actual number of deaths (Fig. 8a) . This ratio seems to be relatively lower (and 337 also lower than 1) for Bayesian models than for statistical models where this ratio was large than one, thus 338 suggesting relatively more accurate predictions with Bayesian models (Fig. 8b) , although these differences are 339 only suggestive due to the small size of the data. A greater number of predictions than we found in this study 340 would be needed for robust significance test. Whether the models included asymptomatic or pre-symptomatics 341 individuals does not seem to affect this ratio (Fig. 8c) . Nevertheless, there was a significant negative correlation 342 between this ratio and the number of days of the first infection in the target country, suggesting that the more 343 data used to make the estimates cover a large period of time, the more accurate the estimates tend to be (Fig. 8d) . Among the above nine studies, seven reported the 95% CI or CrI for their predications and they made ten 353 predictions of the number cumulative deaths (Fig. 9a) . For six of the ten predictions (60%), the ratio of the 354 amplitude of the 95% CI or CrI over the predicted number of cumulative deaths was lower than 1 (Fig. 9a) . Two

predictions had values between 4 and 7 for this ratio (Fig. 9a) . There was no evidence of difference for this ratio 356 among categories of models, nor according to whether the models considered asymptomatic or pre-symptomatic 357 individuals (Fig. 9 b, c) . There was also no correlation of this ratio with the length of period of time, the more 358 accurate the estimates tend to be (Fig. 9d) . 

derivative: Outbreak in India with dynamics of transmission and control

Transmission potential and severity of covid-19 in

To explain or to predict? Statistical science

Quantitative analysis of the effectiveness of public health 578 measures on covid-19 transmission

Pneumonia of unknown cause-china', emergencies preparedness, response. Disease outbreak 604 news, World Health Organization (WHO)

Coronavirus update

Generalized logistic growth modeling of the Covid-19 607 outbreak in 29 provinces in China and in the rest of the world

Linking key intervention timing to rapid decline of 610 the COVID-19 effective reproductive number to quantify lessons from mainland China. International Early estimation of the case fatality rate of covid-19 in mainland China: a data-driven analysis

MERS, SARS and other coronaviruses as causes of pneumonia

Evolving epidemiology and transmission dynamics of coronavirus disease 2019 outside Hubei 621 province, china: a descriptive and modelling study