key: cord-0925973-ln8owe7q
authors: Antulov-Fantulin, Nino; Böttcher, Lucas
title: On the accuracy of short-term COVID-19 fatality forecasts
date: 2022-03-14
journal: BMC Infect Dis
DOI: 10.1186/s12879-022-07205-9
sha: 87112704fa690ee1e480447cdc8f929ed4540bed
doc_id: 925973
cord_uid: ln8owe7q

BACKGROUND: Forecasting new cases, hospitalizations, and disease-induced deaths is an important part of infectious disease surveillance and helps guide health officials in implementing effective countermeasures. For disease surveillance in the US, the Centers for Disease Control and Prevention (CDC) combine more than 65 individual forecasts of these numbers in an ensemble forecast at national and state levels. A similar initiative has been launched by the European CDC (ECDC) in the second half of 2021. METHODS: We collected data on CDC and ECDC ensemble forecasts of COVID-19 fatalities, and we compare them with easily interpretable “Euler” forecasts serving as a model-free benchmark that is only based on the local rate of change of the incidence curve. The term “Euler method” is motivated by the eponymous numerical integration scheme that calculates the value of a function at a future time step based on the current rate of change. RESULTS: Our results show that simple and easily interpretable “Euler” forecasts can compete favorably with both CDC and ECDC ensemble forecasts on short-term forecasting horizons of 1 week. However, ensemble forecasts better perform on longer forecasting horizons. CONCLUSIONS: Using the current rate of change in incidences as estimates of future incidence changes is useful for epidemic forecasting on short time horizons. An advantage of the proposed method over other forecasting approaches is that it can be implemented with a very limited amount of work and without relying on additional data (e.g., data on human mobility and contact patterns) and high-performance computing systems.

Over the course of the COVID-19 pandemic more than 65 international research groups contributed to an ensemble forecast of reported COVID-19 cases, hospitalizations, and fatalities in the US [1] . These forecasts are a central source of information on the further development of the pandemic and used by various governmental and non-governmental entities including the Centers for Disease Control and Prevention (CDC) [2] . A similar initiative has been launched in by the European CDC (ECDC) in the second half of 2021 [3] .

Different forecasting methods [4, 5] rely on different underlying models and assumptions. One may roughly divide forecasting models into three different classes: (i) mechanistic models [6, 7] , (ii) purely data-driven models [8] , and (iii) hybrid models. Most classical epidemic models are mechanistic and aim at describing disease dynamics in terms of interacting individuals in a population. Such models are usually applied to describe the influence of certain factors (e.g., population density, demographics, contact patterns, mobility, etc.) on the dynamics of an epidemic. Data-driven and machine learning models make fewer assumptions about the underlying dynamics and are applicable to a broader range of forecasting problems, but they also come at the cost of less interpretability for policymakers and epidemiologists.

*Correspondence: anino@ethz.ch

Here, we show that a very basic, model-free forecasting approach provides effective short-term forecasts of COVID-19 fatalities. We refer to this method as "Euler forecast" because of its mathematical connection to the Euler method [9, 10] that is used in computational mathematics to calculate the value of a function at a future time step based on the current rate of change.

Different epidemiological models [12] capture different aspects of disease spread and many of these models are based on coupled ordinary differential equations (ODEs). In the susceptible-infected-recovered (SIR) model [6] , the rate of change of S(t), the number of susceptible individuals at time t, is described by the ODE Here, I(t) and N denote the number of infectious individuals at time t and the population size, respectively. The infection rate is β.

We now assume that the epidemic state of a population can be represented by some quantity y(t) and that its evolution (i.e., the rate of change) is described by a function g(y(t), t). That is,

The SIR model (1) can be written in terms of Eq. (2) by setting y(t) = (S(t), I(t), R(t)) ⊤ .

Euler's method [9] is one of the simplest numerical procedures for solving ordinary differential equations of the form (2) for a given initial condition. This method uses a timestep �t > 0 to approximate the solution of Eq. (2) at times t 1 , t 2 , . . . , t n according to9, 10 However, in reality the functional form g(·) that describes the rate of change of y(t) that is relevant for infectious disease surveillance is usually not known. In the following paragraphs, we thus describe practical ways how to estimate COVID-19 fatalities y(t) and their local rate of change ẏ(t) from noisy observation data.

We first collected data on CDC ensemble forecasts between June 2020 and June 2021 [1] . 1 Ensemble forecasts are available for cumulative and weekly incidence and fatality numbers and a forecasting horizon between 1 to 4 weeks. All forecasts use data from the Johns Hopkins Coronavirus Resource Center [11] as ground truth. Forecasts are made for epidemiological weeks which run (3) y(t n+1 ) = y(t n ) + �tẏ(t n ).

Sunday through Saturday. As an example, if forecasts with 1-and 4-week forecasting horizons are being made on June 8, 2020 the corresponding forecasting intervals are June 7-June 13, 2020 and June 7-July 4, 2020 [13] .

We compare CDC and ECDC ensemble forecasts of COVID-19 fatalities with a simple and easily interpretable forecasting method. To do so, let y(t) be the incidence of COVID-19 fatalities at time t. We use ẏ(t) to denote the rate of change of y(t) at time t. Forecasting the incidence y(t + �t) at a target time t + t requires us to find an estimate of this quantity at an earlier time t. A straightforward way to construct short-term forecasts is to (i) use the current rate of change ẏ(t) and (ii) determine a forecast at time t k = t 0 + k t according to the Euler method [9, 10] where t and k = 1, 2, . . . represent a time step (e.g., 1 week) and the number of time steps in the forecasting horizon, respectively. However, observed incidences are subject to observation noise that results from confounding factors including sampling bias, measurement errors, and reporting delays [14] .

A possible way to "de-noise" observed data is to use weekly incidences instead of daily incidence levels. If observational noise can be reduced by averaging over a period of several days, daily errors are less pronounced on a weekly level. However, the local daily derivative is quite sensitive to noise and our incidence correction term may not help in making accurate short-term forecasts. Therefore, we can impose some degree of regularity to reduce the level of noise with the following minimization where y k = y(t 0 + k�t) , w k = w(t 0 + k�t) is a regularized approximation of y k , and is a regularization parameter. In the limit → 0 , the argument of Eq. (5) is minimized if w(t) approaches y(t). In the limit → ∞ , the argument of Eq.

. This optimization process has its equivalent Euler-Lagrange formulation for numerical differentiation [15, 16] . Values of ∈ (0, ∞) yield functions w(t) that are smoothed versions of y(t) with respect to the discrete rate of change w k − w k−1 . Finally, the regularized Euler short-term forecast 2 is given by In the following section, we use both the standard Euler method and the regularized Euler method to generate forecasts of reported COVID-19 fatalities.

Our source codes are publicly available at [17] . 

fatalities in the US. We observe that 1-week CDC ensemble forecast for the majority of data points are not more accurate than 1-week Euler forecasts (Fig. 1a) , which we use as a local-derivative-based forecasting benchmark. Although Euler and CDC forecasts still exhibit a similar structure for a 4-week forecasting horizon (Fig. 1b) , the Euler method is associated with larger deviations from the reported fatalities than the CDC ensemble method. To quantify differences in forecasting errors between the two methods, we use to denote the absolute error between the Euler or CDC forecast y(t) and the ground truth x(t) at time t. Figure 1c , d show the 4-week moving averages of weekly forecasting errors δ(t) (solid lines) of the Euler method (red) and the CDC ensemble (blue) method. As suggested by our above discussion of Fig. 1a , we observe that the error of the Euler method is substantially smaller than that of the ensemble forecast for a 1-week forecasting horizon. In about 61% of the forecasting instances shown in Fig. 1a , the regularized Euler method has a smaller error than the CDC ensemble forecast. The cumulative forecasting errors are 49,925 (Euler) and 52,885 (CDC). Without correction term (7) δ x,y (t) = |x(t) − y(t)| 6), the cumulative forecasting error of the Euler method is 52,660, again smaller than that of CDC ensemble forecast. Note that forecasts without correction correspond to a simple shift of the incidence curve [see Eq. (4)]. For a 4-week forecasting horizon (Fig. 1d) , the cumulative error of the CDC ensemble forecast is 87,717, about 35% smaller than that of the Euler method.

To complement our analysis of CDC ensemble forecasts from June 2020 until June 2021, we have updated the CDC ensemble forecast data in January 2022. We conduct a separate analysis because historical ensemble forecasts can be changed a posteriori [18] . In addition, we have also gathered ECDC forecasts from May 2021 until January 2022 [3] for EU and EFTA countries and the UK.

Based on the second set of CDC ensemble forecasts, we observe that the accuracy of 1-week ensemble forecast (Fig. 2a, b) improved slightly with respect to the regularized Euler forecast. The cumulative for ecasting errors until January 2022 are 93,645 (Euler regularized) and 84,870 (CDC). Without correction term k [w(t) − w(t − �t)] in Eq. (6), the cumulative forecasting error of the Euler method is 94,108. For a 1-week forecasting horizon, the Euler method is associated with larger deviations from the reported fatalities than the CDC ensemble method i n about 51% of the forecasting instances shown in Fig. 2 . On a longer forecasting horizo n of 4 weeks, the Euler method performs better than the CDC ensemble method in only 16% of all cases. This result is not surprising because the Euler method relies on smoothed curve shift and is not designed for longer forecasting horizons. For the comparison with ECDC [3] ensemble forecasts, we use all data that was available in January 2022 to compare the forecasting errors with those of Euler forecasts (Fig. 3a, b) . The cumulative forecasting errors until January 2022 are 28,769 (Euler regularized) and 30,942 (ECDC). Without correction term k [w(t) − w(t − �t)] in Eq. (6), the cumulative forecasting error of the Euler method is 30,353. In about 36% of the forecasting instances shown in Fig. 3a , the regularized Euler method has a s maller error than the ECDC ensemble forecast. Finally, in Appendix Figs. 4 and 5 we show joint comparisons of errors of Euler, regularized Euler, and (E)CDC forecasts.

On 1-week forecasting horizons, regularized Euler forecasts have smaller errors with respect to CDC ensemble forecasts in about 61% of all cases up to June 2021 and in about 49% of all cases up to January 2022. The cumulative errors are worse for CDC up to June 2021 and better if we consider data up to January 2022. In comparison with ECDC forecasts, the regularized Euler method performs better in 36% of the forecasting instances on a 1-week forecasting horizon, while ECDC forecasts are associated with a lower cumulative error up to January 2022. Overall, on a 1-week forecasting horizon simple Euler forecasts can perform similarly to ensemble methods that are composed of a large number of more complex models. In agreement with [19] , our results emphasize the importance of benchmarking complex forecasting models against simple forecasting baselines to further improve forecasting accuracy. Similar conclusions were drawn in a recent study [19] that compared Euler-like forecasts with those generated by Google Flu Trends. Our study also points towards recent findings on algorithm rejection and aversion [20] that found that "people have diminishing sensitivity to forecasting error" and that "people are less likely to use the best possible algorithm in decision domains that are more unpredictable". Finally, in highly uncertain and noisy forecasting regimes, simple methods tend to outperform more complex methods because of a more favorable bias-variance trade-off [21] .

Our results suggest that easily interpretable methods like the Euler method, a model-free local-derivative-based forecasting benchmark, provide an effective alternative to more complex epidemic forecasting frameworks on shortterm forecasting horizons. Simple curve shifts without regularization provide forecasts that are close to CDC and ECDC ensemble forecast, a finding that can help improve existing forecasting methods. For longer forecasting horizons, it is not surprising that CDC and ECDC forecasts that rely on additional input data and epidemiological and statistical models become more accurate than Eulerlike forecasting benchmarks. One clear advantage of Euler forecasting methods is that they are less labor and resource intensive than more complex forecasting models, which often rely on the knowledge of expert groups and require specialized computing infrastructure. In their simplest implementation, Euler forecasts use the currently observed incidence rate as an estimate of the incidence rate in the following week. Regularization methods (6) can help further improve such data-driven forecasts. 

Forecasting errors of Euler, regularized Euler, and CDC ensemble forecasts

Forecasting errors of Euler, regularized Euler, and ECDC ensemble forecasts (data

The COVID-19 Forecast Hub

Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the

European Covid-19 Forecast Hub

Forecasting Covid-19

Comparison of some forecasting methods for covid-19

Modeling infectious diseases in humans and animals

Unifying continuous, discrete, and hybrid susceptible-infected-recovered processes on networks

Applied time series analysis: a practical guide to modeling and forecasting

Institutiones calculi integralis

An interactive web-based dashboard to track COVID-19 in real time

Infectious diseases of humans: dynamics and control

Data submission instructions

Using excess deaths and testing statistics to determine COVID-19 mortalities

Numerical differentiation and regularization

Numerical differentiation of noisy, nonsmooth data. International Scholarly Research Notices

Data processing rules in COVID-19 Forecast Hub

Transparent modeling of influenza incidence: Big data or a single data point from psychological theory?

People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error

The elements of statistical learning

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

We thank the co-organisers (D.Helbing, Z.Ce, D.Dao) of the epidemicdatathon. com 2020 forecasting challenge for inspiring discussions.

See Figs. 4 and 5. 

All data used in this article are available at [1, 11] . Our source codes are publicly available at [17] .

No administrative permissions were required to access the publicly available raw data [1, 3] used in this study.

Not applicable. Manuscript does not contain any individual person's data in any form.

The authors declare that they have no competing interests.