key: cord-0914403-rcdbqj8a
authors: Gustafsson, Fredrik; Jaldén, Joakim; Bernhardsson, Bo; Soltesz, Kristian
title: Identifiability issues in estimating the impact of interventions on Covid-19 spread ⁎ ⁎ The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).
date: 2020-12-31
journal: IFAC-PapersOnLine
DOI: 10.1016/j.ifacol.2021.04.179
sha: 17c8cb26a07ec3db4ebb1d2dbbead214e9ce0d30
doc_id: 914403
cord_uid: rcdbqj8a

The Covid-19 pandemic has spawned numerous dynamic modeling attempts aimed at estimation, prediction, and ultimately control. The predictive power of these attempts has varied, and there remains a lack of consensus regarding the mechanisms of virus spread and the effectiveness of various non-pharmaceutical interventions that have been enforced regionally as well as nationally. Setting out in data available in the spring of 2020, and with a now-famous model by Imperial College researchers as example, we employ an information-theoretical approach to shed light on why the predictive power of early modeling approaches have remained disappointingly poor.

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020) ]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020)]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020)]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020)]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. Fig. 1 . Block diagram for the ICCRT NPI model.

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. 

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020) ]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. Fig. 1 . Block diagram for the ICCRT NPI model.

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

We illustrate how the combination of poor excitation in modelling data and opaque modelling easily lead to the loss of practical identifiability of the resulting model. While being well-known within system theory, these issues have been neglected in remarkably many epidemiological models used to estimate or predict aspects of the ongoing SASRS-CoV-2 pandemic. Several such models have had impact on policy, and ultimately on people's lives. In March 2020 the Imperial College Covid-19 Response Team (ICCRT) issued a report [Ferguson et al. (2020) ], based on an agent model, that resulted in an overnight change of the UK response policy, manifested by a nation-wide lockdown [Boseley (2020) ]. Later the same month, the ICCRT published a subsequent report [Flaxman et al. (2020b) ], in which a Bayesian model ascribed almost all reduction of the viral reproduction number R to lockdowns in all modeled European countries, except for Sweden. The non-pharmaceutical intervention (NPI) that brought R < 1 in Sweden was in contrast ascribed to have had only negligible effect on R in all the other countries. A revision of the second model [Flaxman et al. (2020b) ] was published in Nature [Flaxman et al. (2020a) ]. The model code, including revision history, has been made publicly available [ICCRT (2020)]. When executing the original code, we found that the model suffered sensitivity

The work behind this extended abstract was partially supported by the ELLIIT Strategic Research Area, by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and by the Swedish Foundation for Strategic Research (SSF) via the project ASSEMBLE (contract number: RIT15-0012).

issues, leading to practical loss of identifiability under the data available at the time of publication [Soltesz et al. (2020) ]. This prompted us to conduct a systematic investigation of the model. Here we provide a summary of that investigation.

The basic dynamics of how NPIs affect reported data is illustrated in Fig. 1 . Each NPI is assumed to have an effect on the society that affects the spread, through (presumably) decreasing the effective reproduction number R. Since the infected population is not directly measurable, an observation model for some measurable data is needed. Fig. 1 . Block diagram for the ICCRT NPI model.

The binary indicator function σ k,t,m models NPI k = 1, . . . , K effectuated at time t = 1, . . . , T in country m = 1, . . . , M, where σ k,t,m = ∞ denotes that NPI k was not effectuated in country m during the studied time interval t ∈ [1, T ]).

The modeled NPI categories are:

• Lockdown; • Public events ban; • School closure; • Self isolation;

• Social distancing.

In addition to these, the model ascribes a country-specific effect for the last NPI enacted in each country, modeled as a virtual country-specific NPI.

It is worthy of notice that all five NPI categories were enacted in 10 of the eleven Europeans countries included in [Flaxman et al. (2020a) ]. In the eleventh country, Sweden, four of the categories were enacted, but there was no lockdown. All NPI enactments took place in March 2020, most of them within a single week.

The model relies on three distributions of times between:

• becoming infected and infecting the next person is called the serial interval and is here denoted p II (τ ); • infection and the onset of symptoms is denoted p IO (τ ); • symptoms onset and death is denoted p OD (τ ).

The distribution of time between infection and death, p ID (τ ) = p IO * p OD (τ ), is used to relate the number of infected people to the reported number of deaths.

We can now define the NPI model (1). The infected population I t,m at time t in country m having population N m is given by the recursion

To get some intuition for this formula, consider the special case where p II (k) = δ(k), that is, every infected person is infectious exactly after k days, and only that day, and then spreads the disease to R t,m I t,m /N m susceptible. The effect of NPI k on R t,m is modeled by:

Note that the parameter α k is the same for all countries. This is what pools the M countries together within the model.

Finally, we have an observation model for the mortality data y t,m ,

where IFR denotes the infection-fatality ratio.

In [Flaxman et al. (2020a) ] a Bayesian MCMC framework was used to estimate the parameters R 0,m , m = 1, . . . , M and α k , k = 1, . . . , K. Our contribution has been to recast the model in a linear regression framework where the least squares (LS) method can be used.

The Bayesian model [Flaxman et al. (2020a) ] relies on the following prior distributions:

p II (τ ) = Γ(6.5, 0.62), (2a) p IO (τ ) = Γ(5.1, 0.86), (2b) p OD (τ ) = Γ(18.8, 0.45).

(2c)

Here, Γ(a, b) has mean a, coefficient of variation b, and standard deviation ab.

Priors for the NPI parameters were chosen as α k ∼ Γ(0.5, 1) in [Flaxman et al. (2020b)] , and subsequently changed to

in [Flaxman et al. (2020a) ], with the motivation that α k ∼ U (0, 1.05) when all NPIs are effectuated. That is, there is a possibility that the interventions increase R by a factor e 1.05 , but most of the prior is assigned to a significant decrease. As was noted by Nicholas Lewis in [Lewis (2020) ], the Γ prior has a regularization effect. Suppose that the model needs k α k = 1.75, corresponding to a reduction of 1 − e 1.75 = 0.83 in R t which was the median reduction in the MCMC samples of the Nature model [Flaxman et al. (2020a) ]. Then the prior for α i = 1.76/6 for all i is 0.0023. If one α i = 1.71 dominates and the others are insignificant, α j = 0.01, then the prior is 64.3. That is, the prior gives a strong bias to having one or a few NPIs that explain data. Further, R 0, m ∼ N(2.4, |κ|) where κ ∼ N(0, 0.5). The ICCRT model approach is based on sampling from the parameter priors, evolving (1a) and (1b), and then evaluating the likelihood for each sample in a Bayesian MCMC framework. A simplified description of the methodology essentially consists of the following steps:

(1) Generate random numbers of the K + M involved parameters θ = (α : , R 0,: ). This is the most important step and there are many sampling strategies to achieve it. (2) Simulate the model (1).

(3) Compute the likelihood for the observed mortality data. (4) Generate a random number u = U (0, 1). (5) If the log likelihood ratio has increased more than u, the simulation is accepted, otherwise it is rejected. (6) Continue until a pre-defined number of samples has been obtained, excluding the burn in time before the MCMC has converged to stationarity.

It turns out that the estimation problem can be cast in a Fisherian LS framework. This explicitly reveals identifiability issues of the model, obscured by ambiguous prior choices in the Bayesian MCMC framework.

Equation (1b) is a linear regression in the log domain:

Taking the logarithm also of (1a), and eliminating R t,m , gives 

The diagonal matrix reveals that one linear combination of the parameter vector can be estimated with a variance bound of only 0.0003, which is 200 times better than the worst linear combination Var kα k ≥ 0.0015.

The conclusion is that the sum of the parameters has a CRLB that is 6 to 20 times smaller than its individual components.

The high sensitivity of the ICCRT model is not merely concerning from a theoretical point of view. In [Soltesz et al. (2020) ], we have analyzed how seemingly subtle changes in NPI category definitions result in the ICCRT model [Flaxman et al. (2020a) ] ascribing wildly varying effectivnessess to different NPI categories. The two changes we considered were introduced by the ICCRT modellers in subsequent code versions. They redefined the crowd size associated with a public events ban, and whether a school closure should also encompass high schools and universities.

The perhaps most remarkable artefact of the identifiability issues is that the ICCRT model ascribes almost all reduction of virus spread during the spring of 2020 to the lockdown NPI in the 10 modeled European countries that implemented one. In tho only country that (according to the model) did not implement a lockdown, another NPI category provided an almost as large spread reduction as that caused by lockdowns elsewhere. This other NPI category is either of school closure, public events ban, or the encouragement of social distancing, depending on which of the above-mentioned definition variants is adopted. Notably, neither of these three NPI categories had substantial impact in the other 10 countries. According to the model it was thus fortunate that a lockdown was implemented in all countries where it was effective, and omitted in the single country, where another, and elsewhere ineffective NPI, turned out crucial to bring down R.

Using established principles from systems theory, we have demonstrated that even a seemingly simplistic and datadriven phenomenological model can suffer from severe input sensitivity and identifiability issues.

There are indeed additional properties of the ICCRT model that would justify scrutiny. For instance, the base assumption that changes in R are solely driven by NPIs constrain the model to explain any reduction in R through the NPIs. Another problematic aspect is the stochastic delay, with a mean of about 3-4 weeks, between infection and the death report, which is what is typically directly measured. This adds further uncertainty to the (pseudo) observations of the regression model, and thus the parameter estimates. The perhaps most remarkable model property is, however, the special role ascribed to the last NPI enacted in a particular country. This prohibits prospective use of the model, since it is impossible to tell whether the latest enacted NPI was also the last.

Unfortunately, high sensitivity cannot solely be ascribed to questionable choices underlying the studied model. Instead, practical parameter identifiability is to a large extent inherently limited by available data. All NPIs were effectuated during a period of 22 days, most of them in one single week. The regression tensor therefore has many common leading zeros as well as many common trailing ones for all parameters, indicating poor excitation. Furthermore, several of the involved countries had very little virus spread when the NPIs were effectuated, leading to an almost zero output.

With much at stake during all phases of a pandemic, we conclude that it is crucial to thoroughly scrutinise any SARS-CoV-2 estimation or prediction model, prior to considering its use as decision support in policy-making. Such scrutiny relies on modellers following the practice used by the ICCRT in sharing open source code.

New data, new policy: why UK's coronavirus strategy changed. The Guardian

Report 9 -Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand

Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe

Report 13 -Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries

ICCRT Github repository

Did lockdowns really save 3 million COVID-19 deaths

Sensitivity analysis of the effects of non-pharmaceutical interventions on COVID-19 in Europe

We would like to acknowledge Ericsson Research for hosting our execution of the ICCRT model code, and our collaborators in Lund, Linköping and Uppsala for executing the model code and contributing through fruitful discussions.

This fits the classical linear regression framework, withwhereHere, e m is the m th unit vector. Vectorizing the data for all M countries, we get the more compact formThe solution is given bŷHere, λ = Var(z t,m ) denotes the variance of the transformed data, assuming it to be the same for all times and countries.

The above linear regression model requires knowledge of the number of infected people I t,m to compute z t,m of (6b). It can be estimated by inverse convolution from the modelFrom a statistical viewpoint, this two-step procedure corresponds to a certainty equivalence assumption that does not attempt to give a correct description of the observation noise. However, it is useful for analyzing identifiability of the model.

The lack of excitation in the NPIs constitutes a fundamental limitation. It can be formalized through the Cramér-Rao lower bound (CRLB) that limits the achievable estimation accuracy of any unbiased estimatorwhere J(θ o ) is the Fisher information matrix (FIM) evaluated at the true parameter values θ o , and λ is the variance of the noise on z t,m . Since we have a linear regression, the bound simplifies into

One can note that X t is a matrix with binary entries, rendering a FIM J = t X T t X t over Z + . Identifiability is directly related to the condition number of the FIM. The Nature version of the model corresponds to cond(J) > 1 000, indicating a very high model sensitivity.

Assume we apply all available data up to May 9, 70 days after March 1, and that we can compute the virtual measurement z t,m , having Gaussian noise with variance λ. In this idealistic case, the CRLB dictates Cov α ≥    