key: cord-1043265-ayfdsn07
authors: Maziarz, Mariusz; Zach, Martin
title: Agent‐based modelling for SARS‐CoV‐2 epidemic prediction and intervention assessment: A methodological appraisal
date: 2020-08-21
journal: J Eval Clin Pract
DOI: 10.1111/jep.13459
sha: affe8bce78a60d4fcc5342a8ba0e9b19edcf7761
doc_id: 1043265
cord_uid: ayfdsn07

BACKGROUND: Our purpose is to assess epidemiological agent‐based models—or ABMs—of the SARS‐CoV‐2 pandemic methodologically. The rapid spread of the outbreak requires fast‐paced decision‐making regarding mitigation measures. However, the evidence for the efficacy of non‐pharmaceutical interventions such as imposed social distancing and school or workplace closures is scarce: few observational studies use quasi‐experimental research designs, and conducting randomized controlled trials seems infeasible. Additionally, evidence from the previous coronavirus outbreaks of SARS and MERS lacks external validity, given the significant differences in contagiousness of those pathogens relative to SARS‐CoV‐2. To address the pressing policy questions that have emerged as a result of COVID‐19, epidemiologists have produced numerous models that range from simple compartmental models to highly advanced agent‐based models. These models have been criticized for involving simplifications and lacking empirical support for their assumptions. METHODS: To address these voices and methodologically appraise epidemiological ABMs, we consider AceMod (the model of the COVID‐19 epidemic in Australia) as a case study of the modelling practice. RESULTS: Our example shows that, although epidemiological ABMs involve simplifications of various sorts, the key characteristics of social interactions and the spread of SARS‐CoV‐2 are represented sufficiently accurately. This is the case because these modellers treat empirical results as inputs for constructing modelling assumptions and rules that the agents follow; and they use calibration to assert the adequacy to benchmark variables. CONCLUSIONS: Given this, we claim that the best epidemiological ABMs are models of actual mechanisms and deliver both mechanistic and difference‐making evidence. Consequently, they may also adequately describe the effects of possible interventions. Finally, we discuss the limitations of ABMs and put forward policy recommendations.

In the aftermath of the outbreak of the novel coronavirus, governments around the globe have introduced non-pharmaceutical public health interventions aimed at slowing down the spread of the resultant pandemic. These measures range from relatively mild requirements like wearing face masks, washing hands, or avoiding close contacts to school closures and imposed isolation that are likely to have a detrimental and unpredictable influence on social and economic life. 1 Despite their significant impact, the introduction of many of these measures was not supported with high-quality evidence.

First, conducting RCT would not be feasible for both ethical and practical constraints. Second, significant differences between the coronaviruses that caused the SARS and MERS outbreaks and SARS-CoV-2 (such as the likely airborne transmission 2 and asymptomatic infectiousness of the latter 3,4 ) undermine extrapolation from the data gathered during these previous epidemics. Finally, the current pandemic has not lasted long enough to gather observational data in the amount and quality sufficient for the assessment of the efficacy of alternative public health interventions, since the first reports were published just weeks after the first measures were introduced. 5 One of the many ways to address the issue concerning the impracticality of conducting RCTs and observational studies in the context of an ongoing pandemic is through scientific modelling, in particular epidemiological modelling. Here, we focus on the so-called agent-based modelling (ABM) approach, which differs from more traditional epidemiological modelling in several ways.

ABMs are a form of computational modelling strategy where agents are treated as entities interacting with each other and their environment in a locally defined fashion described by a set of rules.

The overall dynamics of the system are then computed, allowing for the simulation of complex patterns and an understanding of how these patterns arise. 6, 7 ABMs are used in many scientific contexts, including modelling the spread of infectious diseases, and have proven successful in informing policy decisions before. For instance, Eisinger and Thulke 8 modified and then applied a previously developed ABM of the spread of rabies, generating a rule-based model that represented specific spatial and behavioural characteristics of the fox population (eg, with fox families represented as moving within home ranges and young foxes engaging in long-distance migratory behaviour). 6 Whereas the classical differential equation models predicted that vaccinating at least 70% of the fox population would eliminate rabies, the ABM indicated that a successful vaccination strategy could do with much less than 70% of the population being immunized once the spatial arrangements of fox hosts were explicitly considered, saving millions of Euros as a result. Moreover, the ABM also suggested that the classical strategy would fail more often than not, and was successfully applied to deal with the rabies problem. However, despite the promising record of using ABMs in effective epidemiological interventions, its use in informing proposed measures against the novel coronavirus epidemic has raised criticism. [9] [10] [11] Unfortunately for the assessment of healthcare interventions based on this type of epidemiological models, standard evidence hierarchies exclude agent-based models altogether and include theoretical or mechanistic inferences at the lowest level of the hierarchy.

For example, the Oxford Centre for Evidence-Based Medicine 12 and the National Institute for Health and Care Excellence (NICE guidelines) 13 include theoretical and mechanistic reasoning but agent-based models fall beyond their scope. This can be explained by the novelty of agent-based modelling and the limited trust of EBMers in theoretical and, to some extent, also mechanistic reasoning, which, despite being used implicitly to assess the possibility of confounding and the quality of results, 14 is downgraded or rejected as either subjective or fallacious. 15 However, such a view has been challenged by a group of philosophers advocating for improving the practices of evidence assessment in medicine by putting more weight on mechanistic reasoning in causal inference. [16] [17] [18] The position of the EBM+ program [16] [17] [18] is encapsulated by the normative reading of the Russo-Williamson Thesis, 19 which states that causal claims should be based on both difference-making and mechanistic evidence.

The causal claims supported by agent-based models have been interpreted inconsonantly: either as being in line with the potential outcome approach (POA), 20 as delivering theory-driven understanding 21 or as providing mechanistic evidence. 22 Below, we show that all of these apparently inconsistent interpretations are correct, because the best contemporary ABMs bear a resemblance to the actual mechanisms and therefore allow for the counterfactual assessment of intervention efficacy in the target while also delivering an understanding of the phenomena of interest. Our argument proceeds by 

Apart from the compartmental SIR (Susceptible, Infectious, Recovered) framework and its derivatives [23] [24] [25] [26] [27] [28] or regression analysis, 29, 30 most advanced models of the spread of the novel coronavirus are transformed versions of agent-based influenza pandemic models. 11, 31 Such models have been used as evidence for introducing (sometimes severe) public health measures, 32 with the recent change in British policy being the prime example. In this section, we illustrate this approach to modelling the SARS-CoV-2 pandemic with an agent-based model of the epidemic in Australia 31 based on AceMod.

Developed as a "framework for studying influenza pandemics in Australia" 33 (p. 412) . AceMod is an influenza spread model that addresses the need for simulating interventions responding to the outbreaks of future respiratory diseases. While the 2009 swine flu pandemic was the motivation for constructing AceMod, the model was not intended to accurately represent the outbreak of the H1N1 strain, but rather as a generalized framework for studying how an infectious disease spreads through the social interactions of Australians. AceMod utilizes census data to ascribe realistic spatial and social characteristics to almost 20 million agents inhabiting the model world. These agents are divided into different social groups of varying characteristics, with households differentiated proportionally according to statistical data on the prevalence of different types of families (singles, single parents and couples with or without children). These features are ascribed to agents stochastically in a way that replicates the aggregate structure of statistical data. During the daytime, children and students meet in classrooms and at schools, adults go to work and pensioners stay at home. During the nighttime, the agents encounter contacts at households and in their neighbourhoods (eg, at supermarkets, theatres).

The disease can be contracted by an agent in the event of meeting an infected individual in one of these settings. The probability that an agent i contracts the disease in a given step t depends on the number of sick individuals met in that step and the contagiousness of the disease, scaled by K. The modellers assume that the infectivity of the disease decreases linearly over time. Asymptomatic cases are assumed to be 50% less infectious than symptomatic ones, and the flu lasts 5 days within the model. After this period, recovered agents cannot infect others. Additionally, those who experience symptoms do so after an incubation period lasting approximately 3 days. The influenza epidemic is started by agents coming to Australia via international airports and seeded into communities living near the airports at random.

To represent an epidemic of a particular strain of influenza with AceMod, the model requires calibration. Modellers can proceed with this step in two ways, depending on the accessibility of data. In the case of well-studied influenza strains, their infectivity and the ratios of transmission in different contexts are well-recognized, and parameter values can be chosen based on empirical studies. However, if these data are missing, then parameter values have to be calibrated using statistical procedures such as simplex or genetic algorithms to maximize the fit of the model to a benchmark. After constructing and calibrating AceMod, modellers run simulations to obtain the estimates of prevalence, incidence and attack rates, and choose the most common outcome (due to stochasticity, different runs of the model may lead to obtaining slightly different results).

Chang et al 31 ABMs such as AceMod can be seen as consisting of two parts: the rules specifying the behaviour of agents and the creation of the model society, as well as the assumptions characterizing the infectivity of the pathogen causing the epidemic. Given that AceMod is based on 2016 census data and a major change in social behaviours is unlikely to have occurred since then, the model accurately represents the social interactions of present-day Australians. Hence, the former part of the model has been left mostly unchanged, beyond increasing the number of agents to over 24 million to adjust for the growing population. In addition to introducing a social structure sufficiently resembling the contact network of the present population, obtaining accurate predictions of epidemic development and policy assessment requires inputting data on transmission likelihoods that are true for the pathogen causing the modelled epidemic. 34 Most changes in the model are concerned with the assumptions specifying the infectivity of the disease. Even though several features of influenza epidemics are similar to the epidemic caused by the novel coronavirus, they differ with respect to infectivity and attack rates, mortality rates, the average duration of disease, the reproductive number R 0 and the distribution of asymptomatic cases. Therefore, these parameters in the model required recalibration.

The transmission probabilities remained mainly as specified in the The length of the generation period was calibrated to 6.4 days to reflect this difference in the model. Additionally, the likelihood of contracting SARS-CoV-2 but staying asymptomatic was set to be agedependent, and equalled 1/3 for adults while minors were set to be five times less likely to suffer from symptoms than adults. While this assumption is in agreement with the empirical findings that children represent a minor fraction of symptomatic cases, the calibration aimed at reproducing aggregate epidemic curves and may diverge from the actual chances of developing symptoms.

Within the AceMod framework, the reproductive number R 0 is not one of the assumptions inputted into the model. Rather, its estimate results from a simulation of the scenario described by the rules and assumptions, some of which are stochastic. The assumptions considered and, particularly, the parameter denoting contagiousness of the disease (K) have been calibrated such that R 0 stays within the limit of (2.0-2.5), that is, in agreement with empirical estimates of the reproductive number at the beginning of the SARS-CoV-2 outbreak. 35, 36 The set of parameter values that result in the estimate of 

Before proceeding to our argument, let us first make several general remarks about modelling. These remarks should prove essential in clarifying the main issues that are often raised with regard to using simplified models, particularly in the context of policy decision-making. First of all, ABMs are instances of mechanistic models, for they clearly fit the general, also called the minimal, characterization of what a mechanism is: a set of entities whose activities and interactions are organized such that they are responsible for the phenomenon. [37] [38] [39] This definition is broad enough to conceptually unify the debates on biological and social mechanisms under a single notion of a mechanism. Furthermore, such definition leaves open the possibility of integrating biological and social aspects into a mixedmechanism model. 40 It should also be noted that much like any other kind of model, ABMs serve as simplified representations of their target phenomena.

As the AceMod case clearly shows, modellers introduce various simplifications by which they purport to adequately capture the core dynamics of the modelled phenomenon. In this process, they first abstract away from the complexities of the real system by "extracting" certain features that they believe to be of crucial importance and that will then be the focus of modelling, whereas other features that may or may not have a causal influence are disregarded in these early stages. Modelling is an iterative process during which the merits of the model's assumptions are continuously being evaluated, and if required, the assumptions are refined and additional assumptions added. More importantly, some of those extracted features are distorted to the extent that, if taken literally, they would misrepresent the actual state of things. However, introducing such distortions is often made in full awareness, with the ultimate goal of finding out whether the consequences they have for the behaviour of the system make a difference and to what degree. Philosophers often refer to the former-that is, the set of properties retained in a model-as an abstraction, while the latter case-that is, the distortions of the system's features-is called an idealization. 41 However, abstractions and idealizations do not exhaust the conceptual toolbox available to modellers. A popular way to attempt to model a given system realistically is to introduce various approximations. Although there are noteworthy differences between approximations and idealizations, we cannot afford to go into any detail here. In summary, models often effectively disregard, distort and otherwise simplify possibly important details. In light of this, many wonder whether we can gain insight into the modelled phenomenon at all, and if so then how.

Although the SARS-CoV-2 ABM is fairly detailed and precise, it cannot do without some of the simplifications discussed above. Consider some of the following assumptions introduced in the model. On Consequently, we concur with Andersen's claim that "no mechanism model can include all the actual, much less the potential, causal relationships in which such a mechanism may engage in a system" 51 (p. 995). This pessimistic view on simplified models has inspired the method known as exploratory modelling. 52 In cases when the values of parameters and assumptions inputted into the model cannot be established with certainty, researchers can simulate multiple possible worlds to discover the dependencies that are stable across the set of different models. In cases when only a fraction of assumptions are uncertain, researchers conduct sensitivity analyses to check if changes in the values of the parameters lead to changes in their conclusions. 53 The results that remain unchanged despite minor adjustments to assumptions are considered to be robust. 54 This, in turn, leads to choosing those interventions that are most effective across different sets of parameter values, known as robust decision-making. 52 Others prefer to think in terms of the distinction between howactually and how-possibly modelling, referring to models that describe an actual mechanism or a possible mechanism, respectively. 55 There are two general ways to unpack the concept of a how-possibly model. Here we argue that, notwithstanding the simplifications introduced in the discussed influenza and SARS-CoV-2 ABMs, the epidemiologists are, in fact, providing representations of actual mechanisms of the spread of the viruses. This can be supported by exploiting the relevant similarities 56,57 between the SARS-CoV-2 ABM and the actual outbreak. The respects in which an ABM can be judged similar to its target concern the features retained in that model, while the degree(s) of similarity concern the extent to which the model's fea- Two remarks are in order here. First, one may oppose the claim that what is being represented is the actual mechanism by arguing that the mechanism underlying the beginning of the outbreak and the fully-fledged epidemic are distinct. Changes in social behaviour or genetic mutations could undermine the behavioural adequacy of the model. Second, it is possible (at least in principle) that the model represents a false mechanism, but is calibrated to the relevant benchmark such that it reproduces it. For example, there is no data confirming (or disproving) the assumption that children are asymptomatic five times more often than adults. As the modellers admit, this assumption was made not only to account for the lower attack rate among minors, but also to make the model adequate to aggregate-level data. This approach to calibration resembles the estimation of statistical parameters (a.k.a. curve fitting) and is considered dubious. The main line of criticism highlights that it is in principle possible to construct a model that represents a possible mechanism and, using calibration, adjust parameter values so that it reproduces the represented phenomenon, that is, obtains behavioural adequacy despite being false. However, while this criticism is indeed justified regarding models of mechanisms that are epistemically inaccessible in other ways (such as mechanisms in the social sciences 59 ), it is not so in the case of epidemiological mechanisms whose transmission mechanism can be studied empirically and compared to the mechanism represented by the model. This can establish that the mechanism represented by the model is similar (in relevant aspects and to relevant degrees) to the mechanism that generates the outbreak, that is, achieves mechanical ade- Given that AceMod fulfils Glennan's criteria for behavioural and mechanical adequacy, considering our current understanding of the novel coronavirus, we can conclude that Chang's et al 31 model represents the actual mechanism of the spread of the disease in Australia.

Given this, the claims assessing the efficacy of the mitigation measures under consideration are likely to be accurate not only within the model but also about its target. We claim this with several caveats in mind to be discussed in the next section.

It is also important to note that the ABM integrates the biological aspects, expressed by the parameter of infectivity, and the social aspects such as daily interaction regimes. As a result, the ABM should be construed as an instance of a model of a mixed mechanism, a concept elaborated by Kelly et al. 40 Due to exposure patterns, population-level phenomena such as infectious disease epidemics are crucially dependent on human behaviour and social practices. In cases like the current pandemic, effective interventions may best be aimed at the societal level and therefore mechanistic models that integrate social factors, human behaviour and biological aspects (something that the ABM discussed here attempts to do) are arguably best suited for providing understanding and suggesting policy decisions.

Our study defends using ABMs for informing decisions regarding miti- had limited influence on the severity of the epidemic, considering that just one cluster was located at a school. 68 We believe that, considering the diversity in the number and patterns of social interactions across countries, the quality of evidence from ABMs should be assessed on the case by case basis. To do so, one can employ the approach of Parkinnen et al 17 (p. 79) developed initially to evaluate the quality of evidence for biological mechanisms.

In that case, one should consider (a) the quality of the method (ie, con- Additionally, ABMs, much like the compartmental models, are dependent on the assumptions of the modellers. 10 Our claim that AceMod calibrated for SARS-CoV-2 bears similarity to the actual mechanism of the epidemic depends on the accuracy of the empirical results used as an input for this model. We need to repeatedly acknowledge the provisional nature of these empirical results, given the novelty of the pathogen. If the parameter values in AceMod were miscalibrated, then the assessments of intervention efficacy could be wrong. This implies that neither the virus can mutate nor that people can significantly and unpredictably change their behaviour since "the efficacy of implementation depends on people's reactions, [the stability of] pre-existing social norms and structural societal constraints." 9 Furthermore, the effects of epidemiological agent-based modelling are highly dependent on social structure and carefully calibrated to social and economic characteristics. Therefore, the epidemiological ABMs are geographically localized and their conclusions should not be extrapolated beyond their target systems, 71 unless the models and their predictions are calibrated to particular settings. Finally, while AceMod is well-documented in the two publications discussed throughout our paper, neither its code nor detailed documentation regarding its use is published (this unfortunately also applies to some other ABMs of the SARS-CoV-2 epidemic). Given these limitations, the models should be carefully checked for coding errors and other possible flaws before applying their implications in the policy context.

In summary, we have argued that, despite the criticism raised against models being the appropriate vehicle for informing policies, the SARS-CoV ABM is suitable for this purpose because the mechanism described by the model sufficiently resembles the mechanism at work in the real world. Thus, our best contemporary epidemiological ABMs are representations of the actual mechanism of the spread of the virus.

Unfortunately, such models have been left out from methodological discussions and are not explicitly listed by evidence hierarchies. While the need for appraising mechanistic reasoning in medicine is also voiced by EBMers, 72 there is no broadly-accepted view on how to amalgamate evidence of different types. Further research is needed to assess the risk of bias in the epidemiological models that deliver both difference-making and mechanistic evidence. However, considering the current situation and pressing need for rapid and accurate decisions regarding mitigation measures, policymakers should take to heart the advice that "if no randomized trial has been carried out […], we must follow the trail to the next best external evidence and work from there" 73 (p. 74 ). In the current situation, accurately calibrated epidemiological ABMs are the best existing evidence.

Isolation, quarantine, social distancing and community containment: pivotal role for old-style public health measures in the novel coronavirus (2019-nCoV) outbreak

Is the coronavirus airborne? Experts can't agree

Asymptomatic and human-to-human transmission of SARS-CoV-2 in a 2-family cluster

Presumed asymptomatic carrier transmission of COVID-19

Association of Public Health Interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China

Agent-Based and Individual-Based Modeling: A Practical Introduction

An Introduction to Agent-Based Modeling: Modeling Natural, Social, and Engineered Complex Systems with NetLogo

Spatial pattern formation facilitates eradication of infectious diseases

Computational models that matter during a global pandemic outbreak: a call to action

Modelling the pandemic

Report 9: impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand

Developing NICE Guidelines: the Manual. London: National Institute for Health and Care Excellence

The judgements that evidence-based medicine adopts

Evidence: philosophy of science meets medicine

EBM+: increasing the systematic use of mechanistic evidence

Evaluating Evidence of Mechanisms in Medicine: Principles and Procedures

The evidence that evidence-based medicine omits

Interpreting causality in the health sciences

Formalizing the role of agent-based modeling in causal inference and epidemiology

Invited commentary: agent-based models for causal inference-reweighting data and theory in epidemiology

Mechanisms and the evidence hierarchy

Modified SEIR and AI prediction of the epidemics trend of COVID-19 in China under public health interventions

Potential impact of seasonal forcing on a SARS-CoV-2 pandemic

A SIDARTHE Model of COVID-19 Epidemic in Italy

The SARS-CoV-2 Epidemic Outbreak: a Review of Plausible Scenarios of Containment and Mitigation for Mexico. medRxiv

Social distancing strategies for curbing the COVID-19 epidemic

Epidemic Analysis of COVID-19 in China by Dynamical Modeling

Simulating and forecasting the cumulative confirmed cases of SARS-CoV-2 in China by Boltzmann function-based regression analyses

Evaluation of the lockdowns for the SARS-CoV-2 epidemic in Italy and Spain after one month follow up

Modelling Transmission and Control of the COVID-19 Pandemic in Australia

Special report: the simulations driving the world's response to COVID-19

Investigating spatiotemporal dynamics and synchrony of influenza epidemics in Australia: an agent-based modelling approach

Role of social networks in shaping disease transmission during a community outbreak of 2009 H1N1 pandemic influenza

Early phylogenetic estimate of the effective reproduction number of SARS-CoV-2

Secondary attack rate and superspreading events for SARS-CoV-2

What is a mechanism? Thinking about mechanisms across the sciences

The New Mechanical Philosophy

The Routledge Handbook of Mechanisms and Mechanical Philosophy

The integration of social, behavioral, and biological mechanisms in models of pathogenesis

Idealization and abstraction in scientific modeling

Identifying and interrupting Superspreading events-implications for control of severe acute respiratory syndrome coronavirus 2

Superspreading and the effect of individual variation on disease emergence

the role of super-spreaders in infectious disease

Seroprevalence and ethnic differences in helicobacter pylori infection among adults in the United States

Single nucleotide polymorphisms in innate immunity genes: abundant variation and potential role in complex human disease

Ethnic-specific genetic associations with pulmonary tuberculosis

Increase in Clostridium difficilerelated mortality rates, United States

FcγRIIB in autoimmunity and infection: evolutionary and therapeutic implications

SARS-CoV-2 viral load in upper respiratory specimens of infected patients

Mechanisms: what are they evidence for in evidencebased medicine?

Exploratory modeling for policy analysis

Sensitivity analysis of infectious disease models: methods, advances and their application

Robustness analysis

Thinking about mechanisms

How models are used to represent reality

An agent-based conception of models and scientific representation

Modeling mechanisms

The Philosophy of Causality in Economics: Causal Inferences and Policy Proposals

Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand

Australia restricts travelers from mainland China as virus impact spreads

Australia blocks arrival of all non-citizens, non-residents in expanded coronavirus travel ban

Update on coronavirus measures

PM Announces Pubs, Clubs and Cinemas to Close, Schools Stay Open in Stage One Measures-as it Happened. The Guardian

Why Australia Is Not Shutting Schools to Help Control the Spread of Coronavirus. The Guardian

Victoria State Government. Coronavirus update for Victoria

Analysing the combined health, social and economic impacts of the corovanvirus pandemic using agent-based social simulation

Social distancing and supply disruptions in a pandemic

Why a one-size-fits-all approach to COVID-19 could have lethal consequences

Medical scientists and philosophers worldwide appeal to EBM to expand the notion of 'evidence'

Agent-based modelling for SARS-CoV-2 epidemic prediction and intervention assessment: A methodological appraisal

None of the authors reports conflict of interests.

https://orcid.org/0000-0003-1979-0746Martin Zach https://orcid.org/0000-0001-7181-0391