key: cord-0773023-ggalmt3b
authors: Page, Andrew; Diallo, Saikou Y.; Wildman, Wesley J.; Hodulik, George; Weisel, Eric W.; Gondal, Neha; Voas, David
title: Computational Simulation Is a Vital Resource for Navigating the COVID-19 Pandemic
date: 2021-03-13
journal: Simul Healthc
DOI: 10.1097/sih.0000000000000572
sha: 76395b5a03b4b2212b8ef218ff0d92fbddd440ad
doc_id: 773023
cord_uid: ggalmt3b

INTRODUCTION: COVID-19 has prompted the extensive use of computational models to understand the trajectory of the pandemic. This article surveys the kinds of dynamic simulation models that have been used as decision support tools and to forecast the potential impacts of nonpharmaceutical interventions (NPIs). We developed the Values in Viral Dispersion model, which emphasizes the role of human factors and social networks in viral spread and presents scenarios to guide policy responses. METHODS: An agent-based model of COVID-19 was developed with individual agents able to move between 3 states (susceptible, infectious, or recovered), with each agent placed in 1 of 7 social network types and assigned a propensity to comply with NPIs (quarantine, contact tracing, and physical distancing). A series of policy questions were tested to illustrate the impact of social networks and NPI compliance on viral spread among (1) populations, (2) specific at-risk subgroups, and (3) individual trajectories. RESULTS: Simulation outcomes showed large impacts of physical distancing policies on number of infections, with substantial modification by type of social network and level of compliance. In addition, outcomes on metrics that sought to maximize those never infected (or recovered) and minimize infections and deaths showed significantly different epidemic trajectories by social network type and among higher or lower at-risk age cohorts. CONCLUSIONS: Although dynamic simulation models have important limitations, which are discussed, these decision support tools should be a key resource for navigating the ongoing impacts of the COVID-19 pandemic and can help local and national decision makers determine where, when, and how to invest resources.

TheCOVID-19pandemi c has prompted the extensive use of computational models to understand the trajectory of the pandemic, as well as increased public awareness of modeling. Early on in the pandemic, models from the United Kingdom, 1 China, 2,3 and elsewhere 4-6 provided policy professionals with information about the likely scale of the epidemic, the importance of case detection, and the mitigation strategies capable of avoiding overrun of health services. More complex computational models incorporating biological, behavior, social networks, and spatial characteristics of areas have also been developed as the pandemic has evolved. [7] [8] [9] [10] [11] [12] Other models have focused on case detection, contact tracing, and optimal testing regimes [13] [14] [15] [16] ; impacts of on emergency departments and the provision of health care, as well as supply chains for personal protective equipment [17] [18] [19] [20] [21] ; likely impacts of vaccination coverage on subsequent infections 22 and reinfections 23 ; strategies for educational settings [24] [25] [26] ; and impacts on environment and economic indicators. 27, 28 In response to model forecasts, policy makers in many regions implemented a range of interventions, including widespread testing and contact tracing, and nonpharmaceutical interventions (NPIs), such as quarantine, contact tracing, and physical distancing. The combined impact of these interventions has been the reduction of COVID-19 cases to a fraction of what they would have been. It is also likely that these interventions would not have been implemented as confidently had there not been insights available from computational models to inform policy professionals.

These computational models are generally described as dynamic simulation models (DSMs) and offer a way to study real-world complex dynamical systems using artificial complex dynamical systems. Simple DSMs have a long history in infectious disease epidemiology 29 in the form of aggregate differential equation models. 30 More complex DSMs, such as agent-based models (ABMs), have also been used to map epidemic spread. 31 Policy professionals are now beginning to turn to postpandemic recovery. Although public health experts routinely use simple DSMs to understand the epidemiology of disease outbreaks, the usefulness of DSMs for considering policy and service responses to complex chronic disease outcomes is not widely understood. 32 This is despite DSMs having a long history in other disciplines, such as engineering, ecology, physical sciences, and business, where they are considered a key tool in developing and refining hypotheses, priority setting, and guiding decision making. 32 Policy makers can use DSMs to answer the kinds of complex questions that are emerging in the recovery phase of the COVID-19 pandemic. Dynamic simulation models can power decision support tools to help policy professionals determine optimal policy combinations, manage compliance challenges within diverse populations, control secondary outbreaks, and handle downstream psychosocial and health impacts in a postpandemic society.

Dynamic simulation models are computer representations of the real world. Dynamic simulation models include system dynamics models (SDMs), ABMs, and discrete event models. 33 The first two of these are particularly important for the COVID-19 epidemic.

System dynamics models are top-down aggregate models that divide a population (or a system) into "compartments" or "stocks" that represent states of the system being studied. Mathematical representations are then specified to describe how members of one stock change state and "flow" to another stock over time. Investigating how stocks change over time is equivalent to studying how the system dynamically changes its state. This can be visualized using a "stock and flow" diagram, sometimes called a compartment model diagram in epidemiology.

In the context of the COVID-19 pandemic, so-called SEIR models have figured prominently, where a population is divided into 4 states: those susceptible (S), those exposed to the virus (E), those infected (I), and those who die or recover (R; Fig. 1A ). The rate at which individuals in the population move from one stock to another is determined by a set of differential equations, with the manipulation of parameters (changing flow rates between stocks) allowing the investigation of different assumptions and scenarios, and the impacts of selected public health interventions. Traditional SEIR models oversimplify viral spread in a population, overlooking important factors such as symptomatic versus asymptomatic spread, health service capacity, policy responses to suppress infection, and human factors impacting policy compliance. However, SDMs (and ABMs, as described hereinafter) can take account of these important factors by incorporating additional stocks and flows to capture distinct pathways having policy relevance. For example, an enhanced SEIR model might also incorporate stocks relating to asymptomatic spread and new variables to express the effect of human factors on flow rates (Fig. 1B) .

Agent-based models are computer representations of the real world based on the behaviors and interactions of autonomous "agents." In a population health context, these agents are individuals in a population, although in other disciplinary contexts agents could be cars, components of a machine, animals, or corporate entities. 33 Agents change state with time, based on defined mathematical rules, which may be stochastic or deterministic, and the behavior of agents and populations can be observed over time as agents interact in a simulated context.

For example, an individual might be in 1 of 4 states: susceptible, exposed, infected, and recovered ( Fig. 2 ). (Note that this ABM has additional states relating to policy interventions, which are discussed in more detail below). Agent-based models offer a bottom-up approach to modeling a complex dynamical system, with system-level behavior emerging from agent interactions. Agent interactions can be aggregated to represent patterns of population change, permitting the exploration of population-level patterns. However, because ABMs are built with heterogeneous agents, individual life trajectories can also be explored, yielding tangible examples of simulated lived experiences. Similarly, individuals who are outliers on particular measures-for example, in a social network, or on a distribution representing the propensity to comply with physical distancing guidelines-can be identified and studied as "exceptions to the rule," yielding insights about policy effectiveness and guiding sensitivity analyses of parameters.

Unlike SDMs, ABMs can capture the heterogeneity and complexity of individual behavior and social interaction. For example, an individual may be in either a susceptible, exposed, infectious, or recovered state, but this may be affected by individual medical vulnerabilities or by individual values, behaviors, and social contexts. Like people, agents may be more or less introverted in personality, more or less likely to need to travel to work, more or less likely to trust health authority advice, and more or less connected in social networks. Such human factors are vital considerations in assessing the likelihood of an individual being in a given state, are central to understanding viral spread, and are naturally incorporated into ABMs. Capturing specific human factors in models also allows decision makers to investigate the potential impacts of policies targeting specific behaviors on subsequent trends in disease outcomes. Because ABMs allow for the exploration of complex interactions among a range of individual-level factors, there are corresponding challenges in designing and validating such models that are more pronounced than for SDMs.

Traditional approaches for developing and evaluating policies for public health interventions have important limitations when applied to the complex psychosocial problems associated with the COVID-19 pandemic. Current methods assume that relationships between exposures and outcomes are independent, unidirectional, linear, and constant through time. However, complex problems are characterized by interaction and feedback, and complex human problems also involve changing behavior over time, which violate these assumptions. 34, 35 Traditional program logic models and statistical methods also overlook or understate population dynamics, behavioral dynamics, service or workforce dynamics, variation in how intervention impacts change over time, and the nonadditive effects of combining interventions. 36 These limitations can lead to unrealistic expectations about the potential impact of evidence-based interventions in real-world settings.

In comparison with traditional techniques used in health research, DSMs allow for the conceptualization and computational description of complex nonlinear dynamical systems, including interaction effects between independent variables, such as constructive feedback cycles and dampening mechanisms. Dynamic simulation models can capture real-world complexity and are unhindered by the simplifying assumptions of the traditional restricted potential outcomes approach approximated in randomized controlled trials. Dynamic simulation models, developed in collaboration with policy stakeholders, use a variety of sources of evidence, such as research studies, expert and local knowledge, lived experience, and routinely collected data to map and quantify complex problems. 37 The resulting dynamic models can then be used as interactive "what if" tools to test the likely impacts of different scenarios and combinations of interventions over the short and long term, before they are implemented in the real world. 36, 37 This "testing before investing" approach is normative in engineering contexts, and technology has advanced to the point that it can also be used in health policy contexts.

Using DSMs in decision support tools does not preclude the use of traditional program evaluation approaches. Rather, DSMs augment traditional approaches. Dynamic simulation models are particularly useful when they are continually validated using systematic data monitoring to determine the extent to which model outputs correspond with real-world outcomes and how intervention strategies are tracking against forecast outcome targets. Information from monitoring and evaluation can be used to refine model parameters and assumptions to improve predictions and guide decision making in a responsive and iterative way. It is also important that stakeholders who use models for decision making are able to trust the outputs from DSMs and that the assumptions underlying the model structure are robust, transparent, and publicly accessible. 38 This involves building models via a participatory process, with key stakeholders guiding the design and validation of a given model structure, drawing on best evidence and current data, provided by subject matter experts from a range of disciplines. 37, 39 

Incorporating human factors into epidemiological models has not been common in health policy analysis. However, consider how prosocial, political, and personal values affect trust in experts, fear of adverse effects, vulnerability to misinformation, and willingness to comply with public health recommendations. Human values directly influence compliance with public health guidelines, as well as the measures that determine what counts as policy success in grappling with public health challenges (Fig. 1) . With the right tools, the influence of human factors can be incorporated into modeling efforts. Previous models developed by coauthors have demonstrated the importance of human values in modeling of artificial societies, including factors relating to social equity, empathy, prosociality, religion, race, and sexuality. 40 The Values in Viral Dispersion (VIVID) model is an ABM developed by the authors that aims to provide practically useful computational simulations of human factors in managing the COVID-19 pandemic. (The model is available here and see additional model outputs, Supplemental Digital Content 1, http://links.lww.com/SIH/A658). The VIVID model investigates the impact of human factors-including NPIs, compliance-promoting trust in experts, and social networks-on the trajectory of COVID-19 in an artificial population (here, we set the population to 1000 individuals for the purposes of illustration). The VIVID model can also be calibrated to represent any population of policy interest: a city or state, schools or university campuses, business parks, or shopping malls.

The VIVID agents move between 3 states: susceptible, infectious, or recovered (Fig. 2) . Individual agents are assigned a propensity to comply with NPIs and are placed in a social network that governs interpersonal contact. A summary of model parameters is provided in the Table. The VIVID model is able to model the dynamics of disease diffusion through a range of standard network types, including random interaction, interaction based on distance, ring-lattice networks, small-world networks, 41 scale-free networks, 42 networks based on coparticipation in scheduled events (eg, classes in schools), and social networks based on a variety of types of ties (eg, being related, living in the same apartment, shopping at the same supermarket, or attending the same sports event).

Like other ABMs, 7-9,11,12 the VIVID model evaluates 3 NPIs, separately and in combination: quarantine, contact tracing, and physical distancing. Quarantine refers to lockdown, shelter-inplace, self-isolation, and stay-at-home policies, and the VIVID model distinguishes between quarantine for those who are nonsymptomatic and quarantine for those who have COVID-like symptoms. This intervention prevents individuals from interacting, thereby reducing likely exposure and subsequent infection, subject to individual compliance. Contact tracing refers to identifying every individual who has come in contact with a confirmed case of COVID-19 in the previous 14 days and prevents these individuals from interacting in their social network (again, subject to compliance). Finally, physical distancing refers to individuals remaining at least 2 meters from others, avoiding physical greetings and crowded spaces, and practicing good hygiene. The intensity of each of these NPIs can be modified to compare and contrast different intervention scenarios, for example, the proportion of the population complying with physical distancing, an individual agent's propensity to physically distance (expressed as a probability), and the duration (in days) of contact tracing scale up.

The VIVID model permits the user to select outcomes of interest, which will likely differ depending on the policy context. For example, some policy analysts working in health service settings may be focused on the number of infections and deaths to anticipate required capacity. Others may be interested in the impact on disease transmission of changing the intensity of NPI advice. However, others may be most concerned about levels of compliance with NPIs. Here, we illustrate the VIVID ABM by a series of policy questions related to different levels of analysis.

Policy question #1: Which NPIs should be used, and with what intensity, to optimize 2 outcome metrics: (1) to minimize the number infected (valuable when thinking of entire populations) and (2) to maximize a compound outcome metric measuring the distance between positive and negative outcomes, that is (total never infected + total recovered) -(total deaths + total reinfected)? The first metric (infected metric) can go above the population count, which is 1000, because of reinfections, but cannot go below zero. The second metric (compound metric) has a maximum of 1000 but can go below zero if the number of deaths and reinfections exceeds the count of those who were never infected or recovered after infection. If the compound metric is 1000, this indicates that NPIs were effective in preventing deaths and reinfections and a perfect outcome for this populationall agents have either recovered from an infection or have never been infected. If the compound metric is less than 1000, this indicates the number of deaths and reinfections relative to those never infected or recovered. A value of 0 would suggest an equal number of agents in the population who have never been infected or recovered and agents who have died or been reinfected. A value of −1000 would indicate the worst outcome for the population-where all agents have either experienced reinfection or have died.

A first step in answering policy question #1 is to determine the parameters (independent variables) most strongly associated with each outcome metric (dependent variables; see additional model outputs, Supplemental Digital Content 1, http://links.lww.com/SIH/A658). Outputs from the model suggest that the parameters accounting for the most variance in the infected metric (number of infections) are a propensity to engage in physical distancing, individual contact rate, and level of infectivity. In contrast, the parameters accounting for the greatest variance in the compound metric (the distance between positive and negative outcomes) were the age above, which one is more likely to die from an infection. This suggests that limiting exposure of older people by limiting interactions will be associated with the highest number of those never infected (and recovered) and the lowest number of reinfections and deaths and that physical distancing and contact rates among younger age cohorts are relatively less important. Change the metric, which reflects human values and defines what counts as strategic success, and the public health recommendations also change.

A second step in answering policy question #1 is to understand how parameters, such as contact tracing, social distancing, and population demographics, affect outcomes over time (Fig. 3) . Figure 3A uses the infected metric to show a typical infection curve over several model runs, varying network types, social distancing compliance, and whether contact tracing is in place. As one would expect, higher levels of social distancing flatten the curve, preventing critical care overrun, and contact tracing flattens it further. Note that network types play a clear role in the infection curve. A network structure in which agents are connected to spatially nearby agents ("BasedOnDistance" in Fig. 3 ) tends to have the highest curve, and a random network reaches a similar height. The distance network can be interpreted as one where people are connected to those who are similar to them (and hence closely situated in the social and/or geographic space). A "small-world" network structure in which most of an agent's links are locally proximate and interconnected with some long-range links that considerably shorten the path distance interconnecting distant social spaces has a much flatter curve than the other network structures. Public health experts can use network analytics to identify the structure of physical contact networks in groups or regions for which they have special responsibility and can use the VIVID model for planning purposes.

In Figure 3B , we explore the compound metric (the difference between positive and negative outcomes). Because our preliminary analysis found that age demographics accounted for most of the variance for this compound metric, Figure 3B varies age demographics instead of physical distancing compliance. In Figure 3B , while the trends associated with contact tracing and network structures from 3A still hold, we see an extreme dichotomy for different age cohorts (those aged 50-90 years and those aged 20-26 years). The policy relevance of this finding bears on whether NPIs should be advocated equally for all age groups or whether more stringent social distancing and contact tracing policies should be used for specific demographic groups (eg, older people and those who interact with them) based on risk, allowing less stringent physical distancing among demographic groups at lower risk (eg, those of younger age). Like any ABM, the VIVID model can consider the impacts of a wide variety of conditions on outcome measures (see additional model outputs, Supplemental Digital Content 1, http://links.lww.com/SIH/A658, for additional examples for the number never infected and the timing of contact rates on number of infections).

In addition, some pressing public health policy questions call for more granular insights, particularly when it is suspected The minimum likelihood that an agent will report symptoms to a contact tracing body [possible values: (0,1)] Contact tracing reporting compliance max

The maximum likelihood that an agent will report symptoms to a contact tracing body [possible values: (contract tracing reporting compliance min, 1)] Contact tracing quarantining compliance min

The minimum likelihood that an agent who was contact notified (notified of having contacted an infectious agent recently) will quarantine based on this information [possible values: (0,1)] Contact tracing quarantining compliance max

The maximum likelihood that an agent who was contact notified (notified of having contacted an infectious agent recently) will quarantine based on this information [possible values: (contract tracing quarantining compliance min, 14) ] Symptomatic quarantine compliance min

The minimum likelihood that an agent will choose to quarantine when noticing symptoms [possible values: (0,1)] Symptomatic quarantine compliance max

The maximum likelihood that an agent will choose to quarantine when noticing symptoms [possible values: (symptomatic quarantine compliance min, 14)] Testing compliance min

The minimum likelihood that an agent will agree to be tested when selected for random testing [possible values: (0,1)] Testing compliance max

The maximum likelihood that an agent will agree to be tested when selected for random testing 

The VIVID model can also explore the trajectories of individual agents in the model, allowing focused analyses to answer questions such as: policy question #2: How do policy level recommendations relate to experiences of individuals? Figure 4 depicts individual outcomes, with each individual either "alive" or "dead" (shape) and having an individual propensity to comply with NPIs (color). Runs are clustered into 10 facets based on infection severity ("less" or "very" deadly for horizontal facets) and social network type (for vertical facets). Within each of the 10 facets, individuals are located according to infectivity (vertical axis) and the percentage of low-compliant agents in the population (horizontal axis). Each marker represents an individual outcome and corresponds to a set of experiences that can be uncovered and narrated.

Consider the individual life narrative for one particular outlier (indicated by the red arrow in Fig. 4 ). This individual is in the facet representing a very deadly infection spreading in a random contact network. Relative to the other agents in that facet, this agent has a high propensity to comply with NPIs (the dark blue color), was alive at the end of the simulation (the circular marker shape), survived a highly infectious variant of the virus (the vertical axis), and managed all this in a context where almost 40% of the population was low in NPI compliance (the horizontal axis).

By drilling down into the simulated data, we find that this agent was susceptible on day 3, became asymptomatically infectious on day 11, was diagnosed and quarantined on day 26, and eventually fully recovered. Being able to explore individual trajectories provides policy analysts with specific examples of putative "lived experience" and accompanying insights into how to conceptualize individual-level impacts of particular policy combinations.

More generally, the clustering of deaths (relative to agents remaining alive) can also be informative. Figure 4 shows a higher clustering of deaths for the "very deadly" compared with the "less deadly" scenario, even among agents with higher levels of compliance with NPIs in environments also with higher overall levels of compliance (indicated on the x-axis). In contrast, for a "less deadly" virus, lower agent compliance in environments with high levels of compliance shows clusters of agents who remained alive at the end of the simulation, but clustering of deaths among agents with lower compliance in environments with lower compliance. This implies that similar types of individual behavior (eg, a propensity to comply with public health advice) can have a different outcome depending on the overall level of responsiveness to public health advice in the population (and social network) in which the individual resides.

Finally, it is possible to use ABMs to run virtual experiments to address policy question #3: What is the best combination of NPIs given a specific metric that matters to most people affected by policies? Experimenting in this way is usually impossible in real-world situations, and when it is possible, it is typically highly unethical. An artificial society makes virtual experimentation both possible and safe, although ethics challenges about the design and use of models remain.

The VIVID model can also be used to compare specific scenarios, identifying the conditions under which these scenarios would result in optimal outcomes under, say, the compound metric (see additional model outputs, Supplemental Digital Content 1, http://links.lww.com/SIH/A658). For example, are there higher scores on the compound metric when there is (1) stringent shelter-in-place or quarantine for those in higher-risk age cohorts and low social distancing among all other cohorts or (2) high social distancing for the whole population? This comparison is an instance of a policy choice that is common in public health analyses: do we identify high-risk subgroups and intervene to lower the risk for associated individuals, or do we lower the average risk for the entire population? Both essentially shift the risk distribution between subgroups but in different ways expressing different value-laden priorities. The VIVID ABM offers measures and analyses at the levels of population, subgroups, and individuals that generate insights to support complex decision processes of this kind. As noted previously, the VIVID model also allows flexibility in terms of reparameterizing the model for specific populations of subgroups and can be modified to accompany developments in the understanding of COVID-19 pathogenesis given a rapidly evolving understanding of this virus.

In coming months, and indeed in the longer term, officials with responsibility for countless organizations will need to answer strategic policy questions like those discussed previously, in preparation for making difficult decisions relating to the ongoing impacts of the COVID-19 pandemic. Impacts will likely be evident across a range of chronic disease outcomes, social and health service contexts, and education and economic settings. Understanding the psychosocial impacts of changes to human behavior and social interaction will also be important in framing policy responses, and all of this must be navigated within the context of limited resources and a global economic recession.

Well-designed simulations of population health outcomes, calibrated to a particular context and then used as a platform for virtual experimentation, will be central to guide decision making. When human lives are at stake, this is a more ethical way to conduct policy evaluation than trial-and-error approaches. These tools can help local and national decision makers determine where, when, and how to allocate investments and with what intensity. Computational modeling and simulation should be seen by population health and policy professionals as a key resource for making decisions during the COVID-19 pandemic and future public health crises.

Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. London: Imperial College COVID-19 Response Team

Early dynamics of transmission and control of COVID-19: a mathematical modelling study

A conceptual model for the Coronavirus disease 2019 (COVID-19) outbreak in Wuhan, China with individual reaction and governmental action

Modeling the impact of social distancing, testing, contact tracing and household quarantine on second-wave scenarios of the COVID-19 epidemic

Modelling the effects of public health interventions on COVID-19 transmission using R -part 2

Estimating the case detection rate and temporal variation in transmission of COVID-19 in Australia

Maximizing the probability that the 6-week lock-down in Victoria delivers a COVID-19 free Australia

Spatio-temporal simulation of the novel coronavirus (COVID-19) outbreak using the agent-based modeling approach (case study: Urmia, Iran)

OpenABM-Covid19-an agent-based model for non-pharmaceutical interventions against COVID-19 including contact tracing

IHME COVID-19 Forecasting Team. Modeling COVID-19 scenarios for the United States

COVID-ABS: an agent-based model of COVID-19 epidemic to simulate health and economic effects of social distancing interventions

The estimated likelihood of eliminating the SARS-CoV-2 pandemic in Australia and New Zealand under current public health policy settings: an agent-based-SEIR modelling approach

Modelling the impact of testing, contact tracing and household quarantine on second waves of COVID-19

Exploring the effectiveness of a COVID-19 contact tracing app using an agent-based model

Evaluating a SARS-CoV-2 screening strategy based on serological tests

Temporal dynamics of viral load and false negative rate influence the levels of testing necessary to combat COVID19 spread

An agent-based model to evaluate the COVID-19 transmission risks in facilities

Royal flying doctor service coronavirus disease 2019 activity and surge modeling in Australia

COVID-19 healthcare demand projections: Arizona

The use of system dynamics methodology in building a COVID-19 confirmed case model

Computational simulation to assess patient safety of uncompensated COVID-19 two-patient ventilator sharing using the pulse physiology engine

Vaccine efficacy needed for a COVID-19 coronavirus vaccine to prevent or stop an epidemic as the sole intervention

Estimation of the probability of reinfection with COVID-19 by the susceptible-exposed-infectious-removed-undetectablesusceptible model

Simulating preventative testing of SARS-CoV-2 in schools: policy implications

Simulating COVID-19 in a university environment

The Artificial University: decision support for universities in the COVID-19 era

The propagation of economic impacts through supply chains: the case of a mega-city lockdown to prevent the spread of COVID-19

The impact of the pandemic lockdown on air pollution, health and economic growth: system dynamics analysis

A contribution to the mathematical theory of epidemics

Coupled contagion dynamics of fear and disease: mathematical and computational explorations

Strategies for mitigating an influenza pandemic

System dynamics modeling for public health: background and opportunities

From System Dynamics and Discrete Event to Practical Agent Based Modeling: Reasons, Techniques, Tools. Citeseer: Proceedings of the 22nd International Conference of the System Dynamics Society

Applying dynamic simulation modeling methods in health care delivery research-the SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force

Learning from evidence in a complex world

Static metrics of impact for a dynamic problem: the need for smarter tools to guide suicide prevention planning and investment

Turning conceptual systems maps into dynamic simulation models: an Australian case study for diabetes in pregnancy

Call for transparency of COVID-19 models

Decision makers' experience of participatory dynamic simulation modelling: methods for public health policy

US: CMAC; 2021

Networks, dynamics, and the small-world phenomenon

Emergence of scaling in random networks

Introduction to Decision Analysis: A Practitioner's Guide to Improving Decision Quality