key: cord-0452902-eyjuva7g
authors: Huang, Yunhan; Zhu, Quanyan
title: Game-Theoretic Frameworks for Epidemic Spreading and Human Decision Making: A Review
date: 2021-06-01
journal: nan
DOI: nan
sha: 7502cee64b1ce5c7542f805150a823d07762705f
doc_id: 452902
cord_uid: eyjuva7g

This review presents and reviews various solved and open problems in developing, analyzing, and mitigating epidemic spreading processes under human decision-making. We provide a review of a range of epidemic models and explain the pros and cons of different epidemic models. We exhibit the art of coupling epidemic models and decision models in the existing literature. {More specifically, we provide answers to fundamental questions in human decision-making amid epidemics, including what interventions to take to combat the disease, who are decision-makers, when to take interventions, and how to make interventions.} Among many decision models, game-theoretic models have become increasingly crucial in modeling human responses/behavior amid epidemics in the last decade. In this review, we motivate the game-theoretic approach to human decision-making amid epidemics. This review provides an overview of the existing literature by developing a multi-dimensional taxonomy, which categorizes existing literature based on multiple dimensions, including 1) types of games, such as differential games, stochastic games, evolutionary games, and static games; 2) types of interventions, such as social distancing, vaccination, quarantine, taking antidotes, etc.; 3) the types of decision-makers, such as individuals, adversaries, and central authorities at different hierarchical levels. A fine-grained dynamic game framework is proposed to capture the essence of game-theoretic decision-making amid epidemics. We showcase three representative {frameworks} with unique ways of integrating game-theoretic decision-making into the epidemic models from a vast body of literature. {Each of the three framework has a unique way of modeling, conducting analytical analysis, and deriving results.} In the end, we identify several main open problems and research gaps left to be addressed and filled.

The advancement of Information and Communication Technologies (ICTs) and transportation technologies has been connecting entities, including people and devices, in various ways and improving the quality of lives globally. While benefiting from increasing connectivity, we have also experienced several toxic "side effects" that are bought by hidden spreading processes over the underlying networks. Such processes include the spread of contagious diseases over human contact networks and animal populations, the diffusion of viruses or worms over communication and computer networks, and the propagation of rumors/fake news over social media. The recent COVID-19 pandemic, which has been taking a devastating toll on the physical and economic well-being of people across the world, needs more earnest heed to these processes. Certainly, a fundamental understanding of the evolution and control of these processes will contribute to alleviating the threats to the safety, well-being, and security of people and other interconnected systems around the world.

The underlying networks on which these processes spread are usually largescale complex networks composed of intelligent and strategic individuals with different beliefs, perceptions, and objectives. The scale and complexity of the underlying networks, the unpredictability of individuals' behavior, and the unavailability of accurate and timely data pose challenges to the fundamental understanding of the evolution and control of these processes.

The inclusion of human decision-makers in the epidemic spreading process creates challenges. It becomes a significant hurdle for researchers to understand human-in-the-loop spreading processes at a fundamental level. The key to clear this hurdle is to integrate decision models into epidemic models. Mathematical models of epidemic spreading started 200 years ago by Daniel Bernoulli [17] , at the dawn of the industrial revolution. Many papers have been dedicated to the modeling of epidemic spreading over the last 200 years. But not until very recent years has there been studies investigating human decision-making amid epidemics and the effect of human responses on the spreading processes. In this review, we review and present various solved and open problems in developing, analyzing, and mitigating the epidemic spreading process with human decisionmaking. We provide a tutorial on epidemic models and the pros and cons of different epidemic models. We explain in detail how decision models are introduced and integrated into epidemic models in the existing literature. Among various decision models, game-theoretic models have become prominent in modeling human responses/behavior amid epidemics in the last decade. The popularity of game-theoretic models for human-in-the-loop epidemics is primarily due to the following reasons. One rationale is the large-scale nature of the human population. Centralized decision-making becomes intractable in a large-scale network. Game-theoretic models provide a bottom-up decentralized modeling framework that naturally makes the computation and design scalable. The second one is that individuals living amid an epidemic may not be willing to comply with the suggested protocols, and individuals are mostly selfinterested. The third one is that game theory as a mature and extensive field offers a set of relevant concepts and analytical techniques that can be leveraged to study human behavior amid epidemics. In this review, we demonstrate that game-theoretic frameworks are powerful in modeling the spreading processes of human-in-the-loop epidemics. We provide a multi-dimensional taxonomy of the existing literature that has proposed, studied, and analyzed game-theoretic models for human-in-the-loop epidemics. Among existing literature, we showcase three representative frameworks with unique ways of integrating gametheoretic decision-making into the epidemic models. The uniqueness of each of these three frameworks distinguishes them from each other by means of their models, analytical methods, and results.

Despite a recent surge in the literature about game-theoretic models for studying the human-in-the-loop epidemic spreading, a number of open problems and research gaps are left to be addressed and filled. Hence, we use one section to discuss emerging topics. As we continue to witness the devastating toll of the pandemic on human society, this review aims to introduce to more researchers, especially game and dynamic game theorists the subject of game-theoretic modeling of human-in-the-loop epidemic spreading processes. Their contributions to understanding and mitigating the human-in-the-loop epidemic spreading would make a significant societal impact.

Graph Theory: A directed graph (network) is a pair G = (N , E), where N is the set of nodes representing individuals involved and E ⊂ N × N is the set of edges representing connections between two individuals. The size of the network is = |N |. Given G, an edge from node ∈ N to node ∈ N is denoted by ( , ). When ( , ) ∈ E implies ( , ) ∈ E and vice versa, the graph G is undirected. For an undirected network, we denote N = { |( , ) ∈ E} the set of neighbors of node . For a directed network, we denote the out-neighbor set of individual , which is defined as N = { |( , ) ∈ E}. Let ∈ R × be the adjacency matrix for an unweighted graph G with elements = 1 if and only if (1, ) ∈ E. Otherwise, = 0. For an unweighted network, we use as the adjacency matrix with elements ∈ R + . The number of neighbors a node has is called the degree of a node. Given a graph G, ( ) is the proportion of nodes who have degree . The average degree of a graph is denoted by ( ). We define 2 2 ( ). If G is a scale-free graph, its degree distribution follows ( ) ∼ − with ranges from 2 to 3. If G is a regular graph, then each node in G has the same degree.

Notation: For a square matrix , ( ) is the spectral radius of matrix . Given a vector ( 1 , 2 , · · · , ), diag( 1 , 2 , · · · , ) is an × matrix whose elements on the -th row and the -th column is . The identity matrix is denoted by . We use E[·] and P(·) to denote the expected value and the probability of the argument. Given x = ( 1 , 2 , · · · , ), x − denotes the vector x with element removed. Given a graph G and a vector associated with its nodes x = ( 1 , 2 , · · · , ), x N denotes the vector that includes all the elements associated with node and its neighbors. Let Δ be the probability simplex of dimension , i.e., Δ = { ∈ R | 1 + 2 + · · · + = 1, ≥ 0, ∀ = 1, 2, · · · , }. 

To understand the evolution and control of epidemic spreading at a fundamental level, we need to understand both the epidemic model and the decision model, and the art of coupling these two models.

The spreading of infectious diseases has affected human civilization since the age of nomadic hunter-gatherers. Mathematical models of epidemic spreading started to be proposed and studied since the beginning of the industrial revolution, with one of the earliest attempts to model infectious disease transmission mathematically by Daniel Bernoulli [17] . Over the last 200 years, many mathematical models of epidemics have been proposed and analyzed [11, 29, 101, 105, 134] .

Compartment Models: The basic models of many mathematical modeling of epidemics are the well-known compartment models [71] . In compartment models, every subject, based on the status, belongs to some compartment of the population at any given time. Common compartments include the susceptible (S), the exposed (E), the infected (I), the asymptomatic (A), and the recovered who are immune to the disease (R). Depending on the mechanisms of infectious diseases, different compartment epidemic models such as SIS (Susceptible-Infected-Recovered), SIR (Susceptible-Infected-Recovered), and SEIR (Susceptible-Exposed-Infected-Recovered) are studied and analyzed. Some infections, such as the common cold and influenza, do not confer any long-lasting immunity. Such infections can be modeled by the SIS model since individuals can become susceptible again after their recovery. If individuals recover with permanent immunity, the model is an SIR model. The SIR model has been used to study infectious diseases such as measles, mumps, and smallpox [71] . Many variants of compartment models have been proposed and studied in the past decades. For example, Rothe et al. [118] have looked into an SAIRS (susceptible-asymptomatic-infected-recovered-susceptible) to model the COVID-19 pandemic; Erdem et al. [32] have investigated an SIQR (susceptible-asymptomatic-infected-quarantine-recovered) to study the effect of imperfect quarantine on the spreading of an influenza epidemic; Huang et al. [62] have proposed a model that connects the SIS and the SIR models, in which once recovered from an infection, individuals become less susceptible to the disease.

To capture the dynamics of the epidemic-spreading process, we need a dynamic model to describe the evolution of the population in each compartment. In general, epidemic models are categorized into two groups: deterministic epidemic models and stochastic epidemic models, depending on the mathematical formulation. Deterministic models, oftentimes represented by a collection of ordinary differential equations (ODEs), have perhaps received more attention in the literature [71, 101, 105, 134] . Their popularity is because deterministic models can become more complex yet still feasible to analyze, at least when numerical results are sufficient. In contrast, stochastic models, usually repre-sented by Markov processes, need to be fairly simple to be mathematically manageable [11, 29] .

There are, however, several advantages of stochastic epidemic models over deterministic epidemic models when the analysis is tractable. First, the epidemic spreading processes are stochastic by nature. For example, the disease transmission between individuals is more spontaneously described by probabilities than deterministic rules that govern the transmission. Second, stochastic modeling has its deterministic counterpart through mean-field analysis. The stochastic modeling provides a microscopic description of the epidemic process, while the deterministic counterpart is useful to describe the spreading of epidemics at a macroscopic level, e.g., the fraction of the infected population at a given time. When the number of individual is small, or the size of the infection is small in a large community, the mean-field approximation will experience a considerable approximation error, and hence deterministic models may fail to accurately describe the spreading process [134] . Third, deterministic models are incapable of capturing higher-order characteristics of the spreading process, such as variances, which are useful for the understanding of the uncertainties in the estimates.

Overall, deterministic models and stochastic ones are complementary to each other. Deterministic models describe the spreading process at a macroscopic level and are more manageable mathematically, yet subject to assumptions on the spreading processes. Stochastic models approach the spreading processes from a microscopic point of view and offer a detailed description of the spreading process. In the following subsections, we introduce both deterministic and stochastic models using the SIS epidemic models as examples.

Over the last century, increasingly sophisticated deterministic epidemic models have been proposed to capture the spreading processes on growingly complex and realistic networks.

In an SIS model, each individual in the system is either infected or susceptible. An infected node can infect its susceptible neighbors with an infection rate . An infected individual recovers at recovery rate . Once recovered, the individual is again prone to the disease. The simplest deterministic SIS model is introduced by Kermack and McKendrick [71] :

where ( ) is the fraction of the population who are susceptible, ( ) is the fraction of the infected. In (1) , the rate at which the fraction of infected individuals evolve is determined by the rate at which the infected population is recovered, i.e., ( ) and the rate at which the fraction of infected population grows, i.e., ( ) ( ). The latter rate captures the encounter between the fraction of susceptible individuals and the fraction of infected individuals. Simple deterministic models like (1) assume that the individuals in the population are homogeneously mixed; i.e., each individual is equally likely to encounter every other node. Such models have ignored the structure of the underlying network.

Starting from the 90s, new deterministic models have been proposed and studied to accommodate epidemic processes over more complex and realistic network structures. Kerphart and White [70] investigated a regular graph with individuals where each individual has degree . The Kerphart and White model is described by the ODE:

where the rate of infection is ( ) (1 − ( )) which is proportional to the fraction of susceptible individuals, i.e., 1 − ( ). For each susceptible individual, the rate of infection is the product of the infection rate and the number of infected neighbors ( ). The Kermack and McKendrick model (1) and the Kerphart and White model (2) are referred to as "homogeneous" models since they assume that the underlying network has homogeneous degree distributions, i.e., each node in the network has the same degree.

With the emerging occurrence of complex networks in many social, biological, and communication systems, it is of great interest to investigate the effect of their features on epidemic and disease spreading. Pastor-Satorras et al. [104, 105] studied the spreading of epidemics on scale-free (SF) networks. In SF networks, the probability that an individual has degree follows a scalefree distribution ( ) ∼ − , with ranges from 2 to 3. It has been shown that many social networks such as collaboration networks, and computer networks such as the Internet and the World Wide Web exhibit such structure properties. The Pastor-Satorras model further divides individuals into subcompartments based on their degrees, with ( ) representing the proportion of infected individuals with a given degree :

where

represents the probability that any given link points to an infected individual. The rate at which the infected population grows is proportional to the infection rate , the number of connections , and the probability Θ( ) of linking to an infected individual. The model of Pastor-Satorras et al. incorporates the degree distribution of the underlying network and approximates the spreading processes on more complex and realistic networks.

To incorporate an arbitrary network characterized by the adjacency matrix , Wang et al. [137] have proposed a discrete-time model that generalizes the Kerphart and White model (2) and the model of Pastor-Satorras et al. (3) . Mieghem et al. [134] have studied a continuous-time SIS model, called the -intertwined deterministic model that generalizes the results in [137] . The -intertwined deterministic model describes the spreading processes by

where denotes the probability of individual being infected [134] or the proportion of infected individuals in the sub-population [33] . The element in the adjacency matrix represents the connectivity between individual and individual . The -intertwined deterministic model provides a microscopic description of the spreading processes by incorporating the adjacency matrix that fully characterizes the underlying network. Other variants of the -intertwined deterministic model (4) have been recently studied in [72, 102] . To capture the heterogeneity of individuals' demographic or health situation, people consider a -intertwined model with heterogeneous parameters: [102] .

Most studies of deterministic epidemic models are interested in analyzing the system equilibria (the limit behavior of the spreading process as time reaches infinity), characterizing the threshold that determines which equilibrium the system converges to, and evaluating stability properties of different equilibria [70, 101, 104, 134, 137] . Define the effective spreading rate / . One of the primary goals in most studies of deterministic epidemic models is to characterize the threshold . If > , the epidemic persists and at least a nonzero proportion of the individuals are infected. If ≤ , the epidemic dies out. The Kerphart and White model (2) gives a "steady-state" epidemic threshold ,KW = 1/ , which is inversely proportional to the individuals' degree in a regular graph [70] . The model of Pastor-Satorras et al.

(3) provides a threshold ,PS = / 2 , with = ( ) and 2 = 2 ( ). Hence, in SF networks, there will be an absence of threshold if 2 → ∞, meaning even a disease with a low infection rate can cause an outbreak in such a network. The epidemic threshold for the -intertwined deterministic model (4) is specified by ,VPM = 1/ ( ), where ( ) the special radius of the adjacency matrix [134] . For the heterogeneous -intertwined deterministic model (5), the infectious disease will die out when ( − ) ≤ 0. Otherwise, an outbreak will occur. Here,

, and is the adjacency matrix of the underlying network. For a comprehensive review of deterministic epidemic models, their "outbreak" thresholds and stability properties, one can refer to [97, 101] .

In this subsection, we present several well-known stochastic epidemic models based on SIS compartment models. The counterparts for the SIR type of epidemics can be formulated by adding a new compartment R and describing the transition rule between compartments S, I, and R. First, we consider stochastic epidemic models without network structures, which we refer to as stochastic population models.

Consider a population of individuals. Recall that the rate of infected individuals infecting someone else is denoted by . Each infected individual recovers and becomes susceptible at rate . Assume that individuals encounter each other uniformly at random from the whole population. Let ( ) be the number of infected individuals at time . Then, { ( )} ≥0 is a Markov jump process with state space N, with transition rates (1 − / ) from state to state + 1, and from state to state − 1. The transition probability hence is given as

The stochastic population model (6) does not consider network structures and assumes that each individual encounters each other uniformly. It is clear that the stochastic population model (6) is a Markov jump process with an absorbing state = 0, i.e., the state where no individual is infected. When the process enters the absorbing state, the infection dies out. Indeed, stochastic epidemic models with a finite population admit an absorbing state at the origin (i.e., no infection) and a degenerate stationary distribution that has a probability 1 at the origin. The stochastic model gives a prediction that extinction will ultimately occur for any given initial distribution, regardless of the value / . However, the time to extinction does change drastically with / . When / is distinctly less than 1, the "die-out" time is small and almost independent of / . However, the "die-out" time increases exponentially in the number of population if / is distinctly larger than 1 [76] . The "die-out" time can last longer than the entire human history for a small population. Hence, the distribution of the number of infected individuals during the long waiting time before extinction is close to the distribution of the same random variable under the condition that extinction has not occurred [94] . This is called the quasi-stationary distribution of the stochastic model. Analyzing the quasi-stationary distribution for various stochastic models becomes the focus of many papers [6, 76, 94] .

Another common result in the studies of stochastic epidemic models dwells is to determine the mean-field deterministic model associated with the stochastic model when the population becomes large. Indeed, as goes to infinity, the trajectory → ( )/ converges to the solution ( ) of the Kermack and McKendrick deterministic model (1), i.e., lim →∞ sup 0≤ ≤ | ( )/ − ( )| = 0 almost surely. The proof of the convergence is underpinned by the well-known Kurtz's theorem. One can refer to Section 5.3 of [29] for more details. We now introduce the stochastic network models that incorporate network structures. Define ( ) as the epidemic state of individual . The individual is infected if ( ), and healthy (susceptible) if ( ) = 0. Infected individuals recover at rate , while susceptible individuals become infected at rate =1 , i.e., the product of the the infection rate and the number of infected neighbors. Let¯( ) = ( 1 ( ), . . . , ( )) be the entire epidemic state of the whole population. The SIS stochastic network model can be expressed by the following Markov process:

for = 1, 2, · · · , . The stochastic network model (7) has 2 states among which there is an absorbing state (i.e., a state where no individual is infected) reachable from any state¯( ) with non-zero probability. Hence, this model indicates that the epidemic will die out almost surely in finite time. A more meaningful way to describe the spreading process is through probabilistic quantities. Indeed, the probability that the epidemic will not die out at time is bounded as follows [29] 

where is the adjacency matrix of the underlying network, ( ) is its spectral radius. From (8) , how fast the epidemic will die out depends on the effective spreading rate / and the spectral radius of the underlying network. Let denote the time to absorption state (the time when the epidemic dies out). An application of (8) leads to the results regarding the expected extinction of the epidemic [29] : Given arbitrary initial condition¯(0) ∈ {0, 1} , if < 1/( ( )),

If ≥ 1/( ( )), the expected extinction time increases exponentially as the number of individuals increases, i.e., E[ ] = ( ), where depends on the effective spreading rate and the network structure [40] . Loosely speaking, the spectral radius ( ) quantifies "how tightly the underlying network is connected". The results (8) and (9) align well with the intuition that it is easier for an infectious disease to grow on a more tightly connected network.

Letting Δ goes to zero, the dynamics of E[ ( )] can be written as

Alleviating the complication of the term

for all ≠ , one can recover from (10) the -intertwined deterministic model (4) , which re-states below:

is not necessarily true, the -intertwined deterministic model (4) serves as an approximation of the stochastic network model (7) . Indeed, it is shown that the expected values in (4) are upper bounds on the actual probabilities given by (7) [22, 97, 102, 134] . Furthermore, [133] investigates how accurate the deterministic approximations (4) are in describing the stochastic model (7) .

So far, we have introduced main stochastic and deterministic epidemic models that are well studied in the literature. To offer an overview of these models and their connection, in Figure 2 , we present an illustrative taxonomy of the epidemic models we introduced so far. Apart from models that are based on Markov processes and ODEs, there are epidemic models based on stochastic differential equations [5, 46] , purely data-driven approaches [26] , or spatial modeling [63, 111, 116, 132] . Even though the focus of this review is on actual virus epidemics, there has been an abundant number of papers studying the spread of malware, Trojans, worms on mobile wireless networks [7, 9, 74] , social networks [8, 88, 95] , Delay Tolerant Networks [4, 100] , general computer network [80] . These papers introduced many useful techniques and insightful results applied to analyzing models of actual epidemics [7, 74, 80] . In this review, we will introduce several papers that do not necessarily focus on actual epidemics but provide useful insights for models of actual epidemics [66, 70, 73, 129, 130, 148] .

With a basic understanding of the epidemic spreading processes and their stochastic and deterministic modeling, the next section introduces how decisions can be made based on these models and how decision-making can in turn affect the spreading processes.

The epidemic models introduced in the previous subsection have described the spreading process when there is no human intervention. In practice, once an epidemic starts to prevail, human will reacts accordingly by taking interventions. The human intervention creates an considerable impact on the spreading processes. Hence, the modeling of infectious diseases needs to take it into consideration to provide a consolidated understanding of the epidemic spreading processes in the human population. To better understand epidemic spreading processes in human population, we introduce a holistic framework that incorporates epidemic models and decision models, illustrated in Figure 1 .

In the real world, different entities play different roles in making decisions to combat the spread of the disease. Such entities can be individuals, households, and organizations. Decision making consists of the choices of individuals and the strategies of stakeholders at different hierarchical levels. Decision models considered in the existing literature mainly consider two types of decisionmakers: a central planner [12, 13, 25, 27, 28, 54, 57, 66, 73, 75, 81, 82, 87, 97, 98, 103, 107, 108, 112, 122, 136, [139] [140] [141] [142] 144, 146, 148] , which works for social benefits by conducting mechanism design and/or applying enforceable measures directly to the general public, and a collection of individuals with various types [2, 13, 16, 21, 27, 30, 35, 36, 50, 56, 59, 65, 66, 114, 127, 129, 130, 145] who have their own goals. Considering only one central planner simplifies the case in the real world and allows modelers to focus on a particular aspect of decision making amid the epidemic and derive results that explain certain phenomena.

Decisions makers can take either non-pharmaceutical interventions (NPIs) or pharmaceutical interventions or both. For a central planner, non-pharmaceutical interventions include but not limit to requiring mandatory social distancing, enforcing lockdown, quarantining infected individuals, and deploying protective resources such as masks, gloves, gowns, and testing kits. Pharmaceutical interventions are related with the availability of vaccines or antidotes. A central planner's decision may involve vaccine distributions, antidote allocations, treatment prioritization, etc. For individuals, possible non-pharmaceutical interventions are wearing a mask, practicing social distancing, self-quarantine, etc. Pharmaceutical interventions, such as getting vaccinated, seeking for treatment, securing an antidote, are usually adopted by individuals to protect themselves in the epidemic.

To understand the coupling between the decision models and the epidemic models, we need to figure out how non-pharmaceutical and pharmaceutical interventions can be modeled and incorporated into the epidemic models. Generally, non-pharmaceutical interventions help curb the spreading by either reducing the interaction between individuals (e.g., avoiding crowds, social distancing, lockdown, and quarantine) or utilizing protective resources (e.g., wearing a mask and frequent use of hand sanitizer).

In networked epidemic models such as the -intertwined deterministic model (4) and the stochastic network model (7), the interaction between individuals is usually captured by the network topology. In an unweighted network, the adjacency element = 1 means that there exist interactions between individual and individual . Otherwise, = 0. Some existing papers use the adjacency elements ∈ {0, 1} to describe strict measures that completely cut down the interaction between individuals such as lockdown and quarantine [31, 139, 142] . For example, if individual is quarantined, then = 0 for all ∈ N . To capture the effect of measures do not require complete isolation such as social distancing, several papers have considered weighted networks to describe the intensity of interactions [66, 108] . For example, [66] uses a weight coefficient ∈ [0, 1] to describe the intensity of the interaction between individual and . In epidemic models that do not capture the complete topology (e.g., the Kermack and McKendrick model (1), the Kerphart and White model (2) , the model of Pastor-Satorras et al. (3) , and the stochastic population model (6)), the reduced interaction between individuals is modeled by a scaled infection rate , where ∈ [0, 1] is the scaling factor and is the normal infection rate when there is no intervention [12, 27, 75, 103, 107, 114] . For example, in [75] , captures how well people practice social distancing. The better people practice social distancing, the smaller the factor is. An alternative way of modeling interventions such as lockdown and quarantine is to create a new compartment [32, 81, 144] . In [81] , the central planner decides the number of infected individuals to be quarantined to curb the spreading. The authors introduce a new compartment called 'Quarantine' to model infected individuals being selected for quarantine. Once individuals are quarantined, they will not infect susceptible individuals and will re-enter the Susceptible compartment after recovery.

Utilizing protective resources such as wearing a mask and using hand sanitizer helps individuals protect themselves from infection without reducing their interaction with other individuals. Such interventions are also captured by a scaled infection rate [97, 98] with ∈ [0, 1], which describes the fact that when contacting infected individuals while wearing a mask, susceptible individuals will less likely to be infected.

Pharmaceutical interventions refer to the use of preventive medicines, vaccines, antidotes, or effective treatment methods when available. One example of preventive medicines is pre-exposure prophylaxis, which is proved effective in limiting HIV spread [77, 113] . Preventive medicines lower the infection rate of the target diseases. Individuals need frequent use of the medicine (e.g., daily or twice a day) to maintain a lower rate of infection to the disease. Preventive medicines protect individuals for a shorter period than vaccines [113] . Individuals can prolong the protection by constantly taking preventive medicines. Hence, to model using preventive medicines as an intervention, modelers use different ways to describe the cost and inconvenience of using preventive medicines than getting a vaccine. Many researchers have studied the effect of vaccination on the spreading process [2, 11, 16, 49, 59, 82, 86, 112, 129] , which can be modeled in several ways. One way is to reduce the infection rate between an infectious and a vaccinated individual to < while those who are not vaccinated suffer a higher infection rate [2, 11, 16, 59, 82, 112] . How small vac is depends on the efficacy of the vaccine distributed. An-other way to create a new compartment called 'Vaccinated' and the rate at which individuals exit this compartment captures the protection duration of the vaccines [1, 86] . The use of antidotes and the deployment of mass treatment accelerate the recovery process. As a result, The use of antidotes and the deployment of mass treatment are usually modeled by a recovery rateh igher than the natural recovery rate [25] .

Timing plays a significant role in the coupling between the epidemic spreading processes and the decision models. Depending on when decisions are made, decision-making can be categories into pre-epidemic decision-making [20, 51, 53, 57, 97, 112, 123, 130] , during-epidemic decision-making [2, 3, 16, 21, 25, 27, 28, 30, 31, 34, 54-56, 59, 66, 73, 81, 82, 84, 87, 97, 107, 108, 114, 120, 122, 127, 129, 139, 142, 145] , and post-epidemic decision-making [19, 66, 106, 142] . Pre-epidemic decision-making refers to the decisions made before an epidemic happens or at the beginning of the pandemic. Most papers focus on network design/formation problems in which a virus resistant network with guaranteed performance is designed/formed [20, 57, 97, 130] . Some researchers study the optimal design of an epidemic surveillance system on complex networks that helps detect an epidemic at an early stage [51, 53, 123] . Early epidemic detection allows a central planner to kill the spreading at its infancy. Post-epidemic decisionmaking happens at the very end of an epidemic or after an epidemic dies out. The post-epidemic decision-making addresses problems such as how to safely lift restrictions, how to reverse the interventions taken during the epidemic season [19, 66, 106, 142] . This review focuses on during-epidemic decision-making problems where decisions are made while an epidemic is present. The dynamics of the epidemic spreading processes and the dynamics of the decision adaptation may evolve at different time scales. Some interventions such as getting vaccinated (if one obtains life-long protection from the vaccine), taking an antidote, and distributing curing resources are irreversible [16, 56, 59, 87, 129] . For such interventions, decisions were only made once. In other studies, authors assume that decisions are made once and for all and remain fixed over the spreading process or that decisions are made at a much smaller frequency than the epidemic spreading process [25, 107, 120] . These assumptions and the irreversibility of some interventions make it possible to formulate a static optimization [25, 87] or game [16, 56, 59, 107, 120, 129] problem to study the decision-making during the epidemic spreading. The objective functions of these static optimizations or game problems only involve the limiting behavior of the epidemic models (1)-(7), e.g., the infection level at equilibria (when the epidemic model reaches a steady state) [16, 25, 120, 129] , whether the threshold condition is met (i.e., whether the epidemic will eventually die out) [97] .

If the epidemic spreading processes and the decision models evolve at the same time scale, adaptive strategies are employed in which the decisions are adapted at the same pace as the epidemic propagates. For example, a central planner distributes testing kits based on the daily infection data. This creates a real-time feedback loop in the human-in-the-loop epidemic framework shown in Figure 1 . In this case, individuals take interventions such as whether to wear a mask, conduct social distancing, or stay self-quarantined based on currently perceived information; the central planner adapts his/her interventions according to the observed infection status of the whole population in real-time. The strategies of individuals and the central planner is represented by a map that maps the information they received so far to an action that describes the interventions being taken. Depending on the choice of epidemic models, researchers employ different tools to study the human-in-the-loop epidemic framework depicted in Figure 1 . When deterministic models are employed to describe the epidemic spreading processes, optimal control [27, 28, 54, 81, 97, 139] , differential game [27, 66, 73, 114, 122] , or evolutionary game theory [127] have been applied to study the human-in-the-loop epidemic framework. When stochastic models are used, the human-in-the-loop epidemic framework is often modeled by Markov decision processes [41, 108, 143] and stochastic games [3, 30, 31, 55] . Some research papers consider seasonal epidemics and at each epidemic season, decisions are made once. People adapt their decisions based on the payoff of the prior seasonal epidemic. This type of decision-making problem under seasonal epidemics is solved by analyzing repeated games [21, 84] or evolutionary games [84] .

For the human-in-the-loop epidemic framework, decision-makers usually involve a central planner that represents the central authority and individuals that represent the general public [66] . The central planner can be an effective central government when fighting against an epidemic, or a network operator whose users are obliged to abide by the company security policy. An individual can be a individual citizen of a society [134] , a local community [33] , public authorities of different countries [85] or a user in a computer network [148] .

A central planner cares about the welfare of the whole population such as the number of infected individuals in the entire population, the well-being of the economy [12, 13, 25, 27, 28, 54, 57, 66, 73, 75, 81, 82, 87, 97, 98, 103, 107, 108, 112, 122, 136, 139-142, 144, 146, 148] . Individuals concern about their own interests, which include his/her own infection risk, the inconvenience of wearing a mask, and the monetary cost of getting a effective treatment [2, 13, 16, 21, 27, 30, 35, 36, 50, 56, 59, 65, 66, 114, 127, 129, 130, 145] . Due to the selfishness of the individuals, the goal of an individual is not well aligned and sometimes conflicts with the goal of a central planner [31, 66] . For example, the infected individuals, with no infection risk anymore, might be reluctant to take preemptive measures to avoid spreading the disease [31] . It is shown that there will be an increase in the number of infected individuals if they optimize for their own benefits instead of complying with the rules applied by the central planner [66] . Indeed, enforcing a strict protocol can be costly and sometimes impossible for the central planner. As a result, central authorities should ask themselves whether they can offer the public sufficient incentives that are acceptable by the individuals and sufficiently strong to combat the epidemic. A recent example is that, to reach herd immunity, the Ohio state of the United States will give 5 people 1 million each in COVID-19 vaccine lottery to combat the hesitancy of getting a COVID-19 vaccine [47] . Hence, instead of solving an optimization problem or a game problem directly, some papers have looked into the mechanism design problem or the information design problem [147] , on behalf of the central planner, that incorporates both the global state of the whole population and the individual's choice into designing incentives to combat the epidemic [13, 21, 34, 66, 82, 99, 107] .

Decision-makers rely on what information they have to make decisions. The information available to decision-makers at the time when the decision is made plays a crucial role in the human-in-the-loop epidemic framework [28] . For example, the severity of COVID-19 infection, perception of government responses, media coverage, acceptance of COVID-19-related conspiracy theories lead to a change of people's attitude about wearing masks during the COVID-19 pandemic [37, 117, 119] . Many studies assume that perfect information, including the health status of every individual and complete knowledge of the network topology, is available to decision-makers at all times [2, 27, 30, 31, 34, 59, 81, 97, 122] . However, in epidemics, acquiring perfect, accurate, and timely information regarding the spreading process is arduous if not impossible [18] . For example, obtaining an estimate of the number of infected individuals requires testing at scale, which can be challenging to implement in rural areas [90] . Also, testing results can be delayed due to a high testing demand and a long sample analysis time. Hence, some researchers investigated the decision-making based on an estimated disease prevalence from available data [55, 108, 139] and the effect of delayed information in decisionmaking [149] . In game theory, a specific type of strategic games that deal with incomplete information are called Bayesian games. A few papers have leveraged the concept of Bayesian games to deal with the incomplete information when only part of the information or some statistics about the information are revealed to the players [48, 131] .

Individuals can receive information from mass media (global broadcasters) such as TV, radio, newspaper, and official accounts on social media and/or from local contacts such as friends, family members, and connections on social media. During an epidemic, individuals may receive two levels of information: one is statistical information that describes the overall prevalence of the epidemic such the number of positive cases, the number of hospitalized patients, and the death toll; the other is local information such as whether people with close connections are infected or not, risk level in one's neighborhood [45, 78] . [39, 45, 138] investigate how the word-of-mouth type of information spreading affects individuals' behavior and hence, alters the spreading processes.

Individuals may suffer from inaccurate information from unreliable resources. An example would be information obtained from social media or by word of mouth in a spatially or culturally isolated community or neighborhood. The perceived information of an individual may not necessarily reflect the actual prevalence of the infectious disease. Such incomplete or biased information about the epidemic together with strong prior beliefs may impede individuals from taking rational and reasonable responses to protect themselves and others [114] . Information released by central authorities also plays a significant role in individuals' decision-making. Responsible central authorities should therefore not only fight against the epidemic-related misinformation but also conduct information design to curb epidemic spreading and even panic spreading.

Game theory, in a nutshell, is a powerful mathematical tool of modeling how people make strategic decisions within a group [15, 38] . In the last decade, there has been a surge in research studies in game-theoretic decision-making amid an epidemic [2, 10, 13, 16, 18, 21, 23, 27, 30, 31, 34, 50, [55] [56] [57] 59, [65] [66] [67] 73, 78, 82, 84, 99, 107, 114, 115, 120, 122, 127, 129, 130, 141, 145, 146] . The reason behind the surge is three-fold.

First, centralized decision-making becomes less practical for large-scale networked systems such as human contact networks and most computer networks. Computing centralized protection strategies faces the challenge of scalability when they are applied to very large networks [56, 87, 142] . Also, it requires a high level of information granularity for a central authority to make satisfactory centralized decisions for most individuals. The central authority has to gather a huge amount of local information, which not only is challenging to implement but also creates privacy issues and management overheads. In contrast, decentralized decision-making is more reliable and practical since local entities decide their own protection strategies satisfying high-level guidelines provided by the central authority. Second, Self-interested individuals in the midst of the epidemic might not be willing to comply with the suggested protocols [13, 16] . This is because, as we have explained in Section 2.2.4, there is a misalignment of individual interests. Individuals concern less about and societal interests that are major concerns of the central authority. There also exists a misalignment of interests between individuals and interdependencies among the individuals. Each individual has choices, but the payoff for each choice depends on choices made by others. Third, game theory, as a mature and broad field, provides a plethora of useful solution concepts and analytical techniques that can model and explain human decision-making. For example, the self-interested strategy maximizing individual payoff is called the Nash equilibrium in game theory [15] . Through a Stackelberg game framework, a central authority can design incentives for the public individuals to combat the epidemic [13] . Many infectious disease models usually do not incorporate human behaviors that change as the epidemic evolves and the information spreads over the network. Dynamic game theory, which has been applied in many dynamic settings such as management science [14] , labor economics [83] , and cybersecurity [61] , delivers a powerful paradigm to capture dynamic human behaviors [16] . We start the introduction of game-theoretic models in epidemics by presenting a taxonomy in the next section.

The synthesis of game-theoretic models and epidemic models roots in the coupling between decision models and epidemic models. The choice of gametheoretic models depends on multiple factors such as who the decision-makers are, what interventions decision-makers can take, what information decisionmakers know and etc. Existing literature mainly studied the following five types of games: static games [57-59, 65, 99, 107, 120, 129, 130, 141] , discretetime stochastic games [30, 31, 78] , differential games [13, 27, 65, 66, 73, 114, 122] , repeated games [2, 21, 60, 82] , and evolutionary games [10, 50, 110, 115, 145] .

In static game frameworks, researchers have incorporated the epidemic models by only considering the limiting behavior of these models [57, 59, 99, 120, 129, 130, 141] . Here, we use [99] as an example. In [99] , J. Omic et al. captures the risk of infection using the limiting behavior of the heterogeneous -intertwined deterministic model (5) . Let ∞ be the steady state of model

In [99] , each individual decides its own recovery rate by seeking treatment, having antidotes to optimize the trade-off between the overhead invested in recovery and the penalty of infection ∞ . [99] creates a game with players whose goals are to minimize ( , − ) = + ∞ . The coupling between individuals' strategies is captured by the limiting behavior of the epidemic model (11) . Static game frameworks consider once-and-for-all interventions which cannot be revoked and concern about the long-term outcomes such as the infection risk at the steady-state, or whether the disease will die out eventually.

Discrete-time stochastic games and differential game frameworks are introduced to capture the transient behavior of the epidemic process and to model adaptive interventions. The difference between Markov game frameworks and differential game frameworks lies in the choice of epidemic models. Discretetime stochastic game frameworks are built upon stochastic epidemic models such as the stochastic population model (6) and the stochastic network model (7) [30, 31, 78] . Differential game frameworks rely on deterministic epidemic models or stochastic epidemic models that use stochastic differential equations to describe the dynamics of the spreading processes [13, 27, 65, 66, 73, 114, 122] .

Characterizing a Nash equilibrium over the whole horizon is prohibitive for discrete-time stochastic games when the number of individuals increases or the number of stages becomes large. Even structural results are difficult to obtain. Hence, in [30, 31] , the authors introduce a concept called myopic Markov perfect equilibrium (MMPE). The solution concept MMPE implies the assumption that individuals maximize their current utility given the state of the disease ignoring their future risks of infection and/or future costs of taking interventions in their current decision-making. This is a reasonable assumption considering the computational complexity of accounting for future states of the disease during an epidemic. [78] also adopts a similar solution concept where only the current state and the next state of one's infection are considered. For differential game frameworks, the equilibrium can be calculated using the general methods of Isaacs [68] . Using Pontryagin's maximum principle, [114] studied the differential game of social distancing and the spreading of an epidemic under the equilibrium; Khouzani et al. [73] found the optimal way of dissemination security patches in wireless networks to combat the spread of malware controlled by an adversary; Huang et al. [66] characterized the optimal way of reducing connectivity to keep the balance between mitigating the virus and maintaining the economy.

Repeated game frameworks are used to model seasonal epidemics which appear periodically [21, 82] . Interventions will be taken repeatedly at each epidemic season. For example, to protect oneself from influenza, one needs to get a flu vaccine each flu season due to the mutation of the virus or the protection time of a vaccine. Individuals adapt his/her behavior based on the cost/payoff incurred last season. Different from differential games and discretetime stochastic game frameworks, repeated game frameworks do not include the transient behavior of the spreading process [2, 60] .

One common way of individuals making 'best-response' decisions that give the best immediate or long-term payoff. This way of decision-making is adopted by differential game, stochastic game, and repeated game frameworks. Another is the use of 'imitation' dynamics where individuals copy the behavior that is previously or currently most successful [115] . The 'imitation' dynamics governing the time evolution of the fractions of strategies in the population is similar to the replicator dynamics of evolutionary game theory [110] . Evolutionary game theory has been adopted to model the human behavior of imitating other individuals' successful strategy by many previous studies [10, 50, 110, 115, 145] . In the evolutionary game framework for epidemic modeling, the strategy dynamics is coupled with the epidemic dynamics [50, 115, 145] . The focus of evolutionary game frameworks is on the analysis of the coupled dynamics and the interpretation of the behavior of these dynamics [10] . A detailed review of this branch of research can be found in [23] .

Based on the interventions individuals adopt, game-theoretic models in epidemics can be categorized into vaccination game [2, 21, 59, 82, 120, 145] , social distancing game [13, 27, 43, 65, 66, 78, 107, 114] , quarantine game [10, 55] , and mask wearing game [107] etc. There are also papers that study the adoption of preemptive interventions [30, 31, 145] and interventions that changes the recovery rate [58, 141] . Beyond human social networks, many papers studied interventions that can curb the malware or virus spreading in computer networks or wireless communication networks [50, 60, 65, 66, 73, 99, 122, 129, 130] , such as network re-forming [129, 130] , installing security patches [50, 122] , reducing the communication rate [66, 73] and etc.

If all individuals practice socially distancing, wear masks, stick to stay-athome orders, the risk of infection and the infection level of the whole population will be reduced significantly. However, there always exist trade-offs and temptations to defect from the regimen. Hand-washing is tedious, wearing a mask is uncomfortable or annoying, socializing is necessary. When it comes to getting a vaccine, people express concerns about safety and side effects. One commonality of most effective interventions usually exhibits the characteristics that if one takes the intervention, he/her needs to pay for all the cost or inconvenience but everyone else in the population will more or less benefit from his/her behavior. This characteristic creates the coupling between individuals. For example, one can enjoy empty streets and markets without having a higher risk of infection if most people stay at home. Those who choose not to get a vaccine effectively reap the benefits of reduced virus transmission contributed by the people who do opt for vaccination. This behavior is referred to as 'freeriding' behavior [67] . When a significant number of free rides appear, there will be a collective threat to containing the virus.

Different interventions induce different costs and provide benefits in different ways. For example, wearing a mask gives immediate protection and is irreversible, hence induces instantaneous cost and benefit. Vaccination creates longer protection yet induces an immediate cost such as making a payment for the vaccine and experiencing side effects. Hence, different models were used to model different interventions. Vaccination games are usually modeled by static games or repeated games due to the irreversible of getting a vaccine [2, 21, 59, 120] . Revocable interventions such as quarantine, social distancing, wearing a mask are usually modeled by differential games or stochastic games [13, 27, 66, 78, 114] .

Another criterion to classify the literature is to consider the players of the game. Most game-theoretic models in epidemics investigate the interplay between individuals [2, 16, 18, 21, 23, 24, 27, 30, 31, 50, 55, 56, 59, 66, 67, 78, 82, 84, 114, 115, 120, 129, 130, 145] . Individuals can be completely selfish who maximize their own payoff via optimizing their own payoff or imitating the most successful individuals [2, 16, 18, 21, 23, 27, 50, 55, 56, 59, 67, 78, 84, 114, 115, 145] . Some papers incorporate the effect of altruism into their game-theoretic models in which individuals are not completely selfish and care about the well-being of their neighbors [24, 30, 31] . It is shown by Eksin et al. [31] that a little empathy can significantly decrease the infection level of the whole population. The results in [24] by Chapman et al. show that the central planner should promote vaccination as an act of altruism, thereby boosting vaccine uptake beyond the Nash equilibrium and serving the common good. Several papers examine the inefficiency of selfish acts of individuals and the inefficiency is quantified by the price of anarchy [2, 66, 82, 120, 129, 130] . Results from these studies demonstrate that individuals' selfishness becomes a big hurdle to fight against infectious diseases. Hence, some papers introduce the role of central authorities and study how central authorities should create incentives/penalties to achieve social optimum [13, 34, 107] . Another strain of research focuses on the interplay between a central authority and an adversary [73, 122, 141, 146] . The adversary aims to maximize the overall damage inflicted by the malware and the central authority tries to find the best counter-measure policy to oppose the spread of the infection. The conflicting goals between the players are usually captured by a zero-sum dynamic game.

In the existing literature, there is no consensus on which game-theoretic framework to study human-in-the-loop epidemics. The integration between gametheoretic models and epidemic models is done on a case-by-case basis depending on the players involved, what interventions people take(see Section 2.2.1), when interventions are taken (see Section 2.2.3), what epidemic models modelers choose (see Section 2.1), and the underlying network structure. Here, we present a fine-grained dynamic game framework to describe the essence of human-in-the-loop epidemic modeling.

We consider the following discrete-time Markov game with players.

• Players: We consider a population of individuals denoted by N and a central authority. 

for ∈ N . As we can see if all individuals brace for the epidemic by quarantining, i.e., ( ) = 0 for all ∈ N for some ∈ [ 1 , 2 ], the spreading will slow down at a fast rate decided by . Not every individual has the inventive to do so. Remark 1 When there is a presence of a central authority, the central authority cares about the population profile instead of the health status of a particular individual. For example, the goals of the central authority might be to suppress the proportion of infected individuals, reduce the death toll on the general public, and boost up the uptake of vaccines. These metrics can all be reflected in the population profile ( ) for = 0, 1, 2, · · · . Hence, the cost function of the central authority can be a function of ( ), i.e., ( ) = ( ( )). There are two paths the central authority can follow to achieve its goals. The first is designing a penalty/reward function ( ( ), ( )) that rewards (punish) individuals who comply with (violate) the suggested rules such as social distancing, quarantine, or getting vaccinated. The second is through information design such as promoting altruism, raising awareness, health education, etc.

The game between individuals unfolds over a finite or infinite sequence of stages, where the number of stages is called the horizon of the game. Infinite horizon game models epidemics that persist for decades [114] . Some papers consider finite horizon game where the terminal time is decided by when the vaccines are widely available [23, 66, 115] . The overall objective, for each individual, is to minimize the expected sum of costs he/she receives during the epidemic.

Solving such a fine-grained stochastic game is difficult, if not prohibitive, under general solution concepts. The difficulties emerge from three facts. The first is that as the number of individuals increases, the size of the state space |S| for the global state¯increases exponentially. In the human population, the number of individuals in a community ranges from thousands to millions. Hence, analyzing such an enormous number of individuals under the fine-grained dynamic game framework becomes impossible. The second is that individuals do not know the exact information of the whole population¯( ) for every . For some epidemics, it is difficult for individuals to know their own state due to the fact that some infectious diseases do not cause symptoms for some individuals or cause common symptoms that are shared with other diseases. Also to gather information regarding the population profile ( ) for each time and broadcast it to the individuals, the central authority needs to arrange large-scale surveys, polls, and diagnostic tests on a daily or weekly basis. Even so, the population profile ( ) can only be estimated using gathered data. This partial information situation creates a partially observable stochastic game. It is intractable to compute Nash or other reasonable strategies for such partially observable stochastic games in most general cases [52] . The third is if the transition kernel is described by a networked stochastic epidemic model such as (12) , the state dynamics of one individual is coupled directly or indirectly with every other individual through the underlying network. The fact makes it impossible to obtain an appropriate strategy by decoupling [31] . Hence, we propose this fine-grained stochastic game framework to describe the quintessences of the integration model into the epidemic spreading process, with no intention to solve it.

Existing literature usually proposes less fine-grained game-theoretic frameworks in order to obtain meaningful results that help understand the spreading of epidemics under human responses. These papers generalize or simplify the fine-grained dynamic game mainly in three ways. One is to use mean-field techniques by assuming the transition probability of each individual's state only couples with the population profile ( ) and some indistinguishability assumptions [41, 79, 114, 125, 126] . The second is to study a simplified solution concept where each individual considers costs over only a limited number of stages [30, 31, 78] . The third is to consider a continuous -intertwined epidemic model (4), rather than a stochastic one, where a dynamic game is built upon [66] or to apply mean-field techniques in a homogeneous epidemic model such as the Kermack and McKendrick model (1) [114] . In the next subsection, we present three representative papers that proposed simplified frameworks to deliver meaningful results by leveraging the above-mentioned methods. These frameworks choose totally different epidemic models and have their unique ways of integrating game-theoretic decision-making into these epidemic models. [114] In [114] , T. Reluga studies the effect of social distancing on the spreading of SIR type of infectious diseases. The author uses a homogeneous deterministic epidemic model: the Kermack and McKendrick model (1). A differential game framework is proposed in which the interplay between individuals is simplified as the interaction between a specific individual and the aggregate behavior of other individuals.

How social distancing is modeled : Let be one specific individual's strategy of daily investment in social distancing. The population strategy is the aggregate daily investment in social distancing by the population. Borrowing the idea from mean-field games, in the limit of infinitely large populations, i.e., → ∞, , and are independent strategies because changes in one individual's behavior will have a negligible effect on the average behavior. The effectiveness of investment in social distancing is captured by ( ), which is the infection rate given an aggregate investment in social distancing practices. Without loss of generality, we set (0) = 1 when there is no investment. To model the diminishing returns with increasing investment, the author assumes that ( ) is convex and given by

with the maximum efficiency of social distancing (0) = − . Epidemic Models: Epidemic usually start with one or a few infected cases, so (0) ≈ 0. The macroscopic behavior of the spreading process under the aggregate social distancing investment can be captured by a normalized SIR version of the Kermack and McKendrick epidemic model:

where the infection rate and the recovery rate are normalized in order to focus on the effect of social distancing practices on the spreading process.

The cost function: The total cost of the epidemic to the population, , includes the daily costs from infection , the daily costs of infection :

where −ℎ is a discount term with ℎ being the discount rate and the last term is a salvage term representing the cumulative costs associated with individuals are sick at the time mass vaccination occurs ( ). The evolution of individual states: The premise of the game is that at each point in the epidemic, individuals can choose to pay a cost associated with social distancing in exchange for a reduction in their risk of infection. Let p(t) ∈ Δ 3 be the probabilities that an individual is in the susceptible, infected, or recovered state at time t. The probabilities p( ) evolve according to the Markov process

where is the individual's daily investment in social distancing and the transitionrate matrix

The coupling between the population profile ( , , ) and the individual profile p(t) is described by the two processes (13) and (15) (14) into the expected present values V( ; , ). Since the dynamics of V is independent of , there is no need to consider recovered individuals further. Further simplifying the dynamics of V by taking ℎ = 0 (not discount) and = −1 (fixed expected value at infected state), one obtains

which evolves backward in time with boundary condition ( ) = 0. If everyone else invest heavily in social distancing, the individual can become a free rider that earns the benefit (value) without having to invest too much in social distancing because the infection level will remain low. To find a balanced social distancing strategy (Nash strategy), one can simply focuses on (13) and (16) .

To find a balanced strategy is to find the best strategy to play, given that all the other individuals are also attempting to do so. Given such context, a Nash equilibrium solution becomes an appropriate solution concept for the differential game formulated in Section 3.3.1.

( ; , ) in the susceptible state and the associated population level spreading dynamics (13), * is a Nash equilibrium if for any possible strategy , (0; , * ) ≤ (0; * , * ).

A Nash equilibrium is a subgame perfect equilibrium if it is also a Nash equilibrium at every state the system may pass through. Indeed, the Nash equilibrium can be obtained by finding the investment that maximizes the rate of increase in the individual's expected value. 

From Theorem 1, one knows that at the Nash equilibrium, whether an individual invests in social distancing depends on the maximum efficiency of social distancing (0) = , the current infection level of the population , and the expected value in susceptible state . When an individual does invest in social distancing, an individual tends to invest more if any of the following values is higher: the efficiency of social distancing, the current infection level , the value of the susceptible state .

The authors investigate the instantaneous behavior * given the expected value in susceptible state , the infection level , and the susceptible level . Such results can be computed by solving (17) and the results are shown in Figure 3 .

From (17), we know that there are two types of equilibrium strategies including no investment in social distancing ( * = 0, ( * ) = 1) and positive investment in social distancing ( * > 0, ( * ) < 1). The first figure in Figure 3 is a contour plot in × surface about the relative risk ( * ( , )). The values attached to the blue lines represent what the relative risks ( * ( , )) are at the corresponding coordinates ( , ). The line with value 1 attached to it separates the region where the equilibrium strategy will include no investment from the region where the equilibrium strategy requires investment in social distancing. The feedback form of equilibrium strategies, transformed from ( , ) coordinates to the ( , ) coordinates of the phase-space is represented with another contour plot in the second figure of Figure 3 .

The last few 'survivors' tend to social distance more: As we expect, the left figure shows that a larger value of the susceptible state induces a greater instantaneous social distancing. From the right figure, one can see that as the number of susceptible individuals increases, the investment in social distancing decreases, hence the individual's infection rate increases with less social distancing. One can also see that when only a small portion of the population remains susceptible, the biggest investments in social distancing happens. That means the last few 'survivors' tend to social distance to brace for the infection. Two scenarios are investigated in [114] . The first is the infinite horizon differential game that gives the equilibrium behavior when there is never a vaccine and the epidemic spreads until its natural end. The second is the finite-horizon problem that studies the individual behavior in equilibrium when there will be a vaccine introduced at time . For the infinite-horizon case, the epidemic spreading dynamics under the social distancing equilibrium and in the absence of social distancing is plotted in Figure 4 .

Social distancing occurs later but ends sooner than the wide spreading of epidemics: As we can see from the top left figure in Figure 4 , under equilibrium social distancing, social distancing is never used until part-way into the epidemic and ceases before the epidemic fully dies out. That means at the beginning of the epidemic, individuals will not be alert to take any interventions until the epidemic prevails. And social distancing practices are going to be lifted before the epidemic completely ends when the situation gets better.

Social distancing leads to a smaller epidemic but prolongs the epidemic: When comparing the time series data on the top left figure and its counterpart on the bottom left figure of Figure 4 , one can observe that social distancing reduces the scale of the epidemic and prolongs the prevalence of the epidemic. Even though social distancing prolongs the epidemic, practicing social distancing is still important since it helps 'flatten the curve'. "flattening the curve" reduces the number of cases that need intensive care at any given time, giving health-care workers, hospitals, police, schools, and vaccine-developers time to prepare and respond without becoming overwhelmed. Essential facilities (e.g., hospitals and schools) can still function normally with a tolerable level of infection. Slowing and spreading out the tidal wave of cases will save lives. Flattening the curve keeps society going. Now let's shift the focus to the finite-horizon problem where vaccines are universally available at a given time . As is shown in Figure 5 , at an equilibrium, social distancing will last until the very time when vaccines are universally available. When the wide availability of vaccines arrives sooner, social distancing begins sooner. When the vaccine becomes available at = 8.6 (see the left plot in Figure 5 ), individuals save 50% of the cost of infection per capita by practicing social distancing. When the vaccine becomes available earlier, say when = 6.5, 80% of the cost of infection can be saved per capita. Social distancing enlarge the window of opportunity during which mass vaccine can reduce the cost of the epidemic: The earlier a vaccine becomes available, the less the whole society suffers. If a vaccine becomes available at the late stage of the epidemic when most individuals are recovered, the vaccine won't help much reduce the transmission. There exists a limited window during which large-scale vaccination can effectively cut down the cost of infection at the population level. Numerical results in [114] show that equilibrium social distancing can extend this limited window of opportunity.

The modeling in [114] unravels the complexity of the fine-grained dynamic framework from several aspects. The first is the use of a homogeneous SIR deterministic epidemic model: the Kermack-McKendrick SIR model, in which the epidemic process is described by two ordinary differential equations ( can be expressed as = 1 − − ). The effect of social distancing on the spreading process is captured by a scalar function ( ) which is homogeneous to all individuals. The spreading process enjoys a decreased infection rate as the population invests more in social distancing. The use of such a homogeneous epidemic model is a double-edged sword. On one hand, homogeneous epidemic models make analytical results more attainable, but on the other side, homogeneous epidemic models need the assumption that the population is homogeneous and strongly mixed. However, we know that the contact patterns among individuals are highly structured, with regular temporal, spatial, and social correlations. The second is the decoupling of the direct connection between an individual's strategy and the aggregate strategy of the population . This allows one's risk of infection to depend only on the infection level of the population , which implicitly depends on .

Realistically, mass vaccination cannot happen overnight as is assumed in the paper. Vaccination is usually rolled out continuously as it is proved to put into use. This effect can be incorporated into the model by considering a time-dependent forcing. In this game, the individual has complete information about the epidemic including the expected value and the infection level of the whole population. However, in reality, incomplete information (biased or inaccurate information) may drive human behavior away from the equilibria obtained in this paper.

In [31] , Eksin et al. have proposed a Markov game framework using the contact network stochastic epidemic model [96, Ch. 17] in which healthy individuals utilize protective measures to avoid contracting a disease and sick individuals utilize preemptive measures out of empathy to avoid spreading a disease. A solution concept, called the myopic Markov perfect equilibrium (MMPE), is introduced to model human behaviors, which also makes theoretical results attainable for such a framework. Eksin et al. have shown that there is a critical level of empathy by the sick individuals above which the infectious disease die out rapidly. Further, they show that empathy among sick individuals is more effective than risk-aversion from healthy individuals.

The epidemic model : Eksin et al. considers a networked SIS stochastic epidemic model, which is a variant of the networked stochastic model (7) . An individual in the population N susceptible ( ( ) = 0) or infected and infectious ( ( ) = 1) at any given time = 1, 2, · · · . A global detailed description of the population is denoted by¯( ) = ( 1 ( ), · · · , ( )) . The transition kernel Γ can be specified by 01 ( ) and 10 ( ) for ∈ N and = 1, 2, 3, · · · . 01 ( ) is the probability that individual is susceptible at time , but gets infected at time + 1 and 01 ( )

where ∈ (0, 1) is the infection rate, 0 ≤ ≤ 0 is the action of individual , and the action of its neighboring individual is 0 ≤ ≤ 1 for ∈ N . How protective and preemptive measures are modeled: Each term 1− ( ) ( ) ( ) is the probability that the individual is not infected by neighbor . If either individual or his/her neighbor takes protective and preemptive measures such as wearing masks, practice social distancing, or other measures, the probability that individual is infected by neighbor will decrease. An extreme case is either = 0 or = 0 under which individual will never be infected by his/her neighbor . The product of all the terms ∈N (1 − ( ) ( ) ( )) is the probability that the individual is not infected by any interactions. 10 ( ) is the probability that individual is infected at time but recovered at time + 1. This probability is equal to the inherent recovery rate of the disease ∈ (0, 1), i.e., 

where 0 , 1 . 2 are fixed weights. The first term inside the square bracket is the payoffs of not taking any measures including socialization benefits, convenience benefits, and economic benefits etc. The second term captures the risk aversion of susceptible individuals. The risk comes from contacting with infectious neighbors who fail to take serious measures to protect others. The third term is the empathy term that quantifies the risk of infecting others. Hence, the weights 0 , 1 , and 2 are referred to as socialization, risk aversion, and empathy constants. The payoff function is a bilinear function of and for ∈ N . That means given the actions of individual 's neighbors N and their health status N ( ), to maximize his/her payoff, he/she needs to decide whether to resumes normal activity ( = 1) or self-isolates ( = 0) depending on the sign of expression inside the square bracket. If the expression is positive, individual prefers to resume normal. Otherwise, self-isolation is the best choice. The payoffs of the neighbors of individual depend on the actions of their own neighbors. This means if the underlying network G is connected, the payoff profile of the population couples the actions of all individuals. Hence, individuals need to reason about the interaction levels of their neighbors in their decision-making. Such individual reasoning can be modeled using game theory.

Obtaining an analytical solution such as the Nash equilibrium is difficult if one considers accumulative payoffs over a finite period of time or an infinite horizon. Here, Eksin et al. have considered a solution concept called the myopic Markov perfect equilibrium (MMPE). holds for any ∈ [0, 1] and for all = 1, 2, · · · , ∈ N under the Markov process described by (18) and (19) .

The use of MMPE profile carries two implied assumptions. One is the assumption that individuals' actions depend only on the payoff relevant state of the disease. Whether the class of Markovian strategies contains the Nash strategy for all possible strategies is not discussed in [31] . Another is the assumption that individuals make decisions considering the current instantaneous payoff only. Under the assumption of myopic strategies, individuals do not foresee their future risks of infection or infecting others in their decision-making.

The computation of the MMPE strategy profile involves only one stage of the payoff. So, it is more tractable than computing the Nash strategies that consider the accumulative payoffs with states evolving from time to time. Indeed, Eksin et al. show that there exists at least one such strategy profile for the bilinear game captured by (20) . The proof of existence is constructive, which also provides an algorithm that computes an MMPE strategy profile in finite time (Readers who are interested in the proof can refer to [31] ). But unfortunately, even for such a simplified solution concept, no closed-form results in terms of expressing the MMPE action * as a function of the current state¯( ) is obtained. In the next subsection, we present several highlighted results obtained from simulations by the authors.

A little empathy plays a huge role in bounding the basic reproduction number : An important measure for an epidemic process is the basic reproduction number 0 , which measures the spread of an infectious disease from an initial sick individual in an otherwise susceptible host population. Whether 0 > 1 or not is an indicator that the disease is likely to persist when there is a relatively low number of infected individuals. Hence, 0 is an important measure relating the likelihood of disease persistence to network and utility weights. When individuals act according to an MMPE strategy profile, the following bound holds for 0 ,

where ( ) is the proportion of individuals who have degree in the network, and min{ 1 / 2 , }. Here, is the largest degree an individual has in the network. When there are no protective and preemptive measures taken, the bound for the network is 0 ≤ =1 ( ). If the empathy constant is close to zero such that 2 is smaller than 1 0 , one recovers the bound for the contact network models with no protective or preemptive measures. When individuals weigh the risk of infecting others and the costs of taking measures equally, i.e., 0 = 2 , the basic reproduction number 0 is well bounded, i.e., 0 ≤ (1). In a scale-free network with degree distribution ( ) ∼ −2 , the bound becomes

If the empathy weight 2 is negligible ( 2 < 0 / ), meaning individuals do not care about infecting others, then the bound increases logarithmically with the size of the population, i.e., 0 ≤ 1 2 ( + 1). Indeed, the bound is the exact reproduction number for the contact network SIS model with no individual behavior response to disease prevalence [96, Ch. 17] . The effect of individual response on 0 appears as the empathy weight 2 increases. To guarantee 0 < 1, the central authority needs to promote altruism among individuals to guarantee

To show the accuracy of the critical 2 given by (22), Eksin et al. compares the 0 bound given by (21) to simulated 0 value in [31, Figure 4 ]. Risk aversion alone cannot eradicate the disease but it can affect the infection level : While the risk aversion weight 1 does not appear in any of the bounds for 0 , 1 cab affect the infection level of the population under the MMPE strategy even though it is not shown how 1 affects the infection level. The authors show that when the empathy weight 2 is zero, the outbreak threshold condition stays the same, i.e., ( )/ > 1 for any 1 ∈ [0, 1]. This indicates that risk aversion alone cannot help eradicate the disease without the empathy of infectious individuals in this game.

The game framework formulated here by Eksin et al. is similar to the finegrained framework introduced in Section 3.2. Eksin et al. made two simplifications to ensure the analysis and the numerical computation are feasible. One is the use of MMPE as a solution concept to avoid resorting to dynamic programming techniques to solve the problem, which also facilitates the process of showing the existence of the equilibrium. Another one is the assumption that every individual knows the complete state information¯( ) at every stage .

Compared with [114] 

This upper bound can be arbitrarily bad, i.e., in the order of 1/ . That means without enforced measures from the central authority, an individual's response can be inefficient in terms of maximizing the social benefit. Hence, it is essential to couple the decentralized individual responses to disease prevalence with centralized policies such as public announcements, mass organized isolation, or vaccination campaigns arranged by the central authorities.

3.5 Individual Weight Adaptation: An -Player Differential Game [66] In [66] , Huang et al. proposes an -player differential game framework based on the -intertwined deterministic epidemic model (5) . The author mainly fo-cuses on mitigating virus spreading on computer networks. But the modeling, the analysis, and the results of [66] can shed light on the epidemic spreading in the human population. In the differential game, each individual aims to strike a balance between being infected and loss of benefit (economic or social) by reducing his/her connection to neighbors. With a deterministic epidemic model, using Pontryagin's maximum principle, the authors are able to obtain structural results that help understand human behavior under an epidemic.

The -intertwined epidemic model is networked, allowing the authors to capture the local interactions among individuals and their neighbors. The authors also propose a penalty scheme, on behalf of the central authority, to achieve social welfare for the population. 

where N denotes the set of neighbors individual reaches out for. If ( ) is high, individual can reach out less to individual to avoid being infected, i.e., reducing ( ), and restoring its connection to individual after ( ) decreases. The epidemic model (23) is a variant of the -intertwined deterministic model (4) we have introduced in Section 2.1.2 with two differences. One is that here, huang et al. uses weights that can be adaptive to describe the network instead of fixed adjacency matrix elements. Another one is that the transmission is now directional. Individual being able to infect individual does not mean individual can be infected by individual . Such cases appear more in computer networks. In human population, the transmission is often times mutual. Hence, it is more reasonable to use an undirected network to model connections in human population.

The cost function: The authors consider completely selfish individuals associating with two costs. One cost arises from infection captured by : [0, 1] → R + . Another cost is from inefficiency or performance degradation induced by deviating from the original connections for , ∈ N , which is captured by a convex function : R → R + . Individual optimizes over a finite horizon [0, ] with respect to its cost :

where is the original elements in the adjacency matrix. The original weights , , ∈ N describe the connectivity before the epidemic hits. The original weights , , ∈ N are assumed to be optimal for each individual meaning everyone keeps the best way of connecting to others for themselves. Any deviation from the original weights at time will induce inefficiency or performance degradation quantified by ∈N ( ( ) − ). The first term inside the integral of (24) represents individual 's cost of infection. Different from the work by Eksin et al. [31] , the individuals here are completely selfish who only care about their own infection and inefficiency or performance degradation. However, the goal of the central authority is to minimize the aggregated social cost

The terminal time is the time when the mass installation of an anti-virus or anti-malware patch is implemented.

The authors apply Pontryagin's minimum principle to obtain a Nash equilibrium for the -person differential game defined by (23) and (24) . Under the Nash equilibrium strategy, individuals adapt their weights according to the following rule.

The Nash equilibrium strategy profile { * } ∈N, ∈N for theperson differential game defined by (23) and (24) are adapted according to * ( ) =

for ∈ N , ∈ N , where ( ) ( ) (1 − * ( )) * ( ). Here, (·) is the -th component of the costate function x (·) whose dynamics are governed by

where = − / and Λ is a matrix defined as

otherwise.

One can observe from (26) that the weight adaptation * under Nash equilibrium depends on ( ) ( ) (1 − * ( )) * ( ). That means the weight adaptation * is decided by the costate component , which carries the information about the whole population, individual 's health status , and his/her neighbor's health status. To see the dependence more clearly, one can assume that is quadratic, i.e.,

One can draw several intuitive conclusions immediately from (26) and (27). The first is that individual reaches out less to his/her neighbor if the neighbor is infected with high probability, i.e., * decreases when * ( ) increases. The second is that individuals tend to reach out less to neighbors who can get infected easily, i.e., a high leads to a lower * . 

Not just current health status matters, inherent vulnerability also matters: From (26) and (27), we know that individuals cut down their connections with individuals who are currently infected. Beyond that, individuals also lower more weight of connections with individuals who are inherently more vulnerable to the virus even though they may not be infected currently. For example, in computer network, one may reduce data transfer from a server whose security level is very low.

The 'what-the-heck' mentality: The 'what-the-heck' mentality is one that when something undesired happens, one gives up making effort to fix it. Similar mentality also appears in results of (26) and (27) . The term (1 − ( )) in suggests that if individual is infected, he/she just restores all his/her connections with his/her neighbors. Since the network studied here by Huang et al. is directed, infected individuals restoring their connections will not cause more infection. It is worth investigating whether this result still holds or not in Fig. 6 [66] The dynamics of the whole network's infection under the differential gamebased weight adaptation scheme, the optimal control-based weight adaption scheme, and the scheme without weight adaptation when ( ( )) = ( ).

undirected networks. If it holds, this uncaring behavior may lead to a higher infection level in the population.

The weight reduction lasts till the last minute: The authors show that for all ∈ N , ( ) ≥ 0 for all ∈ [0, ] with ( ) = 0 if and only = . It is also shown that for every ∈ N , > 0 for all ∈ [0, ]. That means every individual would more or less cut down his/her connections with neighbors during the spreading process and only restore his/her connections when mass installation of security patches or mass vaccination is available, i.e., * ( ) < for ∈ [0, ) and * ( ) = .

Equilibrium weight adaption outperforms social optimal weight adaptation in terms of lowering infection level of the whole network : Using numerical examples, Huang et al. show in Figure 6 that even though the game-based scheme is inefficient in terms of minimizing the total cost , it outperforms the optimal control-based scheme as we can observe that the infection level under the game-based scheme is always lower than the infection level under the optimal control based scheme. No matter what value is, the equilibrium weight adaption scheme tends to mitigate the spreading of the virus better than the scheme without adaptation.

The game framework formulated here by Huang et al. is, similar to the framework proposed by Eksin et al. in [31] , built on a networked epidemic model. This framework allows the authors to capture the local interaction and the individual-level virus spreading. The authors here choose the deterministic epidemic models allowing them to obtain analytical results in terms of the Nash equilibrium of a finite horizon problem, while in [31] , only the solution for the myopic Markovian Nash equilibrium can be computed numerically. However, since the -intertwined deterministic epidemic model is an approximation of the stochastic model and the accuracy of the approximation is only guaranteed under certain conditions [134] , it is not clear whether the gametheoretic model built upon the approximated spreading process provides an accurate approximation of human behavior. In general, the analysis of whether the equilibrium obtained under the approximated spreading process stays close to the equilibrium obtained under the stochastic process requires significant technical leaps, and it is a topic of active research.

The Nash equilibrium obtained by the authors is an open-loop solution. It is pointed out by [73] that this solution is appropriate in the context of the security in networks, as the instantaneous state of each individual is impossible (or very costly) to follow. But it is still worth studying whether the closed-loop Nash equilibrium and the open-loop one produce the same value of the game and whether the open-loop strategy can be implemented in a feedback way in future endeavors. Another direction of research for this type of framework is to study the long-term behavior of individuals, i.e., the case when is large or infinite.

In the previous section, we presented an overview of the current state of the art involving the game-theoretic models to study human behavior amid epidemics. Game theory has catched many eyes of people who experiences the covid-19 pandemic and influenced policies both directly and indirectly. Since the prevalence of the COVID-19 pandemic, game theory has been on the move from academia to media and the government. The New York Times, a newspaper that can reach tens of millions readers worldwide, has published an article about how game theory can explain the waves we observed from the COVID-19 infection data, how game theory can help prioritize vaccines, and how game theory incorporated with COVID data can help make predictions [128] . The Forbes Magazine has featured an article about how a game theory-based model can enhance stock management of PPE (Personal Protective Equipment) supply in the hospitals of the English National Health Service (NHS) [135] . The Fortune Magazine has featured a commentary on how game theory can solve the vaccine rollout puzzle [91] and help state authorities to minimize the cumulative distance traveled and congestion at facilities, or maximize efficient and equitable access to vaccines. The game theory-based approach for vaccine rollout planning has been proven valuable in after-the-fact analyses of the H1N1 vaccination campaign in 2009 [124] and the response to Haiti's cholera epidemic in 2010 [92] .

Apart from media coverage, stakeholders such as the government and the US army are actively looking for expert advice from game-theoretic perspectives. The Ontario government in Canada has funded Chris Bauch's research project that studies how to re-open Ontario's economy without causing a resurgence by leveraging game theory-based models [44] . The Ontario government is leveraging the research results to prevent, detect and treat COVID-19. Amid the COVID-19 pandemic, the US Department of Defense (DoD) has awarded the University of Michigan in Ann Arbor $6.5 million to study how officials at different levels of government can work together to maximize COVID-19 safe behavior [89] . Beyond that, research results regarding the blood shortages during the COVID-19 pandemic has reached to the Admiral Brett Giroir of the US Department of Healthy & Human Services in a letter sent from California Attorney General Xavier Becerra and signed by Attorney Generals of 21 other states [69] . The letter cited and demonstrated the research results in which Dr. Anna Nagurney has investigated the supply chain network competition among blood service organizations using a generalized Nash equilibrium framework and how it causes the blood shortages during the COVID-19 pandemic [93] . The idea of game theory also influences contact tracing application developers. Inspired by game theory and behavioral science, app developers develop an app called NOVID which uses an approach called "inverts the incentives". The app warns people ahead of time that danger was near instead of telling people they'd been exposed and, for other people's good, quarantining themselves to stop further spread [121] .

Despite of the current impact that the game theory-based model made, there are still many hollows and research gaps in the results presented that need to be well taken care of. This section spotlights several of the main research challenges, to what extent they are currently being tackled, and what needs to be done.

Many game-theoretic frameworks have adopted continuous deterministic epidemic models since these models are easy to analyze and more likely to produce meaningful analytical results. Continuous-time deterministic models are approximations of the epidemic spreading processes in the form of ODEs. The justification of passing from the original stochastic models to a continuous deterministic model in the presence of strategic interventions is barely mentioned in most papers. For example, it is an open problem of how the weight adaptation policies found for the -intertwined deterministic model (23) in Section 3.5 relate to the policies found for its stochastic counterpart (7) , and how the social distancing policies found in Section 3.3 relate to the policies we would find for the stochastic model (6) .

Gast et al. has proposed a mean-field Markov decision process framework, which provides a rigorous justification of the use of a continuous optimal control approach for the virus spreading problem, and shows that the continuous limits provide insights on the structure of the optimal behavior for the discrete stochastic epidemic model [41] . The work by Gast et al. has focused on centralized problems (i.e., optimal control problems). Tembine et al. have investigated the asymptotic behavior of a Markov decision evolutionary games as the size of the population grows to infinity [126] . It is an open research question whether these mean-field results would hold for popular continuous epidemic models when game-theoretic decision-making is involved.

Recognizing the necessity of decentralized decision-making for controlling epidemics, a growing number of researchers have started to pay attention to game-theoretic decentralized solutions. However, most studies of this kind focused on the interactions among individuals [2, 16, 18, 21, 23, 24, 27, 30, 31, 50, 55, 56, 59, 66, 67, 78, 82, 84, 114, 115, 120, 129, 130, 145] . Some studies investigated the inefficiency of selfish decision-making demonstrated by the price of anarchy. Among these studies, some has not considered the role of the central authority [115, 129] . Indeed, the central authority plays a significant role in controlling the epidemic spreading even when the interventions are not enforceable among individuals. When individuals make their own decisions to maximize their payoffs, it is important to consider mechanism design [38] and/or information design [147] to create incentives for individuals and to achieve a certain degree of social optimum.

A handful of papers have studied the role of the central authority in an epidemic when individuals care for their payoffs [13, 107] . [13] proposed a Stackelberg game framework in which individuals play a non-cooperative game, and the central authority influences the nature of the resulting Nash equilibrium through incentives so as to optimize its own objective. In [107] , the authors promote a mechanism design approach, on behalf of the central authority, weighing response costs vs. the social good. Some researchers have studied how information can affect human behavior amid an epidemic [78, 138] . So far, how should central authority disseminate information to help individuals fight against the virus remains an open problem.

Most papers assume complete information in their game-theoretic frameworks [30, 31, 65, 66, 114, 115] . The assumption of complete information in the context of an epidemic study is that every individual knows, when they make decisions, the health status of every other individual in the network, which is unrealistic and impractical. Hence, dynamic games with partial information or incomplete information need to be applied to studying human behavior amid an epidemic. However, papers following this direction of research are rare.

Obtaining information can be costly. For example, if an individual needs to find out whether he/she is infected or not, he/she has to pay for a testing kit or spend time commuting to a testing spot. If the central authority wants to find out the infection level of the whole population, it needs to deploy a large number of resources to conduct mass testing or mass survey. How decision-makers decide when to obtain information and how frequently information needs to be updated become interesting questions worth addressing. In [127] , the authors revealed a counterintuitive fact that the equilibrium level of infection increases as the users' learning rate increases. That means the more frequently the information exchanges, the higher the infection level at the equilibrium. In [64] , the authors consider costly information in a continuous Markov decision process, and the authors show it is not necessary to obtain information very often to maintain satisfactory performance. Understanding the right timing to conduct mass testing to help all decision-makers resist the diseases is essential, especially for those long-lasting low-infection-level diseases. However, this problem has not been touched in the literature yet.

As with all game-theoretic models, human behavior is unlikely to completely agree with the equilibria due to the lack of information or solid prior beliefs. Nevertheless, very few papers have studied the effect of bounded rationality of individuals. Recent studies have included irrationality in evolutionary games when individuals are not entirely rational [109] . Advanced solution concepts such as the 'Quantal Response Equilibrium' have been studied to account for bounded rationality of individuals [42] . In [59] , Hota et al. pointed out that humans often perceive probabilities differently from their true values, and they modeled the misperception of the infection risk among individuals. However, to understand the bounded rationality of human behavior, more studies need to be carried out in the future.

This article has provided a tutorial combined with a review about the integration of decision models into epidemic models. We focus on how game-theoretic models are coupled with epidemic models to understand the human-in-theloop epidemic spreading process. Targeting at readers from the game theory community and the dynamic game community, this review focuses more on introducing the popular epidemic models, the pros and cons of different epidemic models, and how interventions can be modeled in the epidemic models rather than the basics of game theory. We provide a taxonomy of game-theoretic framework for human-in-the-loop epidemic modeling. From the taxonomy, game theorists from different branches can identify topics that are related to them the most. Although this review focuses on disease and epidemics in the human population, similar mathematical formulations, tools, and results can extend directly to many different spreading processes including malware spreading, information dissemination, etc.

In this review, we strive to provide a comprehensive view of the literature. Game-theoretic modeling for epidemics is an emerging topic. There are many useful and important game-theoretic frameworks in the literature that have not been included. We encourage interested readers to keep track of the current of the art. Despite the vast amount of papers studying the game-theoretic modeling for epidemics, there are still many open questions and gaps waiting to be addressed and filled, especially the ones we introduced in Section 4. We believe game theorists and dynamic game theorists can play a significant role, address these questions, and fill the void in theory and practice. Epidemics have taken and are taking a devastating toll on human society. The game-theoretic framework for understanding the human-in-the-loop epidemic spreading will make a real societal impact.

Optimal impulse vaccination approach for an sir control model with short-term immunity

To delay or not: temporal vaccination games on networks

A mean-field approach for controlling singularly perturbed multi-population sis epidemics

Estimating filespread in delay tolerant networks under two-hop routing

Modeling with Itô stochastic differential equations

An introduction to stochastic epidemic models

An epidemic game between contents in a wireless network

A stochastic game approach for competition over popularity in social networks

Optimal monotone forwarding policies in delay tolerant mobile ad-hoc networks

An epidemiological model with voluntary quarantine strategies governed by evolutionary game dynamics

Stochastic epidemic models and their statistical analysis

Modelling a pandemic with asymptomatic patients, impact of lockdown and herd immunity, with applications to sars-cov-2

Optimal incentives to mitigate epidemics: a stackelberg mean field game approach

Mean-field games and dynamic demand management in power grids

Dynamic noncooperative game theory

Vaccination and the theory of games

Essai d'une nouvelle analyse de la mortalité causée par la petite vérole, et des avantages de l'inoculation pour la prévenir. Histoire de l'Acad

A game dynamic model for delayer strategies in vaccinating behaviour for pediatric infectious diseases

Redirecting resources to promote post-pandemic growth

Network formation in the presence of contagious risk

Mean-field analysis of an inductive reasoning game: application to influenza vaccination

Second-order mean-field susceptible-infected-susceptible epidemic threshold

Game theoretic modelling of infectious disease dynamics and intervention methods: a review

Using game theory to examine incentives in influenza vaccination behavior

Optimal curing strategy for competing epidemics spreading over complex networks

Time series forecasting of covid-19 transmission in canada using lstm networks

Optimal control and differential game solutions for social distancing in response to epidemics of infectious diseases on networks

Covid-19 and flattening the curve: A feedback control perspective

Epidemics and rumours in complex networks

Control of stochastic disease network games via influential individuals

Disease dynamics in a stochastic network game: a little empathy goes a long way in averting outbreaks

Mathematical analysis of an siqr influenza model with imperfect quarantine

Epidemiological models and lyapunov functions

An efficient dynamic allocation mechanism for security in networks of interdependent strategic agents

Epidemic spreading on random surfer networks with infected avoidance strategy

Epidemic spreading on weighted networks with adaptive topology based on infective information

Why Coronavirus Conspiracy Theories Flourish. And Why It Matters

Game Theory

The spread of awareness and its impact on epidemic outbreaks

The effect of network topology on the spread of epidemics

Mean field for markov decision processes: from discrete to continuous optimization

Quantal response equilibria

Endogenous social distancing and its underappreciated impact on the epidemic curve

Ontario COVID-19 Rapid Research Fund

Competing spreading processes on multiplex networks: awareness and epidemics

A stochastic differential equation sis epidemic model

Get vaccinated in ohio and get a shot at $1 million, gov. mike dewine announces

On the efficiency of sampling and countermeasures to critical-infrastructure-targeted malware campaigns. SIGMETRICS Perform

Two modes of vaccination program in controlled sir model

Epidemic protection over heterogeneous networks using evolutionary poisson games

Disease surveillance on complex social networks

Solving partially observable stochastic games with public observations

Explainable ai and mass surveillance system-based healthcare framework to combat covid-i9 like pandemics

A closed-loop framework for inference, prediction and control of sir epidemics on networks

Impacts of game-theoretic activation on epidemic spread over dynamical networks

Interdependent security games on networks under behavioral probability weighting

Optimal network topologies for mitigating security and epidemic risks

Game-theoretic choice of curing rates against networked sis epidemics by human decision-makers

Game-theoretic vaccination against networked sis epidemics and impacts of human decision-making

A game theoretic approach for making iot device connectivity decisions during malware outbreak

Dynamic games for secure and resilient control system design

A novel epidemic spreading model with decreasing infection rate based on infection times

Epidemic spreading in random walkers with heterogeneous interaction radius

Continuous-time markov decision processes with controlled observations

Achieving social optimum in dynamic weight adaptation for virus mitigation: A potential differential game approach

A differential game approach to decentralized virus-resistant weight adaptation policy over complex networks

Free-riding behavior in vaccination decisions: an experimental study

Differential games: a mathematical theory with applications to warfare and pursuit, control and optimization

State attorneys general urge FDA to transition to gender neutral screening for blood donations

Directed-graph epidemiological models of computer viruses. In: Computation: the micro and the macro view

Contributions to the mathematical theory of epidemics. ii.-the problem of endemicity

Stability properties of infection diffusion dynamics over directed networks

Saddle-point strategies in malware attack

Maximum damage malware attack in mobile wireless networks

Robust and optimal predictive control of the covid-19 outbreak

On the extinction of the S-I-S stochastic logistic epidemic

Jeehyun and others: Mathematical modeling of HIV prevention measures including pre-exposure prophylaxis on HIV incidence in South Korea

Games of social distancing during an epidemic: Local vs statistical information

Controlling propagation of epidemics via mean-field control

Potential game approach to virus attacks in network with general topology

Suboptimal control and targeted constant control for semi-random epidemic networks

Minimizing social cost of vaccinating network sis epidemics

Stochastic evolutionary game analysis between special committees and ceo: Incentive and supervision. Dynamic Games and Applications pp

Impact of committed individuals on vaccination behavior

Coordination of epidemic control policies: A game theoretic perspective

Non-pharmaceutical interventions and vaccination controls in a stochastic sivr epidemic model. Differential Equations and Dynamical Systems

Distributed algorithm for suppressing epidemic spread in networks

Posting behavior dynamics and active filtering for content diversity in social networks

Game theory and the COVID-19 outbreak: Coordinating our interests at individual to national levels

Testing at scale during the covid-19 pandemic

How game theory could solve the COVID vaccine rollout puzzle

Decentralized beneficiary behavior in humanitarian supply chains: Models, performance bounds, and coordination mechanisms

Supply chain network competition among blood service organizations: a generalized nash equilibrium framework

The quasi-stationary distribution of the closed endemic SIS model

Learning the graph of epidemic cascades

Networks: An Introduction

Analysis and control of epidemics: A survey of spreading processes on complex networks

Efficient containment of exact sir markovian processes on networks

Protecting against network infections: A game theoretic perspective

Tracking message spread in mobile delay tolerant networks

Modeling, estimation, and analysis of epidemics over networks: An overview

Epidemic processes over time-varying networks

Modelling and predicting the effect of social distancing and travel restrictions on covid-19 spreading

Epidemic dynamics and endemic states in complex networks

Epidemic spreading in scale-free networks

Towards safety and sustainability: Designing local recommendations for post-pandemic world

Corona games: Masks, social distancing and mechanism design

Smart testing and selective quarantine for the control of epidemics

Risk perception and effectiveness of uncoordinated behavioral responses in an emerging epidemic

Spontaneous behavioural changes in response to epidemics

A mathematical framework for modeling propagation of infectious diseases with mobile individuals

Optimal vaccine allocation to control epidemic outbreaks in arbitrary networks

Evaluating the cost-effectiveness of pre-exposure prophylaxis (PrEP) and its impact on HIV-1 transmission in South Africa

Game theory of social distancing in response to an epidemic

Evolving public perceptions and stability in vaccine uptake

On the levy-walk nature of human mobility

German and chinese dataset on attitudes regarding covid-19 policies, perception of the crisis, and belief in conspiracy theories

Transmission of 2019-ncov infection from an asymptomatic contact in germany

Conspiracy theories as barriers to controlling the spread of COVID-19 in the US

Equilibria in epidemic containment games

An anonymous covid-19 contact tracing app that warns you when your friends are sick

Differential game-based strategies for preventing malware propagation in wireless sensor networks

Optimal exponentially weighted moving average (ewma) plans for detecting seasonal epidemics when faced with non-homogeneous negative binomial counts

Quantifying and explaining accessibility with application to the 2009 h1n1 vaccination campaign. Health care management science

Covid-19: Data-driven mean-field-type game perspective

Mean field asymptotics of markov decision evolutionary games and teams

Selfish response to epidemic propagation

The Pandemic Is a Prisoner's Dilemma Game

Decentralized protection strategies against sis epidemics in networks

Designing virus-resistant, high-performance networks: a game-formation approach

Game theoretic modeling of cyber deception against epidemic botnets in internet of things

Epidemic spread in mobile ad hoc networks: Determining the tipping point

Accuracy criterion for the mean-field approximation in susceptible-infected-susceptible epidemics on networks

Virus spread in networks

Covid-19: Could Game Theory Help In Tackling PPE Shortages Among Health Care Workers?

Modeling and analysis of conflicting information propagation in a finite time horizon

Epidemic spreading in real networks: An eigenvalue viewpoint

Epidemic propagation with positive and negative preventive information in multiplex networks

Robust economic model predictive control of continuous-time epidemic processes

Optimal resource allocation for competitive spreading processes on bilayer networks

Competition over epidemic networks: Nash and stackelberg games

Distributed topology manipulation to control epidemic spreading over networks

Generalized markov models of infectious disease spread: A novel framework for developing dynamic health policies

Modeling covid-19 pandemic with hierarchical quarantine and time delay

Braess's paradox in epidemic game: better condition results in less payoff

Differential security game in heterogeneous device-to-device offloading network under epidemic risks

Informational design of dynamic multi-agent system

Virus propagation and patch distribution in multiplex networks: modeling, analysis, and optimal allocation

Stability and bifurcation analysis on a delayed epidemic model with information-dependent vaccination