key: cord-0893749-ljvv909q
authors: Mitcham, Jack; Keisler, Jeffrey
title: Multi-attribute COVID-19 policy evaluation under deep uncertainty
date: 2022-03-06
journal: Ann Oper Res
DOI: 10.1007/s10479-022-04592-9
sha: a47cbc16447f6d14657490cb4e9b2a1f52325b04
doc_id: 893749
cord_uid: ljvv909q

Mitigating the impacts of COVID-19 comes with the evaluation of tradeoffs. However, the exact magnitude of the tradeoffs being made cannot be known ahead of time. There are three major concerns to balance: life, liberty, and economy. Here, we create a multi-attribute utility function including those three attributes and provide reasonable bounds on the weights of each. No one set of weights on the utility function can be considered “correct.” Furthermore, the outcomes of each mitigation strategy are deeply uncertain. Not only do we need to take into account the characteristics of the disease, but we also need to take into account the efficacy of the mitigation strategies and how each outcome would be evaluated. To handle this, we use Robust Decision Making methods to simulate plausible outcomes for various strategies and evaluate those outcomes using different weights on the multi-attribute utility function. The simulation is done with a compartmental epidemiological model combined with a simple economic model and a model of liberty costs. Rather than trying to optimize likely outcomes for a particular version of the utility function, we find which strategies are robust across a wide range of plausible scenarios even when there is disagreement about how to weigh the competing values of life, liberty, and economy.

The 2019 novel coronavirus, SARS-CoV-2 (which causes continues to spread in the face of countermeasures such as closing businesses and schools, banning large gatherings, limiting travel, and mandating masks. Many governments are facing cries from their citizens to relax or end the countermeasures for the sake of the economy. Phrases such as "the cure shouldn't be worse than the disease" have been used both in US media and by the President of the United States to insist that damage to the economy from social distancing measures could be more costly than This implies a tradeoff between economic health and physical health, for which there exists ample literature. However, this literature is not being cited in the political sphere when pundits and politicians debate about whether or not to "re-open the economy."

On one hand, there are people who are convinced that the economic damage is worse than the lives lost to , and on the other hand, there are people who are averse to putting a price on lives at all. Neither side appears to be citing a dollar amount they put on each life, or each year of life lost.

Furthermore, there are groups of people who are protesting the countermeasures against COVID-19 in the name of "freedom." Signs at "anti-lockdown" protests read "Freedom is Essential" and "Land of the Free" among other things. (Gabbatt, 2020) This demonstrates there is a third piece to the equation: a hedonic element in which people weigh their personal liberty against both economic well-being and health risks.

The challenge, then, is not only to formulate a utility function for decision-makers to use when evaluating strategies to stop a pandemic but also how to apply such a utility function when the underlying values differ from person to person.

To address this challenge, we will synthesize several methods. First, we create a multiattribute utility function that incorporates the values of life, liberty, and the economy. Then, we outline plausible worldviews, representing different weights on that utility function. Next, we model the pandemic and mitigation strategies using a SEIR system dynamics model and perform global sensitivity analysis to generate a large range of scenarios, analyzing the differences between the strategies to gain insight into what works under what conditions. Last, we use the Robust Decision Making (Lempert, 2019) framework to find mitigation strategies that are robust not only against characteristics of the pandemic but also robust against differing worldviews.

We shall use several well-established techniques which manifest in some unique ways and combine in an interesting fashion. We describe these here very briefly and provide some links to the extant work in these techniques.

The impact of COVID-19 can be characterized by "deep uncertainty." As defined by the Decision Making Under Deep Uncertainty Society, deep uncertainty exists when: parties to a decision do not know, or cannot agree on, the system model that relates action to consequences, the probability distributions to place over the inputs to these models, which consequences to consider and their relative importance. (Decision Making Under Deep Uncertainty Society, 2015) COVID-19 meets all of those criteria. There are many epidemiological models which can be used to predict pandemics, including mathematical models and agent-based models, and they don't necessarily give the same results. (Connell et al., 2009) The parameters of those models are almost completely unknown at the very start of the pandemic, which is when Fig. 1 The RDM process (Lempert, 2019) some policy decisions need to be made. And lastly, there is disagreement about the relative importance of the consequences, which is the main focus of this paper.

When faced with deep uncertainty, robust methods can be used to generate insights for the decision-maker (Keith & Ahner, 2021) . Robust methods are applied over a wide range of situations to stress test and refine decisions by comparing alternatives under different extreme assumptions about factors whose exact values are not easily ascertained (e.g. Jang, 2019; Mild et al., 2015; Wei et al., 2013) . Rather than trying to optimize a decision for some "most likely" scenario, robust methods can be used to investigate which decisions provide a satisfactory outcome across a wide range of scenarios.

One such robust analytical framework to handle decision-making under deep uncertainty is Robust Decision Making (RDM) (Lempert, 2019) . Rather than trying to come up with assumptions and creating a predictive model, RDM focuses on examining a large range of possibilities using exploratory modeling (Bankes, 1993) , and global sensitivity analysis (Wagner, 1995) to illustrate strengths and weaknesses in policy decisions (Fig. 1) .

There are five steps to RDM:

1. Decision Framing 2. Evaluate strategies across futures 3. Vulnerability analysis 4. Tradeoff analysis 5. New Futures and Strategies

The main thrust of this paper is step 4, tradeoff analysis, which is detailed more in the next section. Additionally, we are not only evaluating the strategy across futures (step 2), but also across different worldviews, leading to strategies that are robust against variations in both.

Multi-criteria decision analysis (MDCA) (Belton & Stewart, 2002 ) is a branch of operations research that can help decision-makers decide between alternatives when there are multiple criteria to take into consideration. Multi-attribute Utility Theory (MAUT) (Keeney et al., 1979) holds that one can create a utility function in which each criterion is listed and weighted in order of importance. The result is a composite measure of different criteria that can be used to measure each alternative. This is useful when the criteria being evaluated are not all on the same scale. For instance, comparing the tradeoff between economic outcomes, hedonic value of liberty, and health outcomes, as we're doing here. Public policy alternatives often lie on a Pareto frontier in which one option is best on one criterion and another option is best in a different criterion. This method can be used to help differentiate between options on the Pareto frontier by selecting the alternative with the highest utility value.

This method works best when there is a unitary decision-maker that can set the weights of the utility function. In the case of governments, there generally isn't such a unitary decisionmaker. A government that wishes to uphold democratic ideals will look to represent the interests and viewpoints of all of its citizens to the best of its ability, even when there are disagreements. Nevertheless, MCDA and MAUT can be used as a way to explore differences in individual viewpoints to aid in public policy decision-making.

MAUT is generally concerned with maximizing utility under uncertainty, but this is not the most robust measure (Rosenhead et al., 1972) . Another measure, regret (Savage, 1951) , is often used, especially when using the RDM framework (Groves & Lempert, 2007) . Regret is the difference between the utilities of two different outcomes. While utility is to be maximized, regret is to be minimized. One common way to use regret is to minimize the maximum regret, also known as "minimax regret". Minimax regret is useful when facing large "tail risk", (Anthoff & Tol, 2014) which is common when facing deep uncertainty. It is plausible to have a strategy that is not usually preferred in most cases but is often second-best, and rarely has a large regret value.

As we are using COVID-19 as a case study, an epidemiological model will be necessary in order to evaluate strategies across futures and worldviews. We use a compartmental model known as SEIR (Susceptible-Exposed-Infected-Removed) (Li & Muldowney, 1995) . The SEIR model is often used for diseases for which there is a time gap between being exposed to a virus and being infected by it, as well as a group who are removed (either through recovery and immunity or through death) at the end. Both of these appear to apply to COVID-19 (Lauer et al., 2020) . There are models which are even more detailed than the basic SEIR model used here. RAND Corp, for example, has several extensions to the basic SEIR model in their COVID-19 decision support tool. (Vardavas et al., 2021) . These extended models can provide better predictions in suitable situations, where needed data are available. But in a decision context, such refinements must be balanced against the require richer connections to alternatives and utility they require, and they may be of secondary importance for questions about where stakeholder values are likely to be in conflict or in harmony.

Looking ahead, in Sect. 3 we propose a way to integrate the three approaches. In Sect. 4 we develop a utility function. In Sect. 5, we develop and implement a SEIR model for COVID-19 as a System Dynamics model in Vensim. and populate the integrated model by developing assumptions about values or range epidemiological parameters, economic parameters, policies, and utility functions. This model will be used as a "scenario generator" to perform exploratory analysis. Parameters for each run of the model will be generated through Monte Carlo sampling, and we will evaluate the output of each sample using the utility function developed in Sect. 4 across many runs of the model.

The of this paper is to assist with public policy decisions featuring deep uncertainty with regards to outcomes and value disagreements about how to interpret those outcomes. To achieve this aim, we combine several commonly-used OR methods into a novel modeling framework. One such method is quantitative modeling. Here, we use the aforementioned SEIR model. Another method is Monte Carlo sampling of a quantitative model of a system with large parametric uncertainty. A good example of an application of this method is Anthoff and Tol (2014) . In that paper, the authors use an existing climate model and perform a Monte Carlo simulation of the parameters. They convert all of the outcomes (a statistical life lost, a square kilometer of wetland lost, and so on) to a dollar value, and analyze the results. We diverge in this paper in that we don't use a single value to convert the outcomes to dollars, which leads to another OR method: multi-criteria decision making. We explicitly build a utility function to interpret the output of the simulation. Rather than giving the terms of the utility function a single value, we look at different plausible values for those terms. This adds a second layer to the interpretation of the results. Not only do we look at a large range of possible scenarios and try to pick a strategy that is robust against a large range of them, but we also view the results through the lens of differing worldviews to pick a strategy that is robust against those worldview differences. Ideally, this will lead to a strategy that is not only good for society in the face of deep uncertainty but will also be easier to implement because it is robust against differences in personal values.

The notion of "worldview robustness" was introduced by Lempert and Turner (2021) who used an idealized lake pollution model in a semi-prescriptive fashion. In that paper, they started with worldviews and determined what strategies to employ to balance the tradeoff between economic output and lake pollution based on what those worldviews prefer. Here, we instead start with strategies and consider how each worldview would interpret the results. By applying this idea in a real-life example with real-world parameters, we also illustrate some of its benefits and challenges. Figure 2 is an illustration of the full research framework. The top portion represents the quantitative model developed in Sect. 5. The Policy Decision Variables directly impact both the SEIR parameters and the societal outcomes. For example, the policy decision variable related to mandatory mask-wearing will influence the relevant parameter in the model as well as have the societal outcome of people being required to wear masks. Other assumptions also impact those two concepts, for instance, the number of hospital beds available. The SEIR parameters are then used to run the model, and the outputs from that model describe the portion of societal outcomes related to the disease spread.

Those SEIR outputs are interpreted through the lens of the utility function we develop in Sect. 4.

That entire process is repeated multiple times by randomly varying the parameters to perform global sensitivity analysis, and analyzing the results of the sensitivity analysis, to determine the impact of the policy decisions on the overall utility across different worldviews.

To interpret the results of the model through differing worldviews, we must first build a utility function with plausible ranges on the weights of each term. A utility function should contain terms representing the considerations that drive an individual's preferences over different outcomes. Given the cacophonous public discourse as well as the deep uncertainty, just defining clear measures and a coherent way of combining them is a challenge. However, our stated purpose is to understand the implications of value differences in the context of potential COVID-19 policies, rather than selecting or recommending a single such policy. Hence, we develop a form for the utility to be used that maps preferences relating to the main arguments about policies involving some sort of restrictions. For analyzing similar questions about a richer set of policies and responses, a more complex utility function could be developed.

The proposed utility function has three fundamental objectives: life value, economic value, and hedonic value of liberty. In principle, we could have included any number of other decision criteria. Stone (2012) lists 5 policymaking goals: Equity, Efficiency, Liberty, Welfare, and Security. Here, efficiency is represented by economic value, and life value is a type of security. We explicitly include a liberty term. Outside of our scope, but also interesting would be terms for welfare and equity, which could be useful to include in an extension of the model with a more granular analysis of policy impacts on individuals (as would be appropriate for a prescriptive model guiding policy-makers on what to do). Such analysis would also require more granular epidemiological and economic models. Here, we do not distinguish potentially differential impacts of the pandemic and its solutions across socioeconomic, racial, or gender lines.

One of the important tradeoffs we have to face is the tradeoff between life and the economy. This is perhaps the most controversial part of the utility. Value is placed on life-years routinely in medical settings as well as public policy settings. There are many approaches to put a monetary value on life. Each of these approaches leads to a range of valuations. The tradeoff between life and money depends both on what measure is used and one's personal values.

One method to place a monetary value on life is the Value of a Statistical Life (VSL) (Viscusi & Aldy, 2003) in which the cost of a government policy is weighed against the statistical lives it saves. This is largely based on a "willingness to pay" metric. In other words, how much are people willing to pay to reduce their risk of mortality? The US Department of Transportation uses a VSL measure, which uses a value of $9.4 m per statistical life when making decisions. They use a range of $5.4 m-$13.4 m for sensitivity analysis. (US Department of Transportation, 2016).

Using a "statistical life" valuation makes sense when the lives being saved or lost have the same statistical parameters as the population. However, when a disease like COVID-19 has a nonuniform impact on different segments of the population, VSL is not necessarily the best measure. Keeney (1990 Keeney ( , 1994 turns the VSL around. Instead of putting a dollar value on lives, he asks how lives would a reduction of wealth cause? Poverty leads to increased mortality, he argues, so therefore increasing taxes to pay for a government program to save lives will increase poverty which can cost lives. In this way, instead of comparing dollars to dollars, we can compare lives to lives. This avoids some of the controversy of putting a value on lives. In the 1990 study, he estimates that a fatality might be induced by an economic cost of $3 m to $7.5 m in 1980 dollars. In the 1994 study, Keeney estimates that the number is in the range of $5 m to $50 m in 1990 dollars (Keeney, 1990 (Keeney, , 1994 . Updating the values from Keeney (1990) to 2020 dollars gives us a range of $9.4 m to $23.5 m. The wider range from Keeney (1994) in 2020 dollars is $9.9 m to $99 m. As with the VSL approach, this counts all lives the same, regardless of age, disability, or any other factor. Another way to calculate this value is to look at the years lost. After all, everybody dies. What is actually lost are the years of life between when someone died from a particular cause, and when they otherwise would have died in the absence of that cause. This is common in medical literature, where they use the concept of the QALY, or Quality-Adjusted Life-year (Zeckhauser & Shepard, 1976) . This measure looks at how many years of life are gained by a particular medical intervention adjusted for the quality of those years. For instance, a year of life on a ventilator will be adjusted downwards when compared to a year of perfectly healthy life. This sort of calculation is used when determining how to allocate limited medical resources.

Typical values per QALY are in the range of $100,000 to $150,000, with sensitivity analysis values recommended by Neumann et al. ranging from $50,000 to $200,000. (Neumann et al., 2014) .

If we assume an average life has 40 QALYs left (to put it on par with the "statistical life" saved in the VSL) that provides a range of $2 m to $8 m per life. The lower bound on this range is probably too low as argued in Neumann et al. The upper bound is in the same neighborhood as middle-ground estimates of both the VSL method and Keeney's method of estimating deaths from economic costs.

A common criticism of QALYs is that each year of life should not be adjusted for quality of life (as defined by health economists). The Institute for Clinical and Economic Review (ICER) uses a measure called Equal Value Life-year Gained, or evLYG, in addition to the QALY to determine the effectiveness of interventions. (ICER, n.d.) For the most part, this is not different enough from the QALY to make a difference in our analysis. For COVID-19, the difference between life-years lost and QALYs lost is about 20% according to the Centre for Health Economics in London. (Briggs, 2020) But in any case, the QALY is an upper bound of how much a year of life can be valued, in that one QALY is one life-year lived in good health.

A last point is that regardless of the number of life-years lost or saved, the manner of death matters. When someone is murdered, we as a society do not base the punishment on the number of life-years the victim had left. Thus, we can assume that society additionally places some fixed value on each life saved, regardless of the number of life-years the person had left.

To model this, the life portion of the utility function is of the form:

where V D, V LY are the values placed on each death due to COVID-19 and the value of each life-year lost due to COVID-19 respectively. X D, X LY are the number of deaths due to COVID-19 and the average number of life-years lost per death, respectively. It would be feasible to only put a value on one or the other. For example, one could value each life lost at a fixed number, regardless of how many years of life the person would have otherwise had. On the other hand, one could only consider the number of life-years lost, valuing each life lost at $0 by itself.

To bound the parameters V D and V LY , we consider a little bit wider of a range than most of the literature. This is to allow for the differences between ways to discount life-years lost and to capture a wide range of reasonable positions of valuing not only life-year but the relative importance of the economy. V D should be bounded between $0 and $25 m. The latter value is just above the upper range from Keeney (1990) , and roughly double the Department of Transportation's $13.4 m upper bound for sensitivity analysis.

V LY should be bounded between $50,000 and $500,000. The lower bound is a valuation of the QALY that Neumann et al. has called "curiously resilient" even as they argue it should be updated. The upper bound is far above most measures and is near the upper bound of the Keeney (1990) .

Additionally, the sum of V D + 40V LY should not be higher than $30 m, to avoid doublecounting of life-years (40 being an estimate of the average number of life-years lost in a Value of a Statistical Life calculation). This gives a wide range of plausible values for life in terms of economic value to examine (Table 1) .

People place a value on their comforts and liberties. Philosophers at least as far back as Jeremy Bentham and John Stuart Mill talk about hedonistic utilitarianism. Economists seek to measure this with a Willingness To Pay (WTP) measure (e.g., Potoglou et al., 2010; Viscusi & Zeckhauser, 2003) . Many responses to COVID-19 restrict the liberties of the members of society, and this needs to be accounted for in the utility function.

Rather than pick a particular range of values from WTP estimates, we will just use reasonable upper and lower bounds. A year of having one's freedoms completely restricted is certainly not worth less than losing a year from one's life, so a natural upper bound is the value per life-year, V LY . There exist cases in medical literature in which an outcome can be worse than death, however, we can reasonably stipulate that being isolated at home is not one of those situations. A natural lower bound on this value is $0.

So. if V H is the hedonic value of losing a year of liberty, then

In general, the restrictions placed to mitigate the spread of COVID-19 will be less than a total restriction of freedoms. The "strength" of the restrictions, X R , can be represented by a constructed scale from 0 to 1, depending on the specifics of the policies being proposed, where a 0 means no restrictions, and a 1 is equivalent to having all of your liberties completely curtailed. The last two factors to consider are the length of time these restrictions are in place and the number of people impacted by the restrictions. These can be represented by X RT and X RP , where X RT is in fractional years, and X RP is the number of people.

Then, the total hedonic part of the utility function is:

This will capture the hedonic cost to society caused by the restrictions on liberty.

Valuing the loss to the economy is not as straightforward as it seems. A first-order approximation of economic damage is a decrease in GDP from what we otherwise would expect.

There are multiple issues with only looking at GDP, however. Some of which include:

• Value of life measures don't generally refer to GDP. When the Department of Transportation looks at VSL, they are looking at government expenditures. When Keeney looks at the cost of regulations, he's not explicitly looking at GDP, but rather the wealth of the population. QALYs are generally valued against expenditures from a specific limited budget. None of these cases are a comparison to an increase or decrease in GDP. • GDP doesn't take into consideration inequality. It's not difficult to envision a scenario in which GDP drops by redistributing wealth, leaving the majority of people better off.

Conversely, it is feasible to come up with a situation where GDP increases, but more people die from poverty due to a redistribution of wealth to the wealthiest individual in society. • Government expenditures are part of GDP. This means that the government could go into debt and increase spending to offset a drop in personal income and business investment. This would mask the real economic costs occurring due to COVID-19 mitigation strategies. • The broken window fallacy. The broken window fallacy states that if a shopkeeper has his window broken, it is a boon to the economy, since the glazier gets more business, boosting GDP, who will then, in turn, spend more money, further increasing GDP, and so on. This ignores opportunity costs. Likewise, increased business selling masks may look like a boon to the GDP, but it still represents an economic cost that should be taken into account.

So rather than looking at a specific measure like GDP, we're considering "economic costs" similar to what is seen in (Meltzer et al., 1999) . Note that this does not pass the "clairvoyant test," (Morgan et al., 1990) since there is no number that a hypothetical clairvoyant could look at and say whether or not our prediction would come to pass. In fact, there is no prediction being made at all. Predictions are an anathema to exploratory modeling. Rather, we use the "Bridgeman test" as used in Cooke (2012) , which states, "Every term in a model must have operational meaning, that is, the modeler should say how, with sufficient means and license, the term would be measured." So, while a clairvoyant wouldn't be able to give an exact numerical quantity of economic dollars lost, we would be able to come up with a way to measure each individual piece of the puzzle given the resources to do so.

Another thing worth noting is that economic impact is really just a proxy for other outcomes. When people have more money, they have both improved health outcomes and more access to necessities and pleasures.

Furthermore, economic value is not an independent measure. Both hedonic value of liberty and life value are measured in terms of units of economic value. If, for example, one wanted to downplay the importance of a loss in economic value, this would be equivalent to increasing the weights on life and hedonic values. Conversely, a low value for life and liberty is equivalent to placing a greater importance on economic value. This is a key reason the value of life spans such a large range.

The loss in economic value is the base unit for which the other measures are defined, and can be represented by X E . This consists of a few components. There are direct costs due to the illness (e.g. the cost of doctor's treatments), indirect costs due to the illness (e.g. missing time at work), and costs due to mitigation strategies (e.g. forcing a business to close or reduce services). We give equal weight to each sub-attribute. For example, we are indifferent to whether someone misses a day of work because they're sick, or misses a day of work because of an imposed quarantine, at least for the economic part of the utility. The difference between the two would show up in the hedonic value of liberty, not the economic value.

Lastly, we do not include the economic cost of death since that is already counted in the value of life piece of the utility function. Many of the valuations of life-years are related to economic productivity, and we do not disaggregate the economic value of life from the noneconomic value of life. In other words, if someone misses a day of work because they're ill, that counts as an economic cost. If someone misses work because they died from the illness, that counts as a life cost.

Combining the above pieces gives us the utility function:

In general, we want to maximize utility. All of the above pieces of the utility function were costs, so higher numbers are worse. Thus, we need a negative sign in front of the sum of the above pieces to the utility function so that we are correctly maximizing utility.

Lastly, we are assuming a linear utility function. That is, we value losing 1000 lives as exactly half as bad as losing 2000 lives. This does not necessarily hold in the extreme, as second-order effects take place. For an absurd example, we would value losing every human being on the planet as infinitely worse than losing only half. But short of a catastrophe of that size, a linear utility function should suffice from a societal perspective. We also assume that the terms are independent, which isn't necessarily accurate. For example, we assume that the value of a life remains constant even when the economy is damaged. Anthoff and Tol (2014) uses a VSL of 200 times the annual per capita income, for example. If the pandemic, the countermeasures to prevent the pandemic, or both lower the annual per capita income, that could plausibly change the VSL. We ignore this interaction for simplicity and because it is likely a small effect.

One challenge to decision-making is determining the relative preference between multiple value criteria. These preferences can be combined into "worldviews," which describe a coherent way of interpreting the world. (Churchman, 1968) . For example, the value of lives, the hedonic value of liberty, and the value of economic prosperity are largely a matter of preference, and differing worldviews will put different weights on each. Robust Decision Making can be used to find a set of policies that are not only robust against the scenarios generated by the epidemiological model, but also robust against different worldviews.

With life and liberty utility expressed in terms of units of economic utility, we have 4 feasible extremes of the utility function, which we have grouped into worldviews.

The "Maximum life" utility function places the largest value on life and the minimum value of liberty. Someone with this worldview would prefer to save as many lives and life-years as possible, with no regard to personal liberty. The economy is an intermediate concern. This roughly corresponds to an orthodox liberal view in the United States.

In contrast, "Maximum liberty" places the lowest feasible value on life, and sets the value of liberty to its maximum value (which in this case is the same as the value of life). This corresponds to a person who sees being told to wear a mask as an untenable overreach by the government. The economy is a secondary concern. A motto of such a person might be "give me liberty or give me death."

"Maximum economy" is functionally the same as minimizing the value of life and liberty. Someone who wants to maximize the economy would want to limit the deaths due to the virus since those deaths would hurt the economy. They would support measures to limit freedoms (like forcing people to wear masks or extensive monitoring for contact tracing) as long as it means businesses can stay open. This roughly corresponds to an orthodox conservative view in the United States.

Lastly, the "Minimum economy" worldview means placing the highest possible values on both life and liberty. As in the "Maximum liberty" worldview, this sets the value of a year of liberty equal to the value of a year of life. The difference is the value of life is maximized, and therefore losses to the economy are less impactful. A person who holds this view might want to close businesses but keep parks and beaches open, or want to fund a vaccine but make it optional.

In general, people will not be at one of the four extremes. Rather, they will exist somewhere in the middle. Anyone's preferences could be described by a linear combination of the four worldviews. Therefore, by evaluating scenarios against the above "extreme" worldviews, we can establish upper and lower bounds on the utility.

In order to use the utility function to analyze the results of the simulation model, we need to set a decision frame. For this example, the decision frame is a decision-maker in the state of Massachusetts at the beginning of the COVID-19 pandemic, before most of the properties of the disease were known. We chose Massachusetts because COVID-19 mitigation decisions were largely made at the state level, and we choose the beginning of the pandemic because that's both when uncertainty was the greatest and when some of the most critical decisions had to be made.

Looking at this from the societal level, we invoke Rawls' Veil of Ignorance (Rawls, 2005) to aggregate individual preferences. For instance, one young, healthy person may find their liberty to not wear a mask as worth more than the lives of a thousand strangers. But Rawls' Veil of Ignorance states that decisions should be made as if no one knows whether or not they'll be the one to be affected adversely by the consequences. A young healthy person making a decision about a mask mandate, for example, should make the decision from behind a veil of ignorance, such that they don't know if they're going to be in the high-risk category. In this spirit (though cognizant of its limitations), we are able to compare the value of one person's life with the value of someone else's liberty.

The RDM model consists of three parts: an epidemiological model, an economic model, and a liberty model. The relationships are illustrated in Fig. 3 . The Epidemiological model is used as a "scenario generator" to create a wide range of plausible outcomes, which is combined with the outputs of the Economic and Liberty models and interpreted through a utility function. Sections 5.1, 5.2, and 5.3 will develop the Epidemiological, Economic, and Liberty models respectively.

For the epidemiological model, we have to consider both the characteristics of the disease and the policy levers that can be used to influence the rate of spread. The impact of policies will be summarized in terms of their direct impact on the effectiveness of masking and isolation measures. In Sect. 5.1.1, we describe the disease model, which includes parameters for the direct policy impacts. In Sect. 5.1.2, we define a set of mitigation strategies to be analyzed.

For the sake of simplicity and clarity, we consider static strategies. A better but more involved method might be to use adaptive dynamic strategies, which turn on or shut off depending on dynamically evolving conditions. The use of the utility function to evaluate adaptive strategies is the same as for the static strategies, and that is what we want to highlight here.

To model the disease, we use a System Dynamics model, based largely on the Community Coronavirus Model version 8 by Tom Fiddaman. (Fiddaman, 2020) . It is a SEIR compartmental model implemented in Vensim. Figure 2 shows the structure of the model. Several changes were made to Fiddaman's model to add policy levers that can be modeled. In particular, "mask effectiveness" and "social distancing and lockdown effectiveness" were added. One lever, isolation effectiveness, was already in place in the original model (Fig. 4) .

The version of the SEIR model implemented includes a number of parameters, some specific to the COVID context. These are listed in Tables 2 (disease parameters) and 3 (strategy parameters) below with a brief description of each parameter's role in the model. The table also gives the ranges of potential values we assumed for the simulation. Again, with the RDM model being used for informing the decision process, the ranges are intended The policy levers and strategy parameters outlined in Table 3 will be defined in more detail in Sect. 5.1.2.

The search for robust strategies requires exploration of the range of possibilities rather than probabilistic risk analysis. With 11 parameters, a large number of runs is required to explore the entire parameter space. An alternative to Monte Carlo simulation would be to discretize the parameter distributions and exhaustively calculate results for every combination of the minimum, midpoint, and maximum parameter values. However, this would require 3 11 , or 177,147 runs, and would grow by a factor of three for any other parameter that might be included. And this method would ignore what happens with intermediate parameter values between the minimum and midpoint or midpoint and maximum. Thus, the random inputs in Tables 2 and 3 are simply drawn from a random uniform distribution, while other subtle considerations are not explicitly modeled as uncertain parameters. For instance, the asymptomatic spread parameter in Fiddaman's original model was assumed to be zero. Here, we assume it to be 0.5 (meaning 50% of asymptomatic people contribute to the spread) but do not vary the number in the Monte Carlo simulation. In Fiddaman's original model, he included a seasonality term, which we do not include here since we are looking at a full year and assume the seasonal variance will balance out (Tables 4 and 5) .

Simulation settings: To sample the possibility space, we use a Monte Carlo simulation with two loops. The "outer loop" iterates 100,000 times with a random sampling of the parameters. Within each iteration, there is an "inner loop" of the 5 strategies, each using the same values for the parameters. This leads to a total of 500,000 data points. This was done in Python using the PySD library to control the Vensim model. (The PySD Cookbook-PySD-Cookbook 0.1.0 Documentation). The model simulates one full year of the pandemic. It used a time step of 0.25 (i.e., 6 h), and thus ran for 1460 time steps. Each strategy was held constant throughout each run of the simulation.

The model includes three policy levers, Mask (yes or no), Lockdown (none, light, heavy), and Quarantine (yes or no), that could be combined to form a strategy. There are 12 such combinations, but to start we're only considering 5 of them, in order of increasing severity. We use this as a starting point that will help us identify new strategies later. Modeling all 12 at the start would increase the computation time and make analysis more difficult and cluttered.

• Base: Do nothing and let the disease run its course. Let society self-regulate the response.

• Masks: Mandate mask-wearing by the entire population, but take no other mitigation measures. • Light Lockdown: In addition to mandatory mask-wearing, also shut down large gatherings, and encourage social distancing. • Heavy Lockdown: In addition to everything in Light Lockdown, also close non-essential businesses. • Central Quarantine: In addition to everything in Heavy Lockdown, also centrally quarantine infected individuals, rather than sending them home to recover.

The differences between the strategies are as follows: • Base: Mask effectiveness of 0, isolation effectiveness is between 0.4 and 0.6, lockdown effectiveness is 0 • Masks: Mask effectiveness is between 0.15 and 0.65, isolation effectiveness same as Base, lockdown effectiveness is 0. • Light Lockdown: Mask effectiveness and isolation effectiveness same as Masks, lockdown effectiveness between 0.1 and 0.25 • Heavy Lockdown: Mask effectiveness and isolation effectiveness same as Light Lockdown, lockdown effectiveness between 0.4 and 0.6 • Central Quarantine: Mask effectiveness and lockdown effectiveness same as Heavy Lockdown, isolation effectiveness between 0.9 and 0.95

As with the epidemiological model, the economic model is intentionally simple for purposes of tractability in particular during the integration of the different elements of the RDM model. Below, we define the economic cost terms used and describe the assumptions used in the model and their rationales. We consider both direct and indirect economic costs due to illness and the costs related to the strategies to mitigate the illness. This is similar to Meltzer et al. (1999) which compared the cost of a flu vaccine (which is a cost of mitigation) to the economic cost of pandemic flu.

The direct cost of COVID-19 arises from medical treatment. For those that don't need to be hospitalized, we estimate the direct cost to be $200, per Meltzer's study (ibid.) which found the average cost for outpatient treatment, including prescription drugs, for influenza was $140 in 1999 dollars.

For those that are hospitalized, the cost is much higher, and we estimate this to be $75,000. This is roughly in line with a report from FairHealth which put the cost at $73,300 ("Costs for a Hospital Stay for COVID-19", 2020). Sensitivity analysis showed that there was not a big difference in the results if those numbers were modestly higher or lower, and here the numbers are unlikely to be orders of magnitude different.

The only indirect cost of illness we consider is lost productivity at work. We estimate average production per person to be around the same amount as average wages. In fact, production is higher than wages, but this is balanced out by the fact that some affected people are not employed.

The average daily wage in Massachusetts (which is our decision frame) is estimated to be about $250 a day. The US Bureau of Labor Statistics shows the mean hourly wage in Massachusetts as of May 2019 was $31.58 per hour, which leads to a daily wage of $252.64 assuming an 8-h workday.

We do not consider any differential impact on who gets sick and has to miss work. It may be the case that certain low-wage employees, like fast-food workers and grocery store cashiers, might be more exposed to the virus and thus more likely to get sick than people who can work from home, like software developers. Instead, we let sensitivity analysis handle this. Increasing or decreasing the value by a factor of three has little impact on the analysis.

Each mitigation strategy has a different cost associated with it. In each case, we provide an option on the Tableau dashboard to perform real-time sensitivity analysis on these parameters.

The "no mitigation" strategy will have indirect costs associated with it, mainly in that people will modify their behavior to reduce their risk of getting sick even without a government response. We consider this to be true in every mitigation situation and treat it as a baseline. Thus, the costs of the other strategy should be considered a departure from this baseline, and not an absolute cost.

This is the strategy in which the only thing the government does is mandate mask-wearing. Everything stays open otherwise. In this case, the only economic cost is the cost of the masks per person. We estimate this to be $50, e.g., 200 disposable masks per person at $0.25 per mask. The results are insensitive to large changes in this value either up or down.

With this strategy, masks are also required, but also, large gatherings are prohibited, and some social distancing is enforced. To estimate the overall economic impact, we're using a percentage of state GDP as a baseline. Even though GDP has myriad problems as a measure of economic impact, it at least sets the correct scale. So if, for example, we assume an economic cost of 10% of GDP, we don't mean that GDP will necessarily drop by 10% of what it otherwise would be. Rather, we assume that when the direct and indirect economic costs are added up, they will sum to a value that is 10% of GDP.

We don't assume an absolute cost of a light lockdown. Rather, we set impact to be on the order of 10% for the year, and conduct sensitivity analysis from this baseline.

Heavy Lockdown is the strategy in which bars and restaurants are curbside pickup only, most indoor businesses are closed, and so on. Similar to Light Lockdown above, we do not estimate a single economic cost, but rather check it against a range of plausible inputs. We assume the cost will be on the order of 20% of state GDP.

This strategy has all of the heavy lockdown restrictions, but instead of infected people going home to infect their families, they are centrally quarantined in a facility. The cost of this would be the same as Heavy Lockdown, plus the cost of quarantining each individual. We consider a range of possible quarantine costs across an order of magnitude, from $200 to $2000 per person per day of infection for a length of time equal to the duration of the infection, with a baseline assumption of $500. For comparison, the cost of quarantining an Ebola patient in 2014 was estimated to be $1000. (Hyman, 2014) , and that included police protection and meals.

The last piece of the model is valuing liberty. We've established that the maximum value society can place on a year of one's liberty is the value of a year of life. This maximum value should correspond to the difference between the situation where when liberty is maximally curtailed (within our range of consideration) vs. when it is not curtailed at all. But what percentage of that maximum is it when one has to, for example, wear a mask when leaving the house, or cannot eat in a restaurant? A study after the fact could use a Willingness-To-Accept measure to see how much money people would be willing to accept to live under the lockdown restrictions, but in the absence of such data, we again make assumptions and test them with sensitivity analysis.

For masks, the policy is somewhat straightforward, and in some ways, the liberty impact is easier to estimate. For light and heavy lockdowns, the policies are a bit more nebulous. These strategies are highly simplified for the purpose of demonstrating the utility function. Thus, we estimate the percentage impact on liberty as a direct consequence of the strategy, which is included in the overall model results.

When wearing a mask, one can do almost anything one could do without a mask. It is a minimal infringement on liberty. Here, we consider mandatory mask-wearing to be a restriction on liberty of 2% of the maximum value. If a person valued a year of life to be worth $500,000, this would put a liberty cost of wearing a mask at $10,000. If we interpret that as a "willingnessto-accept" measure, it is a reasonable number. Results are insensitive to minor changes in this value.

Here, we assume that only shutting down large gatherings and enforcing some social distancing is a 10% restriction on liberty. As with masks, one can still do almost anything.

We assume that a heavy lockdown, in which bars and restaurants are take-out only, movie theaters are closed, and so on, is a much bigger restriction on liberty than in the light lockdown scenario. We assume that such a lockdown accounts for 50% of the maximum liberty value.

We assume that individuals who are centrally quarantined experience a 100% loss of liberty.

The data from the simulation was loaded into Tableau for visualization. This was done to create a dashboard where the parameters could be adjusted to see instant updates in the visualization. This would allow a decision team to answer "what if" questions in real-time. The dashboard was uploaded to Tableau Public and can be found here: https://public.tableau. com/profile/jack.mitcham#!/vizhome/CovidProject_15976817050220/UtilityDashboard. This decision support tool has several filters and parameter sliders to aid in sensitivity analysis. A screenshot of the tool is available in the appendix.

Before analyzing the results, we should take a moment and review the manner in which the results will be analyzed. We are interested in several questions:

• What is the spread of utility results for each strategy • Which strategy was preferred for each scenario • How each strategy performs compared to the "best" strategy Then, each of the above will be checked against each worldview. We are looking for robustness not only across scenarios but also across worldviews.

We include both utility and regret in the results. In order to provide decision-makers with qualitative insights using the quantitative model, the entire distribution of results is presented rather than relying solely on summary statistics.

What follows are the results of applying the four worldviews (Maximum Life, Maximum Liberty, Maximum Economy, and Minimum Economy) to the output of the scenarios. Furthermore, two versions of "Maximum Life" and "Minimum Economy" are presented, which shows the impact of valuing lives over life-years or vice versa.

Care should be taken not to interpret a strategy being preferred in a certain number of scenarios as a probability that the strategy will be preferred in real life. All it means is that a strategy is preferred in a larger volume of parameter space in this particular model. It says nothing about the likelihood of those parameters being correct in the real world (Fig. 5) . The following are the results with life maximized such that the value of each life lost is $10,000,000, and the value of each life-year lost is $500,000.

Next are the same graphs with the value per life-year at $200,000, but the value per life lost at $22,000,000 (Figs. 6, 7).

The Maximum Liberty worldview sets the value of each life-year lost at $50,000, the value of each life at $0, and the value of a year of liberty also at $50,000.

The Maximum Economy worldview sets the value of life to the minimum value of $50,000 per life-year, $0 per life, and the hedonic value of liberty at $0. Thus, impacts on the economy are the greatest driver (Figs. 8, 9) .

The minimum economy worldview maximizes both Life and Liberty values. First, we look at the results when both the value of each life-year and the hedonic value of liberty are maximized at $500,000, and the value of each life is $10,000,000. 

Next, we take the above information and analyze each strategy. Under what conditions does each strategy make sense for each utility function?

The "base" strategy in which the government lets the disease take its course is almost never correct. For this to be correct, we should be very sure that the R0 of COVID-19 is less than 2, and even then, most worldviews would prefer masks in a large percentage of the scenarios. The base strategy has a long tail of bad outcomes that are in the plausible range, even if we limit scenarios to those with an R0 under 2.

Even for the "maximum liberty" worldview, which almost always prefers the base strategy over mandatory mask-wearing, the regret for the mask strategy is never large, while doing nothing is a much riskier strategy, even when R0 is under 1.98 as above (Figs. 11, 12) .

For the other worldviews, the preference for other strategies is even stronger. Doing nothing is a very risky strategy in this model.

The mask strategy tends to perform well with every worldview. Interestingly, the Maximum Liberty worldview seems to like masks the best, since it represents such a small infringement on liberty while still saving lives. Even if we assume that a mask mandate is a 10% infringement on liberty, the strategy is still generally preferred to doing nothing. Fig. 7 Utility, regret, and strategy preferences for the maximum liberty worldview

The strategy of having a light lockdown is not generally preferred by any worldview assuming an economic impact of 10% of GDP. In fact, it's not preferred by any worldview with any value for economic impact greater than 0.

The general trend is that if the virus is on the lower end of severity, the Mask strategy is preferred. If the virus is severe enough, the Central Quarantine strategy is preferred. The Light Lockdown strategy is only preferred in a thin slice of the parameter space, where the virus is bad, but not too bad. And exactly where that slice is changed depending on which worldview is looking at it.

The Heavy Lockdown strategy is almost completely dominated by the Central Quarantine strategy. Heavy Lockdown is virtually never the preferred strategy. Even if we make Central Quarantine as unattractive as possible by increasing the cost per patient per day to $2,000 a day, Heavy Lockdown is virtually never preferred.

The Central Quarantine strategy is particularly interesting, because the maximum regret across all scenarios and worldviews is very low, and it provides low variance in outcomes. The economic and liberty costs of a heavy lockdown plus centrally quarantining patients are largely fixed, regardless of the properties of the virus. So, when the virus is particularly bad, Central Quarantine is seen as the best option across all worldviews, but when the virus is not bad (i.e., when you "guess wrong"), the utility cost associated with the strategy is capped. This is the opposite case for the Base scenario, in which no countermeasures are taken. If you "guess right" and the virus isn't bad, there isn't a big utility loss, but the potential downside is almost unbounded.

The Maximum Life worldview is particularly happy to use this strategy, while the Maximum Liberty worldview rarely prefers it. These results are not very sensitive to the Heavy Lockdown economic cost. The Maximum Life worldview still is very often happy with the Central Quarantine strategy even if it ends up having a much larger economic impact than 20% of GDP, while the Maximum Liberty worldview is unhappy with it even if the economic impact is a bit lower than 20%.

We also included the sensitivity analysis on valuing Life-years more highly than Lives. When maximizing the value of life and minimizing the hedonic value of liberty, the distinction largely didn't matter. The ordering of the strategies didn't change in any way. However, when using the Minimum Economy worldview, in which both life value and hedonic value of liberty are maximized, there was a difference. When maximizing the value of each life, and limiting the value of each life-year, we're also limiting the maximum hedonic value of liberty. In this case, the value of life dominates, and the preferences look more like the Maximum Life worldview than the Maximum Liberty worldview. mask mandate alone wasn't enough. The Central Quarantine strategy also had a lot of support across most worldviews in many scenarios, and in every case had the lowest maximum regret. This is because the majority of the utility loss comes from the lockdown itself, regardless of the parameters of the disease. The worst-case scenario when implementing the Central Quarantine strategy is far better than the worst-case scenario when using the mask mandate only strategy. No worldview preferred the Light Lockdown strategy in much of the parameter space. There seemed to be a thin slice of parameter space for which all four worldviews preferred the Light Lockdown strategy, but that particular slice was different for each worldview. On the other hand, no worldview routinely had Light Lockdown as the worst strategy; it was usually on the lower end of the regret measure. There are some counterintuitive results here. For example, we find that the Liberty Maximizing worldview would often prefer a light lockdown or a mask mandate over doing nothing. This is because there is a minimum value they can assign to lives and life-years. Under many sets of parameters, the life cost of doing nothing is much higher than the liberty cost of masks or a light lockdown, even if they minimize the life cost and maximize the liberty cost. This is an interesting result because there has been a widespread backlash against mask mandates. (McKelvey, 2020) However, many of the people who oppose mask mandates also believe that COVID-19 is not particularly deadly or as infectious, or that mask effectiveness is very low, or even 0. In those cases, if you limit the parameters to where COVID-19 is less dangerous than the seasonal flu and mask effectiveness is very low, the Maximum Liberty worldview prefers the strategy of doing nothing in almost every case.

This also relates to what some are calling the "Paradox of Preparation." Dr. James Hamblin said: "if shutdowns and social distancing work perfectly and are extremely effective it will seem in retrospect like they were totally unnecessary overreactions." (Hamblin, 2020) . So, some of the professed support for the "do nothing" strategy can be attributed to people seeing the observed rate of spread in the face of countermeasures and mistake that for the rate of spread in absence of those countermeasures. Additionally, this initial decision is being made at the very beginning of the pandemic when there is deep uncertainty about the parameters of the disease. In the real world, we're only reacting to one realization of the course of the pandemic out of the countless possible futures which could have occurred which needed to be accounted for in the decision-making process.

All of the above is only for the first run of the model. Robust decision-making takes place in a cycle, in which information from one run of the model can help develop new strategies to explore (step 5 of Fig. 1 ). This is similar to the Hybrid Strategies approach in Decision Analysis, first described as part of Kusnic and Owen's (1992) , "Unifying Vision process".

In this case, we note that the Central Quarantine strategy completely dominated the Heavy Lockdown strategy, in which the only difference was the isolation effectiveness. We might consider whether or not the isolation strategy could also be applied to the mask mandate strategy. Thus, keeping everything open, mandating masks, and centrally quarantining anybody found to be positive. Ideally, this would lead to an outcome that dominates a mask mandate alone across all worldviews. We might also consider whether or not a "very light lockdown" might be better. If we could identify the "low-hanging fruit" so to speak, that would have very little impact on the economy or liberty but might help with mitigating the virus. Part of the success of the mask mandate strategy was that it had an impact on the spread of the virus with little associated cost. If other countermeasures with a similar impact-to-cost ratio could be identified, they should be incorporated. Or perhaps neither of those ideas will perform as expected, and that information can be used to generate further new scenarios.

Model uncertainty is an important factor in decision-making under deep uncertainty. No single model should be used as the sole basis for a decision. Several different models should be used, with different plausible structures and assumptions. Only a perfect model can perfectly capture all of the uncertainties, and no model is perfect. A model that's a little bit wrong, or missing only one piece, can create a large deviance in the output. Thompson and Smith (2019) call this the "Hawkmoth Effect" (as a foil to the more popular "Butterfly Effect").

Ideally, this decision process would be iterated through multiple times using different models.

An important limit to this research is that many of the figures provided, e.g., for the value of life, are based on data from the United States. While the overall structure of the RDM model will remain the same, the specific parameters will vary with the decision frame in terms of economy and culture, as well as the biological properties of the pandemic. Lastly, this model aims to inform high-level policy decision-making. It may aid in political message framing to help build consensus by highlighting potential benefits that are recognized across worldviews. Further analysis might characterize constraints such as political capital and logistical feasibility.

We have both theoretical and practical contributions to the literature on robust modeling and the Covid-19 pandemic. The robust approach incorporates a SEIR epidemiological model containing variables for policy levers of interest; random simulation is used to generate a range of policy results, and these are evaluated through a linear multi-attribute utility function structured in terms that reflect the discourse about pandemic policy. Simulation model input values are varied to reflect uncertainty, while utility function parameter values are varied to reflect different worldviews. This approach provides a framework to incorporate robustness against worldview differences into quantitative modeling. Rather than relying on the personal worldview of the modeler or the decision-maker, this framework allows the decision-maker to see how robust the results are in the face of differing worldviews. This has applications outside of pandemic modeling. For example, in climate modeling (e.g., Anthoff and Tol (2014) , the value of life and the value of a square kilometer of wetlands are used in the analysis, and these values may differ from person to person. This "worldview robustness" could be an interesting extension to a lot of existing research.

Furthermore, our particular formulation of this utility function, incorporating liberty into the life-economy tradeoff, could be useful in other research. Any policy problem which includes a value of life calculation, and a possible impingement of liberty could use this utility function to aid in decision-making.

On the practical side, we define several pandemic management strategies in terms of the degree of masking, lockdown and quarantine. We evaluate the range of impacts arising from Fig. 12 Utility, regret, and strategy preferences for the maximum liberty worldview maximizing the value of Liberty, assuming that a mask mandate is a 10% infringement on liberty each strategy using each worldview, and then compare the strategies in terms of utility and regret in order not only the practical but also the political implications of different plans.

This can provide some qualitative insight into the opinion dynamics of epidemic responses. For example, we found that a mask mandate was robust against worldview differences, even to those who want to maximize liberty. This is counterintuitive because one would expect people who want to maximize liberty to not want to be forced to wear a mask. However, they may prefer to wear a mask if they were aware of the potential downsides of doing nothing, especially in the early days of the pandemic when the disease characteristics were unknown. Furthermore, as the pandemic evolves, variants of the model here may address different policy questions that arise such as those involving vaccines, as well as adding more subtle attributes of utility that are emerge in public dialogue as, for example, duration of impacts becomes salient. Thus, not only can modeling for robustness against futures and worldviews aid in decision-making but it can also aid in political communication and consensus-building. The findings here provide a template for analyzing future situations fraught with deep uncertainty and conflicting societal objectives. Data availability The data were randomly generated from the given parameters and the results are available on the Tableau Public link in the text.

Code availability See the references in the text.

The authors have no competing interests.

Climate policy under fat-tailed risk: An application of FUND

Exploratory modeling for policy analysis

Multiple criteria decision analysis: an integrated approach

The systems approach. Delta

Comparison of an agent-based model of disease propagation with the generalised SIR epidemic model. Defence Science and Technology Organisation Victoria (Australia) Air Operations Div

Uncertainty analysis comes to integrated assessment models for climate change…and conversely

Costs for a Hospital Stay for COVID-19

US anti-lockdown rallies could cause surge in Covid-19 cases, experts warn. The Guardian

Google COVID-19 Community Mobility Reports

A new analytic method for finding policy-relevant scenarios

The thing is if shutdowns and social distancing work perfectly and are extremely effective it will seem in retrospect like

The high cost of quarantine

A decision support framework for robust R&D budget allocation using machine learning and optimization

Mortality risks induced by economic expenditures

Mortality risks induced by the costs of regulations

Decisions with multiple objectives: Preferences and value trade-offs

A survey of decision making and optimization under uncertainty

The Unifying vision process: Value beyond traditional decision analysis in multiple-decision-maker environments

The Incubation Period of Coronavirus Disease 2019 (COVID-19) From publicly reported confirmed cases: Estimation and application

In Decision making under deep uncertainty

Engaging multiple worldviews with quantitative decision support: A robust decision-making demonstration using the lake model

Global stability for the SEIR model in epidemiology

The reproductive number of COVID-19 is higher compared to SARS coronavirus

Coronavirus: Why are Americans so angry about masks? BBC News. Retrieved September 6th

The economic impact of pandemic influenza in the United States: priorities for intervention

Selecting infrastructure maintenance projects with Robust Portfolio Modeling

Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis

Updating cost-effectiveness-the curious resilience of the $50,000-per-QALY threshold

Scenario development and practical decision making under uncertainty. Decision Support Systems

Quantifying individuals' trade-offs between privacy, liberty and security: The case of rail travel in the UK

A theory of justice (Original)

Long-term population projections for Massachusetts regions and municipalities

Robustness and optimality as criteria for strategic decisions

The theory of statistical decision

Policy paradox: The art of political decision making

The PySD Cookbook-PySD-Cookbook 0.1.0 documentation

Escape from model-land-Economics E-Journal

A Community Coronavirus Model for Bozeman

Revised departmental guidance on valuation of a statistical life in economic analysis

Modeling COVID-19 Nonpharmaceutical interventions: Exploring periodic NPI strategies. RAND Corporation

The value of a statistical life: A critical review of market estimates throughout the world

Sacrificing civil liberties to reduce terrorism risks

Global sensitivity analysis

Potential optimality and robust optimality in multiattribute decision analysis with incomplete information: A comparative study

Case fatality risk of influenza A (H1N1pdm09): A systematic review

Where now for saving lives? Law and Contemporary Problems

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations