untitled Erica Thompson, Roman Frigg and Casey Helgeson Expert judgment for climate change adaptation Article (Published version) (Refereed) Original citation: Thompson, Erica, Frigg, Roman and Helgeson, Casey (2016) Expert judgment for climate change adaptation. Philosophy of Science, 83 (5). pp. 1110-1121. ISSN 0031-8248 DOI: 10.1086/687942 © 2016 Philosophy of Science Association This version available at: http://eprints.lse.ac.uk/65071/ Available in LSE Research Online: November 2016 LSE has developed LSE Research Online so that users may access research output of the School. Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Users may download and/or print one copy of any article(s) in LSE Research Online to facilitate their private study or for non-commercial research. You may not engage in further distribution of the material or use it for any profit-making activities or any commercial gain. You may freely distribute the URL (http://eprints.lse.ac.uk) of the LSE Research Online website. http://dx.doi.org/10.1086/687942 http://eprints.lse.ac.uk/65071/ Expert Judgment for Climate Change Adaptation Erica Thompson, Roman Frigg, and Casey Helgeson*y Climate change adaptation is largely a local matter, and adaptation planning can benefit from local climate change projections. Such projections are typically generated by ac- cepting climate model outputs in a relatively uncritical way. We argue, based on the IPCC’s treatment of model outputs from the CMIP5 ensemble, that this approach is un- warranted and that subjective expert judgment should play a central role in the provision of local climate change projections intended to support decision-making. 1. Introduction. It is now widely accepted that global warming is real and in large part due to human activities.1 But the big picture of increasing global average temperature reveals few specifics about how climate change will im- pact human lives and natural ecosystems at a local scale. Regional climates are changing, and will continue to change, in their own unique ways. Resi- dents of one region may experience more summer droughts, while others on the same continent experience fewer. One city may face higher storm surges; another, more severe heat waves; a third, reduced monsoon rains. *To contact the authors, please write to: Erica Thompson, Centre for the Analysis of Time Series, London School of Economics and Political Science, Houghton Street, Lon- don WC2A 2AE, UK; e-mail: e.thompson@lse.ac.uk. Roman Frigg, Department of Phi- losophy Logic and Scientific Method, London School of Economics and Political Sci- ence, Houghton Street, London WC2A 2AE, UK; e-mail: r.p.frigg@lse.ac.uk. Casey Helgeson, Centre for Philosophy of Natural and Social Science, London School of Eco- nomics and Political Science, Houghton Street, London WC2A 2AE, UK; e-mail: casey .helgeson@gmail.com. yThis work was supported by the AHRC’s Managing Severe Uncertainty project (AH/ J006033/1), the Spanish Ministry of Science and Innovation (grant FFI2012-37354), and the Natural Environment Research Council (grant NE/M008304/1). 1. The evidence is presented in the most recent report from the Intergovernmental Panel on Climate Change (IPCC; Stocker et al. 2013); the consensus is documented in Oreskes (2007). Philosophy of Science, 83 (December 2016) pp. 1110–1121. 0031-8248/2016/8305-0039$10.00 Copyright 2016 by the Philosophy of Science Association. All rights reserved. 1110 To some degree, we can adapt to these changes by aligning our behavior, institutions, and infrastructure with the new reality. Farmers can plant dif- ferent crops, foresters can adjust their management plans, a water company can add a new reservoir, a city might strengthen its flood protection, centers for disease control can prepare for migrating vectors, city heath departments can build cooling centers for vulnerable residents, and insurers (and reinsur- ers) can recalculate their prices. Adapting, however, requires insight into where the local climate is headed. Is it worth investing in a new irrigation system? That depends on whether recent dry conditions are the new normal or just a streak of bad luck. Longer- term plans that are expensive to change once implemented would better be informed by projections further into the future. Construction of Britain’s new high-speed train infrastructure, for instance, is set to begin in 2017. We would like to design the tunnels to avoid overheating during late twenty- first-century summers, projected to be warmer and drier than today’s and with more intense hot weather events (High Speed Two Limited 2013). Several coastal sites in the United Kingdom are currently being evaluated for new- build nuclear power stations; the choice of site, physical design, and adap- tive management plan are sensitive to long-term projections of local sea- level rise (Wilby et al. 2011). And similar questions arise, of course, in other parts of the world. Our topic is how climate science can best inform these local adaptation decisions. We begin by reviewing a standard approach to local climate change projections (sec. 2) and considering some reasons for caution (sec. 3). An alternative emerges from a sideward look at how scientific knowledge has been incorporated into decisions on issues such as dam safety and regula- tion of toxic substances. The key idea is to introduce a process of struc- tured expert elicitation (sec. 4). We discuss specific issues that may arise in implementing such a procedure for local climate change adaptation de- cisions (sec. 5) and then draw some conclusions (sec. 6). 2. Answers Offered. Science-based information and advice that aim to in- form local adaptation decisions go by the general name climate services. The goods on offer include data sets and climate forecasts (sometimes called products), as well as guidance on how to approach and think about adaptation planning (decision support). A variety of organizations currently develop and deliver climate services, including private consultancies, non- governmental organizations, universities, and government agencies.2 While climate services products vary in what exactly they offer and in how the offerings are presented, the methodology behind them is, in broad 2. For a brief overview of the state of climate services in the United States, see the American Meteorological Society’s (2015) policy statement on climate services. CLIMATE CHANGE ADAPTATION 1111 strokes, often the same: start with output from general circulation models (GCMs), which simulate the whole climate system at a coarse resolution; then use one or more downscaling methods, which interpolate the GCM output to give finer details over the region of interest; finally, downscaled projections may be combined with local historical data. Methods diverge in their selection of GCMs, their approach to downscaling, and how they manage and communicate the uncertainties that pile up along the way. An example of such a product offered by a government agency is The United Kingdom Climate Impacts Program’s UKCP09 project. The project offers a set of quantitative projections for local variables such as the change in temperature of the warmest summer day in London or the change in total precipitation during the wettest winter season in Tewkesbury.3 These pro- jections, at the scale of a 25 km grid, can be directly accessed on UKCP09’s website in the form of probabilities for different timescales (2020s/2050s/ 2080s) and alternative emissions scenarios (high/medium/low). The projec- tions are advertised as relevant for practical decision support, and “worked examples” show how they should be used to tackle concrete problems such as energy use and sustainability in school buildings, overheating risk for buildings, and flood management policy (Met Office 2009). Projects of this kind are not restricted to the United Kingdom. A larger project called weADAPT provides small-scale information based on the CMIP5 archive of GCM runs at particular station locations and using a sta- tistical downscaling method.4 The weADAPT portal describes some in- depth case studies for use and interpretation of the data. For example, one case study looks at Bagamoyo, Tanzania, and offers an answer to the ques- tion, “What do current trends in temperatures, rainfall and mean sea level, and medium- and long-term projections, suggest are the most serious cli- mate risks that government officials in Bagamoyo district need to address?” (Besa 2013). Cal-Adapt offers adaptation information for California, Cli- mate Wizard for the entire United States, ClimateImpactsOnline for Germany, and the list could go on.5 An ever-increasing number of private consultan- cies also offer similar climate change products, usually based on publicly available data.6 In sum, questions about future local climate are, today, mostly answered by using state-of-the-art GCMs combined with downscaling. 3. “UKCP” stands for United Kingdom Climate Projections, and “09” indicates that it was launched for public use in 2009. 4. CMIP5 is the fifth instantiation of the internationally coordinated Coupled Model In- tercomparison Project, which compares the results of different GCMs. 5. For information see, respectively, California Energy Commission (2016), The Nature Conservancy (2009), and Potsdam Institute for Climate Impact Research (2014). 6. An example is JBA Consulting, North Yorkshire, UK. 1112 ERICA THOMPSON ET AL. 3. Do the Answers Provide Actionable Information?. Its popularity not- withstanding, this method has questionable credentials as a provider of ac- tionable local information. This conclusion can be reached from at least three different directions. We comment only briefly on the first two: detailed analysis of individual projects, and recognition of actual predictive failures. Our main focus here is a third path to the same conclusion: making explicit the implications of the most recent report from the IPCC. Each climate projection project is based on a set of methodological as- sumptions, and these can be subjected to scrutiny. Frigg, Smith, and Stain- forth (2013, 2015) analyze the foundational assumptions of UKCP09 and argue that the methods used do not warrant treating model outputs as decision-relevant projections. Similar in-depth analyses could be carried out for other projects. Needless to say, we can only speculate about the outcomes of such an exercise, but given the arguments we present below, we would not expect the verdicts on other projects to differ significantly. Regarding the success of predictions, Mearns (2014) reports telling re- sults from a study evaluating models for the purpose of predicting the future regional climate of North America. Mearns and colleagues compared a number of variations on the methodology described above, in each instance combin- ing one of four GCMs with one of six regional climate models (simulation- based downscaling methods). They evaluated each model pair by comparing its simulation of 1980–2005 with the weather data of that period. Two such model evaluation exercises focused on the North American Monsoon in the Southwest United States and Northern Mexico, and summer precipitation in the Northeast. A surprising result was that the model pair that performed best on the North American Monsoon performed worst on Northeast precip- itation, and few model pairs performed respectably on both. The result suggests that local climate projections are more sensitive to the choice of models than previously supposed. The problem is compounded by the difficulty of identifying any factor (or set of factors) in the models that explains the differences in performance. Furthermore, past success is not necessarily a guide to future success (Reifen and Toumi 2009), making it difficult to learn from a model’s success in reproducing past data. This sug- gests that interpreting regional model outputs as decision-relevant projec- tions is more problematic than previously assumed. In particular, it raises doubts about using any single model, or even the same ensemble or ensem- ble weighting, for general-purpose national climate projections where mul- tiple regions, variables, and timescales are of interest. Our main argument, however, is based on the most recent IPCC report.7 The report includes global mean temperature projections for 2100 with a 7. Our discussion is based on Stocker et al. (2013), secs. 12.2.3 and 12.4.1, and Sum- mary for Policymakers Table SPM.2 (see notes c and d). CLIMATE CHANGE ADAPTATION 1113 clear assessment of their reliability. They give, for instance, a projection of 2.67C–4.87C of global mean temperature rise relative to the late twentieth century, supposing the RCP8.5 scenario, which describes an illustrative “possible future” in which greenhouse gas emissions continue to rise un- abated. The IPCC quantifies uncertainty in a projection using standardized language referring to likelihood intervals, for example, “virtually certain” (>99%), “very likely” (>90%), or “more likely than not” (>50%) (Stocker et al. 2013, Box TS.1, Technical Summary). The 2.67C–4.87C projection is qualified as “likely,” which means that there is a 66%–100% chance of the real-world outcome falling into that range, given the scenario. The range 2.67C–4.87C is arrived at by looking at the CMIP5 ensemble of climate models: a set of state-of-the-art models that have each run the RCP8.5 scenario and supplied a value for the projected temperature change at 2100. This set of single values is then used to fit the parameters (mean and standard deviation) of a Gaussian probability distribution. The range 2.67C–4.87C is the interval that symmetrically spans 90% of that distribu- tion. In the “model world,” therefore, this range is an estimate of the interval into which model runs chosen from some underlying distribution will fall 90% of the time. In IPCC terminology, this interval would be referred to as a “very likely” range. The IPCC authors’ next step is to note that even state-of-the-art climate models share systematic biases (Knutti et al. 2010). All models are limited by the same computational constraints, they are based on the same limited set of parameterizations, and they are calibrated to reproduce features of twentieth-century climate that may be less relevant for twenty-first-century developments. For these reasons, when translating model-world findings into real-world projections, the IPCC authors downgrade the model-derived “very likely” uncertainty quantification for the range 2.67C–4.87C to “likely.” Thus, while there is a >90% probability that a state-of the-art model run will result in a temperature change between 2.67C and 4.87C, there is only a >66% chance—in the authors’ judgment—that the real climate system will produce this outcome. So the IPCC judges that there is a up to an additional 24% probability that the actual climate outcome (for the RCP8.5 scenario) will be outside the central range of the models and that we will experience something entirely different than what the models suggest. This is a clear indication that climate scientists do not believe the raw outputs of model ensembles to represent real-world probability distributions. The reassignment of probabilities described above is based on an infor- mal process of expert judgment: the discussions and iterations between au- thors during the long writing and review stages of the report. We commend the forthrightness with which the IPCC acknowledges the role of expert judgment in their assessment process. At the same time, we recognize some 1114 ERICA THOMPSON ET AL. drawbacks to the (relatively unstructured) way in which they go about it (details below). The main conclusion we draw from the example is that the IPCC’s own results and methods contain the assertion that there is sig- nificant uncertainty about climate projections, and that this uncertainty is best quantified by appeal to expert opinion. This conclusion should not be limited to global mean temperature. In- deed, global mean temperature is the variable in which there is (rightly) the most confidence about climate models’ ability to represent relevant pro- cesses.8 So for the local-scale variables that matter to adaptation planners, we would expect to see a probability downgrade of at least the same mag- nitude as the one given for global mean temperature. In effect the 24% rep- resents an imprecise lower bound to our uncertainty about other variables.9 The conclusion we draw from this is that the case for quantifying uncertainty through expert judgment is even stronger when it comes to local-scale var- iables, and that uncertainty in local climate projections should be assessed through a suitably structured process of expert elicitation. This conclusion stands in stark contrast with current practice, where model results are often interpreted directly as a probability distribution of future real-world cli- mate outcomes (this happens, e.g., in UKCP09). 4. Lateral Knowledge Transfer. The climate services sector faces a situ- ation in which decisions call for scientific input, but there is severe uncer- tainty about the modeling on which scientists can rest their advice. Climate change adaptation, however, is not the only field that satisfies this descrip- tion. Indeed, the situation is comparable in many policy-relevant areas of science and engineering, for example, toxicology, public health, and nu- clear safety. To illustrate what has become a standard approach to the prob- lem, we now have a look at a concrete case in environmental regulation. Our claim is that the example will illustrate important lessons for climate services. Zoning plans for sites that use or manufacture large volumes of toxic substances typically include a buffer zone in which no housing develop- ment can occur. How large should the buffer zone be? A political process can determine the acceptable level of risk for residents of neighboring de- velopments, but what distance limits the risk to this level? The Ministry of 8. See, e.g., Stocker et al. (2013, Technical Summary, 81): “Model agreement and con- fidence in projections depends on the variable and on spatial and temporal averaging, with better agreement for larger scales. Confidence is higher for temperature than for those quantities related to the water cycle or atmospheric circulation.” 9. In passing we note that levels of uncertainty are not expected to decrease significantly in the short and medium term, ever-increasing computational resources notwithstanding (Maslin and Austin 2012; Knutti and Sedlacek 2013). CLIMATE CHANGE ADAPTATION 1115 Housing, Physical Planning and Environment of the Netherlands sponsored a study to help answer this question for a handful of common industrial chemicals, including chlorine, ammonia, and hydrogen chloride (Goossens et al. 1998). The key uncertainties concerned quantities for which there are no direct empirical observations: concentrations and exposure times that are fatal for humans. The standard approach to estimating those quantities models hu- man toxicology using data from animal studies, treating the laboratory an- imal as an analog model for humans, and extrapolating concentrations and exposure times by adjusting for body weight, lung capacity, and so on. This modeling exercise delivers answers, but how much can they be trusted? Hu- mans are not 70 kg rodents, and different species may react through differ- ent physiological pathways altogether. Moreover, extrapolation results are sensitive to debatable scaling assumptions. And of course, there are no data against which to test the resulting predictions (if there were, these would have already resolved the issue). Furthermore, a sole reliance on animal studies fails to take advantage of other relevant sources of knowledge. For many chemicals, there are well-developed, qualitative toxicological models that describe metabolic processes, functional changes in the impacted organs, and clinical express- ion of these organ-function disturbances. There may also be quantitative chemical and biological understanding of their “kinetics” within the body: rates of absorption, distribution, metabolism, and elimination. This knowl- edge is relevant to estimating the required concentrations and exposure times and to judging the reliability of the extrapolations from animal stud- ies, but there is no established quantitative method of feeding everything into a hopper and turning the crank for an all-things-considered best esti- mate. To integrate the available evidence into actionable findings, the experi- menters recruited 27 toxicologists from industry, academia, and government (about five experts per chemical). The experts reviewed the available re- search and provided, through a carefully structured interview, their all- things-considered subjective judgments for the required concentrations and exposure times. The interview process, or elicitation, draws on a large body of research on the formatting and sequencing of questions to reduce overconfidence and minimize the effects of cognitive biases. In this partic- ular case, the experimenters also developed a set of seed questions (with known answers) to evaluate expert competence, and then pooled judgments mathematically, giving experts with higher marks on the seed questions greater weight, yielding a single best-estimate dose-response curve for each chemical. From this, regulatory authorities could derive the buffer zone depth that limits the risk of lethal exposure to the safety standard previously set. 1116 ERICA THOMPSON ET AL. The study just described is an example of structured expert elicitation (SEE), a family of methods designed to incorporate expert knowledge into risk management and strategic and policy decisions. SEE has spread from early applications in defense planning and aerospace engineering to wider use in the nuclear sector (Cooke 2013) and in support of regulatory and other policy decisions (Aspinall 2010; Morgan 2014), especially environmental regulation, food safety, public health, and disaster management. There are a variety of methods; some include deliberation between experts, others (like the above) elicit judgments separately and then aggregate the results, and yet others report a full range of opinion, disagreements and all (for wide-ranging reviews of applications and discussions of the method, see Cooke and Goo- ssens 2000; Goossens et al. 2008; Martini and Boumans 2014). 5. Changing Paradigm. In section 3 we argued that there are severe un- certainties in regional climate projection, and that these uncertainties are best assessed via expert judgment. Section 4 looked at an analogous case of environmental planning where SEE is used to assess uncertainties and synthesize different sources of knowledge into actionable results. Putting these two strands of argument together leads to our core suggestion: that all local climate change projections intended to support decision-making should be produced through a process of SEE. To be clear, SEE does not replace modeling exercises on this approach. Rather, the suggestion is to quit regarding downscaled GCM outputs as the only (or the privileged) source of information and to use model outputs alongside other sources of information. Experts have much to draw on, in- cluding physical understanding of large-scale processes such as the green- house effect, energy conservation principles (sometimes violated by model behavior), historical data sets, projections based on simple models and sta- tistical methods, and reconstructions of climate change in the distant past. Knowledge about model development is also essential, including under- standing of the effects of grid-scale resolution, of including or excluding particular processes (or including them only through ad hoc parameteriza- tion), of mathematical idealizations of conceptual relationships, and of the numerical approximations required to process the mathematical idealiza- tions. Simulations of the recent past have also been statistically compared to corresponding observations and plausible reasons for inaccuracies debated. The relative strengths and weaknesses of alternative model formulations are continually debated as well. All of this knowledge and understanding (and more) can inform expert judgments about local-scale climate projections. There are many variations in SEE methodology and a large literature on its theory and application. If one accepts our proposal, many questions re- main about how best to implement SEE for climate change projections, in- cluding how to pick the experts, what constitutes a reasonable quorum of CLIMATE CHANGE ADAPTATION 1117 relevant experts, and how best to incentivize participation and address con- flicts of interest. Other questions concern the mathematical formalism used to encode an expert’s judgments (e.g., precise vs. imprecise probability) and whether the individual judgments are aggregated (Genest and Zidek 1986) or presented as a group. Cooke (1991, 2014) and Aspinall (2010), for exam- ple, promote a version of the former, while Morgan (2014) generally prefers not to aggregate. We believe that no one method is right or wrong in general. The challenge is finding the right tool for the job, and this is a task for future research. Irrespective of the specifics, there are a few general issues we wish to ad- dress. One knee-jerk reaction to our proposal may be that results produced with SEE are too subjective to inform evidence-based policy. This is wrong. All climate modeling exercises involve subjective decisions that can signif- icantly influence the outcome, from the selection of models to include in an ensemble, to the range of parameter perturbations and initial conditions ex- plored in a sensitivity analysis, to the method of downscaling (not to mention the decisions made in constructing a GCM in the first place). By asking ex- perts to assess and communicate the consequences of these decisions as best as they can, considering all available information, the proposed approach is more transparent about the subjectivity, but it is not more subjective. The al- ternative to SEE is not objective calculation; the alternatives are unstruc- tured expert judgment (as in the IPCC report) or unstructured nonexpert judgment (when model-world results are communicated to policymakers ill-equipped to assess their real-world importance). Reliable objective calcu- lation of probabilities based on nonlinear model output has been argued else- where to be infeasible (Frigg et al. 2014). SEE does not guarantee trustwor- thiness, but it can compensate for (some of) the (known) inadequacies of climate models. Echoing a notorious remark of Churchill’s about democ- racy, we submit that SEE is the worst form of science advising, except for all those other forms that have been tried from time to time. Others seem to have reached a similar conclusion and called for the IPCC assessment process to incorporate SEE (Reilly et al. 2001; Oppenhei- mer et al. 2007; Shapiro et al. 2010; Yohe and Oppenheimer 2011). IPCC author teams have so far made their collective judgments through an infor- mal procedure that does little to address cognitive biases and adverse group dynamics such as deference to high-status participants. And despite the re- cent emphasis on “traceable accounts,” the expert judgment process is still largely opaque to outsiders. Standard SEE practice requires publishing the interview protocols, raw elicitation results (anonymized), and any proce- dures for facilitated deliberation and/or mathematical aggregation along with the final results. We support the incorporation of SEE into the IPCC process and recommend that the same ought to be done in the case of local climate projections. 1118 ERICA THOMPSON ET AL. Even those who agree in principle with our conclusion might worry that SEE will offer no new insights in the case of local climate change projec- tions because the outcome is in effect a foregone conclusion. We know, the objection continues, that models underestimate uncertainty and so SEE will simply downgrade all probability distributions, the only question being by how much. The worry is misguided. First, even where the need to downgrade is clear from the start, the degree is not, and the degree matters. Second, this is not the only thing that can happen. In some contexts (e.g., weather forecast- ing) models can overestimate uncertainty and experts will sharpen the prob- ability distribution. In other cases uncertainties are asymmetrical and can lead to a skewing rather than a widening of the distribution, and in yet other cases the verdict could be that the distribution is systematically biased and needs to be “moved.” Moreover, SEE need not enter the process only at the end of resource-intensive modeling projects, but should also inform their aims and methodology earlier on. This could constrain the choice of vari- ables, timescales, and spatial scales for which it is worth trying to provide decision-relevant model information. Undoubtedly, there are some such lim- its beyond which projections are simply not plausible; arguably, many cli- mate services products already exceed those limits. If experts judge large swaths of model output useless or misinformative for decision-making, some- thing has gone wrong at an earlier stage. 6. Conclusion. For the purpose of local climate change decision support, direct model outputs can be misleading. The IPCC’s published expert judg- ment about modeled global mean temperature projections implies that prob- abilities derived directly from model output on smaller-scale variables are inappropriate inputs to decision-making. Better models and faster comput- ers cannot by themselves make the problem go away, and provision of local climate change projections should instead be reconceptualized as a problem of eliciting expert judgment based on a number of sources including but not limited to model output. Many open questions remain about how best to im- plement structured expert elicitation for local climate projections. REFERENCES American Meteorological Society. 2015. “A Policy Statement of the American Meteorological So- ciety: Climate Services.” AMS Council. https://www2.ametsoc.org/ams/index.cfm/about -ams/ams-statements/statements-of-the-ams-in-force/climate-services1/. Aspinall, Willy. 2010. “A Route to More Tractable Expert Advice.” Nature 463 (7279): 294–95. Besa, Monica, C. 2013. “Using Climate Information to Support Adaptation Planning and Policy- Making: A Practical Case Study in Bagamojo District, Coastal Tanzania.” weADAPT, the Collaborative Platform on Climate Adaptation. California Energy Commission. 2016. “Precipitation: Decadal Averages Map.” Cal-Adapt. http:// cal-adapt.org/precip/decadal/. CLIMATE CHANGE ADAPTATION 1119 Cooke, Roger, and Louis Goossens. 2000. “Procedures Guide for Structured Expert Judgment.” Project Report, European Commission Nuclear Science and Technology. Cooke, Roger M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford: Oxford University Press. ———. 2013. “Uncertainty Analysis Comes to Integrated Assessment Models for Climate Change . . . and Conversely.” Climatic Change 117 (3): 467–79. ———. 2014. “Deep and Shallow Uncertainty in Messaging Climate Change.” In Safety, Reliabil- ity and Risk Analysis: Beyond the Horizon, ed. Raphael D. J. M. Steenbergen, Pieter H. A. J. M. van Gelder, Simona Miraglia, and A. C. W. M. Vrouwenvelder, 13–25. Boca Raton, FL: CRC. Frigg, Roman, Seamus Bradley, Hailiang Du, and Leonard A. Smith. 2014. “Laplace’s Demon and the Adventures of His Apprentices.” Philosophy of Science 81 (1): 31–59. Frigg, Roman, Leonard A. Smith, and David A. Stainforth. 2013. “The Myopia of Imperfect Cli- mate Models: The Case of UKCP09.” Philosophy of Science 80:886–97. ———. 2015. “An Assessment of the Foundational Assumptions in High-Resolution Climate Pro- jections: The Case of UKCP09.” Synthese 192 (12): 3979–4008. Genest, Christian, and James V. Zidek. 1986. “Combining Probability Distributions: A Critique and an Annotated Bibliography.” Statistical Science 1 (1): 114–35. Goossens, Louis H. J., Roger Cooke, Andrew R. Hale, and Ljiliana Rodić-Wiersma. 2008. “Fifteen Years of Expert Judgement at TUDelft.” Safety Science 46 (2): 234–44. Goossens, Louis H. J., Roger M. Cooke, Fred Woudenberg, and Pieter Van Der Torn. 1998. “Expert Judgement and Lethal Toxicity of Inhaled Chemicals.” Journal of Risk Research 1 (2): 117– 33. High Speed Two Limited. 2013. “London-West Midlands Environmental Statement.” Vol. 5, “Technical Appendices: Resilience to Impacts from Climate Conditions.” https://www.gov .uk/government/uploads/system/uploads/attachment_data/file/259614/Volume5_Climate _Change_Resilience_Report_CL-003_000.pdf. Knutti, Reto, Reinhard Furrer, Claudia Tebaldi, Jan Cermak, and Gerald A. Meehl. 2010. “Chal- lenges in Combining Projections from Multiple Climate Models.” Journal of Climate 23: 2739–58. Knutti, Reto, and Jan Sedlacek. 2013. “Robustness and Uncertainties in the New CMIP5 Coordi- nated Climate Model Projections.” Nature Climate Change 3:369–73. Martini, Carlo, and Marcel Boumans, eds. 2014. Experts and Consensus in Social Science. New York: Springer. Maslin, Mark, and Patrick Austin. 2012. “Uncertainty: Climate Models at Their Limit?” Nature 486 (7402): 183–84. Mearns, Linda. 2014. “Credibility of Regional Climate Model Projections of Future Climate.” Lec- ture, Knowledge and Models in Climate Science: Philosophical, Historical, and Scientific Per- spectives, University of Western Ontario, October 2014. Available at http://youtube.com /watch?vpuAM5bHH4VGo. Met Office. 2009. “Worked Examples.” UK Climate Projections. http://ukclimateprojections .metoffice.gov.uk/23102. Morgan, M. Granger. 2014. “Use (and Abuse) of Expert Elicitation in Support of Decision Making for Public Policy.” Proceedings of the National Academy of Sciences 111 (20): 7176–84. The Nature Conservancy. 2009. “ClimateWizard.” http://www.climatewizard.org/AboutUs.html. Oppenheimer, Michael, Brian C. O’Neill, Mort Webster, and Shardul Agrawala. 2007. “The Limits of Consensus.” Science 317 (5844): 1505–6. Oreskes, Naomi. 2007. “The Scientific Consensus on Climate Change: How Do We Know We’re Not Wrong?” In Climate Change: What It Means for Us, Our Children, and Our Grand- children, ed. Joseph F. C. DiMento and Pamela Doughman, 65–99. Cambridge, MA: MIT Press. Potsdam Institute for Climate Impact Research. 2014. “ClimateImpactsOnline.” http://www .climateimpactsonline.com/. Reifen, Catherine, and Ralf Toumi. 2009. “Climate Projections: Past Performance No Guarantee of Future Skill?” Geophysical Research Letters 36 (13): L13704. Reilly, John, Peter H. Stone, Chris E. Forest, Mort D. Webster, Henry D. Jacoby, and Ronald G. Prinn. 2001. “Uncertainty and Climate Change Assessments.” Science 293 (5529): 430–33. 1120 ERICA THOMPSON ET AL. Shapiro, Harold T., et al. 2010. “Climate Change Assessments: Review of the Processes and Pro- cedures of the IPCC.” Technical Report, InterAcademy Council. Stocker, Thomas F., Dahe Qin, Gian-Kasper Plattner, Melinda M. B. Tignor, Simon K. Allen, Ju- dith Boschung, Alexander Nauels, Yu Xia, Vincent Bex, and Pauline M. Midgley, eds. 2013. Climate Change 2013: The Physical Science Basis. Working Group I Contribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge: Cam- bridge University Press. Wilby, Robert, Robert J. Nicholls, Rachel Warren, Howard S. Wheater, Derek Clarke, and Richard J. Dawson. 2011. “New Nuclear Build: Adaptation Options over the Full Life-Cycle.” Pro- ceedings of Institution of Civil Engineers: Civil Engineering 164 (4): 129–36. Yohe, Gary, and Michael Oppenheimer. 2011. “Evaluation, Characterization, and Communication of Uncertainty by the Intergovernmental Panel on Climate Change—an Introductory Essay.” Climatic Change 108 (4): 629–39. CLIMATE CHANGE ADAPTATION 1121