Robustness analysis and tractability in modeling University of Groningen Robustness Analysis and Tractability in Modeling Lisciandra, Chiara Published in: European Journal for Philosophy of Science DOI: 10.1007/s13194-016-0146-0 IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below. Document Version Publisher's PDF, also known as Version of record Publication date: 2017 Link to publication in University of Groningen/UMCG research database Citation for published version (APA): Lisciandra, C. (2017). Robustness Analysis and Tractability in Modeling. European Journal for Philosophy of Science, 7(1), 79-95. https://doi.org/10.1007/s13194-016-0146-0 Copyright Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons). Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum. Download date: 06-04-2021 https://doi.org/10.1007/s13194-016-0146-0 https://research.rug.nl/en/publications/robustness-analysis-and-tractability-in-modeling(abd4213a-6f42-4ac5-b394-d641dbaa2765).html https://doi.org/10.1007/s13194-016-0146-0 Euro Jnl Phil Sci (2017) 7:79–95 DOI 10.1007/s13194-016-0146-0 ORIGINAL PAPER IN PHILOSOPHY OF SCIENCE Robustness analysis and tractability in modeling Chiara Lisciandra1 Received: 12 November 2015 / Accepted: 29 March 2016 / Published online: 3 May 2016 © The Author(s) 2016. This article is published with open access at Springerlink.com Abstract In the philosophy of science and epistemology literature, robustness anal- ysis has become an umbrella term that refers to a variety of strategies. One of the main purposes of this paper is to argue that different strategies rely on different crite- ria for justifications. More specifically, I will claim that: i) robustness analysis differs from de-idealization even though the two concepts have often been conflated in the literature; ii) the comparison of different model frameworks requires different justifi- cations than the comparison of models that differ only for the assumption under test; iii) the replacement of specific assumptions with different ones can encounter spe- cific difficulties in scientific practice. These claims will be supported by a case study in population ecology and a case study in geographical economics. Keywords Robustness analysis · Modeling · Idealizations · Tractability assumptions 1 Introduction In the philosophy of science and epistemology literature, robustness analysis has become an umbrella term that refers to a variety of strategies. Probably also due to the fact that different taxonomies have been put forward by various authors quite closely in time (Kuorikoski et al. 2010; Weisberg and Reisman 2008; Woodward 2006), a cer- tain confusion has emerged in the literature, as well as an overlap in the terminology. � Chiara Lisciandra c.lisciandra@rug.nl 1 Department of Economics, Econometrics and Finance, Faculty of Economics and Business, University of Groningen, PO Box 800, 9700, AV, Groningen, The Netherlands http://crossmark.crossref.org/dialog/?doi=10.1007/s13194-016-0146-0&domain=pdf mailto:c.lisciandra@rug.nl 80 Euro Jnl Phil Sci (2017) 7:79–95 Scientists often refer to one sense or the other without specifying their source, or without mapping their terminology to that of other scientists. More importantly, the epistemic virtues underlying different senses of robustness analysis have been often left implicit and their legitimacy as confirmatory tools remains to be clarified. It is one of the purposes of this paper to distinguish different epistemological argu- ments behind distinct uses of robustness analysis in theoretical models. In so doing, this work identifies certain gaps between the ideal characterization of robustness analysis and its application to scientific practice.1 In what follow, I will first show that robustness analysis differs from de- idealization even though the two concepts have often been conflated in the litera- ture. Secondly, that there are different justifications for using robustness analysis, according to whether it is considered as a strategy to compare different modeling frameworks or models that differ only as to the assumption being tested. Finally, that in scientific practice it can be difficult to introduce single changes in a model without altering its main structure. If robustness analysis were a ‘surgical’ opera- tion, in which controversial aspects could be replaced by different ones with no other relevant changes, then the role of a single assumption could be evaluated and the invariance of the results assessed. However, it is often the case that the intimate con- nection between simplifying assumptions and mathematical tractability is such that variations can often only be introduced by altering the overall structure of the model. This argument will be supported by a case study in population ecology and a case study in geographical economics. In what follows, possible solutions to the previous shortcomings and related difficulties will be examined. This paper is organized as follows. In Section 2, I introduce the argument for robustness analysis and present different taxonomies proposed in the literature. In Section 3, drawing on a model in population ecology, I explain how robustness anal- ysis differs from de-idealization. In Section 4, I examine the goal and the import of robustness analysis as a strategy to compare different mathematical approaches to describing the same phenomenon. In Section 5, I discuss a case study from geograph- ical economics, which reveals some possible practical difficulties in using robustness analysis. In Section 6, I conclude by pointing out some challenges that robustness analysis faces in actual scientific practice, which feature as candidate directions for future research. 2 Robustness analysis Suppose that we have a theoretical model, based on a set of initial assumptions, from which we can derive a number of predictions. If the initial assumptions are unrealistic representations of a real-world phenomenon, it is natural to ask how the predictions 1The class of models that will be analysed in this paper are abstract, theoretical models, i.e. models of phenomena which are often difficult to quantify or measure (see, e.g., Marchionni et al. 2016). Robustness analysis is particularly relevant here, as it is considered to be a strategy to test the final predictions of a model, when an empirical test is not in the hands of scientists. Euro Jnl Phil Sci (2017) 7:79–95 81 can apply to the real-world phenomenon, where these unrealistic assumptions do not hold. Intuitively, a way of proceeding is to replace the initial assumptions with slightly different ones, in order to observe whether the result holds true across con- ditions. Invariance of the result would suggest that the unrealistic assumptions were irrelevant to the final result; variation of the result would show that the predictions were not independent of the specific initial assumptions. This method of testing whether a result is invariant under different initial assump- tions is known as robustness analysis. The idea behind this strategy is that confidence in the predictions of a model increases if the predictions are invariant to small changes in the assumptions from which they are derived. In a slightly more formal notation, robustness analysis can be described as follows: we start from a model M, which consists of a core assumption C and an auxiliary assumption A1, from which the result R follows. If the same result R occurs respectively under different auxiliary assumptions A2, A3, A4, etc., we can conclude that changes in the assumptions do not influence the final result. In other words, if the result is invariant across condi- tions, we have an indication that it is the core of the model that is driving the result rather than the auxiliary assumptions (Kuorikoski et al. 2010). As an example of robustness analysis, consider Schelling’s segregation model (Schelling 1978), which describes the dynamics that lead to racial segregation within social groups.2 Schelling’s model starts from a simplified representation of its tar- get system: a checkerboard, standing for a certain metropolitan area, and dimes and pennies, standing for the individuals of two different groups, for example Blacks and Whites. The main rule of the game is that the individuals on the checkerboard move from one place to another until the composition of their neighborhood meets their preferences. As it turns out, regardless of their initial distribution in the metropolitan area, Black and White citizens will end up being clustered in different parts of the city, as a consequence of their preference for having at least half their neighbors of their own color.3 With respect to robustness, the fact that segregation is shown to follow from different initial positions provides a robust result, which does not depend on one spe- cific assumption, i.e. a particular representation of the distribution of the individuals in space. The robustness of Schelling’s result has been tested under a number of different assumptions, other than the initial position. For example, Bruch and Mare (2006) have shown that segregation occurs under different structures of neighborhood, and alternative choice functions. Muldoon et al. (2012) have shown that segrega- tion occurs even when the individuals prefer to be in the minority group of their 2Racial sorting is only one specific case of segregation. More generally, Schelling’s model applies to any situation where the individuals have preferences that tend to generate clusters within social groups, i.e different tastes, social status, sex, language, age, etc. 3Interestingly, the model predicts segregation not as a consequence of the preference of the individuals for segregation itself, but as a by-product of their preference for having some neighbors of the same ethnic group. 82 Euro Jnl Phil Sci (2017) 7:79–95 neighborhood. In the literature, alternative mathematical approaches to Schelling’s model have even been explored that are based on analytical methods instead of sim- ulations (Zhang 2004). The comparison of the results achieved via simulated models versus analytical ones is a further example of robustness analysis, this time applied across different mathematical treatments of the problem under analysis. The different senses of robustness analysis illustrated above can be traced back to a classification drawn by Weisberg and Reisman (2008), according to which: 1) parameter robustness refers to variations in the initial conditions or in the value of the parameters of the model; 2) structural robustness refers to changes in the variables included in the model; 3) representational robustness refers to modifications in the mathematical structure in which the model has been implemented, as in the example of the analytical versus the simulated version of Schelling’s model. In the context of economic modeling, a further distinction has been introduced by Kuorikoski et al. (2010), who refer to robustness analysis as a strategy to assess the role of different tractability assumptions, i.e. different mathematical formula- tions of the same factor in a model. The label tractability assumptions was originally introduced in the literature by Hindriks (2005, 2006, 2012) to indicate assumptions imposed if the problem at issue cannot be solved or is significantly more difficult to solve without them. From this brief introduction, it should already be evident that a variety of strategies have been described as robustness analysis.4,5 The attempt to connect one account with another sheds light on important differences between them that the use of a similar vocabulary has so far obscured.6 In the philosophy of science literature, robustness analysis as a confirmatory strat- egy is at the centre of a contentious debate; the core of the dispute is about whether this practice is appropriate to guide the comparison between a theory or a model and the empirical world. According to the critics (Cartwright 1991; Odenbaugh and Alexandrova 2011; Orzack and Sober 1993; Sugden 2001; Stegenga 2009) robustness 4Yet another distinction has been proposed by Woodward (2006) to take account of the variety of uses of robustness analysis in different scientific domains. He classifies four kinds of robustness: 1) infer- ential, 2) derivational, 3) measurement and 4) causal robustness, according to whether the analysis is conducted 1) to extrapolate inferential relations from the data through changes in the assumption space; 2) to derive theoretical results from models with different assumptions; 3) to measure physical entities with a family of experimental tools; or 4) to assess the stability of a generalization as invariant under interventions. 5 Note also that robustness analysis is used in other areas of science with yet a different meaning: e.g, in the study of complex systems, robustness analysis is a method of quantifying the effect of uncertainty at the level of the parameters on the final predictions; in statistics, robust estimators are those unaffected by outliers in the data. A classic reference on robustness in econometrics is Leamer (1983) and in climate sciences Parker (2011) and Pirtle (2010). In what follows, I will focus on robustness in theoretical models and not on the literature in statistics. 6For example, robustness analysis in climate models is often cited as an example of derivational robust- ness, whereas Woodward classifies it as a case of inferential robustness. For other examples, see Justus (2012). Euro Jnl Phil Sci (2017) 7:79–95 83 analysis is not a method of boosting confidence in an hypothesis. Above all, they maintain that robustness analysis is a non-experimental method of inquiry, at odds with the principles of scientific method. According to these principles, our hypothe- ses should be tested against the empirical evidence rather than against a priori reasoning. According to its advocates, on the other hand, robustness analysis can be an effec- tive guide in scientific practice (Kuorikoski et al. 2010; Weisberg and Reisman 2008). If a process or event is shown to be invariant across a range of assumptions, then sci- entists can omit the details of the problem without this undermining the final result. This turns out to be a crucial feature in all those areas of science where scientists cannot know or specify the exact configuration of the system under scrutiny. A way to reconcile the different positions is to consider the merits that each one has. In line with its critics, it can be said that robustness analysis has lower confirmatory power than empirical testing. In line with its advocates, however, robustness analysis remains the preliminary strategy to assess the results of abstract mathematical models. In this paper, I will first spell out different epistemic arguments underlying differ- ent cases of robustness analysis. In so doing, I will investigate the extent to which robustness analysis is a viable strategy to be used by scientists. In the example of Schelling’s model, it is straightforward to vary, for instance, the initial position of the agents on the checkerboard and to observe the result across conditions. Schelling’s model is often characterized in the literature as an example of a ‘toy-model’, to indi- cate a very abstract representation that idealizes away the characterizing features of the real world phenomenon. In Schelling’s case, as well as in other examples, the single parts of a model can be replaced with different ones as if they were Lego build- ing blocks. But is robustness analysis an option in scientific practice, when modelers have to deal with more complex model structures? I will show that in the case of complex models, whose components are in relation with each other partly to satisfy analytical requirements, it becomes more difficult to break them down into single units that can be exchanged with different ones. Even though this objection does not undermine the validity of robustness analysis under ideal conditions, it suggests that an evaluation of robustness analysis is still needed for those cases where the ideal conditions do not obtain. 3 Robustness analysis and de-idealization The previous section lists a number of ways in which robustness analysis has been described in the literature. What is common to a variety of cases is that robustness analysis is a strategy to increase confidence in the results of theoretical models; also, that confidence increases through changes in the assumptions of a model and obser- vation of the consequent effects. The main differences between cases are the elements that are manipulated and the logic behind their manipulation. Let us start from the question of why an assumption should be replaced with a dif- ferent one. In the philosophy of science, the literature on idealizations, abstractions, 84 Euro Jnl Phil Sci (2017) 7:79–95 approximations, simplifications etc. is now extensive but it is uncontroversial across different accounts that unrealistic assumptions are not problematic per se. With a simple example, consider negligible assumptions, i.e. assumptions that represent fac- tors that are irrelevant for the phenomenon under study (Mäki 2009, 2011). A model should not be criticised for being unrealistic with respect to negligible assumptions. A model is by definition a partial representation of the target system; what makes a model adequate, at least according to certain accounts, is that it isolates the causal mechanism that is relevant for the phenomenon under study (Mäki 2009, 2011). In its most basic sense, robustness analysis is conducted precisely to test whether the result of a model depends on the putative causes rather than on possible confounders. To do so, the assumptions are replaced by different ones, so to test that changes in the final result depend on variations in the causal factors and not on confounding factors. Note that this is the sense of robustness analysis that corresponds to what Weisberg and Reisman call parameter robustness analysis. A further reason why an assumption might require modifications is that it leaves out aspects of the target system that might be relevant for the phenomenon under study. In such a case, one way to proceed is to relax the problematic assumption so as to consider the result under a more accurate representation of the target system.7 To illustrate the case, I will refer to Weisberg and Reisman’s discussion of robust- ness analysis. In their paper The Robust Volterra Principle, they present a body of results from the Lotka-Volterra model, i.e. a population abundance model for a preda- tor species and a prey species, consisting of a pair of coupled first-order differential equations: dV dt = rV − (aV )P (1) dP dt = B(aV )P − mP (2) where V is the population of the prey, t is time, P is the population of the predator, r is the intrinsic rate of increase in prey population, a is a measure of the capture efficiency, B is a measure of conversion of the captured prey into more predator, and m is the death rate of the predator population. One of the most important properties of the Lotka-Volterra model is known as the Volterra principle, which shows that the introduction of an external cause of death in the system, such as a pesticide that equally affects the prey and the predator, determines a relative increase in the abundance of the prey population. In the Lotka-Volterra model, unrealistic assumptions are, e.g., that populations are treated as continuous, even though populations are discrete; that they grow indefi- nitely and continuously in time (see Colyvan (2013) for this and similar examples). These assumptions are said to be of the tractability kind, to indicate that they are 7Note, however, that this does not imply that the only way to validate a model is gradually to add assump- tions that better characterise the phenomenon of interest. There might be highly idealized models that are adequate on a number of grounds (see e.g. Batterman 2008; Grüne-Yanoff 2011; Knuuttila 2011; Suárez 2010). Here, however, we are considering the situation in which the replacement of an assumption is called for, because it omits aspects of the target system that should be included in the model. Euro Jnl Phil Sci (2017) 7:79–95 85 mainly adopted for reasons of mathematical tractability: without these assumptions, it would not be possible to derive the solution of the problem.8 Suppose it were not possible to quantify the error that these assumptions entail. In this case, one way to increase confidence in the validity of the model is to test the results under more realistic assumptions than the initial ones. Consider how the Lotka-Volterra model includes time as a tractability assumption in population dynam- ics. According to Colyvan and Ginzburg (2003, 72–3): ‘Our view, in fact, calls for discrete equations, where time is treated discretely. These equations, however, are notoriously hard to deal with. We therefore continue to use differential equations, but we bear in mind that these are idealizations of the underlying finite, discrete-time model’. Here, not only is it expected that both mathematical treatments, i.e. discrete and differential equations, would provide results that are largely in agreement, if differ- ential equations are a good enough representation of the phenomenon under analysis; more importantly, confidence in the validity of results increases once a more accurate mathematical treatment is provided for the system under analysis. When it is so, then the more accurate formulation becomes the benchmark by which to assess the valid- ity of the results; it is not the invariance of the result that increases confirmation of the result itself. This is because, if the results were not consistent, this would reveal certain problematic aspects of the original assumption. But the replacement of a particular assumption with a more realistic one is based on different epistemological criteria than the standard argument for robustness anal- ysis. When we replace an unrealistic assumption with a more realistic assumption, what we are doing is known in the literature as the de-idealization of a model. This method has been largely discussed in the philosophy of science and in epistemology and it raises questions that are tangential to robustness analysis (see e.g. Batterman 2008; Cartwright 2006; Mäki 2011; McMullin 1985; Odenbaugh and Alexandrova 2011). Examples of such questions are, how de-idealization should proceed, or what is the appropriate level of idealization according to the different nature and aims of the inquiry. Robustness analysis is a different strategy from de-idealization. In robustness anal- ysis, it is because different assumptions, none of which is more realistic than another, all determine the same result, that we claim that the unrealistic aspects of the assump- tions do not compromise the validity of the model. This is the sense in which ‘our truth is at the intersection of independent lies’, as the famous biologist Richard Levins affirmed when introducing the notion of robustness analysis in the literature (Levins 1966, 423). When we replace a tractability assumption with a more realistic one, we 8As with the concept of robustness analysis, so also ‘tractability assumption’ is a heavily loaded term. According to Colyvan 2013, the effect of the assumptions adopted for reasons of mathematical tractability is often negligible; for instance, nobody would criticize the result of the Lotka-Volterra model because it relies on continuous populations. Hindriks (2006), on the other hand, regards tractability assumptions as those whose effects are presumably non-negligible for the final result. In turn, Mäki (2011) has questioned the non-negligible features attributed to tractability assumptions. Overall, I will use the term in its general sense to indicate assumptions that omit aspects of the target system in order to facilitate mathematical tractability. 86 Euro Jnl Phil Sci (2017) 7:79–95 are not working within a network of models or assumptions, each of which controls for different aspects of the problem under consideration. Here, our truth is neither at the intersection of different lies, nor are these lies independent. Levins’ idea refers instead to a situation in which we have a collection of models, which either stand or fall together, since each of them tackles specific aspects of the problem under analysis. Despite its different underlying justifications, robustness analysis has often been referred to as a strategy to increase confidence in the validity of a model by showing that the result is invariant under more realistic representations of the system under analysis. Weisberg and Reisman’s notion of structural robustness reflects this intent, as exemplified by the analysis of Schelling’s result against refined utility functions, or of the Volterra principle under the density dependence parameter (Weisberg and Reisman 2008). A further example comes from evolutionary game theory. Here, a standard objec- tion to the validity of certain results about the emergence of cooperative behaviors is based on their alleged lack of robustness with respect to the individuals’ cog- nitive constraints (Skyrms 1996; Sugden 1986; D’Arms et al. 1998). A limitation on the kind of possible strategies that can be transmitted across generations is the cognitive load they impose on individuals; thus, a result will not be consid- ered significant if it is not robust under a model that takes these limitations into account. In the above cases, confidence in the validity of the result increases as more realistic models are adopted. Robustness analysis, by converse, is not a process of ‘concretization’ of the model. In robustness analysis, unrealistic assumptions are replaced by other unrealistic assumptions in order to test the extent to which the final result of the model depends on them. On the one hand, this implies that the possibil- ity cannot be ruled out that the further unrealistic assumptions might be affecting the final result. On the other hand, however, in abstract mathematical models, both from physics and economics, it is often the case that the level of theoretical abstraction is such that it is difficult to assess their validity in terms of the accuracy with which they represent the target system. Credible and unrealistic aspects are intertwined with one another to an extent that makes it inappropriate to talk about de-idealisation when replacing any of them with different ones. In these circumstances, robustness analysis does not deal with the realism or truth of the assumptions. In these cases, the under- lying idea is that if a result is invariant across conditions, then the result does not strictly depend on the particular way in which the assumptions represent the target system, and thus on their falsifications. Regardless of the position that one takes on the argument for robustness analy- sis, conflating it with de-idealization creates a terminological as well as a conceptual confusion. Terminologically, if robustness analysis is taken as a synonym for de- idealization, the original definition by Levins no longer applies. Conceptually, the overlap of the two notions obscures the fact that the alleged confirmatory power of the two methods relies on different grounds. In the next section, by looking at an example of how scientists conduct robustness analysis, we unravel some of the philosophical confusion but at the same time some practical problems come to light. Euro Jnl Phil Sci (2017) 7:79–95 87 4 Across-models robustness analysis In the previous section, I introduced the problem of tractability assumptions in the Lotka-Volterra model. Ideally, if there are no other ways to justify their adoption, tractability assumptions should be replaced by more realistic ones. However, de- idealization is often not an easy matter. In the Lotka-Volterra model, mathematical assumptions are adopted precisely because it is not clear how to proceed otherwise. In these circumstances, it is also difficult to replace them with other tractability assump- tions. Continuous populations, infinite populations and continuous time require to be exchanged with assumptions such as discrete or finite populations, or discrete time. A different way to proceed in these cases is by comparing models that differ from one another along multiple lines. This type of robustness analysis corresponds to what Weisberg and Reisman (2008) define as representational robustness analysis, i.e. a test of the invariance of predictions across different mathematical approaches. With respect to representational robustness, the predator-prey interaction has been analyzed both via differential equations and computational simulations and it has been shown that the Volterra principle holds in both cases. By deploying different modeling frameworks, i.e. a population-level model (differential equations) and an individual-based model (simulations), Weisberg and Reisman (2008) compare two different mathematical approaches for the analysis of predator-prey interaction. Let us consider in more detail the purpose of this comparison. In biology-related disciplines, individual-based models are becoming increasingly common despite the lack of analytical results. This is mainly because the degree of specificity they enable scientists to achieve is higher than that achievable via previous standard analytic treatments, such as differential equations as in the Lotka-Volterra model. The question for robustness analysis is what to expect from a comparison of the results from the population-level and the individual-based models. Are the assumptions of one mathematical framework being tested, using another framework that does not take the same assumptions into account? What exactly is it, that is being compared across cases? Consider again a mathematical assumption in the Lotka-Volterra model, such as that populations are continuous, not discrete. Is the individual-based model testing the effect of this assumption on the result? Strictly speaking, the effect of the continuous populations assumption in the Lotka-Volterra model can be tested when adopting the assumption of discrete populations, which is possible once the original tractability problem has been solved. On the one hand, the fact that an individual-based model which is based on dis- crete populations gives the same result as the Lotka-Volterra model is an indication that the Volterra principle can also be derived under the assumption of discrete popu- lations. On the other hand, however, when translating the Lotka-Volterra model into an individual-based model, many aspects of the initial model change. These changes come within an entirely new modeling ‘package’, whose assumptions will have to be tested in turn. Note that the more aspects have been changed, the further we are from analyzing the effect of one specific assumption. The above claim can be illustrated with an example from Schelling’s model. Suppose that a modeler were interested in testing whether two different network 88 Euro Jnl Phil Sci (2017) 7:79–95 structures have different impacts on segregation. The two network structures differ from one another in the number of neighbors that the individuals take into account when making their decision to move on the checkerboard. Suppose that, apart from the network structures, the two models were alike. By simulation, the result proves to be invariant across conditions, which indicates that the differences between the two assumptions do not have relevant effects on the final result. When the comparison is between an individual-based model and a population- based model we are in a different situation. Here we are not testing the effect of one single change in the assumptions. In this case, we are comparing two models that differ not only in the assumption that we wanted to test, but in that one plus many others. This is because–at least at the moment–we are only able to include discrete populations within a entirely different modeling structure. This means that, whenever we are testing the invariance of results across conditions, we always test the original tractability assumption plus a number of other assumptions that are implied by the new one. Hence, whether or not the results are in agreement, we cannot conclude that this provides an indication about the role of the original tractability assumption. The result has to be taken as determined by the model as a whole, not as a case of robustness analysis where a single or a few assumptions have been replaced with a different one to assess their impact on the result. An objection to the above claim is that it does not really matter that several ele- ments change from one model to another, if the result is invariant across conditions. In other words, if two models differ in many respects, and still the result is invariant, this provides an even stronger indication of the validity of the result, regardless of whether the target system is more accurately represented by one model or the other. Notice, however, that the argument just outlined is grounded on other considera- tions than the standard argument for robustness analysis. If we take the comparison of entirely different whole models as an instance of robustness analysis, then the con- firmatory power of this strategy no longer derives from what it has hitherto claimed to be (Kuorikoski et al. 2010; Levins 1966; Odenbaugh 2011). Robustness analysis has been described as a practice of building models of the same phenomenon, which differ slightly from one another, so as to identify which assumptions are necessary for deriving the final result. This is done on the basis that the results that are robust across conditions depend on the shared, rather than on the different assumptions. According to Lehtinen (2016, 2): ‘If a result is robust, only the assumptions that over- lap between the models could be needed for its derivation, and the other assumptions are thus dispensable.’ This is not the case for the simulated version of the Lotka-Volterra model. Here, Weisberg and Reisman (2008) had to introduce new factors, such as a density depen- dence parameter, in order to get results, which were only comparable with the population-based model. In fact, a situation in which very different models provide the same result is quite a fortunate case, probably an exception in science. At that point, the problem becomes that of assessing which result is more accurate on the basis of the different merits of each model. The example above illustrates in simple terms a problem under discussion among scientists working with complex simulation models. In the assessments of climate sciences models, we find that experts are cautious about the possibility of comparing Euro Jnl Phil Sci (2017) 7:79–95 89 the results of models that differ from one another in a number of different elements. According to Parker (2006, 350): ‘Complex climate models generally are physically incompatible with one another–they represent the physical processes acting in the climate system in mutually incompatible ways’. According to Lenhard and Winsberg (2010, 258): ‘The complexity of interaction between the modules of the simula- tion is so severe that it becomes impossible to independently assess the merits or shortcomings of each submodel’. In conclusion, the question of how to compare substantially different models dif- fers from robustness analysis defined as a method of testing the effect of controversial assumptions by replacing them with single different ones. The comparison of dif- ferent modeling frameworks needs further investigation and the subject opens new challenges that are already attracting the efforts and the attention of scientists, chal- lenges that are however different from the comparison of models that differ only with respect to the assumption under analysis. In these cases, the differences between models are several and such that it is not clear how to map the different components with one another. When it is so, is it an open question whether, and on what grounds, the robust results are mutually supporting each other. In the next section, I will return to the initial problem of how to replace a particular assumption in isolation, this time with a case study in geographical economics. 5 Robustness analysis and tractability assumptions In the previous sections, I have first shown that robustness analysis differs from de-idealizations; and then, that when the replacement of a particular assumption in isolation is not a viable option, an underdetermination problem occurs concerning how to compare models that differ from one another along multiple lines. In this section, I will present a case-study from geographical economics which again pro- vides insights into the actual process of model manipulation and into the possibility of changing single assumptions in isolation. In the literature on robustness analysis, a paradigmatic case study comes from the literature in geographical economics (Kuorikoski et al. 2010; McCann 2005; Neary 2001). Geographical economics is at the centre of a debate between economists and philosophers of science precisely because of the tractability assumptions on which it is based (see below). Broadly speaking, geographical economics is a sub-field of economics that studies the relation between economic activity and spatial location. The model at the centre of the debate is known as the Core-Periphery model; it was formulated by Paul Krugman in 1991 and earned him the Nobel prize for economics in 2008. The Core-Periphery model investigates the conditions under which an economic activity agglomerates in a certain region (the core), as against the conditions under which it disperses (the periphery). Various factors influence this process. The forces affecting geographical concentration depend on the advantages of being in a region with good access to the market as against the advantages of being in a region where competition is lower and there is no risk of market congestion. A key factor is the cost of transporting goods from the place of production to that of delivery. The higher 90 Euro Jnl Phil Sci (2017) 7:79–95 the transportation costs, the nearer the economic activities to the place of demand and, contrariwise, the lower the transportation costs the farther the economic activity from the centre. In the history of geographical economics, Krugman’s contribution was crucial in determining a paradigm shift from previous theories of international trade, which were based on tariff costs. The advancement in the field was attained thanks to the introduction of an ‘iceberg’ costs function, which is so called because it is based on the principle that part of the goods ‘melts away’ when transferred from the place of production to the place of delivery. Even though the ‘iceberg’ formulation is obviously a theoretical construct, i.e. it is not based on direct observation, still it is considered to be appropriate mainly for two reasons: first, it reflects the idea that goods are costly to transport; secondly, it enables the formulation of transport factors not as a separate component of the model but as part of the goods themselves. This is the sense in which the ‘iceberg’ cost function is a tractability assumption. Since it would be problematic to introduce additional factors into the model to account explicitly for the diminishing value of the goods, the mathematical trick is to do as if a lesser amount of goods would arrive at their destination. In the words of Krugman: ‘In terms of modelling convenience, there turns out to be a spectacular synergy between [...] market structure and ‘iceberg’ transport costs: not only can one avoid the need to model an additional industry, but because the transport cost between any two locations is always a constant fraction of the free-on-board price, the constant elasticity of demand is preserved’ (Krugman 1998, 11). In the geographical economics literature, some of the features of the Core- Periphery model are a matter of debate. According to Fingleton and McCann (2007, 168) for instance: “Geography enters these [economic geography] models specif- ically and only via the Krugman adaptation of the Samuelson (1952) model, the properties of which are implausible and counter to most observed evidence.” In response to this and other difficulties, in subsequent formulations of the Core- Periphery model, geographical economists have tried to measure how sensitive the predictions are to the ‘iceberg’ assumption. To do so, the attempt has been made to test the results under different functions that do not show the same problematic aspects. The main problem is that the functions that differ from the ‘iceberg’ func- tion, by not showing the same contended properties, are difficult to implement in the Core-Periphery model, which is what a test of robustness would require. One of the most controversial aspects of the Core-Periphery model is the convexity of the price function. The convexity of the price function is derived from the way in which price, value and quantity of goods are defined in geographical economics and combine together in the Core-Periphery model. It is not guaranteed that if a certain feature such as ‘price increases convexly with distance’ needs to be tested, then a model can be built where the feature ‘price is concave with distance’ is introduced while all the rest remains as before. The convexity of transportation costs follows from the mathematics of the model as a whole. In fact, one of the reasons why the ‘iceberg’ cost function was initially intro- duced was indeed to enable the mathematical tractability of a certain problem. Euro Jnl Phil Sci (2017) 7:79–95 91 This assumption accommodates the analytical requirements of the model, such as increasing returns to scale, imperfect competition and constant elasticity of substi- tution. Because of the very features of this assumption, it is particularly difficult to replace it with a different one, and leave the rest of the model untouched. A further consideration is that a model that differed from the initial one, by show- ing concave price with distance, would not maintain the crucial properties of the Core-Periphery model, i.e concentration of the economic activity in the Core versus dispersion in the Periphery. This is because high transportation costs exert a coun- terbalance to agglomeration, which is crucial for the interplay of centrifugal and centripetal forces in equilibrium formation (McCann 2005). A different strategy would be to prove that other theoretical frameworks, not based on ‘iceberg’ transportation costs, produce similar results to those of the Core- Periphery model, thereby providing independent evidence. However, according to McCann: ‘It is almost impossible to provide direct comparisons between models with the ‘iceberg’ assumption and those with other sets of transport costs assumptions embedded in them. [...] This is because these more traditional transport costs func- tions are analytically incompatible with new-economic geography models’ (McCann 2005, 312).9 Theories in international trade that differ from the geographical economic model have not been as successful as the Core-Periphery model in terms of equilibrium analysis, so that at the moment there is no theoretical alternative available, with which the results of the Core-Periphery model can be compared. This brings us back to the problem discussed in the previous section: when the results of different models embedded in different mathematical frameworks are tested against one another, a way has to be found to map their constituents for the comparison of the results to be meaningful. In line with this analysis, the reaction of the scientific community to the shortcom- ings of the ‘iceberg’ cost function was indeed an effort to build models that were not based on the same problematic assumption. For instance, according to Isard: ‘The first advance [in space economy] would involve dropping the iceberg assumption regarding transport cost.’ (Isard 1999, 383). Also, in 2009, the World Bank Devel- opment Report was dedicated to geographical economics and there we find: ‘By using techniques that essentially assumed away the internal workings of transport [...] the more critical policy-related aspects also have been assumed away.’ (World Bank Report 2009, 185). These remarks are to show that the response of economists to the shortcomings of the model was to make progress on how to avoid tractability assumptions in the first place. 9 Note that there is also a conceptual difference between geographical economics and the theories of international trade. Theories of international trade are said to be aspacial, insofar as the effect of distance only comes into play as a tariff cost. Models in geographical economics, by contrast, include geography as a component that determines increasing costs in the goods traded. This also explains why the two frameworks do not have corresponding elements that can be clearly referred one to another. 92 Euro Jnl Phil Sci (2017) 7:79–95 The case-study discussed above shows a particular way in which robustness anal- ysis can go wrong. A single counterexample does not undermine robustness analysis across the board. The strategy can still be successful in evaluating models that differ in some specific aspects from one another. The question is to what extent the very concept of tractability assumptions poses a limit to the possibility of replacing them in isolation. This depends on how we interpret the concept of tractability assump- tions. If the set of tractability assumptions is stretched so far as to include all kinds of assumptions, insofar as they represent factors in a way that has to be tractable in some sense, then the problems highlighted above are restricted to some extreme cases. However, if there is something specific about tractability assumptions, in that they have a particular mathematical role in a model, as in the geographical economics case or in the Lotka-Volterra example, then their replacement with specific differ- ent ones might be problematic for the very reasons why they have been used at the outset. Consider for instance how Colyvan defines this kind of assumptions: “These idealisations are usually invoked in order to employ familiar and well-understood mathematical machinery.” (2013, 1339). Implied in this statement is that the reason why we do not use a different mathematical machinery is that it is not as well under- stood as the one that we use. Or, consider how Morrison discusses mathematical abstractions: “In situations like this where we have mathematical abstractions that are necessary for arriving at a certain result there is no question of relaxing or correcting the assumptions in the way we de-idealize cases like frictionless planes and so on; the abstractions are what make the model work.” (2009, 110). Here, the replacement of a mathematical assumption is not even considered as a possibility. Overall, several different accounts have been put forward in the literature con- cerning the status of tractability assumptions and their role in a model. There is a continuum of cases that ranges from assumptions considered to be innocuous, to assumptions that cannot be relaxed, to assumptions that are assumed in spite of their unrealistic features. The claim defended here does not hinge on the peculiarities of any specific position. Across cases, the possibility has to be considered that, if we conceive of models as systems of inter-connected parts, then it is reasonable to expect that changes in certain aspects of the model will in turn determine further changes in other aspects of the model, in a chain of related effects. This is especially the case for assumptions introduced partly for the purpose of satisfying certain analytical require- ments dictated by the formal structure of the model. The adoption of this kind of assumptions turns out to play a crucial part in a variety of cases of model building. Thus, it is particularly urgent to think of novel criteria for the assessment of models that rely on their use. 6 Conclusion The aim of this paper was to investigate the epistemic goal of robustness analysis in theoretical models and to spell out the details of the procedure in scientific practice. In philosophy of science, robustness analysis is defended as a method of testing the invariance of a model’s results under different assumptions. As argued in this paper, several different reasons underlie the replacement of an assumption with a different Euro Jnl Phil Sci (2017) 7:79–95 93 one. One reason is that an assumption represents a possible confounding factor; by changing it, a modeler tests whether the result depends on the mechanism identified as responsible for the phenomenon and not on possible confounders. Another rea- son is that an assumption omits aspects of the target system that might be relevant for the phenomenon under study. In this case, a possible strategy is to replace that assumption with a different one, so as to assess the results across conditions. In this paper, two examples have been presented–one in population ecology and one in geo- graphical economics–in which the replacement of an assumption with a different one requires to change the model in more than the aspect under test. Comparing substan- tially different models, however, is based on other considerations than comparing models that differ only with respect to a few assumptions. Just as different experi- mental practices might lead to different results, thereby raising the question of how to interpret these results (Stegenga 2009), the same is true of predictions deriving from models with different initial assumptions. A view on model validation that is becoming prominent in the philosophy of science literature maintains that families of models, rather than single models, should be used as a basis for assessment of the final results (Knuuttila 2011; Muldoon 2007; Wimsatt 2007). However, the standard argument for robustness analysis does not necessarily apply to these situations, thus making this an area of research where philosophical work is particularly needed. The problem of how to compare results deriving from structurally different models is one of the most interesting questions that the debate on robustness analysis has opened to today’s scientific practice and promising works are expected to come from this research area in the near future. Acknowledgments Thanks goes to Mark Colyvan, Francesco Guala, Frank Hindriks, and to the partic- ipants of the workshop on robustness analysis that was held at the University of Helsinki in September 2014, i.e. Alessandra Basso, Lorenzo Casini, Jaakko Kuorikoski, Aki Lehtinen, Caterina Marchionni, Cedric Paternotte, Jonah Schupbach, Kent Staley and Jacob Stegenga. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, dis- tribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. References Batterman, R.W. (2008). Idealization and modeling. Synthese, 169, 427–46. Bruch, E.E., & Mare, R. (2006). Neighborhood choice and neighborhood change. American Journal of Sociology, 112, 667–609. Cartwright, N. (1991). Replicability, reproducibility, and robustness - comments on harry collins. History of Political Economics, 23, 143–155. Cartwright, N. (2006). The Vanity of Rigour in Economics: Theoretical Models and Galilean Experiments, In P. Fontaine, & R.J. Leonard (Eds.), The ‘Experiment’ in the History of Economics (pp. 135–153). London: Routledge. Colyvan, M., & Ginzburg, L. (2003). The galilean turn in population ecology. Biology and Philosophy, 18, 401–414. Colyvan, M. (2013). Idealisations in normative models. Synthese, 190, 1337–1350. http://creativecommons.org/licenses/by/4.0/ 94 Euro Jnl Phil Sci (2017) 7:79–95 D’Arms, J., Batterman, R., & Górny, K. (1998). Game theoretic explanations and the evolution of justice. Philosophy of Science, 65, 76–102. Fingleton, B., & McCann, P. (2007). Sinking the Iceberg? On the Treatment of Transport Costs in New Economic Geography, In B. Fingleton (Ed.), New Directions in Economic Geography (pp. 168–204): Edward Elgar Publishing. Grüne-Yanoff, T. (2011). Isolation is not characteristic of models. International Studies in the Philosophy of Science, 25, 1–19. Hindriks, F. (2005). Unobservability, tractability, and the battle of assumptions. Journal of Economic Methodology, 12, 383–406. Hindriks, F. (2006). Tractability Assumptions and the Musgrave-Mäki Typology. Journal of Economic Methodology, 13, 401–423. Hindriks, F. (2012). Saving Truth for Economics, In A. Lehtinen, J. Kuorikoski, & P. Ylikoski (Eds.), Economics for Real (pp. 43–64). London: Routledge. Knuuttila, T. (2011). Modelling and Representing: An Artefactual Approach to Model-based Representa- tion. Studies in History and Philosophy of Science, 42, 262–271. Krugman, P. (1991). Increasing returns and economic geography. Journal of Political Economics, 99, 483–499. Krugman, P. (1998). What’s new about the new economic geography. Oxford Review of Economic Policy, 14, 7–17. Kuorikoski, J., Lehtinen, A., & Marchionni, C. (2010). Economic modelling as robustness analysis. British Journal for the Philosophy of Science, 61, 541–567. Isard, W. (1999). Further thoughts on future directions for regional science: a response to fujita’s remarks on the general theory of location and Space-Economy. The Annals of Regional Science, 33, 383–388. Justus, J. (2012). The elusive basis of inferential robustness. Philosophy of Science, 79, 795–807. Leamer, E. (1983). Let’s take the con out of econometrics. American Economic Review, 73, 31–44. Lenhard, J., & Winsberg, E. (2010). Holism, entrenchment, and the future of climate model pluralism. Studies in History and Philosophy of Modern Physics, 41, 251–262. Lehtinen, A. (2016). Allocating Confirmation with Derivational Robustness. Philosophical Studies. doi:10.1007/s11098-016-0625-3. Levins, R. (1966). The strategy of model building in population biology. American Scientist, 54, 421–431. Mäki, U. (2009). MISSing the World: Models as Isolations, Representations and Credible Worlds. Erkenntnis, 70, 29–43. Mäki, U. (2011). The Truth of False Idealizations in Modeling, In P. Humphreys, & C. Imbert (Eds.), Representations, Models and Simulations (pp. 216–233). London: Routledge. Marchionni, C., A. Basso, & C. Lisciandra (2016). Hypothetical Models in Social Science: their Features and Uses, In L. Magnani, & T. Bertolotti (Eds.), Springer Handbook of Model-Based Science. McCann, P. (2005). Transport costs and new economic geography. Journal of Economic Geography, 5, 305–318. McMullin, E. (1985). Galilean idealization. Studies in the History and Philosophy of Science, 16, 247–273. Muldoon, R. (2007). Robust simulations. Philosophy of Science, 74, 873–883. Muldoon, R., Smith, T., & Weisberg, M. (2012). Segregation that no one seeks. Philosophy of Science, 79, 38–62. Neary, P. (2001). Of hype and hyperbolas: Introducing the new economic geography. Journal of Economic Literature, 39, 536–561. Odenbaugh, J. (2011). True lies: Realism, Robustness and Models. Philosophy of Science, 78, 1177–88. Odenbaugh, J., & Alexandrova, A. (2011). Buyer beware: Robustness analyses in economics and biology. Biology and Philosophy, 26, 757–771. Orzack, S.H., & Sober, E. (1993). A critical assessment of levins’s the strategy of model building in population biology (1966). The Quarterly Review of Biology, 68, 533–546. Parker, W. (2006). Understanding pluralism in climate modeling. Foundations of Science, 11, 349–368. Parker, W. (2011). When climate models agree: The significance of robust model predictions. Philosophy of Science, 78, 579–600. Pirtle, Z., Meyer, R., & Hamilton, A. (2010). What does it mean when climate models agree? a case for assessing independence among general circulation models. Environmental Science and Policy, 13, 351–361. Samuelson, P. (1952). The transfer problem and transport costs: The terms of trade when impediments are absent. Economic Journal, 62, 278–304. http://dx.doi.org/10.1007/s11098-016-0625-3 Euro Jnl Phil Sci (2017) 7:79–95 95 Schelling, T. (1978). Micromotives and macrobehavior. New york: W.W. Norton. Skyrms, B. (1996). The evolution of the social contract. New York: Cambridge University Press. Stegenga, J. (2009). Robustness, discordance, and relevance. Philosophy of Science, 76, 650–661. Suárez, M. (2010). Scientific representation. Philosophy Compass, 5, 91–101. Sugden, R. (1986). The economics of rights cooperation and welfare. Oxford: Blackwell. Sugden, R. (2001). Credible worlds: the status of theoretical models in economics. Journal of Economic Methodology, 7, 1–31. Weisberg, M., & Reisman, K. (2008). The robust volterra principle. Philosophy of Science, 75, 106–131. Wimsatt, W. (2007). Re-engineering Philosophy for Limited Beings: Piecewise Approximations: Harvard University Press. Woodward, J. (2006). Some varieties of robustness. Journal of Economic Methodology, 13, 219–240. World Bank. (2009). World Development Report 2009: Reshaping economic geography. Washington: The World Bank. Zhang, J. (2004). A dynamic model of residential segregation. Journal of Mathematical Sociology, 28, 147–170. Robustness analysis and tractability in modeling Abstract Introduction Robustness analysis Robustness analysis and de-idealization Across-models robustness analysis Robustness analysis and tractability assumptions Conclusion Acknowledgments Open Access References