key: cord-0475747-ulbd6d8a authors: Yang, Vicky Chuqiao; Sandberg, Anders title: Collective Intelligence as Infrastructure for Reducing Broad Global Catastrophic Risks date: 2022-05-06 journal: nan DOI: nan sha: c4c17248cacf2cefb60fca485c5cbb6caff70d50 doc_id: 475747 cord_uid: ulbd6d8a Academic and philanthropic communities have grown increasingly concerned with global catastrophic risks (GCRs), including artificial intelligence safety, pandemics, biosecurity, and nuclear war. Outcomes of many risk situations hinge on the performance of human groups, such as whether democratic governments and scientific communities can work effectively. We propose to think about these issues as Collective Intelligence (CI) problems -- of how to process distributed information effectively. CI is a transdisciplinary perspective, whose application involves humans and animal groups, markets, robotic swarms, collections of neurons, and other distributed systems. In this article, we argue that improving CI can improve general resilience against a wide variety of risks. Given the priority of GCR mitigation, CI research can benefit from developing concrete, practical applications to global risks. GCR researchers can benefit from engaging more with behavioral sciences. Behavioral researchers can benefit from recognizing an opportunity to impact critical social issues by engaging with these transdisciplinary efforts. Recent years have witnessed increasing concerns for humanity's survival and prosperity in the long-term future. Scholars are increasingly concerned about global catastrophic events. Examples of such events include nuclear wars, climate catastrophes leading to large-scale failures in agriculture, pandemics, and powerful artificial intelligence (AI) not aligned with human values (see Ord, 2020 for a summary). Other scholars are also concerned with future technological innovations that enable a single individual to have great destructive power, such as DIY biohacking tools that can kill millions (Bostrom, 2019) . Scholars studying these potentially catastrophic events and how to mitigate them have formed a transdisciplinary field called global catastrophic risks (GCR), and a subset focused on events that can lead to human extinction is referred to as existential risks. An associated, more practice-focused field called effective altruism investigates how an individual can do the most good to help others. These areas are not only of concern to academics but have also attracted considerable attention and action in philanthropic efforts. Discussion in GCR has been centered around which risks are likely to occur and how to circumvent each one. A critical component crosscuts a wide number of risks but remains less explored-GCR reduction hinges on the better collective performance of human groups. For example, mitigating global pandemic risks include coordinating individual lifestyle changes, the research efforts of the scientific community, and policies of nation-states. Our ability for effective collective decision-making and collective action are infrastructures for responding to a wide variety of, if not all, GCRs. The vulnerability of having a weak infrastructure, such as in collective decision-making and collective action, reduces the general stability of the system. Such a system can be compared to a tightrope walker-many forces can lead them to fall, be it a gust of wind, or a shake of the body. Compared to focusing on which forces will tip the tightrope walker over, and preventing these forces from happening, it is more useful to identify tools that can help improve the general stability of the system. For example, adding a balancing pole (Fig 1) , which increases the toques needed that lead to a fall, expands the walker's ability to adjust their center of gravity through small pole movements. Similarly, in GCR mitigation, it is important to consider building general tools that improve a (Heukeshoven, 1946) . The balance pole used by tightrope walkers serves as a device for improving the ability to adaptively balance and adjust to perturbations. The role of Collective Intelligence in Global Catastrophic Risk mitigation is similar to that of the balance pole to a tightrope walker-but performed by a multitude of actors. system's ability for coordination and collective decision-making. This view is echoed in a perspective piece for collective behavior to be considered a "crisis discipline" (Bak-Coleman et al., 2021) . While the mitigation efforts center on resolving technological challenges, a significant minority of the GCR community is concerned with human collective decision and action (Liu, Lauta, & Mass 2018) , and typically frame them as coordination problems that can be approached in a game-theoretic framework such as in prisoner's dilemma. While informative in many applications, it can be too narrow and restricting to think about these complex phenomena of collective decisions and action exclusively as problems of coordination. Firstly, this framework typically considers a small number or type of agents with a fixed set of explicit rules. While many hard problems we face are among a large number or type of agents, and the mechanisms of interaction may be implicit or change over time. This distinction is at the core of some most important human endeavors. According to Harari (2014) and others, Homo Sapiens have accomplished astonishing achievements because humans can cooperate in large numbers and in flexible ways. "In large numbers" is in contrast to chimpanzees and other mammals, which can only cooperate in small groups where one develops a personal relationship with their counterparts. "In flexible ways" is as opposed to bees and ants, which can work in large numbers but with more rigid roles. Second, besides the consideration of group size and flexible ways of interaction, another issue with viewing the human collective through the lens of coordination is that it tends to dwell on the negative outcomes of bad coordination like tragedy of the commons. These are important to avoid, however, making it the sole focus risks neglecting the upsides from the human collective, where the group is more capable than the sum of the individuals within it. The focus on avoiding bad coordination may lead to neglecting how to achieve that. Instead of thinking about these human-collective issues as an issue of coordination, we would advocate for thinking about them more broadly as an issue of Collective Intelligence, one of how to process distributed information effectively in order to reach common goals. (NICHD, 2015) . (B) Fish school respond to the presence of a diver (van de Vendel, 2017) . (C) A prediction market for who will be elected president of France in 2022 (Predictit, 2022) . (D) Elections. Picture shows women voting in the United States in 1920 (Voting, 1920) . (E) A swarm of robots (Carrillo-Zapata, 2019) . (F) Great wildebeest migration at Serengeti National Park, Tanzania (Tung, 2019) . The issue of how a collective work together is a central question in many applications in human society and beyond. Examples include human groups, such as in mitigating existential risks, elections, prediction markets, and juries; animal groups, such as in deciding the direction for migration; robotic swarms, such as designing rules for individual robots such that the collective can perform certain tasks; and neurons, such as the brain making coherent sense of the world while each neuron responds to different, and sometimes conflicting stimuli (see Fig. 2 for illustrations of these examples). In all these diverse applications, a shared problem is when information is distributed across the individuals in the system, how to process distributed information effectively and aggregate it together. This distributed information processing can take a wide variety of forms spanning a wide range of complexity. Some examples of more explicit aggregation mechanisms include majority vote in political elections, and quorum voting in African buffalos when deciding on new grazing locations (one signals preference by standing up and facing the favored direction) (Prins, 1996) . The more implicit mechanisms involve the schooling of fish, where each fish's preferred direction of movement interacts with that of its neighbors to generate movements of the school. As a result, the school can collectively sense the temperature gradient, while no individual fish is able to sense such information (Puckett, Pokhrel, & Giannini, 2018) . Another example of more implicit aggregation is prediction markets, where individuals bet on the likelihood of an outcome occurring, and the information is embedded in the prices. Researchers have come together to study this phenomenon, crosscutting application domains, and the transdisciplinary field is referred to as Collective Intelligence. We refer to this field as transdisciplinary, instead of interdisciplinary, because it studies a shared phenomenon that manifests in many disciplines, spanning computer science, neuroscience, robotics, animal behavior, and many social sciences. It transcends the disciplinary boundaries, while interdisciplinary often refers to combining two fields to discover new things in either one. With many rigorous research efforts in the past two decades in Collective Intelligence, researchers have found many factors that help or hurt the performance of human groups in solving problems and making collective decisions, predictions, or estimations. While research does not yet offer recipes for constructing a good human collective, as collective performance is the outcome of complex interactions of many variables. These findings point to a few general directions for improving the collective performance of human groups, and we offer a brief and non-exhaustive summary of relevant findings below. Collective intelligence. Similar to individual general intelligence, that one individual can excel at a wide range of tasks, such as math and music, Woolley and colleagues (2010) find a similar property for human groups, named collective intelligence. When asking small groups to perform a wide range of tasks, including brainstorming, sudoku, and unscrambling words, the performance on a subset of the tasks gives a good out-of-sample prediction. In other words, some groups outperform others on a wide variety of tasks, suggesting the equivalent of general intelligence for groups. This finding has been replicated by later studies (Riedl et al., 2021) . The studies look into what makes some groups more effective than others. Among the peer-reviewed studies (see Riedl et al., 2021 for a summary), the consensus is that the social process is more important than the skill of individual members. Social perceptiveness, the ability of individuals to identify social cues, is a key factor in improving group performance. This is manifested in group behaviors such as even conversational turn-taking-each group member speaks roughly equal amounts. Consequently, groups with a higher proportion of women tend to have higher collective intelligence, as being female is correlated with greater social perceptiveness. The IQ of individual members plays a much less, and some argue negligible role. Besides the social perceptiveness of group members, research finds benefit in having a diverse group of individuals. The diversity referred to here is diversity in knowledge and cognitive models. Abundant research has found in lab experiments, theoretical and computational models, and real-world problem-solving scenarios, the phenomenon of a "diversity bonus"-that a diverse group performs better than a homogenous group (Page, 2019; Aminpour et al., 2021) . Some even find a diverse group of nonexperts can outperform a homogeneous group of experts (Hong & Page, 2004) . For a group of diverse agents to work together, an important aspect is cognitive alignment, such as commitment to group goals and shared beliefs (Krafft, 2019) . Another line of work finds that maintaining diversity in the group is hard-conformity and traditional market forces are against it. Mann & Helbing (2016) propose alternative incentives to reward accurate minority predictions for maintaining diversity in the group-especially rewarding the minority which is right when the majority is wrong. Committed minorities. There has been much evidence that the presence of committed minorities, a small fraction of individuals who are little affected by the opinions of others, can lead to substantial changes in the collective behavior of a group. Most notably, Centola et al. (2018) find through human subject experiments that a critical mass of committed individuals (around 25% in their experiments) can tip social conventions towards the direction of these committed individuals. This critical transition is also predicted by an abundance of theoretical studies (such as Xie et al., 2011) . It would require further investigation to understand the precise critical mass needed in different scenarios, however, the powerful effects of committed minorities on groups can be a fruitful direction in thinking about how to elicit (or prevent) social change. Social influence. An area under debate is whether and how individuals should communicate with each other in a collective. Experimental studies have found that social influence can lead to both positive and negative effects on collective performance (see Jayles et al., 2017 for an example of positive effect, and Lorenz et al., 2011 for an example of negative effect). On the one hand, letting individuals exchange information can lead to the loss of independent information, and worse collective performance (also referred to as groupthink). On the other hand, communication may help individuals deliberate and discover better answers. The conflicting findings are likely an outcome of the effect of social influence depending on several other variables. A study predicts it depends on the proportion of individuals using social information for their decisions, and whether committed minorities are present (Yang et al., 2021) . Others find they also depend on the social network structure and adaptability (Almaatouq et al., 2020; Becker, Brackbill & Centola, 2017) . The bottom line is that most people blindly following others does not lead to good outcomes in most scenarios. Better sensors. One important component for improving group performance is to let individuals gather better information. This is especially important in forecasting and prediction tasks, such as election forecasts. This can be achieved by asking better survey questions. For example, in a method called "surprisingly popular" (Prelec, Seung & McCoy, 2017) , instead of asking people what they think, researchers also ask people what they think other people think. This helps discover surprising phenomena hidden by social norms, such as Trump winning the 2016 US election. Another example is to use individuals' social circles as better sensors )-instead of asking individuals whom they will vote for, ask whom their friends will vote for. Overall. The literature in this field is vast, and here we highlight a small subset of findings relevant to the performance of human groups. One theme that appears in these scientific studies is that individuals' ability to perform the task matters less than how the individuals interact. Most CI research on human groups, such as those summarized above, focuses on improving estimation accuracies, the effectiveness of teamwork, and collective decision speed. Many aspects of these investigations have implications for GCR mitigation. Below we outline examples of current GCR mitigation efforts and discuss how CI research can be applied to mitigating GCR. CI has been increasingly considered as a useful part of crisis response (Vivacqua & Borges, 2010; Büscher & Thomas, 2014) , and can be useful in GCR mitigation. Mitigation of GCR can naturally be divided into prevention (prevent risks from occurring), response (react before or during the hazard to limit the damage), and resilience (survive and rebuild), with a key role of adaptive collective responses to implement these mitigation measures (Cotton-Barratt, Daniel & Sandberg, 2020) . CI can be applied in each of these layers, in particular by enhancing predictive and adaptive ability (See Fig. 3 for a conceptual illustration of the relationships of these concepts). For prevention, identification of a possible risk and action to reduce its probability (or at least severity) are needed. The identification step is where committed minorities have historically played a clear role, often starting in the scientific community. For example, Clair Cameron Patterson built up a network concerned about the spread of lead in the environment, and the atmosphere scientists of the Crutzen-Rowland-Molina group discovered and documented ozone depletion. Their persistent research, documentation, and outreach provided the signal needed for scientific societies and later international organizations to react to the risk. Currently, the planetary defense community working on impact risks of asteroids and the AI safety community are examples of committed minorities having the specialized concern needed to determine the risk and propose preparedness actions. These are examples of how the scientific and policy community can act with CI, by having specialized networks examining potential threats and-if the evidence becomes compelling-amplifying their signal through a social influence mechanism. Predicting GCRs ahead of time is challenging because they represent unprecedented problems, or occur in domains with very little data. There has been significant interest in estimating the likelihood of different GCRs, especially due to the need for prioritization and making trade-offs among risks. This has been done using a variety of methods that aggregate expert or public opinion (Beard, Rowe & Fox, 2020) . At the simplest, these methods consist of taking the median estimate, while more sophisticated approaches give weightings based on consistency (Prelec, 2004; Frank et al., 2017) or past predictive performance, involve structured deliberative processes to build a consensus, or are prediction markets where (updateable) bets are made. One mechanism for prediction that has been discussed at length in the CI literature is prediction markets, where participants bet on future outcomes, and the prices of bets indicate the joint belief in their likelihoods (Wolfers & Zitzewitz, 2006) . Here, distributed and diverse knowledge is aggregated through a price mechanism. In theory, attempts to manipulate the price can even increase the accuracy by injecting capital exploitable by informed participants (Hanson & Opera, 2009 ). Existing prediction markets contain bets on various natural or anthropogenic disasters. In an experiment, a prediction market was applied to forecasting epidemic disease rates and performed more accurately than predictions from extrapolating historical trends (Li, Tung & Chang, 2016) . However, prediction markets have so far been limited by anti-gambling laws in many countries, lack of liquidity/participants, and the time preferences of participants. Similar to research on prediction markets, work on forecasting tournaments, which is not limited by antigambling laws, has produced methodologies for eliciting short-and medium-term forecasts that are consistently better than individual expert forecasts, demonstrating CI. These methodologies involve weighted aggregation of many individual estimates, identifying top performers and grouping them together into teams, and supporting learning, updating, and debiasing (Tetlock & Gardner, 2016; Tetlock, Mellers & Scoblic, 2017 ). Inspired by the success of forecasting tournaments and prediction markets, metaculus.com is a community for group forecasting using many of the same aggregation mechanisms, plus others like gamified scoring and reputation to encourage higher accuracy. The site covers a vast variety of real-world events, including disasters all the way up to existential risk. The site also did well compared to both the public and experts in forecasting during the COVID-19 pandemic (Recchia, Freeman & Spiegelhalter, 2021) . Often preparation, rapid detection, and response are more feasible than predicting the precise moment a GCR will happen. Rapid detection can draw on the distributed and sensitive nature of CI to find early signals of a hazard, amplify the warning, and trigger pre-planned responses. One experiment demonstrated how rapid social mobilization using social media and an incentive mechanism to solve an "impossible" search problem (finding ten weather balloons located across the US) was able to succeed within 9 hours. Here the winning team CI was enabled by an incentive mechanism that rewarded not just direct search but recruiting likely candidates, plus information sharing across the participants. Other teams recruited suitable pre-existing communities (Pickard et al., 2011) . Similar experiments have demonstrated fast finding of people (<12 hours) globally (Rutherford et al., 2013) . These results suggest that rapid, CI-mediated crisis response such as finding pathogen carriers or dangerous objects is possible. After the initial forecast, adaptivity may become essential. Decision-making and governance relevant to global catastrophic risk are complex, and naturally take place inside a collective cognitive system that is part of the greater system: complexity, reflexivity, and uncertainty cannot be avoided (Fisher & Sandberg, 2022) . Responding to GCR requires adaptive governance. In evolving disaster situations, flexible solutions and the ability to innovate on the fly can be crucial. The responses need to be more adaptive than the hazard, which typically implies a great need for generating diverse solutions, rapid spread of successful solutions, and forming tight feedback loops to help evaluate and improve the solutions. CI has a clear advantage here over centralized control systems that tend to act slowly and with less information. Such top-down approaches may be more suited for pre-or post-disaster preparation and systemic recovery where timescales are long, optimized global solutions perhaps possible and desirable, and especially when a severe GCR needs to be avoided at all costs. Adaptive governance has demonstrated successes in managing commonpool resources in a decentralized fashion (Ostrom, 1990) . Safety can be thought of as a special common pool resource that can be managed in this way. GCR raises the stakes by making the threat global and hence requires global adaptive governance, a challenge for CI. One component that can improve adaptive governance in a GCR situation is effective data-sharing, a lesson learned especially from the COVID-19 pandemic. Data aggregation sites such as ourworldindata.org played a key role in providing various agencies and networks with vital information while they struggled to get data from governments in useful formats. To enable effective CI, constructing better data sharing infrastructure may be necessary. Another component that can improve adaptive governance is resiliency. Resiliency can be viewed as the capacity of a system exposed to a shock or stress to adapt and survive by changing its non-essential attributes and rebuilding itself (Downes, 2013) . Resiliency, both during and after a catastrophe, is often aided by collective memory: adaptations to past disasters remain distributed among community members (and sometimes, stigmergically (Marsh & Onof, 2008) , in the environment like Japanese tsunami stones reminding of long-term danger). Collective memory may have a limited range (Fanta, Šálek & Sklenicka, 2019) but can both exceed individual lifespans and allow distributed reactions. It also interacts with other collective attitudes such as risk-taking and trust (Viglione et al., 2014) . Another form of collective risk cognition affecting resiliency is via insurance pricing, where insurers set premiums based on aggregated and estimated risk, creating incentives. For example, in response to tsunami risks, insurance incentives can incentivize homeowners and builders to build where the risks are reduced. Many forms of adaptations to risk in human society take the form of institutions, whether formal (such as fire services, disaster agencies, the Intergovernmental Panel on Climate Change) or informal (such as cautionary tales, safety practices). They can be seen as integrated systems of rules that structure social interactions (Hodgson, 2015) : hence a way of structuring or creating CI that (when done well) promotes resilience. This institution-forming often occurs in a reactive way when disaster strikes, which may not be acceptable for the most serious risks. CI is often described as spontaneously emergent, but this is rare. Usually, it is enabled by the existence of certain structures-suitable abilities, platforms, and incentives that make forming a shared cognitive system easier and allows it to have strong effects. Thus, there is a need to use the findings of CI research to design and improve these structures-such as the types of institutions and market incentives-for adaptive governance. Systems can be stable by resisting shocks directly ("toughness"), by returning to their equilibria through internal feedback (homeostasis, such as how mammals regulate body temperature), by dynamic stability (constant nudging the system to stay in an intrinsically unstable equilibrium, for example, riding a bicycle), to adaptive stability (the system changes structure to respond to the shocks, for example, buildings add safety stairs and smoke alarms to improve resilience to fires). In human systems facing disaster, sufficient toughness can rarely be achieved but feedback and adaptation are both possible and common. CI represents the distributed, bottom-up approach to achieving these without central control. A central issue in GCR mitigation is how to reconcile individuals with conflicting goals and interests. This is especially manifested in concerns derived in game-theoretic analyses, such as prisoners' dilemma and tragedy of the commons. Collective intelligence offers a broad and, we think, hopeful, perspective on this issue. These differences may first appear as an obstacle to coordination, while in the collective intelligence perspective, these differences, if harnessed with the right aggregation mechanism, is a source of strength. Take the example of a flock of fish. Each fish senses their local environment-food, temperature, potential predator, etc., and has a different preferred direction of where to go next. Nevertheless, the collective effectively aggregates information from all fish through local interactions in movement (Lopez et al., 2012) . The collective's movement responds to a much wider environment, more than the range of any individual fish. Thus, collective intelligence can be generated from different individuals acting on different information and wanting different outcomes. The differences should be seen as a resource, and the key question should be how to harness and aggregate the individuals' differences in productive ways. Despite many examples demonstrating CI, groups are not always intelligent. Research has found many examples where the collective fails or performs worse than an individual (Lorenz et al., 2011; Koriat, 2012) . A key to making progress in understanding collective intelligence is to identify conditions under which group performance can be improved. While many existing studies find effects of individual variables to help or hurt collective performance, with the large number of factors included in collective intelligence, it is also necessary to have a holistic framework to investigate the interaction of these variables. In the words of Elinor Ostrom (2009), "Without a common framework to organize findings, isolated knowledge does not accumulate." The collective intelligence problems behind existential risk problems such as AI governance and nuclear war mitigation, do not yet have solutions. But it is hopeful because nature has solved many hard collective intelligence problems. Besides the example of fish, each neuron in our brain receives different, and sometimes conflicting information, but the brain harnesses the inconsistent signals and makes coherent sense of the world. It would be beneficial for researchers of human groups to look across disciplinary boundaries and learn from nature's approaches to collective intelligence to help inform societal challenges. Besides learning from solutions in other application domains, taking a transdisciplinary perspective is also useful for identifying key processes involved in many existential risks. For example, what fundamental processes make AI, nuclear war, and pandemics dangerous? Research has suggested candidates-many risks having the common denominator of disrupting agriculture (Denkenberger et al., 2021) ; the pace of regulatory innovation being far slower than that of technological innovation (Marchant, 2011) ; dangerous technology being accessible to a small number of actors (Bostrom, 2019) . These exercises of identifying commonality across various risk domains help recognize key characteristics underlying humanity's greatest dangers. With the understanding of commonalities across risks, there is hope for finding crosscutting improvements to many problems at once. The study of GCR mitigation is an emerging field, and great challenges remain-each leading to important global safety implications. We encourage CI researchers to think more about using CI in practice, especially in GCR scenarios representing perhaps the most important societal challenges. We encourage GCR researchers to expand their approach, which is centered on studying the physical sciences behind each risk scenario, to further engage with the behavioral sciences, especially the study of collective intelligence, to formulate useful theoretical frameworks for the reduction of risks across domains. We encourage general behavioral researchers to become aware and take advantage of the transformative opportunities of making an impact on the most important societal challenges through engaging with these two transdisciplinary efforts. Adaptive social networks promote the wisdom of crowds The diversity bonus in pooling local knowledge about complex problems Stewardship of global collective behavior An analysis and evaluation of methods currently used to quantify the likelihood of existential hazards Network dynamics of social influence in the wisdom of crowds The vulnerable world hypothesis Collective intelligence in crises Toward controllable morphogenesis in large robot swarms Experimental evidence for tipping points in social convention Defence in depth against human extinction: Prevention, response, resilience, and why they all matter Long Term Cost-Effectiveness of Resilient Foods for Global Catastrophes Compared to Artificial General Intelligence Safety (No. vrmpf) How do we know about resilience? An analysis of empirical research on resilience, and implications for interdisciplinary praxis How long do floods throughout the millennium remain in the collective memory A Safe Governance Space for Humanity: Necessary Conditions for the Governance of Global Catastrophic Risks. Global Policy, forthcoming Validating Bayesian truth serum in largescale online human experiments Human social sensing is an untapped resource for computational social science A manipulator can aid prediction market accuracy Sapiens: A Brief History of Humankind On defining institutions: rules versus equilibria Groups of diverse problem solvers can outperform groups of high-ability problem solvers Tightrope walker Margret Zimmermann over Köln in 1946 How social information can improve estimation accuracy in human groups The wisdom of crowds in action: Forecasting epidemic diseases with a web-based prediction market system Governing Boring Apocalypses: A new typology of existential vulnerabilities and exposures for existential risk research From behavioural analyses to models of collective motion in fish schools How social influence can undermine the wisdom of crowd effect Optimal incentives for collective intelligence Stigmergic epistemology, stigmergic cognition National Institute of Child Health and Human Development (NICHD) (2015), A neuron in an insect brain Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds A simple computational theory of general collective intelligence When are two heads better than one and why? The Precipice: Existential Risk and the Future of Humanity Governing the Commons: The Evolution of Institutions for Collective Action A general framework for analyzing sustainability of social-ecological systems The Diversity Bonus Timecritical social mobilization Who will be elected president of France in 2022? A Bayesian truth serum for subjective data A solution to the single-question crowd wisdom problem Ecology and Behaviour of the African Buffalo: Social Inequality and Decision Making Collective gradient sensing in fish schools How well did experts and laypeople forecast the size of the COVID-19 pandemic Quantifying collective intelligence in human groups Targeted social mobilization in a global manhunt The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business Superforecasting: The Art and Science of Prediction Bringing probability judgments into policy debates via forecasting tournaments Great wildebeest migration crossing Mara river at Serengeti National Park -Tanzania Insights from socio-hydrology modelling on dealing with flood risk-roles of collective memory, risk-taking attitude and trust Collective intelligence for the design of emergency response Women learn to vote at NCR in Dayton on Oct Interpreting Prediction Market Prices as Probabilities Evidence for a collective intelligence factor in the performance of human groups Social consensus through the influence of committed minorities Dynamical system model predicts when social learners impair collective performance The authors thank Rory Greig for helpful discussions and feedback on an earlier version of the manuscript.