key: cord-0060256-wn2jodjv
authors: Bovaird, Tony; Loeffler, Elke
title: Developing Evidence-Based Co-production: A Research Agenda
date: 2020-07-31
journal: The Palgrave Handbook of Co-Production of Public Services and Outcomes
DOI: 10.1007/978-3-030-53705-0_36
sha: 752ab34bcad9ca0f5275311a605fca887fc057ed
doc_id: 60256
cord_uid: wn2jodjv

This chapter discusses how a more evidence-based approach might be taken to co-production. It outlines a range of developments which mean that public policy and practice urgently require a richer knowledge base and suggests that a fuller understanding of co-production must embrace its key characteristics, namely that it is holistic, contextual, multi-faceted, bridge-building, creative, dynamic, complex, emergent, behaviour-influencing and tacitly understood. It summarises key research recommendations from each chapter of the Handbook, highlighting the potential for a long-term systematic research programme into weakly-evidenced aspects of co-production. It then outlines how co-production might be evaluated, focusing on the whole causal chain from co-production inputs to outcomes. Finally, it explores a more creative approach to this research agenda, incorporating action research and tapping of tacit knowledge.

summarises key research recommendations from each chapter of the Handbook.

This leads to a discussion of how co-production might be evaluated, focusing on assessing not only its impacts on quality-of-life outcomes and public governance principles but also on the whole causal chain from inputs to those quality of life and governance outcomes.

Finally, we explore how this research agenda might be furthered by a more creative approach, incorporating action research and the tapping of tacit knowledge.

The Challenges to the Academy: Issues for the Next Generation of Co-production Research

No academic discipline remains stationary for long and this is even more the case with multi-disciplinary knowledge areas, such as public management or the policy sciences, drawing as each does on several disciplines which are themselves either in a state of flux or which are jostling with each other for priority within the prevailing fashions that are shaping scholarship in that branch of knowledge.

Consequently, there are always challenges to the academy, most of which are ignored for long periods of time but eventually become recognised and even accepted-sometimes suddenly (giving rise to the appearance of 'punctuated equilibria'), sometimes only very gradually (to the extent that many scholars hardly realise how much their subject area has changed during their working life, e.g. the creeping dominance of statistical studies in public administration from the 1980s onwards).

However, the current wave of challenges to dominant modes of thinking in public management and policy are particularly dramatic and have important implications for co-production research. Here we highlight those challenges which seem to us to be most pertinent to making co-production research relevant for the turbulent era in which we live, work, study, govern and are governed.

Among the current challenges are the need for our research and growing knowledge base to throw light on the following actual and potential characteristics of co-production:

• Holistic-the holistic life of citizens means that they do not break down the outcomes they seek to achieve into neat bundles, but rather expect that their personal 'outcomes mix' is understood, respected and, as far as possible, delivered through the co-production activities in which they engage. Knowledge about co-production therefore needs to respond to its inherently multi-disciplinary character, in which a range of stakeholders must all focus on holistic quality of life, rather than simply seeking to maximise the outcomes in which they have a special interest (as is true of all approaches to outcome-based commissioning). This necessitates a much more holistic research approach to understanding and reporting the quality-of-life outcomes of service users and communities. • Contextual -all citizens are different, so 'one-size fits all' approaches to co-production will be inappropriate (a longstanding lesson from the quality management of public services). This is even more the case when it comes to the strengths, assets and resources which citizens can offer in the co-production process-indeed, it seems likely that people differ even more in their 'capabilities' than they do in their 'needs'. In future we therefore need 'needs and capabilities assessments', not just needs assessments, and appropriate tools are needed to allow such assessments to be undertaken accurately and expeditiously. • Multi-faceted-citizens tend to identify with and prefer to interact with people with whom they have symbolic affinities, as well as actual common interests (this is one aspect of 'identity politics'). This means that coproduction may be limited by artificial barriers to joint action and to overcome this, we need to understand the affinities felt by citizens. This adds to the demands on our research, which more commonly classifies citizens by their socio-economic characteristics and interests. • Bridge-building -'identity politics' also means that co-production can be co-opted by governments, parties and interest groups who wish to use it for nefarious ends. For example, there have been concerns that digital co-production such as contact tracing apps developed to reduce the risk of infection in the context of Covid-19 may be misused by authoritarian governments. Consequently, our understanding of co-production must not only reveal how it appeals to the different affinities felt by our target groups, but it must also show the potential ways in which it can be bridge-building and inclusive, able to overcome the fragmented ideologies or 'elective roots' (linked to identity politics) which now characterise our hyper-politicised world (Freeden 2019 (Freeden -2020 . • Creative-as co-production activities develop and experiences are reflected upon, the limitations of old, bureaucratic ways of pursuing publicly-desired outcomes become more evident. For example, professionals working in health and social care have become increasingly aware of the importance of the 'social model' whereas their training often focussed on the 'medical model' in many aspects of health and social care. These emerging lessons require creative responses to break out of the moulds of outdated practices. Moreover, as Cormac Russell points out in Chapter 9, in the final analysis, in a democracy, citizens are the primary inventors, with professionals only there to support that invention, never to replace, demean or overwhelm it. However, we have to recognise that this creativity is not always for positive goals-participatory mechanisms (seeking to divine the 'will of the people'), such as referenda or citizen assemblies, can be creatively hijacked to impose a straitjacket on decisionmaking, negating the possibility of learning from mistakes, uncovering of disinformation campaigns ('fake news'), or adapting to changing circumstances (Freeden 2019 (Freeden -2020 . Our understanding of co-production has to embrace both positive and negative aspects of creativity-in particular, can co-production help people more easily to distinguish relevant information from disinformation with regard to innovative experiences, so that their views, decisions and actions can correspond more closely to Rousseau's concept of the 'adequately informed general will')? • Dynamic-since co-production is inherently about interactions, and these develop over time, often involving more people as they develop, coproduction is necessarily a dynamic process, ever changing in its processes as well as its outcomes. It is therefore poorly understood by linear inquiry methods which use static data. • Complex-co-production involves multiple actors with multiple interconnections, seeking multiple individual and collective goals, and learning emerging lessons from each other-this makes it likely that co-production takes place within a complex adaptive system, with the corollary that the effects of an action within any given time span are non-predictable. In the complex knowledge domain, all we can know is the set of potential outcomes which are attainable and how our interventions can alter this set-but not which of these outcomes is likely to eventuate from any specific intervention. • Emergent -co-production often is not pre-planned but emerges as a natural way for stakeholders to interact with each other in the complex system in which they find themselves. Consequently, research into co-production should not be based purely on pre-planned research programmes-as Ewert and Evers suggest in Chapter 7, piloting coproduction programmes can pave the way for a democratic experimentalism. Instead of merely testing what is planned in traditional fashion, there should be more research using participative and collaborative interaction, flexible in approach and adaptive to changing circumstances. • Behaviour-influencing -decisions are not purely rational, as Tinna Nielsen's chapter in this volume makes abundantly clear. Neither the decisions of co-producers, nor the decisions of policymakers considering the adoption of a co-production approach, are made on purely rational grounds. Consequently, their preferences are quite malleable, so positive experiences can change their minds and their behaviour quite readily, under the right circumstances (Banerjee and Duflo 2019), including experiences of co-production activities. • Tacitly understood-much knowledge which is relevant for excellent practice (and, ironically, this includes the practice of excellent research in the generation of explicit knowledge!) is tacit rather than explicit-'we know more than we can tell'. Tacit knowledge is, by definition, hard to disseminate. Co-production, relying on citizen action and not just citizen voice, offers a key way of disseminating citizens' knowledge, by having them co-deliver alongside members of public service organisations and other stakeholders. Similarly, training of citizens and staff as more effective coproducers may be best done through 'sitting with Nellie', on the grounds that 'actions speak louder than words'.

Potential for a Long-Term Research Programme into Co-production

In this section, we will discuss some of the key proposals made by authors in the Handbook for further research. Many of these respond directly to the challenges outlined in the previous section. Others respond to longstanding issues in the research on collaboration in public governance. Yet others respond specifically to issues in public consultation, participation and citizen engagement. We group these proposals essentially under the main headings of the Handbook:

• Definitions, theoretical and conceptual frameworks.

• Co-commissioning.

• Co-design.

• Co-delivery.

• Co-assessment.

• Management of co-production.

• Governance of co-production.

• Co-production research methodology.

The definition of co-production has been a topic on which a lot has been written over the past forty years. However, as Chapter 2 makes clear, we believe that we might usefully call a halt to the search for the 'perfect definition'. Rather than delving ever more deeply into nuances of co-production, we suggest that research studies will be more productive if they accept one clearly set out definition, chosen from the range of current definitions of coproduction, then explore different ways of achieving co-production according to that definition and throw light on their actual and potential results in terms of quality-of-life outcomes, public governance principles and improvements to system resilience. This will enable research to suggest how to make coproduction, thus defined, more cost-effective. This will not necessarily throw up optimal answers-but in a complex adaptive system the search for an 'optimal solution' is, in any case, a chimera.

We mentioned earlier that the preferences of service users and communities need to be understood as malleable. Peter Jackson (Chapter 4), looking at individual co-production, contrasts the economist's standard assumption of exogenously determined, stable, and homogeneous tastes and preferences with more modern theories assuming that they are endogenous, fluid, flexible and heterogeneous and emergent, having been shaped by the consumer's complex set of social interactions. This helps to explain why the propensities of citizens (and staff) to co-produce can differ sharply from context to context and serves to emphasise that co-production often takes place in the complex knowledge domain. However, little is known about how this variation in citizen preferences occurs and how it can be influenced-another reminder that public service marketing is still in its infancy, compared to private sector marketing. Klijn and Stevens in Chapter 8 of this Handbook argue that branding of public services and public spaces can build on insights about the role of emotions, rather than pure rationality, in making decisions and choices and go on to discuss how co-production can contribute to public service branding, suggesting some research avenues into how public service branding, using visual images and quick messages, can change behaviour. Nielsen in Chapter 5 takes a different tack, exploring how 'inclusion nudges' can change behaviour by working around and behind existing values, beliefs and prejudices but suggests that nudging does not create the sense of empowerment that is at the core of most successful and sustainable community and societal changes, so that more work is needed on creating the enthusiasm and feeling of empowerment that comes from the interactive and relational kind of co-production.

Bertelli and Cannas tackle the long running issue in the literature as to whether co-production must always be voluntary or might, in some circumstances be mandatory. They raise two important research questions in relation to the law and co-production-first, what other legally recognisable concepts, apart from proximity, might legitimate co-production as a matter of law? Secondly, how can the law both legitimate co-production as a policy tool and at the same time permit its flexible application by policymakers, given how valuable is the discretion which co-production leaves in the hands of public managers and citizens in working together to achieve an objective? Arguing that the legal formalisation of co-production can ultimately be a doubleedged sword, e.g. if it means that co-producers could be held legally liable for the externalities which they create through their co-production, Bertelli and Cannas suggest the need for research into how to balance protection from 'truly problematic action' by co-producers against disincentives for citizen participation. Further, they suggest that, since administrative law is specific to the country in which it has emerged, future research will be needed into the range of legislative tools for implementing the co-production paradigm, and how these might vary across countries, including the trade-offs that might be necessary between the legal status and practicality of these co-production mechanisms.

A number of authors in this section of the Handbook, including Strokosch and Osborne, and Klijn and Stevens, call for further research into the potential for co-production to result in value destruction, rather than creation. Klijn and Stevens are partly concerned about the costs to citizens of their contributions to co-production, a topic which has clearly been very under-researched to date and for which there is currently no convincing conceptual framework. (The recent article by Garlatti et al. [2018] highlights some of the directions which such research might take in public services, while a similar discussion on the costs of co-produced research is provided by Oliver et al. [2019] ).

As we highlighted in Chapter 2, understanding the change management necessary to operationalise greater and more cost-effective co-production remains relatively underdeveloped and further research on this would seem an urgent priority. Torfing, Toft Kristjansen and Sørensen in Chapter 18 consider this issue both from a macro and micro-perspective, suggesting the need for research on how ideas, resources and authority can be brought together through democratic processes of collaborative interaction in order to solve pressing societal problems. They identify the need theoretically to consolidate the distinction between co-production and co-creation through the development of conceptual taxonomies and discussions of the grey area where co-production shades into co-creation, together with more theoretical work on platforms and arenas to better understand the new forms of generative governance. At the micro level, they suggest the need for research both about the role of co-creation in politics and how neighbourhood organisations can enhance local participation and solve local problems through new forms of community-level co-design. A key issue in change management is how to scale up co-production and here Robert, Donetto and Williams in Chapter 16 suggest that the example of how EBCD has come to be widely viewed as an improvement methodology offers lessons on how to 'scale up' co-production, although with accompanying warnings to avoid co-optation and loss of fidelity.

Co-commissioning is the area of co-production which is closest to political decision-making, as it concerns societal priorities. As highlighted in Chapter 1, the role of politicians in co-production has so far been little researched, although there is often a supposition that many politicians may somewhat resent sharing power in decision-making with citizens, even though it is those very citizens who elected them in the first place. The account of participatory budgeting (PB) by Escobar in Chapter 15 demonstrates that there have so far been few case studies where politicians have whole-heartedly welcomed citizens as co-determiners of large elements of the government budget. Nor has political backing for PB always grown with experience, as Escobar's account of the Brazilian experience highlights. It will be important to explore in more depth why this is and what strategies might be pursued to develop more political backing. Escobar also calls for more research into the most effective and sustainable approaches to PB, its impact on institutions and services, and outcomes achieved for citizens and communities, all of which he sees as crucial to inform democratic innovation and reform driven by public authorities and civil society.

Ongaro, Mititelu and Sancini in Chapter 14 consider how co-production can be built into the commissioning cycle to make it a co-commissioning process. They suggest that this requires a learning attitude, although this is rarely enabled or facilitated by the environment of public sector organisations. They therefore recommend exploring the conditions under which learning can occur, combined with an open and flexible approach to strategic planning, utilising an open place-based platform for multi-stakeholder learning in which political astuteness works with concomitant, but also competing values, so that co-commissioning can be seen as integral to strategically managed approaches to co-production.

Discussing micro-level commissioning, Musekiwa and Needham focus on the extent to which health professionals are willing to share control over health interventions-their empirical work highlights how slowly health professionals develop trust in the decisions made by their service users, although they detect some signs that there can be positive learning over time. This key issue clearly needs investigation in more detail and at larger scale.

Co-design is clearly a highly popular form of co-production. Robert, Donetto and Williams have long been involved in extensive co-design initiatives but in Chapter 16 they express concerns that the distinction between co-design and other forms of co-production may have become too 'fuzzy'. They therefore propose robust forms of quantitative and experimental evaluation of contemporary co-design in the health context, advising that 'an eye should always be kept on potential tensions around different evaluation paradigms, as well as overlapping interests in studying mechanisms of action', while recognising in equal measure the unique-and importantly radical-features and potential of both co-design and co-production.

Remesar in Chapter 17, exploring the role of co-design in the transformation of public space, proposes that co-design procedure should be considered a social innovation process and suggests as objectives in an action-research process the delivery of proper information, setting up a creative procedure, designing a strategy inclusive of both processes and the different proposals of all relevant residents, and ensuring that the process and its outcomes are seen as belonging to neighbours and not to the facilitating team.

Torfing, Toft Kristjansen and Sørensen in Chapter 18 press for more research both on the role of co-creation in politics and the ability of neighbourhood organisations to enhance local participation and to solve local problems through new forms of community-level co-design. They suggest that empirically we need comparative studies of actual processes of policy-related co-creation and co-design as they unfold in time and space in order to identify the barriers and drivers and reflect on how barriers can be overcome and drivers expanded.

Moore and Evans in Chapter 19 suggest that quality co-design requires more understanding of the difficulties of working with citizens to change the ways in which decisions are made and implemented. However, they highlight the obstacles involved, given that co-design has emerged from many, often non-communicating, disciplines and there is still little shared understanding, particularly in government. As co-design challenges the established ways in which policy is made and services are designed, delivered, monitored and evaluated, they see the fundamental stumbling block to implementing co-design in practice as the lack of support by politicians and senior bureaucrats, partly because the 'policy elite' are largely econocrats or legally trained and not sufficiently open to exploratory and experimental approaches. Consequently, research needs to explore how to convince politicians and their political advisors of the merits of co-design processes.

Elke Loeffler in Chapter 20 highlights the need not only for more research on how to achieve the potential benefits of co-delivery but also on its potential governance pitfalls (see also Chapter 31 by Steve Smith), particularly in public safety, where the increasing availability of surveillance technology may be infringing the public's rights to privacy to a degree which goes beyond the appropriate balancing of public safety outcomes and public governance principles. She ends with a call for further research into the change management process needed to turn the huge potential of citizens' co-delivery into quality of life improvements for citizens, overcoming the barriers posed by the fact that public service systems are still not designed, organised, incentivised or experienced in making use of the rich potential of citizen contributions.

In Chapter 21, Trish McCulloch argues that development and innovation in the field of desistance requires a clearer understanding of the interdependence of individual, group and collective co-production in efforts to support desistance, and a willingness to invest in and experiment with more inclusive, empowering and disruptive forms. She notes that a key lever for change has been the rise of peer support and peer activism as a transformational mechanism for individual, social and system change, epitomised in the Independent Living Movement, with its user-led, collective, rightsbased, paradigm-changing, empowered, empowering and advocacy-oriented construction of peer support. Consequently, she suggests, the key challenge for would-be professional co-producers is to work out how to enable and support this necessarily peer-led movement without trying to monopolise it or undermining it as a genuinely co-productive endeavour and a mechanism for system and social transformation-an important area for future research.

Benjamin Clark in Chapter 22 urges that, although the promise of coassessment through technological means has tremendous power to shape society and provide for better services for more people, governments need to recognise the lesson from research that there is not a one-size-fits-all approach but rather that different levels of engagement come with different applications. Moreover, both policymakers and researchers will need to have an open mind (to the potential of digital co-assessment, as future technological advances will likely enable new and different kinds of co-production that we cannot even imagine today.

Dave Mckenna in Chapter 23 argues for the use in future research of the 'wheel of co-assessment'-co-focussing, co-directing, co-detecting, cojudging and co-effecting-as an analytical framework for determining the extent and likely effectiveness of co-assessment in any audit, inspection or scrutiny process, distinguishing between processes that are fully co-productive and those only partially co-productive-but he highlights the need to explore co-judging, in particular, as the pivotal element of co-assessment. He also suggests use of three scenarios by Linders (2012) as a framework for thinking about how co-assessment might develop in the context of digital and social technologies-namely, citizen reporting, open book government and selfmonitoring.

A number of themes for future research have been suggested in relation to the management of co-production.

In Chapter 24, Schlappa, Imani and Nishino suggest, based on the application of their conceptual framework to a case study, that leading in co-production processes requires a 'light structure' in which citizen and professional co-producers can deliberate on their goals and motivations not only individually but also in groups and with wider stakeholders. They also recommend the use of other relational leadership perspectives, and in particular exploration of the role of emotions (a theme also highlighted by Durose and colleagues in Chapter 35) in leadership-followership relations, to generate additional analytical frames to the social constructionist stance which they take.

In Chapter 25, Tuurnas suggests some avenues for future research with regard to staff motivations to co-produce. First, what types of co-production take staff furthest out of their comfort zones and why? Her analysis here is later backed up by Brandsen's call for more understanding of the conditions under which professionals are likely to engage in co-production. Second, there is a need to develop the categorisation of the core co-production skills for public service personnel, such as reaching out beyond the 'loudest voices' among citizens, and to explore how these co-production skills are handled in the formal education of staff in various public services. Third, there is a need to understand the transformation which co-production can bring about for staff in their work, especially where they are refocusing their identity.

In Chapter 26, Trui Steen urges research into the interplay between factors which explain citizens' engagement in co-production, namely motivation, ability, context and opportunity, in order to understand and manage the conditions for co-production as a 'package' rather than as entirely separate variables-in particular, do extrinsic motivators crowd out intrinsic motivation, how do co-producers' perceptions of their capacity relate to their motivation and how do citizens' willingness and capacity to participate in co-production projects depend on their environment and the efforts made by government to engage citizens? Adding to this, in their chapter, Schlappa, Imani and Nishino recommend specific research into how the motivation for volunteers and service users differs across the Four Co's of co-production.

In Chapter 27, Taco Brandsen concludes that co-production is a solution that will work well for certain people but that we still don't know what its effect is likely to be on vulnerable and marginalised people, so we need more experiments and pilots to explore this. This responds to the suggestion by Jackson in Chapter 4 that those service users who are vulnerable and dependent may not be able to engage directly in acts of co-creation, so individual co-production relationships are more relevant where clients are free to make a series of reasoned choices on their engagement with a service. We need to be keenly aware that co-production has distributional consequences-and could even be seen as giving 'government a licence to dump on users while claiming improvements in effectiveness'.

Echoing the call by Loeffler and Bovaird in Chapter 2 for more research into the inter-relationship of co-production and resilience, Coaffee et al. in Chapter 28 argue that working with and co-producing knowledge alongside policy makers in the area of resilience involves fundamental questioning of the different cultures of universities, local citizenship and policy communities, rather than 'simply rolling out classical models of superficial community and external stakeholder consultation where local citizens or businesses become co-opted or 'responsibilised' to fulfil state-type roles'. Consequently, coproduction is a necessary ingredient in research looking to transform conventional and institutionalised power relations in decision-making to understand better how we can respond flexibly to all manner of risks and crisis, and build resilience into our responses.

As Gazley observes in Chapter 12, there is plentiful evidence of the need for greater safeguards on co-productive activity-e.g. governments may coopt citizen efforts, and both co-producing citizens and co-producing institutions may lose sight of or fail to deliver on public values. Victor A. Pestoff in Chapter 29 argues that exploring the impact of governance on the condition and well-being of either staff alone or users/citizens alone is insufficient-to understand the importance of governance models for promoting participative public governance it is necessary to study their impact on both the main actors in co-production, the staff and users/citizens. This reinforces the recommendation by Phinney and Sandfort in Chapter 11 for research on how exactly public organisations can move from governance characterised by an emphasis on market mechanisms and citizens as customers, backed up by the conventional authority of hierarchies and performance management, to a governance approach prioritising collaboration and co-production.

Wouter Nieuwenhuizen and Albert Meijer in Chapter 30 ask: 'Do we make a service more efficient by giving citizens more responsibilities, therefore partly neglecting lawfulness and integrity?' They conclude that the necessity for democratic involvement in and societal debate on ICT-based co-production is higher when digital technologies are transformative or substitutive, because risks to values such as lawfulness, equality and participation are higher in those cases. They finish with a call for action: 'A strong emphasis on public values is needed if we want to develop forms of co-production that are not only effective and efficient but also proper and responsive' and they warn that the risks of digital co-production for public values are substantial, so it is important that society find solutions to incorporate more inclusive democracy in practices of ICT-based co-production.

Steve Smith in Chapter 31 highlights how sustaining co-production programmes hinges on creating resilient systems and relationships that can weather crises-he suggests that this is likely to mean sustainable organisational funding, ongoing staff capacity building and training (e.g. in oversight and monitoring of co-production programmes, together with learning new engagement and outreach strategies), and support and appropriate incentives for engagement of community members. Further research needs to explore how to make such interventions effective.

Scolobig and Gallagher in Chapter 32 observe that little research has focused on the social and power dynamics emerging during knowledge coproduction processes and highlight that inclusion of some stakeholders may slow down decision-making, or privilege the solutions of already powerful actors, rather than fostering fairness in decision-making. They therefore urge more research on how and why stakeholders form coalitions in co-production, with special emphasis on the role of trust between stakeholders, in order to understand better if the formation of coalitions contributes to conflicts being addressed (given that coalitions may also have negative effects, such as reducing the range of alternatives, as well as helping to shape compromises).

The final section of this Handbook has been on the research agenda for co-production, focusing mainly on the methodologies which might be used to give us a deeper understanding of the practice and the potential of co-production.

In Chapter 33, Kang and Van Ryzin urge the further use of experimental methods in co-production research, and note some gaps in the existing literature as well as issues for future research. They suggest that experimental research on co-production should widen out from its current main focus on environmental issues to the important areas of education, health and policing and indeed further to social care, transportation, housing, arts and culture. In particular, they identify the need for more studies measuring actual coproduction behaviours rather than intentions. This could usefully involve more lab experiments and more quasi-and natural experiments that involve exogenous change in real-world policies or administrative practices, e.g. changes in laws or rules, ideally with a comparative element. They also suggest more attention to stimulation of co-production via group-and meso-level institutional designs and structures, rather than individually targeted micro-level cues, information and incentives. They highlight the potential use of game theory experiments or natural experiments that can possibly probe the more institutional-level factors influencing the co-production of public goods and services. They speculate that experimental methods could also examine potential downsides associated with co-producing public services, probing for social, cultural or political moderating factors that shape how people perceive or behave in response to the co-production of public services in specific policy areas and governance contexts.

In Chapter 34, Vivian Ramsden, Tanya Verrall, Nicole Jacobson and Jackie Crowe-Weisgerber suggest the need to consider innovative and optimal ways to analyse the data and subsequently to measure the success of co-produced research. Moreover, they emphasise the value of mixed methods in future research, reinforcing the recommendation of Torfing, Toft Kristjansen and Sorensen in Chapter 18 that methodologically we must challenge ourselves to combine qualitative case studies with the quantitative measuring of key variables in comparative case analysis, which could contribute to the discovery of competing combinations of factors that facilitate co-creation in different sectors and at different levels.

In Chapter 35 on co-production research with service users and communities, Catherine Durose, Beth Perry and Liz Richardson emphasise the importance of humanising the experiences of very diverse co-researchers, allowing time for differences to be meaningfully explored. Moreover, they highlight that, although co-production can be more ethically aware (e.g. taking greater account of issues of power, rights and responsibilities and roles of all stakeholders) and more egalitarian and democratic (based on respect for partnership with community members), it also entails complex relationships of power and accountability and raises distinctive ethical challenges, so it requires researchers to remain reflexive about their own research practice and strategies for ethically concerned research. They recommend that the emotional labour in co-produced research should not only be acknowledged but also collectivised (e.g. providing space for discussion of the challenges of co-produced research and collective reflection on positions, identities and political and ethical commitments in the process). Since the value of co-production remains contested within the academy, there needs to be an incentive to challenge these barriers, e.g. by reconfiguring the metrics by which research is currently assessed. They provide a series of design principles that can offer a basis for continued reflexivity and underpin wider participation in future research. They end by emphasising that co-production isn't about devaluing science but reevaluating other ways of knowing, that what works best will vary according to context and that there is a need to continue to test and learn about different forms of co-production.

It is clear from the chapters in this Handbook that most co-production initiatives still have an inadequate evidence base, hence the suggestions for further research which we have summarised in this chapter. It will clearly be important to boost this evidence base, ideally in a way which is coordinated, at least to the extent that researchers link the implications of their own research clearly to those of different research projects in the field, making clear how their results reinforce or cast doubt on previous findings or throw up new findings. In order to achieve this, it would be valuable if researchers also made clear the evaluative framework within which they work and how their results fit within this evaluative framework.

In our own research on co-production, we have sought to highlight the hypothesised causal chain linking inputs to outcomes, i.e. the overall theory of change, including the following elements (Loeffler and Bovaird 2018) :

• Increased quality-of-life outcomes arising from co-production.

• Improvements to achievement of public governance principles arising from co-production. • Increased service quality from user/community involvement. • Increased service efficiency resulting from reduced organisational inputs or increased organisational outputs. • Reduction in service user inputs.

• Increases in social capital arising from co-production.

These elements potentially form a virtuous circle, rather than a linear causal chain, since improved quality-of-life outcomes may have valuable positive effects on citizen capabilities and therefore on future citizen inputs to coproduction (and similarly they may form a vicious circle if the outcomes are unfavourable).

Although the final outcomes-both quality-of-life outcomes and achievement of public governance principles-might be expected to be the most important in the long term, each of these elements is important in its own right for some key stakeholders. The priority of these elements for researchers will therefore depend on their context and the research questions which they are investigating-but it will always be important to locate research results within this overall framework. Moreover, research is needed into how the priorities between these elements of the circular causal chain are established in the political process and how this might best be influenced by citizens themselves.

Clearly, evaluation of co-production initiatives should not just explore 'input/outcome' ratios, since these are affected by so many contextual factors-it is essential that attention is paid to testing the hypothesised pathways to outcomes, in order to learn why co-production works better in some circumstances than others, to throw light on the risks involved and how they can best be managed by the many stakeholders involved, and how resilience (of users, communities, service providers and the overall service system) can be achieved.

A key issue which has so far been very under-researched is the costs of co-production (Loeffler and Bovaird 2018) , taking into account the costs to public service organisations, including:

• Inputs of front-line staff.

• Managerial inputs.

• Inputs of local councillors or other politicians.

• Investments in ICT-enabled forms of co-production.

• Investment in public infrastructure.

However, these costs represent only one side of the picture-evaluations also need to explore the costs imposed on service users and communities by coproduction, such as:

• Increased time inputs to learn about co-production opportunities.

• Increased time inputs for preparatory and training activities.

• Increased operational inputs resulting from more intensive co-production activities. • Increased monetary expenditure by co-producing citizens.

• Increased 'psychic' and emotional costs.

Evaluating all the elements in the circular causal chain, and comparing the results to the costs imposed, would allow us to check if indeed there is an evidence-based cost-effective pathway to outcomes through co-production activities. Favourable outcome/cost or output/cost ratios are not enough-it is essential to demonstrate that there is a plausible pathway by which coproduction could cause the improvements (Bovaird 2012) . In practice, we still have few examples of the construction and testing of pathways to outcomes for co-production activities, although some cases such as Services for Young People in Surrey County Council have provided valuable templates, allowing the risks involved in this major transformation of the service to be better understood and more carefully managed (Tisdall 2014) .

In many cases, these pathways to outcomes will demonstrate that the interconnections between different elements in the circular causal chain are so dense and so strong that we are dealing with a complex adaptive system, one of the challenges to the academy discussed early in this chapter. This has major consequences for evaluation, warning us that simple cause-and-effect analyses are likely to be inappropriate and that much more experimental and integrated approaches to learning will be necessary, with evaluations which are much more tentative and contextual.

One of the 'challenges to the academy' highlighted earlier in this chapter was the need for creative responses to the lessons emerging from the new and innovative approaches to involving citizens in the co-production of public services and outcomes. Creative research has two quite different dimensions-doing our 'normal' kind of research very differently or doing very different kinds of research. It has become very evident from the chapters in this Handbook that both approaches have been tried in the different studies which are recounted here. For example, the growth in use of research co-produced with citizens (see Ramsden and colleagues in Chapter 33 and Durose and colleagues in Chapter 35) and the recent drive towards experimental research in public administration (see Van Ryzin and Kang in Chapter 33) show how social science research is already employing new methods to pursue longstanding research objectives. Creative use of existing techniques was also evidenced by Robert and colleagues in Chapter 16 in the use of EBCD (where evaluators commented that 'the co-design 'work' was more creative and relational compared to usual staff-led -or externally driven-quality improvement initiatives') and by Coaffee and colleagues in Chapter 28 in the use of novel ways to analyse data and a World Café style format ('to provide an open and creative conversation … that surfaces collective knowledge, shares ideas and insights, and allows everyone to gain a deeper understanding of the opportunities and challenges …').

However, it is also clear that the authors of the chapters in this Handbook are frustrated by the difficulties in doing co-production research and keen to find new ways to explore its meaning, its practice and its potential.

Two key avenues, which are distinctively separate but nevertheless interrelated, for doing rather different kinds of research into co-production emerge from some of the chapters in this Handbook-action research and the tapping of tacit knowledge.

Action research is exemplified in a range of chapters, including those by Remesar, Phinney and Sandfort, Coaffee and colleagues, Ramsden and colleagues and Durose and colleagues. While several of these are about co-production of research, action research has also been central to other co-production initiatives. Indeed, it is typified by transformation processes such as the Offenbach job centre improvement programme, using the Coproduction Star, which we discuss in Chapter 2, where a range of officers of the public employment agency developed experimental labs with service users and recorded the lessons learnt along the way. As Russell comments in Chapter 9, 'citizens are the primary inventors', given that they are closer to the change process than anyone else and have to cope with it, while fitting it practically into their everyday lives. As Loeffler and Bovaird in Chapter 2 suggest, much more research is needed on the change management processes by which co-production can be promoted further in practice in public service organisations.

Similarly, Durose and colleagues summarise the approach of participatory action research as 'a simultaneous and collaborative process of taking action, doing research and reflecting critically on both', or, more pithily, as a commitment to 'learning by doing'. This approach to research is particularly important for co-production, given that co-production is dynamic and emergent, often operating in the complex knowledge domain, so that the alternative of 'learning by thinking' has severe limitations. As Phinney and Sandfort highlight, a key aspect of action research is its up-front intention to have practical impact-for example, in their Discover Together case study their analysis also drew attention to a set of tools and resources for helping public organisations move towards greater engagement and citizen responsiveness. Given the interest by public service organisations in moving towards more co-productive approaches, it seems likely that co-production research in the future will make more use of action research methodology.

A further driver of creativity in co-production research is the role that coproduction can play in tapping and making use of tacit knowledge on the part of citizens. In his chapter, Russell suggests that this tacit knowledge is often 'swept aside' in the rush to use rational thought processes, based on explicit knowledge. However, access to tacit knowledge is a key benefit of coproduction, so it is surprising how little research has so far been done into how it can be accessed.

Tacit knowledge is knowledge that cannot be codified in words or other explicit ways, building on the dictum of Polanyi (1967) that 'we can know more than we can tell'. This concept has deep roots in philosophy and, more recently, in management. Indeed, Robert Pirsig (1974) , in one of the most influential modern books on quality management, argues that 'quality' is a phenomenon that lies beyond language and number, so that, although everyone is capable of identifying quality, it is not possible precisely to define or measure or describe it to others-later, Pirsig (1991, 64 ) goes on to argue: 'Quality is a direct experience independent of and prior to intellectual abstractions'.

What works and what does not work depends strongly on context and on social relationships. In complex and even in complicated knowledge domains tacit knowledge may be as good as it gets, so that harnessing it becomes central to public service and public outcome improvement. Bovaird and Loeffler (2016) distinguish between 'technical' non-explicit knowledge and 'social' non-explicit knowledge held by service users and community members. They argue that the greatest challenge for professional public services is to get access to the 'technical' non-explicit knowledge held by service users and their communities, since this knowledge is generally only accessible by sharing in the life experience of those people, which for professionals is highly demanding of time and emotional capital. Consequently, where this 'technical' non-explicit knowledge on the part of citizens is high, co-production is likely to entail that power over decisions should lie more with the service users and communities, who have most direct access to this knowledge. Non-explicit 'social relationship' knowledge, on the other hand, is an area where exchange between service professionals and citizens (including service users) is likely to be easier, so that a fully joint approach to decision-making may be more appropriate. A key lesson is therefore that when tacit knowledge is important, learning about co-production is a joint, socially constructed activity, not simply one for linear, rational research. Being part of the doing may therefore be intrinsic to learning and helping others to learn.

Finally, tacit knowledge offers a strongly positive lesson for those coproducing public services and publicly-desired outcomes-its importance suggests that, whatever the appearances, modern public administration and social policy remains an arena in which responsible, experienced people can and must exercise agency, they do not and should not simply follow regulatory rules or algorithms; they can and should use their understanding of the local world around them to improve people's welfare, not just accept top-down analyses of 'strategic options'; and they can and must collectively construct a better understanding of what needs to be done to achieve positive social change, rather than simply 'implement the corporate strategy'. Research into co-production which cannot access and suggest ways to both refine and share this tacit knowledge is missing much of the point of doing research into co-production.

This Handbook has demonstrated that the concept of co-production can be interpreted as covering many issues that are currently very important in public management and policy. Indeed, we had thought of naming this chapter: 'Coproduction: the theory of everything?' or even, picking up a theme from the chapter by Coaffee and colleagues: "Co-ubiquity"-where to start and where to end?'.

However, both suggestions seemed too glib-while it may be true that there has been greater multiplication of co-words than of rigorous evidence on their effects, the chapters in this Handbook demonstrate that there has nevertheless been a slow build-up of knowledge around co-production, which has helped it to become more plausible to policy makers and practitioners as an approach, and has promoted its wider academic consideration. Moreover, the suggestions in this chapter highlight ways in which it should be possible in the next few years to deepen and extend this knowledge base about coproduction, partly by bringing to bear knowledge from a much wider range of disciplines and perspectives and partly by using co-production itself as one of the key tools for research.

So perhaps, as we reach the end of this Handbook, we might speculate on why co-production has become such a major theme in the literature over recent years. We believe that it is, at least partly, because it makes us face squarely the realisation that knowledge and capabilities are often more widespread but, in some ways, more limited than has been commonly assumed over the past century. Indeed, we believe that this twin realisation is so dramatic that it is likely to have a revolutionary effect on the social sciences view of the world over the next decades.

On the one hand, we have come to realise that the knowledge (both explicit and tacit) and the capabilities for everyday living possessed by most of our people are essential for the effective working of society, public services and indeed much of the market system, although they had previously been remarkably ignored by many scholars and by most politicians and by government officials responsible for public policy. The significance of these contributions by citizens through co-production is being increasingly uncovered by research and built into the practices of those on the front line of public services. Perhaps key research questions over the next decade will be why it took so long for this realisation to dawn, and how could we have wasted the capabilities of our citizens for so long by not making them an integral part of our public service systems?

At the same time, we have come to realise how limited is our knowledge about and capability to intervene effectively in complex social, economic and public service systems. This has highlighted how misguided we have been to become so arrogant about the superior effectiveness of professional and 'expert' inputs to public services-essential and valuable though these inputs are, they could never deliver the high aspirations and the level of certainty which we increasingly came to demand from them, tackling 'wicked problems' in complex adaptive systems. This has opened up the gates, from another direction, to appreciation that contributions from other sources-namely, co-production by service users and communities-can be hugely valuable, albeit in different and complementary ways to the inputs from public service organisations.

As we have observed elsewhere (Bovaird and Loeffler 2012) , co-production partly harks back to some of the philosophical roots of public service as necessary to public well-being: 'To everyone according to their needs, from everyone according to their ability'. However, it also provides a fresh perspective on the future reform of the modern welfare state by focusing on people's social abilities and their community actions, not just on their personal capabilities and needs. This therefore also recognises the limits of the state: 'It takes a village to raise a child'. Indeed, given its potential role in accessing tacit knowledge and underused capabilities, and helping us to cope with wicked problems, co-production could be transformative, as many authors in this Handbook have commented, and as the Covid-19 crisis has highlighted in many communities. We hope that the contributions in this Handbook will help in taking forward the demanding research agenda discussed in this chapter and so ensure these high aspirations for co-production are realised.

Good Economics for Hard Times: Better Answers to Our Biggest Problems

Attributing outcomes to social policy interventions -'gold standard' or 'fool's gold' in public policy and management? Social Policy and Administration

From engagement to co-production: The contribution of users and communities to outcomes and public value

Learning about public service co-production when real quality is that which lies beyond language and number. Paper presented at the 38th Annual Research Conference of the Association for Public Policy Analysis and Management

A new ideological era

Co-production and cost-efficiency: An awkward couple? A structured literature review. Paper presented at the International Symposium Resilience, Sustainability and Innovation in Public Services

From e-government to we-government: Defining a typology for citizen coproduction in the age of social media

Assessing the effect of co-production on outcomes, service quality and efficiency

The dark side of coproduction: Do the costs outweigh the benefits for health research? Health Policy and Systems Research 17

Zen and the Art of Motorcycle Maintenance

The Metaphysics of Quality

The Tacit Dimension

The Transformation of Services for Young People in Surrey County Council