key: cord-0035867-j3ljwap9
authors: nan
title: Building on Experience: Learning Lessons from Past Framework Programmes
date: 2006
journal: A New Deal for an Effective European Research Policy
DOI: 10.1007/978-1-4020-5551-5_4
sha: 178b05b40a3bd594194e23fd8ad3165b2431b505
doc_id: 35867
cord_uid: j3ljwap9

nan

Building on past experience via ex-post evaluation is an increasingly important element in the design of research programmes and policies, notably at the European level. 1 Being aware of the strength and weaknesses of past Framework Programmes, knowing their impacts and realising in which fields potential impacts could not be achieved, is a precondition for the improvement of any new programme.

Based on the evidence of ex-post evaluations, this chapter looks back at past Framework Programmes and shows what lessons were helpful for the design of the 7th Framework Programme. The chapter starts with some methodological considerations linked to the evaluation of Framework Programmes (Section 1). Turning to the outcomes and impacts of Framework Programmes, the accumulated evidence convincingly shows that the Framework Programme has had a clear positive impact not only on participants' immediate competitive position, but also on their capabilities, behaviour, and ability to compete. It has improved Europe's scientific, technological, and innovative performance (Section 2). And it has strengthened Europe's human capital (Section 3) and integrated European research infrastructures (Section 4), and enhanced the coordination of Member State research policies and actions (Section 5). It has also generated wider economic, social, and environmental benefits (Section 6).

Identifying the outputs and economic, social, and environmental impacts of individual research projects is complex. The first reason is the difficulty of attribution, which concerns the key question at the heart of every impact study: to what 1 The Union, in response to its desire to regulate better, calls for transparency and accountability, as well as budget constraints, wants to know what impacts past policies and programmes have generated, and how it can increase those of future ones. The importance attached to learning lessons from the past and building on experience is reflected in the Commission's impact assessment guidelines. 2 Over the years, an extensive and highly diverse body of Framework Programme ex-post evaluation literature has emerged. It consists of studies carried out for the European Commission focussing on the pan-European Framework Programme experience, and studies carried out for the Member States concentrating mainly on the programme's national impact. The spotlight of Commission studies was sometimes on the Framework Programme as a whole, sometimes on just one or a few Specific Programmes. Past Framework Programme evaluations were sometimes of an ad hoc and one-off nature, and sometimes carried out repeatedly within the context of the formal Framework Programme ex-post evaluation system extent are measurable effects really the result of -that is causally attributable to -the project under review? 3 Usually it is not just the individual research project financed which has produced the solution to a particular problem or a certain impact. Project participants may have received funding from other sources as well. In the case of the Framework Programme, for instance, account has to be taken not only of purely national direct and indirect research funding, but also of other funding schemes supporting international collaboration such as bilateral agreements, COST, and EUREKA. Fig. 4 .1 shows the evolution of European cooperative research funding schemes, which have grown from 6 per cent of total government expenditure on R&D in 1985 to 15.5 per cent in 2003. 4 In addition, impacts may have resulted not only from intended effects directly related to programme goals but also from indirect and unintended effects going beyond programme goals. Next, factors other than research itself may have contributed to an impact. 5 Finally, attribution is hindered by the usually long period 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 of time between the funding and carrying out of research on the one hand, and the results of that research becoming visible on the other hand. 6 The second reason for the complexity involved in identifying the outputs and impacts of individual research projects is the need to assess additionality or what it is that would have happened anyway even without public support. 7 Three important kinds of additionality are input additionality (did public support increase research and innovation inputs? Or did it just replace and crowd out inputs that would have been made anyway?); behavioural additionality (did researchers change their behaviour (importance attached to research and research collaboration, project management) as a result of receiving public support and participating in the project?); and output additionality (would the same outputs have been obtained without the policy action?).

In addition to the methodological problems discussed in the previous section, puzzling together a consistent picture of past Framework Programmes and their outputs and impacts is hampered by the programme's main characteristics having evolved rather substantially over time. The Framework Programme constitutes a moving target, and this first of all in terms of its rationale. The start of the Framework Programmes constituted a radical break with what came before. Until the late 1970s, European research policy was of an ad hoc nature. Because it was tied to particular sectors (agriculture, coal, nuclear energy, steel, etc.), it was also fragmented. Research policy was mainly a national affair, even though the Founding Treaties already provided the Community with a responsibility in the field. 8 The 1st Framework Programme (1984) (1985) (1986) (1987) , and the inclusion of a separate chapter on RTD in the Single European Act (1986), heralded the shift towards a legally solidly grounded integrated European research policy focussing on the competitiveness of European industry and the quality of life of European citizens.

This apparent unity of purpose has not precluded a certain evolution of the programme's rationale, however, accommodated by the fact that as some observers have noted the overall Framework Programme objective is extremely permissive. 9 The reason for this evolution is that each Framework Programme was prepared against a different background. The rationale for the 1st Framework Programme was the perceived technology gap. 10 The 2nd Framework Programme (1987) (1988) (1989) (1990) (1991) was intended to strengthen the research base of European industry in response to fierce Japanese competition. Developing information and communication technologies was high on the political agenda. The 3rd Framework Programme (1990) (1991) (1992) (1993) (1994) was developed against the background of efforts to integrate the European market. The conceptualisation of the 4th Framework Programme (1994) (1995) (1996) (1997) (1998) took place during the period of the Maastricht Treaty (1992) and the White Paper on Growth, Competitiveness and Employment (1993) . The 5th Framework Programme (1998) (1999) (2000) (2001) (2002) put increased emphasis on socio-economic values. And the 6th Framework Programme (2002) (2003) (2004) (2005) (2006) was designed to help build the European Research Area (Table 4 .1). 11 Together with its rationale, the Framework Programme's content has also evolved. The number and content of Specific Programmes has changed from one Framework Programme to the next. The number of thematic priorities has multiplied. And new instruments have been introduced.

Another indicator of substantial change has been the growth of the Framework Programme budget, which reached about Before (1975) (1976) (1977) (1978) (1979) (1980) (1981) (1982) (1983) Ad hoc approach Energy oriented -FP1 (1984) (1985) (1986) (1987) Coordination of Community RTD actions Energy and ICT oriented Environment, international cooperation human capital and mobility FP2 (1987) (1988) (1989) (1990) (1991) Information society ICT oriented Biotechnologies, marine resources, dissemination FP3 (1990 FP3 ( -1994 Industrial Programmes, different and not always compatible databases were in use managed by different Commission services in charge of a particular Framework Programme component, making it difficult to arrive at one integrated dataset for a single Framework Programme. As instruments and rules of participation evolved from one Framework Programme to the next, the labels attached in the databases to Framework Programme participants also changed, which makes it difficult to analyse, for instance, the evolution of the participant type structure (e.g. evolution of industrial participation). Some data on project outputs have been collected in the past via end-of-project reports, but these have tended to be incomplete, and this is not helped by the frequent perception that reporting such outputs represents a significant administrative burden on Framework Programme participants.

The European Commission has been involved in research programme evaluation since the late 1970s. Yet it was only in the early 1980s, when evaluation became a legislative requirement, that the institutionalisation of Commission research evaluation practices picked up pace. 15

The ex-post evaluation system The Framework Programme ex-post evaluation system was introduced in the mid-nineties. 16 Its two main components are yearly monitoring exercises and five-yearly in-depth assessments (the so-called "Five-Year Assessments") carried out at overall and usually also specific programme level. The annual monitoring exercise is intended to be rather light and enable a quick response to issues arising from ongoing programme implementation. The objective of the Five-Year Assessments, on the other hand, is to provide input for policy formulation and decision-making on the basis of feedback obtained from programme implementation. 17 The current Framework Programme ex-post evaluation system has obvious strengths, for example its independence and legitimacy. However, the literature has also identified some important weaknesses which go beyond the issues plaguing all research evaluations (time lag, attribution, additionality, difficulties of measuring qualitative effects, etc.). According to some evaluation experts, the main problem affecting the current system relates to FP design: "The intervention logic that connects the high-level and operational goals of the FP is poorly articulated, making an overall evaluation of the FP difficult. The Framework needs more systematic planning, clearer objectives and a stronger link to an evidence base. This would ease evaluation and, arguably, improve FP performance". 18 This problem is of course not specific to EU programmes. The same difficulties apply when it comes to establishing quantifiable ex-ante objectives for programme evaluation at the national level.

Other (potential) problems have been identified mainly with regard to the available evidence base and the use of expert panels. A strong and timely evidence base constitutes the main tool for Five-year Assessment expert panels on which to base their assessment and recommendations. Yet a number of studies have highlighted weaknesses in the evidence base available. 19 Sometimes thematic expost evaluations have not (yet) been completed, or they focus on different issues, cover different periods, or have been carried out according to different methodologies. That means that panels must work with rather fragmented evidence, or with what is still possible to do in the very short run. Especially in recent years substantial efforts have been made to address this issue.

In Framework Programme ex-post evaluation, use is often made of external expert panels. As mentioned above, they contribute to independence and legitimacy. However, some observers have considered them to be time-and 17 The careful timing of the Five-Year Assessments allows for the combination of an ex-post evaluation of the previous Framework Programme with a mid-term appraisal of the ongoing one to formulate recommendations for the next Framework Programme. To read more about the Framework Programme ex-post evaluation system, see, for instance, Luc resource-intensive. 20 Though strict rules apply, others have claimed that it is difficult to avoid completely conflicts of interest. 21 Some analysts have suggested that panel members may be pressurised into promoting unrelated agendas and specific interests. And with regard to the 2000 Five-Year Assessment, one scholar noted that "many of the recommendations drew not so much on an evaluation of past FP activities, but on the collective opinions and assessments of the panel member concerning the general structure, and organisation of RTD in Europe". 22 It should however be borne in mind that these problems are not unique to the Framework Programme, but exist in many other industrialised countries' R&D programmes as well.

The above observations on Framework Programme evaluation are all well known and well documented. Indeed, most of the remarks come from the Framework Programme evaluation process itself. But this is also one of the greatest strengths of the system, because transparent and constructive criticism is one of the main vehicles for promoting programme improvement. And indeed many important improvements to the Framework Programme in the past have emanated from comments made at the evaluation stage.

Topics, sources, and methodologies Above we have argued that, except for the recent introduction of impact assessment, the Commission's research evaluation system has remained relatively stable over the past decade. Greater change has characterised the evaluation studies carried out. New topics are being explored, studies are based on new kinds of sources, and use is made of innovative methodologies.

In past Framework Programme ex-post evaluations, substantial attention used to be paid to analysing participant characteristics (e.g. type of institutional actor, country of origin, region of origin, etc.) and R&D inputs. At the same time, much emphasis was put on counting project outputs in order to arrive at total and average (per project) numbers of publications, patents, and so on. This has not disappeared. But attempts are now made to profile programme participants in more innovative ways. This includes analysing their scientific (e.g. numbers of publications, numbers of citations, citation impact scores) and technological (e.g. numbers of patents) quality, the nature of their participation (one-time vs. repeat participation), the nature of their networking behaviour (stable vs. changing partnerships), and so on. From merely counting project outputs, the emphasis has also shifted to assessing 20 Framework Programme aggregate impacts on Europe's scientific and technological performance, and on research capacity, or on the European economy and society.

The study of such more challenging topics can be embarked upon because of the mining of new data sources. Framework Programme ex-post evaluations used to be based mainly on surveys (interviews or written questionnaires) and end-ofproject reports. However, attempts are now made, for instance, to create cleaned and consolidated participant databases at the individual scientist level rather than just the institutional level, and to link them with bibliometric and patent databases. The use of bibliometric and patent data almost by definition also entails the use of new methodological approaches and techniques. Within this context, mention should be made of the use of econometric models to estimate the impact of the Framework Programme on the European economy, as was done in the impact assessment on the Commission's proposal for the Seventh Framework Programme. 23

The way forward Against this background, several steps are envisaged to improve the Framework Programme's evaluation system. A clearer formulation of the intervention logic in the 7th Framework Programme proposal with clear and measurable objectives and the monitoring of their progress will facilitate the ex-post evaluation of the 7th Framework Programme. A comprehensive effort will be made under the next Framework Programme to ensure that the ex-post evaluation will be based on a wide range of completed, focussed and methodologically standardised ex-post evaluation studies. 24 Continued efforts will also be made to explore new topics, sources, and methodologies. There is, for instance, a great need to assess better whether project outputs/impacts were the result just of a research project being carried out, or of a research project being carried out at European level. Important questions also remain regarding the optimal mix of partners in a project, or the optimal size of networks. Also, what constitutes a project of critical mass? Does it relate to the number of partners? To the level of funding? How does critical mass interrelate with flexibility, with cohesion, with excellence? Are repeated Framework Programme participation and the stability of networks across Programme calls and even across different Framework Programmes good or bad, and when do they become excessive? To answer these questions, further efforts will have to be made to consolidate and link databases. New methodologies will also need to be explored, especially those being able to shed a light on questions of causality. For instance, does participation in the Framework Programme increase the quality of an individual scientist or institution, or does it attract already excellent scientists and institutions? As much as possible, methodological development in the field will be supported by including relevant research topics in the Framework Programme's work programmes. It will also be supported through Commission networking. Networking is first of all required in-house, so that methodologies may be standardised. But external networking is also needed so as to achieve complementarity between European Commission and national level Framework Programme evaluations. 25 Those networks also allow sharing and comparing hard evidence of the impact of research policies at regional, national and EU levels and will help identify what is done best at each level, and how to design the S&T governance model in the most efficient and effective way. 

The added value of European funded collaborative research projects is among other things to be found in the fact that they allow for pooling financial resources and knowledge across national frontiers. This allows research projects to attain the appropriate size or "critical mass" needed to achieve scientific breakthroughs. In other words, it helps break down the wasteful fragmentation of research efforts across Europe. 27 Programme research projects has evolved over time in terms of number of participations and funding. The average number of participations per project increased from 4.7 in the 2nd Framework Programme to 6.5 in the 5th Framework Programme, while the average Commission funding per project increased from E1.2 million to E1.4 million. At the same time, average EU funding per participation decreased from E256 000 to E196 000. The 6th Framework Programme appears to have initiated a trend towards decisively larger projects with more participations per project, and with higher funding per project and per participation (Table 4 .2).

So the trend appears to be towards larger projects. But a better understanding has to be gained of what "critical mass" means, and of how it can be achieved. For example, does it refer to the number of participants in and the size of the funding allocated to a collaborative research project? If that is the case, what do we know about the ideal number of participants and the ideal size of funding? Is the relation between project scale and project achievement a simple linear one, or is there a minimum efficient project scale above which there is a disproportionately higher level of achievement? 28 Is it possible to make general statements about it, or is it Participation pattern by type of institution Available data indicate that over time the pattern of participation in collaborative research projects by type of institution has become more balanced. 30 Whereas business enterprises initially accounted for the largest share of funding and participations, that is no longer true. Higher education institutes and research centres now also account for sizeable participation and funding shares (Figs 4.4 and 4.5). 31 Several factors could explain the growth of the collaborative research participation and funding shares held by higher education institutes and research centres. The wider trend in OECD countries is to put renewed emphasis on funding basic research. 32 In line with this, the 6th Framework Programme included a more basic research-oriented activity called New and Emerging Science and Technology (NEST), as well as a Networks of Excellence instrument less attractive to business enterprises. It has been argued that work programmes have shifted away from the strong applications focus of the 5th Framework Programme, and have become more academic in tone. 33 Higher education institutes and research centres may be 29 In this respect, see also DTI -Office of Science and Technology, Targeted Review of Added Value Provided by International R&D Programmes, May 2004, p. 8: "The literature deals poorly with the question of 'critical mass,' providing no ready-reckoner to check whether more or better capacity is necessary." 30 It is not possible to trace in a definitive manner across Framework Programmes the evolution of the collaborative research participation structure by type of institution. In the first section of this chapter, we discussed how painting an adequate picture of the Framework Programme and its evolution over time has been hampered by a relative lack of data. One reason is that instruments and rules of participation have changed across Framework Programmes. Therefore, definitions of especially industrial types of participants have also changed. This makes it more difficult to analyse the evolution of the participant type structure. The result is that a certain margin of error is associated with the statistics reported below. 31 See European Commission, Second Report on S&T Indicators, 1997 -Report, Luxembourg, 1997, pp. 520-522, for a discussion on the drop between FP2 and FP3 and the seeming stabilisation between FP3 and FP4 of the industrial share of participations, and of the drop between FP3 and FP4 of the industrial share of funding. In European Commission, Five-Year Assessment 1999-2003, 15 December 2004, p. 6, the assessment panel, commenting on FP6, states having encountered "significant evidence of industrial discontent and reports from different regions of industry dropping out in unusually high numbers from the Sixth Framework Programme", but does not find supporting statistics. Nevertheless it calls for the next FP to have a stronger industrial focus. 32 less sensitive to the perceived increase over time in the cost of applying to and participating in the Framework Programme. 34 Some observers question the appropriateness of the Framework Programme's mix of participants in terms of type of institution. The supposedly too low participation of SMEs in the Framework Programme, for instance, is a constant in the Framework Programme ex-post evaluation literature. 35 It has led to the establishment of SME participation targets of 5-15 per cent in the 4th Framework Programme (depending on the thematic area), and of 10 per cent in the 5th Framework Programme. In the 6th Framework Programme, at least 15 per cent of the budget of the first and second Specific Programmes was foreseen for research-performing SMEs. The concern about SME participation relates of course to the important role SMEs play in the European economy. There are between 8 and 12 million SMEs in the EU-15, and a further 2.5 million in the new Member States. They account for over 99 per cent of all enterprises, and for two-thirds of all employment in the enterprise sector. Some 30 per cent of SMEs -thus some 10 million enterprises in the EU-15regularly develop, apply, or acquire technology. Less than 3 per cent is involved in leading-edge research, however. 36 Judging the appropriate level of SME or broader business enterprise participation in FP-funded collaborative research is difficult. Theory does not offer much insight beyond the one that it is quite natural for SMEs to be more irregular participants in the Framework Programme than large firms. They often lack a permanent R&D activity and staff. Given their scale, the relative cost of participation is higher. And there is no learning effect from regular participation. This argues against uncritically stimulating SMEs to participate. 37 On the other hand, there are no appropriate national or international points of comparison as the thematic coverage and instruments of these schemes differ from those of the Framework Programme.

Participation pattern by country of origin Framework Programme funding is awarded not on the basis of the nationalities of the applicants, but on the basis of their scientific excellence and the European added value of the research proposal. Framework Programme funding is important to the Member States, however, especially the smaller ones, so that national stakeholders take an understandable interest in the amounts they receive.

The Framework Programme evaluation literature does not dwell much on the countries of origin of Framework Programme participants. It just observes that national Framework Programme participation shares are broadly in line with Member States' sizes and RTD capabilities (e.g. number of researchers). 38 This is confirmed by an empirical check. Fig. 4 .6 displays for each EU-15 Member State (the highest ranking being 15 and the lowest 1) its ranking for three variables: the number of shared-cost action participations in the 5th Framework Programme, the shared-cost action funding received under the 5th Framework Programme, and the share of European GDP in 2000. It is clear that a Member State's Framework Programme participation is more or less in line with its share of European GDP. In other words, Member States accounting for large shares of European GDP participate most often in the Framework Programme and obtain most Framework Programme funding while the reverse is true for Member States accounting for small shares of European GDP. An important exception is Greece, which participates in the Framework Programme to a larger extent and obtains more funding from the Framework Programme than one would expect on the basis of its share of European GDP. The reverse is true for countries like Belgium, Austria and Finland. Similar results are obtained when the comparison is made not with the share of European GDP but with the number of researchers ( Fig. 4.7) . 39 Almost completely opposite results are obtained when we rank Member States in terms of their intensity of Framework Programme participation, in other words in terms of how many Framework Programme participations and how much funding are obtained per unit of GDP, or per researcher (Figs 4.8 and 4.9). Then it becomes clear that smaller Member States participate more intensely in the Framework Programme than larger ones. Fig. 4 .10 shows that the relative importance of established Member States in terms of number of participations has not changed very much across Framework Programmes. New Member States, however, have usually been able to rapidly increase their number of participations across Framework Programmes (See e.g. Sweden, Austria and Finland in Fig. 4 .10).

In addition to intra-European collaboration, the Framework Programme supports international collaboration beyond Europe's borders. The scale of this international collaboration has grown substantially across Framework Programmes. The number Participation pattern by region of origin The pattern of Framework Programme participation can be traced not just at national level, but also in Europe's regions. Some scholars argue that the regional level is the one at which the mechanisms of innovation and resulting economic benefits can be best understood. A close look at the map of Europe reveals the great diversity in research capacity and innovation performance among its regions. 40 R&D "hot spots" generating above average high-tech employment rates exist alongside pockets of poverty and entrepreneurial deprivation. 41 R&D inputs -the presence of universities, higher S&T education enrolment, patterns of overall and particularly business R&D investment and expenditure, and so on -matter, but regional innovation performance also hinges strongly on the functioning of the region as an innovation system. Less favoured European regions are of special concern to policy-makers as they display a regional innovation paradox. This refers to the apparent contradiction between the comparatively greater need to spend on innovation in lagging regions and their relatively lower capacity to absorb public funds earmarked for the promotion of innovation compared to more advanced regions. 42 40 Andrés Rodríguez-Pose, The Dynamics of Regional Growth in Europe: Social A pilot study based on 15 years of Framework Programme data (1987) (1988) (1989) (1990) (1991) (1992) (1993) (1994) (1995) (1996) (1997) (1998) (1999) (2000) (2001) (2002) has analysed in more detail the regional distribution of Framework Programme participation and funding. 43 There have been a number of interesting findings, in particular for Europe's less favoured regions. Successive FP programmes have created networks of research in less favoured regions which are at the base of the diffusion of knowledge spillovers and which undoubtedly have eased the diffusion of knowledge from the centre to the periphery. In terms of study findings, the study confirms firstly and not surprisingly that, in absolute terms, participations and funding are concentrated in the "usual suspects": London and the South-East in the United Kingdom; Île de France and Rhône-Alpes in France; the Stuttgart-Karlsruhe-Lower Bavaria axis in Germany; the Randstad in the Netherlands; the greater Brussels area in Belgium; and the North of Italy. 44 Secondly, within peripheral countries, European research activity is often, though not always, concentrated in those areas with the highest levels of GDP per capita and the greatest concentration of researchers and research activity (e.g. Helsinki, Lisbon, Madrid, Athens). Thirdly and most interestingly, a significant number of peripheral regions obtain a share of European research funds which is larger than one would expect on the basis of their overall R&D effort, the size of their economy, or their relative wealth (e.g. Scotland in the United Kingdom; Catalonia and Murcia in Spain; Steiermark in Austria). Put differently, the European Framework Programme offers less favoured regions an important alternative source of research funding. Given the fact that they often face a more limited choice of funding possibilities than more advanced regions, this study indicates that the Framework Programme plays an encouraging role in reducing the research investment gap in less favoured regions. From the viewpoint of the more advanced -in terms of research endowment and innovation performance -regions it can be said that successive Framework Programmes have supported excellent research without being detrimental to cohesion (Fig. 4.11) .

The degree of concentration of collaborative research participation In the previous two sections, we have looked at the distribution of Framework Programme participations and funding across countries and regions. This section considers how participation is dispersed at the level of individual organisations. Looking across Framework Programmes, it appears that some institutions have been very effective in competing for Framework Programme funds. Geuna found on the basis of 1st, 2nd, and 3rd Framework Programme data that the distribution of Framework Programme university participations is skewed. Many institutions are present in the system only one or a few times, but a small number of universities achieve a high number of participations. 45 summary of a series of Framework Programme ex-post evaluations comes to the same conclusion: many organisations' participation in Framework Programme is short-lived, but there is a core of frequent participants. 48 Finally, a recent network analysis of partial data on the 6th Framework Programme found skewed distributions of connectedness, with a central core of highly linked organisations and a periphery of less well connected entities. 49 A certain degree of concentration of participation in a selective funding scheme based primarily on scientific excellence is inevitable, and a certain degree of continuity of participation is indeed to be welcomed. The literature identifies a number of factors that may promote a certain degree of concentration of participation in (internationally collaborative) research programmes. Large institutions are more likely to apply multiple times to research programmes as they have more staff, and to internationally collaborative research programmes as they are more likely to have prior experience with international research collaboration. 50 Large, well-known institutions are better placed than small, lesser-known ones to put together research consortia. 51 More than small institutions, large organisations have the institutional capacity to handle complex application and project management procedures, and to do so multiple times. 52 Research project proposals submitted by consortia involving a large, well-known institution with a well-established research reputation are more likely to get selected. 53 A lock-in effect then ensues. This results first of all from the fact that successful applicants can familiarise themselves even more with complex application and project management procedures, which remain stable to a certain extent over time. This facilitates later applications. Successful participation, on the basis of which scientific and technological outputs can be produced, further consolidates the participant's scientific reputation and reinforces the information signalling effect to both potential consortia participants and proposal evaluators on the occasion of a next application. In the literature, this phenomenon is referred to as the "Matthew effect" and cumulative advantage. 54 It is not possible to judge now whether the degree of concentration of Framework Programme participation is appropriate. A study would be required of the evolution 48 of the degree of concentration of participation over time. A comparison would have to be made with the degree of concentration of participation in Member State, intergovernmental, and foreign (collaborative) research schemes. And an assessment would have to be made of the relationship between programme participant and network characteristics on the one hand, and project outputs and impacts on the other hand.

In the meantime, the Commission is aware of the importance of not discouraging small innovative institutions from participating regularly in the Framework Programme. The Commission's repeated efforts on the one hand to streamline and simplify the Framework Programme bureaucracy, and on the other hand to provide measures to encourage applicants with particular needs, such as SMEs, are aimed at ensuring that this does not happen.

Collaborative research networks described Framework Programme collaborative research projects are implemented via cross-border, cross-sector, interdisciplinary Framework Programme networks. These networks embody the unique European added value of the Framework Programme, which is to bring together different types of high-quality participants from different countries and with complementary expertise in productive partnerships, thus overcoming the sectoral and national fragmentation of the European research system. Over the past two decades, the number of links established between participants in FP-supported collaborative research projects has increased substantially to reach 160 000 under the 5th Framework Programme (Fig. 4.12) . 55 Collaborative research project participants do not just establish collaborative links within their own sectors (businesses just collaborating with businesses, universities just with universities, etc.). FP-funded collaborative research networks succeed in bringing together and creating links between different types of institutions. Over time, the typical configuration of a Framework Programme project has evolved, as expected given the aforementioned changing overall Framework Programme participant structure. 56 At the heart of these networks, however, still lies science-industry collaboration. This is a feature greatly appreciated by Framework Programme shared-cost action participants. 57 It is also a key component of innovation systems. 58 The average number of different Member States per project has increased from Framework Programme to Framework Programme. Starting from an average of 3 different Member States per project in the 2nd Framework Programme, interim data for the 6th Framework Programme show an average of 6.5 ( Over time, the configuration of intra-European transnational collaboration has also changed (Figs 4.13-4.15 ). The number of privileged collaboration partners has increased across Framework Programmes. No less than eight Member States are now situated at the core (Austria, Belgium, Greece, Ireland, Netherlands, Portugal, Spain, and Sweden). The groups of "intermediate" and "peripheral" countries have each become more integrated. Countries in these two groups increasingly collaborate amongst themselves thus reducing dependency on the core and diversifying collaboration patterns. Under the 5th Framework Programme, this was reflected in the emergence of two distinct sub-networks in the group of "intermediate" countries -Southern European (Greece, Italy, Spain, and Portugal) and Northern European (Netherlands, Denmark, Sweden, and Finland). Finally, a strong cohesion effect is evident. In the 6th Framework Programme, formerly "peripheral" countries -Austria, Ireland, and Luxemburg -joined the two inner circles of collaboration. In summary these three trends highlight a shift from projects dominated by just a few core countries and relying on geographical proximity to projects with a more balanced national representation and detaching themselves from traditional ties. This illustrates the integrating effect of European collaborative research. Modern mainstream innovation theory recognises the importance of networks bringing together different types of actors (e.g. science and industry) in regional, national, and supranational systems of innovation. However, an important tradeoff exists. Some authors question the value of uncritically promoting transnational collaboration at a time when, they would argue, competitive advantage is mainly determined by advances made by excellent single-country research teams. 59 Even if we accept the value of networking, important questions remain to be addressed. One question concerns the optimal size of networks, as some have suggested 59 Is network stability beneficial? Certainly one of the aims of Framework Programme support is to tackle the fragmentation of the EU research system by encouraging the formation of transnational consortia. If the networks thus created are durable, then, one could argue, a real contribution has been made to the long-term restructuring of research in Europe. Continuity over successive Framework Programmes can allow complex research fields to advance within a stable cooperative structure. However, there are also some important caveats. It may be that, once established, certain networks would have continued without further Framework Programme funds. In such instances, there would be no "behavioural additionality" from repeated Framework Programme support (although there may still be input or output additionality). At the same time, a potential danger of too much stability is that it may make it correspondingly difficult for new or more peripheral institutions to enter the game. 65 This in turn may have negative implications for the innovativeness of research undertaken while raising issues in relation to competition. Finding the right balance between stability and dynamism is therefore important and merits ongoing analysis.

From financial to knowledge returns -the multiplier effect of collaborative research via networks A Member State's net returns from the Framework Programme are traditionally calculated by subtracting its contribution to the Framework Programme budget from the combined value of its Framework Programme participations. However, such a restricted financial accounting approach does not capture all the "knowledge returns" flowing from Framework Programme participation. Participating in a Framework Programme collaborative research consortium offers access to an EU-wide knowledge exchange network. In other words, a single project participant benefits from and thus accesses the funding of all project participants combined.

The question is how to calculate these knowledge returns. This is an issue which so far has barely been explored and deserves further investigation. The preliminary, relatively straightforward approach we propose here is to estimate a country's knowledge returns by subtracting its contribution to the Framework Programme budget, not from the combined value of its Framework Programme participations, but from the combined value of the Framework Programme contracts in which it participates. For example, if a country contributes E1 to the Framework Programme budget, the combined value of its Framework Programme participations is E2, and the combined value of the Framework Programme contracts in which it participates is E10, then that country's net financial return is E1 while its net knowledge return is E9, and thus the multiplier effect is of a factor 9.

Calculations carried out by DG Research show that for each Euro invested in the 5th Framework Programme, countries' net knowledge return averaged E19.8 for the EU-25 (Fig. 4.16) . Participation in the Framework Programme was thus a winwin situation for all parties involved: all countries enjoyed positive net knowledge returns under the 5th Framework Programme. However, the size of these returns tended to be inversely related to a country's number of Framework Programme participations. Countries with a smaller number of Framework Programme participations (e.g. smaller EU Member States, and the then Acceding Countries) enjoyed higher net knowledge returns than countries with a larger number of Framework Programme participations (e.g. the bigger Member States). This probably happened because a smaller number of Framework Programme participations translates into a pattern of widely dispersed single participations per project, while a larger number of Framework Programme participations translates into a pattern where regularly two or three participations from the same country can be found in the same project, which partially dampens the net knowledge return. successive Framework Programmes (Fig. 4.17) . The focus of the first two Framework Programmes, for instance, was clearly on energy and ICT, which together accounted for 75 per cent of funding under the 1st Framework Programme, and 65 per cent under the 2nd Framework Programme. On the other hand, the 6th Framework Programme was organised around seven thematic priorities. 66 And the 7th Framework Programme will centre around ten thematic priorities. 67 The coverage of the Framework Programme is comprehensive, and does not appear to leave out any important thematic priority. Though The Framework Programme thematic priorities also correspond more or less to those of major US and Japanese research support programmes. 70 Within this context, it is legitimate to ask whether the Framework Programme's increasingly comprehensive coverage may have led to a degree of fragmentation in the sense that Framework Programme resources may be spread too thinly across too many priorities. This is always an important consideration because, to make a real difference, the level of support in a particular S&T area should be sufficient to achieve critical mass and a minimum efficient scale of research. However, there are no indications as yet in the Framework Programme evaluation literature or elsewhere that this is a major cause for concern.

A rigid predefined thematic priority structure can make it more difficult to accommodate important scientific advances and new needs which emerge during the life-time of the programme. For this reason, there have been a number of calls to introduce greater flexibility in the Framework Programme. Programme experience that adjustments to meet new needs, or to reflect new scientific advances, required a tortuous and time-consuming legal process. 71 In the recent ICT Five-Year Assessment, the Panel was of the opinion that, in a fast-moving area such as ICT research, the programme should be willing and able to respond rapidlyif justified -to changes in the economic, technological, or even policy context. 72 It cannot be denied, however, that such flexibility is gradually being built into the programme. The 6th Framework Programme, for instance, has been able to fund research on, for example, SARS and avian flu. Under the EU-supported SEPSDA (Sino-European Project on SARS Diagnostics and Antivirals), scientists from across Europe and China have spent the past years screening over 8 000 existing drugs to find prime candidates for combating SARS, and identified a number of promising anti-SARS compounds. 73 The European Commission recently also announced that it would be making a further E20 million available for research into avian and pandemic influenza. Relating to animal health, the relevant call for proposals would cover issues such as developing vaccines for avian species, improved diagnosis and early warning systems, and technology transfer to third countries. Relating to human health, it would cover issues such as clinical research on pandemic influenza vaccines, better understanding of the influenza virus, and strengthening support to surveillance. 74 In terms of quality, FP-supported collaborative research has generally been found to be of a high standard. 75 Industrial participants have reported that Framework Programme projects are of relatively high scientific and technological complexity, while university participants have identified major benefits in the form of knowledge enhancement. 76 A first important factor underlying the high quality of FP-funded research is the well-functioning Framework Programme proposal assessment and project monitoring system. 77 For instance, it has been suggested that the strong correlation between the number of participations by country in the 4th Framework Programme and the number of researchers in that country provides some evidence that the proposal selection system is working as it should. 78 Another crucial factor is that Framework Programme participants themselves are generally of high quality. A recent study on the networks resulting from funding under the 6th Framework Programme in the field of ICT concluded that networks under this Framework Programme attract knowledge leaders from corporate and academic centres of excellence, evidenced by the rate at which relevant patent holders and institutions participate. 79 A recent bibliometric study on participants in the 5th Framework Programme in the fields of the life sciences and nanotechnology convincingly demonstrated that the level of Framework Programme participants is generally above world level and that they reach citation impact scores higher than what could be expected on the basis of the journals in which they publish or the fields in which they are active. 80 A substantial number of Nobel Prize laureates in medicine, chemistry, and physics are involved in research projects and networks under the 6th Framework Programme, many of them oriented towards medical purposes. 81 Although Framework Programme collaborative research is of high quality, it has been observed that it can often be too mainstream and insufficiently risky and "at the frontier". The Five-Year Assessment 1995-99 found that the Framework Programme should retain its overarching emphasis on excellence, but make extra efforts to ensure that "risky" projects are not excluded. 82 Efforts were made to address this in the NEST programme under the 6th Framework Programme, and while this was welcomed by the subsequent Five-Year Assessment panel, they felt it was still a too narrow approach. The panel therefore recommended that an ambition to fund more risky projects should be embedded in the Framework Programme objectives, instruments and implementation as well as in the assessment of achievements and impacts. 83 Under the 7th Framework Programme, risky frontier research will be supported through the planned European Research Council.

Before assessing the various impacts generated by FP-funded collaborative research, it is important to establish whether or not Framework Programme collaborative research funding schemes have generated effects that are additional to what would have happened anyway in the absence of such funding schemes. In other words, in the absence of the Framework Programme collaborative research component, would projects have been carried out at all or would they have been carried out differently?

The consensus appears to be that Framework Programme collaborative research funding schemes are characterised by a high degree of additionality. out or they would have been carried out differently, often entailing substantial losses in terms of efficiency and effectiveness. Typical is the conclusion of a UK study on the EU Framework Programmes, which reported a broad agreement that Framework provides considerable added value. 84 It expands the funds available to national researchers over and above that which is available to them through national research funds alone. It provides UK participants with access to foreign researchers and research outputs in a way that national funds cannot. By pooling financial resources, it boosts investment in important research topics, and by pooling competencies it increases the likelihood of a breakthrough in a given area. Complex issues are resolved more quickly and more thoroughly as a result of larger projects and portfolios and multiple projects across successive Frameworks. 85 Supportive evidence is wide-ranging and convincing. Almost 95 per cent of respondents to a survey of Norwegian participants in the 5th Framework Programme answered that EU-funding was very important for getting the project started, 90 per cent that it was important for the size of the project, and over 80 per cent that it was important for how fast the project could be carried through. 86 Among Irish participants in the 4th Framework Programme responding to a survey, 82 per cent would not have proceeded with their project without Framework Programme funding. Of the 18 per cent who would have continued, over 70 per cent would have done so with reduced objectives and reduced funds, while close to 40 per cent would have done so over a longer timescale, and with fewer partners. 87 An Austrian study found very high additionality of the EU Framework Programmes. Of those questioned, 70 per cent said they would not have continued their research project had it been rejected. Of the remaining 30 per cent who would have continued the project without EU subsidy, only 14 per cent said that they would have done so without any adjustment. Typical adjustments entailed a modified goal orientation, a reduced number of project partners, and a different project volume. 88 Of Finnish participants in the 5th Framework Programme, 70 per cent would not have carried out the project without EU funding, 40 per cent would have carried it out on a smaller scale, and 35 per cent would have carried it out more slowly. 89

Evaluations of past Framework Programmes have shown that impacts are important and wide-ranging. The evidence is considerable that the Framework Programme helps to improve participants' research capabilities, and modifies their behaviour, in addition to generating significant scientific and technological outputs, and contributing to competitiveness and innovation by boosting productivity and encouraging the development of new products and processes.

Impact on capabilities, behaviour, and ability to compete FP-funded collaborative research projects strengthen participants' research capabilities by enhancing their knowledge base, and improving the skills of staff. 90 Behaviour is also permanently changed resulting in a better ability to compete. For example, participants able to achieve short-term networking goals are more likely to carry out research through networks in the future. 91 An analysis of the Third Community Innovation Survey (CIS-3) shows that Framework Programme participating enterprises are more likely to engage in innovation cooperation with other partners in the innovation system, such as other firms and universities (Fig. 4.18 ). Impact on scientific performance The extension of the knowledge base is to some extent codified and made explicit through scientific publications. Framework Programme projects generate many publications and thereby contribute directly to Europe's total output in terms of scientific publications, an important indicator of scientific performance.

The publication of peer-reviewed scientific work is an important goal for Framework Programme participants, especially those from universities and research institutes, now accounting for most Framework Programme participations and funding. For instance, 94 per cent of British public sector researchers participating in the 4th and 5th Framework Programmes rated refereed publications in journals and books as important outputs. 92 A similar survey in Austria found that around 80 per cent of respondents considered publications an important or very important goal of participation. 93 And a study in Ireland showed that Framework Programme participants from the public sector ranked publications as one of the ten most important goals. 94 Framework Programme participants generally achieve their publication goals, and thus generate large numbers of scientific publications. For instance, 92 per cent of Finnish university participants in the 4th Framework Programme (and 91 per cent of such participants in the 5th Framework Programme) had achieved their publication goals. 95 And 85 per cent of 3rd/4th Framework Programme public sector participants declared that they had achieved their publication goals. 96 In a study of Irish participants in the 4th Framework Programme, 83 per cent of university/research institute respondents listed publications in refereed journals as important outputs. 97 Over 80 per cent of Austrian university participants in the 4th Framework Programme responding to a survey (and over 70 per cent of such research institute participants) had already reached their publication goals or expected to do so within the next three years. 98 A recent IST impact study obtained for publications a score for goal achievement of over 3 (important goal achieved as expected). 99 In the majority (79 per cent) of projects under the 4th and 5th Framework Programmes with British participation surveyed, at least one peer-reviewed publication was produced, while in around 10 per cent of those projects more than 20 such outputs were produced. 100 An analysis of impact variables resulting from research projects in the fisheries and aquaculture domain of the FAIR programme under the 4th Framework Programme identified 711 publications in 219 peer-reviewed journals by the participants of 82 projects, an average of 8.7 peer-reviewed publications per project. 101 Participants in the BRITE-EURAM II programme under the 3rd Framework Programme reported 3 621 publications for 454 projects, an average of 8 publications per project. 102 Participants in the transport programme under the 4th Framework Programme reported 3 766 publications for 269 shared-cost projects, an average of 14 publications per project. 103 The scientific publications resulting from FP-funded collaborative research projects are often intra-European international co-publications, which points once more to the integrating effect of these projects. Direct evidence for this comes from a recently completed bibliometric study on participants in the 5th Framework Programme in the life sciences and nanotechnology, which found that for those Framework Programme participants the growth in intra-European international co-publications was much larger than that in international copublications with US authors or in the overall number of scientific publications. 104 This may explain the fact that also at the aggregate level the share of intra-European international co-publications in Europe's international co-publications has increased, while that of international co-publications with the United States has decreased. 105 The scientific publications resulting from Framework Programme collaborative research are generally of high quality. It is well known that the citation impact score of international co-publications is generally higher than that of purely national publications. 106 Direct evidence, however, also comes from the aforementioned recently completed bibliometric study on participants in the 5th Framework Programme, which found that the citation impact scores of those Framework Programme participants were higher than could be expected on the basis of the journals in which they published, and that they scored above world level when allocated to specific fields. 107 Impact on technological and innovative performance Framework Programme participation enhances the development and use of new tools and techniques; the design and testing of models and simulations; the production of prototypes, demonstrators, and pilots; and other forms of technological development. 108 One finds confirmation of a strong impact on the innovative performance of European firms in analyses of the Community Innovation Survey (Fig. 4.19) . 109 The results show that firms participating in the Framework Programme, irrespective of their size, tend to be more innovative than those that do not participate. Although no causal links can be "proven" by these results, they nevertheless provide a strong indication that public funding for research strengthens innovation performance. 110 FP-funded collaborative research projects also generate a large number of patents, once more exerting a positive influence on an important S&T indicator. A 2002 assessment of nearly 1 900 non-nuclear energy research and demonstration projects under the 4th Framework Programme found that they had resulted in about 400 product/process innovative companies as a proportion of companies with FP funding product/process innovative companies as a proportion of companies without FP funding Fig. 4.19 . FP participants are more likely to produce product/process innovations Source: DG Research, Eurostat Data: Eurostat Note: Results here are for firms in the manufacturing sector patent applications. 111 Corroborating evidence is again found in the Community Innovation Survey which shows that Framework Programme participating enterprises are more likely to apply for patents than non-participants (Fig. 4.20) . In Germany, for example, FP-funded firms submit three times as many patent applications as non-participating firms. Again, no causal links can be "proven" by these results. Nevertheless they provide a strong indication that public funding for research strengthens innovation performance.

Programme not only strengthens participants' mid-to long-term ability to compete, but also directly enhances their competitive position. A wide range of ex-post evaluation studies show that as a result of Framework Programme participation firms are able to realise increased turnover and profitability, enhanced productivity, improved market share, access to new markets, reorientation of the commercial strategy, enhanced competitive position, enhanced reputation and image, and reduced commercial risk. 112 Companies that applied for a patent as a proportion of companies with FP funding Companies that applied for a patent as a proportion of companies without FP funding 

One of the important effects of Framework Programme collaborative research projects is the enhancement of skills of participating research staff, which helps to strengthen research capabilities. Some Framework Programme actions also explicitly target the training, mobility, and career development of researchers, notably the "Marie Curie" actions. These commenced under the 3rd Framework Programme and now account for about 10 per cent of the 6th Framework Programme's budget. The purpose of the Framework Programme human resources schemes is to enable researchers to participate in top transnational teams and high-level projects, and benefit from training and knowledge sharing. This should ultimately lead to better R&D, while also having a positive impact on the attractiveness of the EU as a place to pursue a scientific career for European and third country researchers.

The Framework Programme human resources schemes are very much in demand. In the period 1994-2002, almost 12 000 researchers undertook international research training funded by the Marie Curie fellowships under the 4th and 5th Framework Programmes. 113 The 6th Framework Programme is characterised by high numbers of applications as well. 114 Widely appreciated and prestigious, it is useful to take a closer look at the characteristics of and impacts achieved through the Marie Curie fellowships. 115 Most fellows (79 per cent) applied to the scheme to gain international experience, and a substantial share of them (28 per cent) would not have gone abroad without a fellowship (which has a bearing on the question of additionality). In addition, the scheme enabled many fellows (50 per cent of post-graduate and about one-third of post-doctoral researchers) to gain their first international experience. Fellows mostly came from Spain (16 per cent), Italy (14 per cent), and Germany and France (13 per cent each). They mostly went to the United Kingdom (28 per cent), France (17 per cent), Germany (12 per cent), the Netherlands (9 per cent), and Spain and Italy (6 per cent each). This turned some countries (e.g. the United Kingdom, Denmark, Netherlands, Norway) into clear "host" countries (hosting more fellows than sending) and other ones (e.g. Slovak Republic, Iceland, Hungary) into clear "home" countries (sending more fellows than hosting), with the reminder reaching a rough balance. Most fellowships by far took place in an academic setting, though the 5th Framework Programme included a special fellowship programme for industry in which 8 per cent of respondents to a survey participated. About one-fifth of fellows were engaged in university-industry collaboration during their fellowship.

Marie Curie fellows identified as most important impacts for themselves having gained international research experience, having had dedicated time to carry out research, and having developed research skills. Marie Curie fellowships generated a number of tangible research outputs. Scientific publications were produced, their number depending on the duration of the fellowship and the experience of the fellow. The longer they stayed, the more likely they were to publish. Of those on fellowships of up to 6 months (for 94 per cent at postgraduate level), 57 per cent produced at least one publication. Among fellows on fellowships of between 6 and 12 months (for 65 per cent at postgraduate level), the per centage rose to 77 per cent. And among fellows on fellowships of at least 12 months (for 92 per cent at postdoctoral level), it rose to 90 per cent. Looking only at postdoctoral fellows, the average publication rate was 3.1 publications per year for a fellowship at a university, 2.6 for one at a research centre, and 1.2 for one in industry. Of respondents who had already concluded their fellowship, 7 per cent reported that their research project had resulted in one or more patents (one-third owned by the fellow and two-thirds owned by the host organisation). Supervisors were of the opinion that in 43 per cent of cases, the work of the fellow had led to the creation of new technologies.

After their fellowship, a majority of fellows eventually returned to their home country, though planning a new mobility experience within five years. They generally returned to positions of more responsibility under more stable contracts. They took with them the networks established and reinforced through their fellowship. The scheme strengthened existing connections for 34 per cent of postgraduate researchers, 40 per cent of postdoctoral researchers, and 73 per cent of senior researchers. Over 70 per cent of fellows reported that their stay abroad generated new contacts influential for subsequent career progression, and 86 per cent maintained contacts with their Marie Curie host institution after the fellowship.

If Europe's research teams are to remain at the forefront of all fields of science and technology, then they will have to be supported by state-of-the-art infrastructures. The successive Framework Programmes have been active in this sense, promoting the development of a fabric of research infrastructures of the highest quality and performance in Europe, and their optimum use. The term "research infrastructures" refers to facilities and resources that provide essential services to the research community in both academic and/or industrial domains. Research infrastructures may be "single-sited" (single resource at a single location), "distributed" (a network of distributed resources, including infrastructures based on Grid-type architectures), or "virtual" (the service being provided electronically). Examples of research infrastructures include large-scale research installations, collections, special habitats, libraries, databases, integrated arrays of small research installations, highcapacity/high-speed communications networks (e.g. Géant), networks of computing facilities (e.g. Grids), and infrastructural centres of competence, which provide a service for the wider research community based on an assembly of techniques and know-how.

To develop a strategic approach for research infrastructures at European level, comprehensive and up-to-date information about the current pattern of existing infrastructures is essential. To this end, the Commission organised a Survey of European Research Infrastructures between December 2004 and January 2005. Existing research infrastructures of clear European dimension, as well as a certain number of infrastructures under construction, were invited to participate The first round of the survey resulted in feedback from 585 existing research infrastructures and 157 under construction, that is, in total 742 research infrastructures were reported.

The technical review panel for the access to research infrastructures action under the 5th Framework Programme concluded that it succeeded in providing researchers with access to unique or outstanding infrastructures in a broad range of disciplines, and thus contributed to the production of high quality research. 116 It also noted that the Framework Programme helped to develop interdisciplinary approaches between several areas as well as cooperation between European researchers, while giving a European dimension to several national facilities and, in several cases, organising more efficiently their networking. Through infrastructure RTD projects, it also contributed to improving the technical performance of several categories of infrastructures and thus to enhancing European scientific competitiveness in several fields.

A questionnaire to participants who received 5th Framework Programme funding for transnational access to infrastructures, carried out as part of this technical review, found that 88 per cent of respondents would have been unable to carry out their project at this research infrastructure without EU support. Of these, 71 per cent indicated that they would have been unable to pay for travel and subsistence, while 41 per cent could not have afforded the user fees.

Although the vast bulk of public R&D spending in Europe is still carried out at national or regional level, these substantial resources remain fragmented and poorly coordinated. As was noted in the Communication on the European Research Area, "the European research effort as it stands today is no more than the simple addition of the efforts of the 15 [now 25] Member States and the Union. This fragmentation, isolation and compartmentalisation of national research efforts and systems and the disparity of regulatory and administrative systems only serve to compound the impact of lower global investment in knowledge". 117 Before the 6th Framework Programme, successive Framework Programmes focussed on fostering cooperation between research actors at "project" level by bringing together universities, research agencies, and companies. They were not used as vehicles for promoting the better coordination of national R&D programmes. This changed under the 6th Framework Programme when following the ERA initiative the first steps were taken to move cooperation forward to the next level by coordinating at programme level. The main tools proposed under the 6th Framework Programme for promoting the coordination of national and regional research at programme level were the ERA-NET scheme and the application of Article 169 of the EC Treaty.

In early 2002, EU research ministers recognised the importance of the mutual opening of national research programmes. CREST followed this impulse by launching five pilot actions for the mutual opening of national programmes in March 2002. 118 During the course of 2003, CREST came to the conclusion that the exchange of good practices between national programme managers was very useful, but that there was a need for a framework for further discussion. The ERA-NET scheme was considered to be the most suitable mechanism for pursuing the first stages of coordination.

The ERA-NET scheme was a highly innovative component of the 6th Framework Programme. 119 Its objective was to contribute to the creation of the European Research Area by facilitating practical initiatives to coordinate regional, national and European research programmes in specific fields, and to pool fragmented human and financial resources in order to improve both the efficiency and the effectiveness of Europe's research efforts. It provided support for creating close, long-term links between national research programmes with shared goals. The scheme's participants were programme funders and managers working in national and regional ministries and funding agencies rather than universities or enterprises. Ultimately ERA-NETs were expected to lead to collaboration of major significance, including the strategic planning and design of joint research programmes, the reciprocal opening of national research programmes to researchers from other member countries, and the launch of fully transnational programmes jointly funded by more than one country. Interest in the scheme was large, which is evidenced by the fact that by 2006 

The above sections have dealt mainly with impacts that are produced within a relatively short period of time, and that are "manageable" from a data collection point of view because they are situated at the project level. Much more complex, however, is the assessment of the Framework Programme's wider impacts. It is well known that the effects of R&D can be spread out over very many years, and it is therefore important that research projects and programmes are not judged narrowly in terms of there immediate outputs. Moreover, the benefits of research are not confined to the R&D actors themselves, or to the beneficiaries of public funding (such as Framework Programme participants), but spill over across many actors, sectors of society, and the economy.

Measuring the economic impact of research is extremely difficult. Among the complicating factors are the long lead time between R&D activity and the eventual economic exploitation of the results, and the problem of unambiguously attributing effects on a firm's turnover to specific research projects. Even if one manages to Legal Nevertheless, efforts have been made to try and evaluate the wider economic impacts of the Framework Programme. One approach that has been usefully employed in the past is mathematical modelling. For example, a recent UK study estimated the impact of the Framework Programme on the United Kingdom's total factor productivity using a model developed at the OECD. It was found that application of the formula generated an estimated annual contribution to UK industrial output of over £ 3 billion, a manifold return on UK Framework activity in economic terms. 123 The European Commission's Joint Research Centre at Ispra subsequently extended the analysis to all Member States for which data were available. The results seem to indicate significant effects on total factor productivity ( Fig. 4.21) . For example, for Finland, first estimates suggest that 0.9 per cent of the value added of industry per annum is attributable to funding from the Framework Programme, while many Member States record even higher contributions. It is estimated that on average E1 of Framework Programme funding leads to an (long-term) increase in industry added value of between E7 and E14, depending on the assumptions and parameters used. This increase will be spread over a number of years, because there is always a time lag before R&D spending has its economic effects. Of course, it is also important to stress that econometrics is not an exact science, and that such results must be treated with a large degree of caution. Other evidence on the positive macro-economic impact of the Framework Programme comes from the projections regarding the macro-economic impact of the 7th Framework Programme carried out via the Némésis model and discussed in a later chapter in this book (Chapter 6), as well as from BRITE-EURAM impact studies on a selection of projects. In the latter, it was found that nearly 1 600 new jobs were created and another 1 000 safeguarded. 124

It is not possible to present -like in the case of macro-economic impacts -in an aggregate way the many social and environmental societal challenges to the meeting of which FP-funded research projects have contributed directly. Only an extensive enumeration of case studies would be able to provide a proper insight into the vast range of areas in which valuable contributions have been made. It is important to note, however, that ever since the 5th Framework Programme the Framework Programme has been more explicitly positioned as having to contribute to the solution of such societal problems. In other words, ever since the 5th Framework Programme, research has become more mission oriented than curiosity driven. Some examples of the contribution made by the Framework Programme to societal and environmental goals include the following:

• Developing the sophisticated analytical tools needed to predict and optimise the net environmental, economic, and social impacts of new policy measures. For example, 3E -a powerful computer model of interdependent economic, environmental, and energy variables -enabled EU negotiators at Kyoto to calculate the cost implication of different policy scenarios on the spot; • Producing forecasting systems to give early warning of flooding, improving water treatment technologies, and planning tools to protect coastal zones against erosion and pollution; • Creating many of the new technologies on which sustainable development will depend -for instance, new sensor systems to monitor levels of pollutants in the rivers that supply our drinking water; • Strengthening the environmental research infrastructures which underpin European work in priority areas like climate change, encouraging coordinated transnational use of complex data sets, and state-of-the-art facilities. The inclusion of socio-economic research in the Framework Programme has surely made it easier for European researchers to help meet the many social and environmental challenges Europe is facing. Through the Targeted Socio-economic Research (TSER) Programme under the 4th Framework Programme, the Key Action "Improving the Socio-economic Knowledge Base" under the 5th Framework Programme, and the Thematic Area "Citizens and Governance in a Knowledgebased society" under the 6th Framework Programme, efforts have been made to improve the understanding of the major structural changes taking place in European society, to identify ways of managing these changes, and to involve European citizens more actively in shaping their own future. Issues related to the emergence of a knowledge-based society and new forms of relationships between citizens on the one hand and between citizens and institutions on the other are being addressed. The activities carried out in these programmes mobilise the wealth and diversity of European research in the economic, political, and social sciences, and the humanities.

Wider social and environmental benefits have also been generated by the fact that policy-makers are increasingly well-informed and policies are increasingly evidence-based. This is the result of three different trends. First, there is the aforementioned inclusion of socio-economic research in the Framework Programme. Second, there is the inclusion in the Framework Programme of the so-called "scientific support" for policies. Third, there is the increasing willingness to base the development of new policies on the results of FP-funded research projects.

Research serves more and more as the knowledge-base referred to in key policy documents. 125 The key internal provider of such scientific support for EC policies is the Joint Research Centre (JRC). Its role as a Community reference for EU policy in scientific and technological questions is epitomised by the increasing volume of legislation which is based on the work of (and which mentions) the JRC. 126 Thus European BSE (Bovine Spongiform Encephalopathy) research, to which the Community contributed E90 million in the period 1996-2003, provided the basis for close to 300 scientific opinions in support of almost 40 pieces of Community legislation in the fields of consumer protection, public health, and risk management. This research was proactive and explorative in nature, and thus enabled a rapid response to increased consumer food threats. In fact, the announcement in March 1996 of the links between BSE and the new variant Creutzfeld-Jacob Disease was the result of Community-funded collaborative research. Even more recently, with the outbreak of the SARS epidemics, Community action proved extremely adaptive and prompt. Some 70 projects under the Environment and Sustainable Development Programme of the 5th Framework Programme were explicitly referred to in various 125 See notably European Commission, The Overall Socio-Economic Dimension of Community Research in the European Framework Programme, Luxembourg, 2003. 126 In 2003, the JRC provided scientific and technical support to over 80 pieces of EU legislation. More widely, it should be noted that the JRC implements its mission through Direct Actions under the nuclear and non-nuclear Framework Programmes (FP). Thematically, the focal points are spelled out in the Specific Programmes (nuclear and non-nuclear parts) and are further defined in the Multi-Annual and Annual Work Programmes. To a lesser degree the JRC also uses Indirect Actions under the FPs, additional work for customer DGs, Enlargement Actions and Third Party Work to implement its mission. See also: http://www.jrc.ec.europa.eu EU policy documents. This provides an indication of the actual exploitation of research results for policy support. 127 A case in point is also the European Climate Change Programme (ECCP), which explicitly recognised the role of research in the energy and environmental fields. The preparatory work that led to the adoption of the Directive establishing a scheme for greenhouse gas emission allowance trading was supported by the results of the PRIMES and POLES models developed under previous Framework Programmes. 128 A number of societal issues (ethics, gender issues, environmental issues) have also been put on the agenda via the Framework Programme proposal evaluation process. Since the 5th Framework Programme, each project's environmental targets and objectives must be detailed in the project proposal. This plays an important role in the evaluation and selection of projects to be funded. Nonetheless, the detail provided on these targets and objectives is seldom sufficient to allow for subsequent systematic monitoring and evaluation at the programme level. Therefore, it is difficult to quantitatively assess the environmental effects of these projects (e.g. in the form of reduced emissions, or decreased health risks, etc.) at an aggregate level. Even so it is widely recognised that most RTD projects funded by the Framework Programme generate either directly or indirectly positive environmental impacts.

As discussed in the beginning of this chapter, many industrialised countries outside Europe participate in the Framework Programmes. Some -including the United States, Canada, Israel, and Australia -have already signed S&T agreements with the EU. For these countries cooperation with Europe is attractive. A study commissioned by the Australian Government Department of Education, Science and Training (DEST) on Australian science and technology cooperation with Europe, for instance, concluded that Australian researchers want to work with Europe because it is seen as a site of leading-edge collaboration, and that there is a large potential for expanded collaboration.

International S&T cooperation with developing countries via the Specific International Scientific Cooperation Activities (INCO) programme of the Framework Programme established in 1983 starts from the concept of sustainable development and the idea that poverty and social marginalisation can be overcome successfully by investing in human and institutional resources. based on dialogue and promotes the development of long-term durable research partnerships, with four main regions: Latin America, Asia, and Africa; the Mediterranean countries; the Western Balkans; and the Newly Independent States. 129 Though difficult to quantify, the impact of the Framework Programme in these regions is fourfold. First, research projects contribute directly to meeting local challenges. Positive impacts concern, for instance, scientific job creation, substitution of imported goods, improved food quality, and environmental protection from rapid industrialisation and urbanisation. Impacts of Asian and African INCO-funded projects in the fields of agriculture, food, and health concern improvements in farmer incomes, gender equality, public health, education, employment, protection against erosion, conservation of natural resources, and training and professional improvement. Second, Framework Programme projects also strengthen the local capabilities to meet local challenges. 130 Third, Framework Programme projects inform local policy development. 131 And fourth, Framework Programme projects promote regional collaboration.

Assessing the impacts of research programmes is complex, and even more difficult in the case of the Framework Programme. Nevertheless, a number of definite things can be said about past Framework Programmes. Across Framework Programmes, the collaborative research component has been characterised more by growth in the number of projects and participations than by growth in the average size of the project, raising the issue of the optimal size of projects and critical mass. The degree of industrial participation has become smaller, while that of universities and research institutes has increased. The by-country ranking in terms of participations and funding is more or less in line with that in terms of GDP or number of researchers, although smaller countries prove to be more "efficient" in obtaining funding than larger ones. The by-country ranking in terms of Framework Programme participations has not changed very much across Framework Programmes. The same finding is valid for the regional distribution of participations and funding: the regions with the highest income levels and/or the highest research activity are the front runners. However, a large number of peripheral regions are allocated a larger share of European research funds than could be expected on the basis of their R&D effort, economic importance, and so on. Regarding individual participants, it was shown that a relatively small number of organisations participate in a relatively large number of projects across calls and across Framework Programmes, collecting most participations and most funding. Collaborative research projects establish an everincreasing number of cross-sectoral and cross-border collaborative links. Networks remain remarkably stable after the end of projects and have a multiplier effect on a country's returns from the Framework Programme. Collaborative research has been characterised by an ever-increasing number of thematic priorities, avoiding major gaps in terms of coverage, though this does not seem to have caused fragmentation. Framework Programme collaborative research is generally considered to be of high quality, though not sufficiently risky and at the frontier. Community-funded collaborative research is characterised by a very high degree of additionality. In the absence of Framework Programme funding, projects would not have been carried out at all or would have been carried out in a very different manner with reduced objectives and a smaller number of partners. Framework Programme collaborative research has clear impacts on capabilities, behaviour, and competitivity, and on scientific and innovative performance. The Framework Programme also contributes to human resources development and has a structuring effect through its coordination of national research programmes.

We have seen in this chapter that learning from past experiences and identifying the strengths and weaknesses of previous Framework Programmes draws upon the evidence of ex-post evaluations. And such evaluations rely to a large extent on good and robust data. In the past few years, the Commission has undertaken important efforts to improve the completeness and consistency of cross-Framework-Programme-participation data. These ongoing efforts need to be continued and consolidated. Special attention should also be paid to developing innovative, including qualitative, approaches for assessing project outcomes. 132 When conceiving its new Framework Programme, the Commission did not only look backwards to the experiences gained with previous Framework Programmes. It also started an open broad-ranging dialogue with the key actors who would be affected or involved. This will be the subject of the following chapter. 132 In this context, it needs to be noted that the Commission is supported by the Member States. 

European Commission, Five-Year Assessment

Co-operation Pinpoints Potential Treatments for SARS

News Alert -Commission Releases New Funds for Avian Flu Research

European Commission, Five-Year Assessment

European Commission, Five-Year Assessment

The Impact of the EU Framework Programmes in the UK

Evaluation of Norway's Participation in the EU's 5th Framework Programme

The 4th Framework Programme in Ireland -An Evaluation of the Operation and Impacts in Ireland of the EU's Fourth Framework Programme for Research and Development

Austrian Participation

Finnish Participation in the EU Fifth Framework Programme and Beyond

Austrian Participation

The 4th Framework Programme

Final Report -Assessment of the Impact of the Actions Completed Under the 3rd and 4th Community Framework Programmes for Research; Survey for the Five-Year Assessment of Community Research Activities

The 4th Framework Programme

Austrian Participation

European Commission, Five Year Assessment Report Related to the Specific Programme Competitive and Sustainable Growth Covering the Period

Bibliometric Analysis

Indicateurs de Sciences et de Technologies Édition

National Characteristics in International Scientific Co-Authorship Relations

Proceedings of the Biennial Conference of the International Society for Scientometrics and Informetrics

A Bibliometric Analysis of International Scientific Cooperation of the European Union (1985-1995)

Bibliometric Analysis

Impact Assessment of the Marie Curie Fellowships Under the 4th and 5th Framework Programmes of Research and Technological Development of the EU

A Mobility Strategy for the European Research Area" and "Researchers in the ERA: One Profession, Multiple Careers

The remainder of this section is based on van de Sande et al., Marie Curie Fellowships