key: cord-0041964-d9rlu0ym
authors: Gouglas, Dimitrios; Marsh, Kevin
title: Prioritizing investments in new vaccines against epidemic infectious diseases: A multi‐criteria decision analysis
date: 2019-07-01
journal: nan
DOI: 10.1002/mcda.1683
sha: 6ad866c07f8cd10139fc9dbe2fa6ea225bdc876f
doc_id: 41964
cord_uid: d9rlu0ym

BACKGROUND: In 2016, the Coalition for Epidemic Preparedness Innovations (CEPI) launched a call for proposals (CfP) for vaccine development against Lassa, MERS, and Nipah. CEPI is faced with complex decisions that involve confronting trade‐offs between multiple objectives, diverse stakeholder perspectives, and uncertainty in vaccine performance. OBJECTIVE: This study reports on a multi‐criteria decision analysis (MCDA) and its testing on CEPI decisions. METHODS: Consultations with CEPI's Scientific Advisory Committee (SAC) and document reviews helped identify and structure the criteria against which to evaluate proposals. Forty four subject‐matter experts assessed performance of 18 proposals on multiple criteria. SAC preferences were elicited via a survey employing an adapted swing‐weighting technique and were incorporated into measures of value and cost‐to‐value. A Monte Carlo simulation estimated overall value and ranking probabilities by value and by cost‐to‐value for each proposal. RESULTS: Reviewer assessments and SAC preferences varied significantly. Despite this uncertainty, 14 preferred proposals emerged from the analysis and SAC recommendations on the basis of value and cost‐to‐value. In some cases, SAC recommendations deviated from the analysis because of: less emphasis on cost‐to‐value if budgets seemed underestimated by applicants, more emphasis on the likelihood of generating vaccines for target pathogens versus platform potential against unknown pathogens, and emphasis on funding a diversity of platforms per pathogen. CONCLUSIONS: Despite vaccine performance uncertainty and stakeholder preference heterogeneity, MCDA distinguished between options in a way that broadly corresponded to decisions. Divergence between the MCDA and the SAC point to potential updates needed to the model such as platform diversity trade‐offs.

Following the successful vaccine research and development (R&D) response to the 2014 West-African Ebola epidemic (Grobusch & Goorhuis, 2017) , the World Health Organization (WHO) prioritized 11 epidemic infectious diseases (EIDs) most likely to cause severe outbreaks in the near future (WHO, 2016) . Vaccines can prevent EID outbreaks from becoming humanitarian crises (CEPI, 2016a; Kieny et al., 2016) . However, market incentives have failed to sustain R&D efforts in this area (Plotkin, 2016) . A new entity, the Coalition of Epidemic Preparedness Innovations (CEPI), was set up in 2016 with a US$1 billion investment target to support the development of vaccines, contributing to the world's preparedness for EID outbreaks (CEPI, 2016a).

One of CEPI's first business plan targets was to advance the development of two to three vaccine candidates against priority EIDs from preclinical through to end of early clinical safety and immunogenicity testing (Phase IIa) by 2022 (CEPI, 2016a) . By doing so, CEPI aimed to address the "just-in-case" R&D preparedness gap associated with lack of Phase IIb/III-ready EID vaccines in advance of epidemic outbreaks.

Just like any funder of pharmaceutical R&D, CEPI is faced with the challenge of prioritizing limited resources in order to meet inherently risky R&D targets (Aurentz, Kirschbaum, & Thunecke, 2011) . Evidence suggests that the average probability of successfully advancing a vaccine candidate from preclinical through to end of Phase II is less than 10% (Pronker, Weenen, Commandeur, Claassen, & Osterhaus, 2013) . In addition, the large costs (DiMasi, Grabowski, & Hansen, 2016; and long timelines (Russell & Gronwall, 2012) involved in developing vaccines make investment decisions in this space tremendously complex. This complexity is compounded by the fact that where commercial objectives are lacking, such as in the field of EID vaccines, commercial value alone is an insufficient criterion to making pharmaceutical R&D investment decisions (Antonijevic, 2015; Cioffe, 2011; Perez-Escobedo, Azzaro-Pantel, & Pibouleau, 2012; Phillips & Bana e Costa, 2007) .

To address its business plan targets in line with mission and scope, CEPI launched a competitive call for proposals (CfP) in late January 2017 to support the development of vaccine R&D in three priority EIDs: Lassa Virus, Middle East respiratory syndrome coronavirus (MERS-CoV), and Nipah virus. The rationale was to invest in vaccine R&D projects that would improve the likelihood of generating vaccines relevant for use in response to these EIDs; as well as improve the likelihood that the platform technologies supporting these vaccines would be suitable for use in vaccine development against newly or unexpectedly emerging EIDs.

Evaluating proposals received in response to the call faces several challenges. First, pharmaceutical R&D portfolio management involves considering multiple criteria, including organizational capabilities, technical and manufacturing feasibility, development timelines and costs, and alignment with target product profiles (TPPs; Aurentz et al., 2011; Bode-Greuel & Nickisch, 2008; Seget, 2005) . For instance, the WHO has been advocating for use of ideal TPPs, or preferred product characteristics (PPCs) tailored to EID outbreak preparedness needs, to determine use potential of vaccines and to guide R&D priorities in the field (WHO, 2018).

Second, stakeholder opinions varied on the relative importance of different objectives. CEPI's Board has the ultimate decision-making authority on all CEPI R&D investments. An independent, multimember Scientific Advisory Committee (SAC) advises the CEPI Board and Secretariat on R&D investments and makes technical recommendations for project funding. The composition of the SAC is diverse, and at the time of the deliberations described here included nine representatives of governments and regulators, seven industry members, eight academics, and four representatives of non-profit R&D organizations (CEPI, 2016b) .

In this context of multiple trade-offs and heterogeneous stakeholder perspectives, multi-criteria decision analysis (MCDA) has the potential to improve the quality of decision making Viergever, Gouglas, & Tromp, 2017) . MCDA has become increasingly popular in health valuation Thokala et al., 2016) and its applications are numerous across a variety of areas in health (Adunlin, Diaby, & Xiao, 2015) and associated decision problems (Drake, de Hart, Monleon, Toro, & Valentim, 2017; Marsh et al., 2017; Thokala et al., 2016) . MCDA can offer a rational and transparent approach to priority setting, simultaneously considering all relevant criteria to avoid ad hoc decisions (Baltussen & Niessen, 2006) . Where there are multiple stakeholders with diverse perspectives, MCDA can make relevant conflicts explicit, helping decision makers understand them and consider their impact on decisions (Phillips & Bana e Costa, 2007; Timmis, Black, & Rappuoli, 2017) .

The use of MCDA has been increasingly advocated in vaccine R&D (Barrochi, Black, & Rappuoli, 2016; Timmis et al., 2017) . However, to our knowledge, only one other MCDA framework has been applied to support the prioritization of vaccine R&D (see for instance Phelps et al., 2014; Kloeber, 2011; Madhavan et al., 2012; Madhavan et al., 2013; Madhavan et al., 2015) . This framework places emphasis on different attributes of burden of disease, which are difficult to apply in the CEPI context, given the sporadic and unpredictable nature of EIDs.

Moreover, it does not lend itself easily to the estimation of value of vaccine R&D, which is adjusted for the probability of success (PoS) of early stage, risky vaccine candidates; nor does it assume sources of preference and constraints that are relevant to the CEPI decision context. An MCDA framework was developed in accordance with ISPOR Good Practice guidelines to inform the prioritization of EID vaccine R&D proposals and support CEPI CfP decisions and was tested against the SAC recommendations. This study reports on the application of the MCDA framework. Results are anonymized because of confidentiality restrictions associated with ongoing contract negotiations between CEPI and developers of selected proposals for funding.

The analysis focused on 18 full proposals that were selected by the CEPI SAC for an extended review following on an initial review of 33 preliminary proposals (CEPI, 2017a). The 18 proposals had a combined budget of over US$700 million and were reviewed by CEPI between March and May 2018. Seventeen proposals were at the preclinical development phase and one proposal was at clinical phase 1, with the aim that CEPI funding would advance them to the end of clinical phase 2. Proposals covered three different types of diseases: Lassa, MERS, and Nipah. Proposal budgets ranged from US$22 million to US$68 million, with a median cost of US$35 million. Proposal timeframes through to end of clinical phase 2 ranged from 4 to 6 years, with a median timeframe of 5 years. Due to confidentiality restrictions, individual proposal budgets and timeframes are not reported here; however, it is these budgets and timeframes that have been used to generate values in the framework presented below. Proposal names and disease classifications have been anonymized throughout the remainder of this manuscript. Proposals have been labelled as P1 to P18 and platform types are labelled 1-3. Seven proposals covered disease 1; seven proposals covered disease 2; and four proposals covered disease 3.

The goal was to undertake a quantitative valuation and ranking of the 18 proposals against criteria that were of interest to the SAC. It was assumed that not more than 14 proposals could be funded, given the resources available. The remainder of this section provides a step-by-step overview of the modelling approach adopted (more details provided in Data S1).

Step 1 To narrow down the list of criteria, and combine the criteria into a value framework, members of the SAC and CEPI secretariat staff were asked first in an email survey and then in a group discussion in January 2017 to determine: whether all factors relevant to CfP decisions had been captured by the criteria; the relationship between the criteria, and whether any of the criteria should be removed or re-grouped if overlapping, or irrelevant. Following this engagement, overall value (Vi) was estimated as described in

Equation (1).

Where: Table 1 . Each of these criteria is defined as a probability on a measurement scale 0-100%.

where:

C1 i = experience and track-record: Likelihood that the applicant is sufficiently competent to deliver on the proposed activities of the project, for a given proposal i.

C2 i = feasibility: Likelihood that the development of the candidate vaccine through phase II is technically feasible, for a given proposal i.

C3 i = manufacturing scalability and speed: Likelihood that the vaccine candidate is manufacturable and scalable in timeframes and volumes to respond to outbreaks, for a given proposal i.

C4 i = use potential for CfP target pathogens: Should a vaccine candidate be successfully developed and manufactured, the likelihood that it will meet CEPI's Target Product Profile and will be relevant for use in an emergency, for a given proposal i.

C5 i = use potential for new pathogens: Should a vaccine candidate be successfully developed and manufactured, the likelihood that the platform technology supporting the candidate vaccine will be suitable for use in vaccine development against newly emerging pathogens for a given proposal i.

The value framework presented in Equations 1-3 was presented to and approved by the SAC in February 2017, together with proposed criteria descriptions, measurement scales, and appraisal questions for reviewers (see Table 1 ). Overall likelihood that the project will generate a vaccine that is relevant for use in response to one of the CfP target pathogens (0-100%)

• Probability of successful vaccine development from preclinical through phase II (criteria C1 to C3) times the probability of use for CfP target pathogens (C4) O2. Likelihood that the platform technology will be suitable for vaccine development against new pathogens

Overall likelihood that the platform technology supporting the vaccine will be suitable for use in response to newly emerging and/or unexpected pathogens (0-100%)

• Probability of successful vaccine development from preclinical through phase II (criteria C1 to C3) times the probability of use for new pathogens (C5) does not have the potential to support vaccine development against a pathogen (C5)-so these criteria were combined multiplicatively.

Step 2 For each criterion C1-C5, reviewers were asked to define the most likely worst-case and best-case performance of proposals on a scale of 0-100% (see Data S1 for details). In order to determine the degree of homogeneity in the assessments provided by the different reviewers, an inter-reviewer assessment variability test was conducted. Specifically, for each criterion C1-C5, and for each performance estimate (worst-case, most likely, best-case), the following steps were undertaken. First, the performance mean across all reviewers assessing a given proposal was calculated. Second, the difference between this mean and each reviewer's performance estimate on the given proposal was calculated. Third, steps 1 and 2 were repeated for all proposals.

Fourth, for each reviewer, the average deviation of his or her performance estimate from the performance mean across all of his or her assessed proposals was estimated. Fifth, on the basis of Cicchetti's (1994) classification, reviewer variability was determined as good if this average deviated less than 20% from the performance mean, and excellent if it deviated less than 10% from the performance mean across all of his or her assessed proposals. Seven of the 44 reviewers were found to have at least one average worst-case, most likely, or best-case performance estimate against C1-C5 that deviated more than 20% from the equivalent performance mean across all their assessed proposals. In total, the estimates for which such deviations were observed accounted for only 3% of the total number of worstcase, most likely, and best-case performance estimates collected.

The impact of removing these results from the analysis was tested and found to not substantially change the performance of proposals.

Step 3. Estimating partial values (PV O1 , PV O2 ) Partial value functions were elicited for O1 and O2 from each SAC member using an online survey (24 respondents out of 29 survey recipients). The functions were defined using a mid-value splitting method-a widely-used decomposed scaling technique also known as the bi-section method (Von Winterfeldt & Edwards, 1986 )-by eliciting the value mid-point on a 10%-60% performance range (point a in Figure 1 ).

In the questioning procedure applied for the elicitation of partial values on each of O1 and O2, SAC members answered up to six pairwise choice questions that iteratively approached this value mid-point. For instance, for O1 the first question was:

"Consider the following two proposals, each with different starting likelihoods of generating a vaccine that will be relevant for use in response to one of the CfP target pathogens. Imagine you are

given the opportunity to improve the performance of one of these proposals. Which of the following options would you prefer?

• Option A: Improve Proposal A so that the likelihood that it generates a vaccine that will be relevant for use in response to one of the CfP target pathogens increases from 10% to x%

• Option B: Improve Proposal B so that the likelihood that it generates a vaccine that will be relevant for use in response to one of the CfP target pathogens increases from x% to 60%

In the first question, x was set as the mid-point in the performance range (35%). If a respondent was indifferent, the partial value function was considered linear, and no further questions were asked. If a respondent chose option A or option B, the value of x was updated according the logic defined in Section 2 of the Data S1.

The pairwise choice questions identified a to be within a range.

It was assumed that a was the mid-point in this range, on the basis of which partial value functions could then be defined (see Section 2 of Data S1).

Step 4. Estimating weights (W O1 , W O2 )

Weights were elicited for O1 and O2 using the trade-off method (von Winterfeldt and Edwards, 1986) , in each case for a range of performance of 10%-60%, from each SAC member using an online survey (24 respondents out of 29 survey recipients). An iterative pairwise comparison was used to identify the value of b, such that respondents would be indifferent between improving O1 from 10%

to y% and improving O2 from 10% to 60%. Specifically, the following question was asked via an online survey:

FIGURE 1 Partial value function of likelihood of generating a vaccine for one of the CfP target pathogens: an illustration "Considering the following two proposals, which of these would you prefer?

• Proposal A ○ Likelihood of generating a vaccine that will be relevant for use in response to one of the CfP1 target pathogens = y% ○ Likelihood that the technology will be suitable for use in vaccine development against newly emerging/unexpected pathogens = 10%

• Proposal B

○ Likelihood of generating a vaccine that will be relevant for use in response to one of the CfP1 target pathogens = 10%

○ Likelihood that the technology will be suitable for use in vaccine development against newly emerging/unexpected pathogens = 60%"

The initial value of y was set at 35%, and varied depending on responses as defined in Section 2 of the Data S1. After six questions, the value of b was defined within a range defined in Section 2 of Data S1. It was assumed that the value of b was the mid-point in this range.

Section 2 of the Data S1 provides more details on how b was used to estimate weights for O1 and O2.

Step 5. Eliciting time preference (r)

Time preference was estimated using a choice exercise designed to identify the value of c, such that SAC members were indifferent between a z% chance of successfully delivering a proposal within 5 years, and a 100% chance of doing so within 10 years. The following question format was implemented with SAC members in an online survey:

"Considering the following two proposals, which of these would you prefer?

• The value of z in the first question was set at 55%, and then varied depending on responses in a manner described in Section 2 of Data S1. After up to six questions, the value of c was identified within a range described in Section 2 of Data S1. It was assumed that c took the value of the mid-point in this range. Section 2 of the Data S1 provides more details on how c was used to estimated to estimate the discount rate.

Step 6 Dealing with uncertainty Both reviewer performance inputs and SAC preferences were subject to significant variations. This uncertainty was incorporated into the MCDA via Monte Carlo simulation. The model was run 10,000 times, each time drawing from the different inputs, as follows:

• Performance inputs: For criteria C1 to C5, each iteration randomly selected one reviewer and randomly selected a performance estimate from their performance distribution.

• SAC preferences: Each iteration randomly drew the partial value, weights, and time preference of a single SAC member's distributions.

The mean and 95% confidence intervals of performance on C1-C5, O1, O2, and V were estimated for each iteration of the simulation.

Comparison of proposals within each iteration allowed a ranking of proposals, which, when analysed across all iterations, allowed the estimation of the rank probability of a proposal.

Step 7 

Criteria performances of the 18 vaccine R&D proposals are presented in Table 2 . The uncertainty in performance means that there is substantial overlap in the confidence intervals around most proposals' performance on: the likelihood of generating a suitable vaccine for one of the CfP target pathogens (O1) and on the likelihood that the platform technology will be suitable for vaccine development against new pathogens (O2). Assuming that not more than 14 projects can be selected, Figure 3 plots the likelihoods of proposals ranking in the top 1-14, on the basis of discounted value versus cost-to-value. The consideration of budgets did not affect most of these ranking outputs in the analysis, for funding (proposals 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, and 18 Second, the SAC placed less emphasis on cost-to-value, as in some cases proposed budgets were considered unrealistic. One implication of this was that some proposals that had small budgets but whose overall value was otherwise low were not prioritized. Third, the SAC's deliberation pointed towards structural implications for the MCDA. A novel combined multiplicative-additive model structure was adopted. Few MCDA applications in healthcare have multiplicative components despite concerns that health technology assessment does not meet the analytical requirements of additive models (Marsh, Sculpher, Caro, & Tervonen, 2018) .

The multiplicative component of the model implicitly gave equal weight to criteria C1-C5, whereas the SAC deliberation seemed to emphasize some of these criteria more than others (e.g., C2 and C4).

Fourth, the SAC's recommendations could imply alternative weights to those used in the model; specifically that an even greater weight was given to O1 than what was elicited through the survey.

Weights in the model were elicited using an iterative comparison of improvements in pairs of criteria. This method was chosen due to the small sample size providing insufficient power for a discrete choice experiment; and a desire to elicit ordinal data in a survey format (Tervonen et al., 2017) . The result was that, on average, more weight was given to O1. Though there was also significant variation in SAC member weights. Given this variation, one possibility that would reconcile the SAC recommendations with the result of the MCDA is that SAC members who gave a higher weight to O1 were more influential in the deliberation.

In conclusion, the analysis reported in this study demonstrates that it is possible to use a MCDA to support the prioritization of vaccine R&D investments in a complex decision context characterized by outcomes uncertainty, variance in expected performance of vaccines, and heterogeneity of stakeholder preferences. With the intention to aid, rather than replace deliberative stakeholder processes or prescribe decisions, the findings illustrate how MCDA can help differentiate investments, and support decision making.

Application of multicriteria decision analysis in health care: A systematic review and bibliometric analysis

Optimization of pharmaceutical R&D programs and portfolios: Design and investment strategy

Revitalizing portfolio decision-making at Merck Serono S.A.-Geneva

Priority setting of health interventions: The need for multi-criteria decision analysis. Cost Effectiveness & Resource Allocation

The BARDA BAA Medical Countermeasures Acquisition Process

Multicriteria decision analysis and core values for enhancing vaccine-related decision-making

Value-driven project and portfolio management in the pharmaceutical industry: Drug discovery versus drug development-Commonalities and differences in portfolio management practice

Coalition for Epidemic Preparedness Innovations preliminary business plan 2017-2021

Interim scientific advisory committee (SAC)

Interim board meeting 11 and 12

Policy documentation

Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology

Portfolio selection and management in pharmaceutical research and development: Issues and challenges

Evaluation of the product development partnerships (PDP) funding activities. Web site

Innovation in the pharmaceutical industry: New estimates of R&D costs

Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA

European commission research & innovation: eligibility and evaluation criteria

A report to the Norwegian Agency for Development Cooperation (Norad): A review of MFA/Norad's support to global health product development

Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimisation study. The Lancet Global Health

Safety and immunogenicity of a recombinant adenovirus vector-based Ebola vaccine. The Lancet

IMI2 evaluation form for coordination and support actions: Single-stage and two-stage submission procedure

The need for global R&D coordination for infectious diseases with epidemic potential

Current and cutting edge methods of portfolio decision analysis in pharmaceutical R&D

on behalf of the Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health

on behalf of the Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health

King L (Eds.) on behalf of the Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine

The National Academies Press

Multi-criteria decision analysis to support healthcare decisions

Multiple criteria decision analysis for health care decision making-Emerging good practices: Report 2 of the ISPOR MCDA emerging good practices task force

The use of MCDA in HTA: Great potential, but more effort needed

Product development partnerships III Fund (PDP III)

Multiobjective strategies for new product development in the pharmaceutical industry

A priority-setting aid for new vaccine candidates

Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing

Vaccines for epidemic infections and the role of CEPI

Risk in vaccine research and development quantified

New vaccines against epidemic infectious diseases

U.S. medical countermeasure development since 2001: A long way yet to go

Pharmaceutical project and portfolio management: Effective organization and decision making for long-term growth

MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment

Multiple criteria decision analysis for health care decision making-An introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force

Improving accountability in vaccine decision-making

Multi-criteria decision analysis to support healthcare decisions

Decision analysis and behavioral research

An R&D blueprint for action to prevent epidemics: Plan of Action

Target Product Profiles (TPPs)

Multiple criteria decision making (MCDM): From paradigm lost to paradigm regained? Journal of Multi-Criteria Decision Analysis

How to cite this article: Gouglas D, Marsh K. Prioritizing investments in new vaccines against epidemic infectious diseases: A multi-criteria decision analysis

We thank the CEPI CEO, Dr. Richard Hatchett, the CEPI Deputy CEO, Dr. Frederik Kristensen, the CEPI Director of Vaccine Science, Dr. Gunnstein Norheim, and all CEPI staff for their support in the implementation of the study. We are grateful to all members of the CEPI Interim Scientific Advisory Committee who have contributed in the design of the model and who have responded to the preference elicitation exercise included in this study.

Mr. Gouglas 

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent was obtained from all individual participants included in the study.

Dimitrios Gouglas https://orcid.org/0000-0002-4299-8601