key: cord-0962867-wgtocdy2 authors: Hollingsworth, Rosalind; El Guerche‐Séblain, Clotilde; Tsai, Theodore; Vasiliev, Yuri; Lee, Sam; Bright, Helen; Barbosa, Paula title: Assessment of the benefits of seasonal influenza vaccination: Elements of a framework to interpret estimates of vaccine effectiveness and support robust decision‐making and communication date: 2020-09-03 journal: Influenza Other Respir Viruses DOI: 10.1111/irv.12786 sha: 597657cf6517f7869df981c58540f6024ce17789 doc_id: 962867 cord_uid: wgtocdy2 Systematic reviews and meta‐analyses confirm that influenza vaccination reduces the risk of influenza illness by between about 40% and 60% in seasons when circulating influenza stains are well matched to vaccine strains. Influenza vaccine effectiveness (IVE) estimates, however, are often discordant and a source of confusion for decision makers. IVE assessments are increasingly publicized and are often used by policy makers to make decisions about the value of seasonal influenza vaccination. But there is limited guidance on how IVE should be interpreted or used to inform policy. There are several limitations to the use of IVE for decision‐making: (a) IVE studies have methodological issues that often complicate the interpretation of their value; and (b) the full impact of vaccination will almost always be greater than the impact assessed by a point estimate of IVE in specific populations or settings. Understanding the strengths and weaknesses of study methodologies and the fundamental limitations of IVE estimates is important for the accuracy of interpretations and support of policy makers’ decisions. Here, we review a comprehensive set of issues that need to be considered when interpreting IVE and determining the full benefits of influenza vaccination. We propose that published IVE values should be assessed using an evaluative framework that includes influenza‐specific outcomes, types of VE study design, and confounders, among other factors. Better interpretation of IVE will improve the broader assessment of the value of influenza vaccination and ultimately optimize the public health benefits in seasonal influenza vaccination. Epidemiological factors account for some of these differences but often study design 3 and the assessment of different outcomes make it difficult to compare studies within and across different seasons. 4 The strengths and weaknesses of IVE study design have been extensively reviewed elsewhere. [5] [6] [7] [8] [9] [10] [11] But methodological deficiencies in the evaluation of influenza vaccine effectiveness, and misinterpretation of the outcomes, 12,13 pose a serious challenge to the use of seasonal influenza vaccination as a public health tool. 14 This paper reviews the critical considerations that should be made when interpreting IVE and proposes an evaluative framework to be used to interpret the outcomes of IVE studies as a basis for determining the full benefit of influenza vaccination. In sum, the contextual variability of IVE calls for the use of a common evaluative framework to ensure the consistency of the assessment of IVE studies outcomes and to ensure that the strengths, weaknesses, and limitations of IVE data are fully appreciated by the reader. To account for differences in reported IVE estimates, WHO recommends that reporting of IVE studies include sufficient details on study participants, data collection, and analyses to enable readers to judge the validity of each study. 14 With this in mind, and expanding on these ideas, we propose that an evaluative framework could be developed, which would include the following elements (as summarized in Table 2 and discussed below), to ensure the limitations of estimates of IVE, as an indicator of public health benefit, are fully appreciated and effectively communicated: For each of these outcomes, IVE may differ. This is evidenced by the variability of IVEs against different endpoints, from different systematic reviews or meta-analyses, as shown in Table 1 . IVE studies of asymptomatic influenza are important for assessing vaccination effectiveness against disease dynamics but are difficult and expensive to conduct. IVE studies of symptomatic influenza are important for assessing the impact of vaccination on the burden of disease, since even for those not seeking medical attention influenza is socially and economically burdensome. However, these studies are expensive and time-consuming since they often require active surveillance and testing. Medically attended IVE studies are the most common because they can be the least logistically challenging, but they are prone to bias and often do not/cannot capture the effectiveness that influenza vaccine may have on other population outcomes. Likewise, IVE studies against hospitalizations and severe outcomes tend to show the highest IVE, but they are subject to selection bias because hospitalized individuals may not be representative of the entire population. It is important to note that studies usually focus on outcomes clearly related to influenza (influenza-like illness, hospitalization for influenza/pneumonia), but ignore the broader impact of influenza and vaccines on outcomes where the role of influenza may be less apparent, such as exacerbations/destabilization of non-communicable diseases (NCDs) (eg, COPD, diabetes) or as a trigger of serious events, such as myocardial infarction or stroke. To recap, IVE is specific to an outcome, and each outcome assesses a different impact of vaccination. When evaluating IVE, it is important to be mindful of the outcomes reviewed and thus the relevance of the outcome to the population or setting when subsequently communicated. Experimental In observational studies, VE is estimated by rate ratios or hazard ratios of events occurring in vaccinated versus unvaccinated persons over time5. But these methods cannot control for bias to the same degree as prospective randomized clinical trials. 30 However, observational studies can estimate the impact of vaccination programs on the entire exposed population. In other words, experimental studies answer the binary question 'does the vaccine work?' (yes or no), whereas observational studies address 'how well a vaccine works'. A hybrid experimental-observational method, referred to as "pragmatic clinical trial" (PCT) 31 has also recently been used to estimate vaccine effectiveness. This study design investigates randomized groups prospectively, but measures endpoints from routinely collected data or vital statistics. The primary advantage of pragmatic/ hybrid clinical trials is that they can be designed to be more reflective of real-world vaccine experience, with research questions (such as outcomes, patients populations, and so on), that are more relevant to policy makers, clinical decision makers, and others as they seek to optimize immunization programs. This design is currently being used to compare the effectiveness against LCI of licensed egg-based inactivated influenza vaccines against two other types of licensed vaccines (cell-culture inactivated and recombinant). 32 The large number of study designs used to assess IVE underscores the imperfect nature of each. The RCT is the only design that controls for selection bias and con- Cohort studies, the gold standard design for estimating incidence rates, relative risks, and attributable risks, proceed in a logical sequence from exposure to outcome. They allow for hypotheses about causality of the exposure on the outcome and can provide an estimate of the VE. However, for influenza disease, at an attack rate of 5 to 10%, the duration and cost of studies can be prohibitive. Case-control studies identify cases (outcome) and then ascertain exposure status. This design is the most appropriate for diseases with low incidence rates or with long duration between exposure and outcome incidence. The main advantages of case-control studies are that they require a smaller sample size than cohort studies, they can be relatively inexpensive, and they can be relatively short in duration. They can also be used to assess VE in real time, from sentinel screening and surveillance. But since case-control studies do not measure incidence rates, VE is estimated from the odds ratio. When the relative risk of exposure is small (<5%), the odds ratio can approximate the relative risk. The main weaknesses of case-control studies are that the exposure status is subject to recall bias, and the strength of causal association (odds ratio) is not as not as deterministic as the relative risk. With large databases of patient registries, electronic health records, insurance data, web/ social media, or records of pharmaceutical products sales, observational data not collected under experimental conditions can be used to estimate "real-world" effectiveness. Advantages and weaknesses of these estimates are mainly linked to the quality and completeness of the data collected. A summary of the advantages and disadvantages of main study designs are summarized in Table 3 . Identifying factors that are strongly associated with vaccination and with disease risk is key to interpreting IVE estimates. At least two sets of factors play an important role in determining the likelihood of IVE against illness: 1) host characteristics (such as age, underlying health conditions, and level of pre-existing immunity to circulating strains of influenza), and 2) vaccine characteristics, including the match between circulating stains of influenza virus and influenza vaccine strains. 33 In years when vaccine strains are mismatched with circulating strains, IVE will be lower. These factors alone can explain most of the observed inter-season variability in benefit. 21 Several factors that confound comparisons between studies must be considered. The main factors that can confound IVE are identified in Table 4 . Some of these are discussed below. IVE is dependent on the match between vaccine strains and circulating strains. IVE will be lower in years when there is a mismatch. Generally, reassortant viruses must be adapted in eggs in order to produce high-yield candidate vaccine viruses. 38 During this process, mutations of influenza viruses may result in altered antigenicity. While antigenicity is continually tested, it can be difficult to accurately predict and these mutations may affect vaccine match with circulating strains. Vaccines are now available that are not manufac- Vaccine effectiveness is viral type/subtype specific. VE is typically highest for H1N1 and lowest for H3N2 (Table 1) to reduced VE. [41] [42] [43] VE for B types may be similar to H1N1 or slightly lower (see Table 1 ). Viral/ epidemiological Strain match-circulating strains can be genetically diverse, differing between seasons, geographic regions, and countries. Antigenic drift can occur after vaccine strain selection. Potential for adaptive mutations in HA during virus growth and passage in eggs or cell culture. Type/subtype-specific effectiveness-predominant circulation of a subtype as effectiveness is highest for H1N1 and lowest for H3N2. Transmissibility and virulence of viruses can vary (from mild to severe disease). Vaccine-type effectiveness-VE is brand-specific as production methods (split virus, sub-unit, etc) and processes (purification, inactivation, etc) differ between manufacturers. Sampling/ methodological Healthy user effect. Timing of enrollment into study during influenza season in relationship to vaccine uptake and incidence of influenza. Waning seroprotection Herd effects. Vaccination history. Accuracy of link between record of immunization in databases and cases. Vaccines are produced by several manufacturers, and none are identical. Differences in production methods (split virus, sub-unit, etc) and processes (purification, inactivation, etc) may mean that the nature of the vaccine itself may impact VE, making VE brandspecific. The use of quadrivalent vaccines versus trivalent vaccine may also affect the IVE estimates since more strains (2 lineages of B) are covered by quadrivalent vaccines. The DRIVE project1 seeks to achieve high quality, brand-specific effectiveness estimates for all influenza vaccines used in the EU each season. However, with frequent use of a number of different influenza vaccines in any given population, few vaccine registries and reliance on patient recall, developing valid brand-specific IVE estimates will be challenging. One of the main confounders in observational studies is the healthy user effect. This refers to a situation where patients in better health conditions are more likely to adhere to the annual recommendation for influenza vaccination. If not adjusted for comorbidities or health seeking behavior, the healthy user bias can overestimate IVE. Several approaches to control for this effect, such as by using propensity scores and other means, exist in the literature. Post-vaccination antibody titers decline over time, and although titers usually remain at levels above those associated with protection for the duration of an influenza season, 48 Vaccination history has become one of the latest confounders to come under scrutiny in test-negative design studies. Vaccine protection from influenza is not binary (all or nothing) but an odds ratio will not account for an incomplete protective effect of vaccination. 55 There are also questions about whether differences in estimates of IVE can be explained by factors like immune or vaccine history. For example, an unvaccinated individual may be afforded some protection from influenza due to prior exposure to influenza or previous vaccination history; thus, estimates of IVE are relative and not absolute measures of effectiveness. The accuracy of immunization records from clinical or administrative databases that are associated with cases and comparison groups is critical. Invalid immunization history may compromise the validity of a study, and reliance on patient recall may also confound estimates. While the need to monitor vaccine performance by estimating IVE is recognized, no single methodology is perfect, and the headlinegrabbing point estimates communicated during and at the end of each season hide the considerable public health benefits of vaccination, even in years with modest vaccine effectiveness. Annual studies of IVE have limitations, are difficult to generalize, and, if considered in isolation, will not provide a comprehensive picture of the public health impact of influenza vaccines. However, development and implementation of a common evaluative framework may offer a consistent and objective approach to the assessment of IVE estimates, and help to ensure that the strengths and weaknesses of the data can be appreciated. We recommend that before VE estimates are used to support policy, they should be evaluated against the criteria outlined Table 2 . Using a framework to support the development of vaccine policy will improve understanding of the impact of seasonal vaccination programs and thus may drive much-needed urgency in influenza prevention efforts. Similar frameworks and approaches could also be applied to assess VE for other vaccines (eg, SARS-CoV-2). The authors wish to acknowledge Shawn Gilchrist, President of S Gilchrist Consulting Services Inc, for providing his professional writing services in the development of this manuscript. https://orcid.org/0000-0002-8066-1081 Guideline on Influenza Vaccines Nonclinical and Clinical Module Development of Robust and Innovative Vaccine Effectiveness Estimates of influenza vaccine effectiveness in primary care in Scotland vary with clinical or laboratory endpoint and method-experience across the 2010/11 season Impact of influenza vaccination on healthcare utilization -A systematic review 2019 Challenges in estimating influenza vaccine effectiveness Bias of influenza vaccine effectiveness estimates from test-negative studies conducted during an influenza pandemic Caution required in the use of administrative data and general laboratory submissions for influenza vaccine effectiveness estimation Reply to Skowronski, De Serres and Orenstein A dynamic model for evaluation of the bias of influenza vaccine effectiveness estimates from observational studies A comparison of the test-negative and the traditional case-control study designs for estimation of influenza vaccine effectiveness under nonrandom vaccination On the bias of estimates of influenza vaccine effectiveness from test-negative studies Global seasonal influenza disease and vaccination: a paradox with substantial public health implications World Health Organization. Development of robust and innovative vaccine effectiveness. D4.6 Guideline for interpretation of influenza vaccine effectiveness result. Geneva: World Health Organization Evaluation of influenza vaccine effectiveness: a guide to the design and interpretation of observational studies. Geneva: World Health Organization Efficacy and effectiveness of influenza vaccines: a systematic review and meta-analysis Vaccines for preventing influenza in healthy adults Variable influenza vaccine effectiveness by subtype: a systematic review and meta-analysis of test-negative design studies World Health Organization. Vaccines against influenza WHO position paper Crude vaccine effectiveness" is a misleading term in test-negative studies of influenza vaccine effectiveness Influenza control in the 21st century: optimizing protection of older adults Influenza vaccines: from surveillance through production to protection Influenza vaccine effectiveness in the United States during the 2015-2016 Season Influenza vaccine effectiveness among high-risk groups: a systematic literature review and meta-analysis of case-control and cohort studies Interim estimates of 2018-19 seasonal influenza vaccine effectiveness -United States /16 I-MOVE/I-MOVE+ multicentre case-control study in Europe: Moderate vaccine effectiveness estimates against influenza A(H1N1)pdm09 and low estimates against lineage-mismatched influenza B among children US Centers for Disease Control and Prevention. CDC Seasonal Flu Vaccine Effectiveness Studies Effects of influenza vaccination in the United States during the 2017-2018 influenza season Interim estimate of 2019-2020 seasonal influenza vaccine effectiveness -United States Annual public health and economic benefits of seasonal influenza vaccination: a European estimate Epidemiologic methods in immunization programs Generating evidence for comparative effectiveness research using more pragmatic randomized controlled trials Pragmatic Assessment of Influenza Vaccine Effectiveness in the DoD (PAIVED): Methods. Poster Abstracts Vaccine Effectiveness -How Well Does the Flu Vaccine Work? Comparing influenza vaccine efficacy against mismatched and matched strains: a systematic review and meta-analysis Mismatch between the 1997/1998 influenza vaccine and the major epidemic A(H3N2) virus strain as the cause of an inadequate vaccine-induced antibody response to this strain in the elderly Effectiveness of influenza vaccines in preventing severe influenza illness among adults: A systematic review and meta-analysis of test-negative design case-control studies Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: Results from the 2005-2006 season of dual A and B vaccine mismatch in Canada Efforts to improve the seasonal influenza vaccine. Vaccines Relative effectiveness of the cell-cultured quadrivalent influenza vaccine compared to standard, egg-derived quadrivalent influenza vaccines in preventing influenza-like Relative effectiveness of cell-cultured and egg-based influenza vaccines among elderly persons in the United States The impact of influenza epidemics on mortality: introducing a severity index Mortality associated with influenza and respiratory syncytial virus in the United States Mapping the antigenic and genetic evolution of influenza virus Frequency and impact of confounding by indication and healthy vaccinee bias in observational studies assessing influenza vaccine effectiveness: a systematic review Influenza in the tropics The test-negative design for estimating influenza vaccine effectiveness HepA, flu vaccines Do antibody responses to the influenza vaccine persist year-round in the elderly? A systematic review and meta-analysis Modest waning of influenza vaccine efficacy and antibody titers during the 2007-2008 influenza season Rapid decline of influenza vaccine-inducedantibody in the elderly: is it real, or is it relevant? Decline in influenza vaccine effectiveness with time after vaccination Vaccine effectiveness of 2011/12 trivalent seasonal influenza vaccine in preventing laboratory-confirmed influenza in primary care in the United Kingdom: evidence of waning intra-seasonal protection Intraseason waning of influenza vaccine protection: evidence from the US Influenza Vaccine Effectiveness Network Duration of influenza vaccine effectiveness: A systematic review, meta-analysis, and meta-regression of test-negative design case-control studies Immune history and influenza vaccine effectiveness. Vaccines (Basel) Epidemiology and prevention of vaccine-preventable diseases. The Pink Book: Measles World Health Organization. International travel and health. Seasonal influenza Herd protection against influenza Original antigenic sin: a comprehensive review Impact of repeated vaccination on vaccine effectiveness against influenza A(H3N2) and B during 8 seasons Repeated seasonal influenza vaccination among elderly in Europe: Effects on laboratory confirmed hospitalised influenza Repeated annual influenza vaccination and vaccine effectiveness: review of evidence Effects of repeated annual inactivated influenza vaccination among healthcare personnel on serum hemagglutinin inhibition antibody response to A/Perth/16/2009 (H3N2)-like virus during 2010-11 The Impact of prior season vaccination on subsequent influenza vaccine effectiveness (VE) to prevent influenza-related hospitalizations over four influenza seasons in Canada the IFPMA Influenza Vaccine Supply (IFPMA IVS) task force. Assessment of the benefits of seasonal influenza vaccination: Elements of a framework to interpret estimates of vaccine effectiveness and support robust decision-making and communication