key: cord-0703600-8t43h7ha authors: Balasubramani, G.K.; Zimmerman, Richard K; Eng, Heather; Lyons, Jason; Clarke, Lloyd; Nowalk, Mary Patricia title: Comparison of local influenza vaccine effectiveness using two methods date: 2021-01-21 journal: Vaccine DOI: 10.1016/j.vaccine.2021.01.013 sha: 10010160147f0cf494408b9852ae0d6e1e014a8f doc_id: 703600 cord_uid: 8t43h7ha BACKGROUND: In some settings, research methods to determine influenza vaccine effectiveness (VE) may not be appropriate because of cost, time constraints, or other factors. Administrative database analysis of viral testing results and vaccination history may be a viable alternative. This study compared VE estimates from outpatient research and administrative databases. METHODS: Using the test-negative, case-control design, data for 2017–2018 and 2018–2019 influenza seasons were collected using: 1) consent, specimen collection, RT-PCR testing and vaccine verification using multiple methods; and 2) an administrative database of outpatients with a clinical respiratory viral panel combined with electronic immunization records. Odds ratios for likelihood of influenza infection by vaccination status were calculated using multivariable logistic regression. VE = (1 − aOR) × 100. RESULTS: Research participants were significantly younger (P < 0.001), more often white (69% vs. 59%; P < 0.001) than non-white and less frequently enrolled through the emergency department (35% vs. 72%; P < 0.001) than administrative database participants. VE was significant against all influenza and influenza A in each season and both seasons combined (37–49%). Point estimate differences between methods were evident, with higher VE in the research database, but insignificant due to low sample sizes. When enrollment sites were separately analyzed, there were significant differences in VE estimates for all influenza (66% research vs. 46% administrative P < 0.001) and influenza A (67% research vs. 49% administrative; P < 0.001) in the emergency department. CONCLUSIONS: The selection of the appropriate method for determining influenza vaccine effectiveness depends on many factors, including sample size, subgroups of interest, etc., suggesting that research estimates may be more generalizable. Other advantages of research databases for VE estimates include lack of clinician-related selection bias for testing and less misclassification of vaccination status. The advantages of the administrative databases are potentially shorter time to VE results and lower cost. Since 2011, the US Flu VE Network has been estimating influenza vaccine effectiveness (VE) using a test negative design that requires specimen collection from patients seeking medical care for an acute respiratory illness. Recruitment takes place in outpatient settings such as urgent care centers, primary care offices and emergency departments. Consenting, enrolling and swabbing for PCR testing for presence of influenza is frequently performed by research personnel who are not members of the clinical staff. With the onset of the COVID-19 pandemic, the ability to enroll patients with acute respiratory illness was abruptly curtailed. Many institutions temporarily ceased all but essential research, personal protective equipment (PPE) was in short supply and was being reserved for the protection of health care workers, while vast amounts of resources, human and otherwise, were diverted to the containment, treatment and prevention of SARS-CoV-2 infection. In addition, the ability to test for influenza and other respiratory viruses locally, was hampered by the health system's decision to prioritize SARS-CoV-2 testing over respiratory viral panels (RVPs) (Graham Snyder, MD, personal communication, 2020). The timeline for adding SARS-CoV-2 to the currently used multi viral testing https://doi.org/10.1016/j.vaccine.2021.01.013 0264-410X/Ó 2021 Elsevier Ltd. All rights reserved. platforms, and the duration and severity of the novel coronavirus pandemic remain unknown. It is unlikely that SARS-CoV-2 will displace influenza in the coming influenza season and the viruses will most likely co-circulate. Given that the need to determine influenza vaccine effectiveness continues, other methods of determining influenza vaccine effectiveness should be explored. Previous studies have used administrative databases to estimate influenza outpatient visits [1] , track influenza outbreaks [2] and calculate influenza VE in specific population subgroups such as pregnant women [3] and older adults [4] . In this study, VE estimates using methodology from the US Flu VE Network (research database) are compared with VE estimates using data from a clinical surveillance software system (administrative database). The advantages and disadvantages of both methods of estimating influenza VE are discussed. The study took place in emergency departments, hospital-based clinics and outpatient primary and urgent care sites that are part of a large (40 hospital, 700 doctor's offices and outpatient sites) integrated health system located in western and central Pennsylvania. Data used for this analysis were collected from three sources: 1) the health system's clinical surveillance software system which extracts virology test results from the EMR (Theradoc); 2) the Pennsylvania Statewide Immunization Information System (PA-SIIS); and 3) research data from local outpatient facilities and emergency departments participating in the US Flu VE Network. An IRB-approved honest broker extracted data from Theradoc on a cohort of Allegheny County residents who received an outpatient RVP test at a hospital-based clinic or emergency department of one of the general acute care hospitals in the health system during the study period that included the 2017-2018 and 2018-2019 influenza seasons. The RVPs were performed on a nasopharyngeal swab using the Genmark Luminex platform. RVP tests may be conducted year-round, but are more routinely conducted in October through April on patients presenting with acute respiratory symptoms. For patients with more than one visit with RVP tests 14 days apart, only data from the first visit were included. If visits occurred greater than 14 days apart, then data from both visits were included. Analysts created annual Boolean indicators for influenza from the RVP results. If no positive influenza result was observed, the final specimen collection date was saved to confirm there was no infection as of that date. This list also contained basic demographic data of race, sex and age. This list of patients was combined with immunization records from PA-SIIS. In cases where an individual had more than one vaccination in a given influenza season, the immunization records were reduced to a single record per patient per influenza season by selecting the first vaccination date. This dataset is henceforth called the ''administrative" database. The ''research" database was derived from participants who were recruited from ambulatory, urgent care clinics and emergency departments during the 2017-2018 and 2018-2019 influenza seasons for the US Flu VE Network study. Detailed study methods on the US Flu VE Network have been described elsewhere [5] [6] [7] [8] [9] . Briefly, patients aged !6 months presenting with an acute respiratory infection (ARI) including cough within 7 days of symptom onset were enrolled at participating outpatient healthcare facilities, including community physician offices, urgent care centers and emergency departments. Patients who had received antiviral medication in the 7 days before enrollment or had been enrolled in the prior 14 days were ineligible. Following informed consent, study staff collected respiratory specimens (nasal and throat swabs from patients aged !2 years or nasal swabs only from patients aged <2 years) for influenza virus testing (including virus type and subtype) by reverse-transcription polymerase chain reaction (RT-PCR). Demographic data were obtained from interview. Vaccination status was based on documented receipt of each year's influenza vaccine from PA-SIIS. The two databases were considered to be independent because they primarily included patients from different clinical sites and because enrollees who had clinical RVP testing and were enrolled in the US Flu VE network study accounted for <4% of the total administrative database. The influenza circulation period, defined as the dates between the first and last influenza positive research enrollment during each season, was determined for each year in both the administrative and research databases. Subjects with influenza testing performed outside the influenza circulation periods were excluded from analyses. The enrollment period details are shown in Table 1 . (See Figs. 1a and 1b) for the flow diagrams of participants in each database. Summary statistics of the demographic and clinical characteristics were determined for the administrative and research databases. Baseline characteristics were compared between vaccinated and unvaccinated patients using Chi-square tests for categorical variables. A test-negative study estimates VE by comparing the odds of vaccination among RT-PCR confirmed influenza cases to the odds of vaccination among controls. Using odds ratios obtained from multivariable logistic regression models, VE estimates were calculated as (1-aOR) Â 100. A series of logistic regression models was conducted with RT-PCR confirmed influenza A and B as the dependent variables and vaccination status as the independent variable. The primary analyses determined VE for all influenza; subgroup analyses determined VE for influenza A/H1N1, influenza A/H3N2, and influenza B (both lineages combined due to small numbers of cases). The logistic regression models were adjusted a priori for age group (6 months-17 years, 18-49 years, 50-64 years and 65+ years), sex, race (white, non-white), influenza season (2017-2018, 2018-2019), prior vaccination status for the immediately preceding year and whether the visit took place in the emergency department. The VE and its 95% CI reported for the two databases were also stratified by age group and by season. Thus, age group was not adjusted for in the age-stratified model and season was not adjusted for in models which stratified seasons. The significance of the difference between administrative and research database VE was identified through the effect of interactions in the logistic regression model. An indicator variable was created for database and the interaction of this binary indicator and the vaccination status was included in the model. Data were analyzed using SAS version 9.4 (SAS Institute, Cary, NC, USA). Statistical significance was defined as a two-sided P value < 0.05. The University of Pittsburgh IRB approved the study. There were significant differences in demographic characteristics between the administrative and research databases as shown in Table 2 . For example, compared with those in the administrative database, research participants were more likely to be younger, white and less likely to have been seen in the emergency department. Thus, the first difference between the methodologies is reflected in the demographics of the two populations. Table 2 also compares vaccinated with non-vaccinated participants and influenza cases and non-cases within each database. In the administrative database, the vaccinated compared with the unvaccinated differed by age, race, (69% vs. 51% white; P < 0.001), previous vaccination status (65% vs. 19%; P < 0.001) and number of emergency department enrollments (66% vs. 77%; P < 0.001). Among the individuals in the research database, compared with the unvaccinated, the vaccinated were older, more often white than non-white, female, previously vaccinated, and less frequently enrolled through the emergency department (all P < 0.001). In the administrative database, those infected with influenza were younger, non-white, not vaccinated in the previous season or the current season, enrolled in the 2018-2019 season and enrolled in the emergency department (all P < 0.01). Those in the research database with influenza were more likely to be non-white, male, not vaccinated in the previous season or the current season, and enrolled in the 2017-2018 season (all P < 0.05). Influenza circulation differed in the two seasons; influenza A/ H3N2 was the predominant strain in 2017-2018, although there was circulation of A/H1N1 and influenza B, whereas in 2018-2019, A/H3N2 and A/H1N1 circulated in nearly equal proportions with a small proportion of influenza B. Table 3 shows the unadjusted and adjusted VE estimates from each database for any influenza, influenza A, A/H1N1, A/H3N2, and influenza B for 2017-2018, 2018-2019, and for both seasons combined. Using the administrative database, significant VE estimates were observed for any influenza, and influenza A for both seasons individually and combined, and for both influenza A/H1N1 and A/ H3N2 for 2017-2018 and both seasons combined. VE for influenza and 64%), influenza A (60% and 64%), influenza A/H1N1 (78% and 72%) and influenza A/H3N2 (78% and 56%). VE point estimates for 18-49-year-olds in the research database were significant for any influenza (41%), influenza A (35%), and A/H3N2 (36%). Table 4 shows the demographic characteristics and VE estimates of patients in each of the databases split into emergency departments and outpatient clinics. In the emergency departments, patients in the administrative databases were significantly older (P < 0.001), more often white (50.5% vs 43.9%; P = 0.002) and more often female (58.5% vs. 49.1%; P < 0.001) than those in the research database. Furthermore, when interaction terms were used, VE estimates from the administrative database were significantly lower than from the research database against all influenza (46%, 95% CI = 29%, 59% vs. 66%, 95% CI = 52%, 76%; P < 0.001) and any influenza A (49%, 95% CI = 33%, 62% vs. 67%, 95% CI = 53%, 77%; P = 0.002). Three differences between the databases for those enrolled in outpatient clinics were noted. Administrative database enrollees were significantly older (P < 0.001), and more often vaccinated both in the enrollment season (51.5% vs. 43.8%; P = 0.002) and in the prior season (47.7% vs. 40.3%; P = 0.002). In this comparison of influenza vaccine effectiveness estimates from a test-negative case-control research study and administrative data sources, subjects differed by demographic characteristics of age, race and site of enrollment. The administrative database included older patients, more non-whites and more patients enrolled in the emergency department. With this limited sample size, the majority of comparisons found no significant differences in VE estimates between the two databases; however, differences were found for influenza B in 2017-2018 and with a larger sample size may have been found for others, given the percentage point differences between estimates. When the databases were further divided into emergency department and outpatient clinics, new patterns emerged. In emergency departments, research enrollees were younger, more often male and non-white, and VE against any influenza and all influenza A was significantly higher than was found in the administrative database. Whereas, in the outpatient clinics, age distribution was more evenly distributed in the administrative database and vaccination rates appeared to be higher than in the research database, but there were no significant differences in VE estimates. These differences may be attributed to the fact that recruitment for this research study was limited to two emergency departments (one a pediatric hospital) among several included in the administrative database and to a greater emphasis on research enrollment at urgent and primary care centers. However, given that influenza VE is frequently lower among older persons [10] , it is not unexpected that VE estimates would be lower in a population that was significantly older. The results of this study and examination of some of the literature suggest advantages and disadvantages of each type of data for determining influenza VE as shown in Box 1. Administrative database analysis for influenza vaccine effectiveness has some clear advantages that lie primarily in the potential sample sizes [4, [10] [11] [12] and relative cost per patient. Administrative databases can include large single health system data [2] or can combine data across large geographic areas [1] , thus may more accurately estimate VE for a country or region. These large datasets would be expected to be sufficiently powered to produce narrow confidence intervals and thus instill confidence in their VE estimates. Large sample sizes, with rich EMR data, would also allow for inclusion of instrumental variables (those not collected at enrollment) in the analysis [10] . The per-patient cost of acquiring data on large numbers of patients would be lower because the cost of influenza testing would be part of clinical care and would not be included as part of the study. Furthermore, there would be no need to hire research assistants to screen, consent and enroll participants. Administrative data collection does not carry risk of infection to the research staff that in-person enrollment does. Finally, the results of clinical influenza testing are likely to be rapidly available because of their importance to clinical care and infection control. Conversely, use of administrative databases for VE analyses has some disadvantages. For example, administrative databases may represent the subgroup of individuals who seek care at the specific types of facilities included in the database, such as the hospital-based clinics in this study. This situation may result in certain demographic, health or healthcare-seeking-behavior characteristics and may limit general applicability of VE estimates. There may be limited information available about patients who are included without consent [4] ; the quality of the data received may vary; there may be delays in completing administrative databases [4] , especially if data are being combined from several health systems; or there may not be indicators for factors that may affect VE in some groups, such as frailty among older adults [13] . Regarding influenza testing, there may be several types of tests used with varying sensitivity and specificity, influenza subtyping may not be available, and there may be policies that favor or disfavor influenza testing in certain settings that may introduce bias [14] . A study of inpatient testing in our locale did not find any association between influenza vaccination and clinical testing. However, clinical testing was significantly higher during the peak and post peak influenza periods than earlier in the season and higher among younger hospitalized patients with an ARI [15] . In administrative databases, vaccine verification is limited to electronic sources. It is not known how many health systems are linked electronically to their state's immunization registries, what the lag time is for making those data transfers or the completeness of the registries' data. Lack of an automatic feed to the EMR, or incomplete registry data could introduce bias by vaccination status misclassification [16] . Using research databases to determine influenza VE also has advantages and disadvantages. The primary advantage of research databases is control. For example, the researcher can determine the type or types of influenza testing to be used based on test characteristics, the clinical setting for recruitment, whether or not to oversample certain subpopulations. The researcher can set standards for data quality and ensure that those standards are met with training and monitoring of research staff and monitoring data completeness and quality as they are being collected. The data may be more informative and complete because subtyping of influenza virus can be conducted, and manual, as well as electronic, vaccine verification can be employed. Because misclassification of vaccination status can produce substantial biases in VE estimates, manual verification may be worth the extra effort and expense. The primary disadvantages of research databases are cost and scale. Significant human and other resources are needed to train research staff; identify, approach, and enroll participants while risking infection; transport research specimens; and analyze them. These specimens may be batch analyzed, delaying return of results. Distance of sites from the research offices or the testing labs may limit geographical reach of the study and the number of sites may be limited by costs. No matter which method is used, speed to release of VE estimates depends on the relative timeliness of all parts of the process from data collection to analyses to compiling the report. If administrative data were readily available, they might be the best source for rapid influenza VE estimates, with the caveat that their generalizability may be limited. Currently the US Flu VE Network provides mid-season VE estimates using self-reported vaccination status that are published in the Morbidity and Mortality Weekly Report. Earlier VE estimates based on administrative databases may be useful in a severe influenza season, or in the present era of potential co-circulation of SARS-CoV-2, when treatment and infection control measures might differ. A hybrid approach that uses clinical specimen collection and EMR data from a specified group, such as the Royal College of General Practitioners Research and Surveillance Centre [17] may be possible in areas where PCR testing for influenza is routinely conducted in outpatient settings. Strengths of this study include the test-negative design that utilizes the highly specific molecular methods to diagnose influenza, limiting misclassification of the outcome status. The sample size was limited, particularly for substrain analyses, as we found that point estimate differences of 40 were not significant, due to overlapping confidence intervals. A post hoc power calculation using our data and a hypothetical value for the overlap between databases indicated that an N of 5490 would be needed to detect differences in VE between databases with 80% power at P < 0.05. In general, post hoc power calculations are not advisable [18, 19] . The number of potential confounding factors included in regression modeling was limited due to the limited data available using administrative testing and vaccination databases. We checked for confounding for month of the influenza season for influenza status and for vaccination status. We found that month and season were both significant for influenza status in both databases. But season was significant for vaccination status only in the administrative database. Thus, we included season in the regression for both databases for uniformity for comparing VE between the two models. Administrative database analysis Lower cost; no need to screen or enroll participants, RVP performed and charged as part of clinical care Population distribution may be dependent upon type of patients seen at clinics included in database, no oversampling possible (selection bias) Influenza testing results potentially reported rapidly Data quality is not guaranteed. No risk of exposure of research staff to infectious diseases Sensitivity and specificity of the PCR test cannot be guaranteed because the time between symptom onset and testing is not known and may lead to underestimation of VE. Potentially large sample sizes [4, [10] [11] [12] Limited information about patients is available without consent or is missing from the administrative database (Information bias, unidentified confounders) [4] Can use instrumental variable analysis to improve estimates and remove bias [10] Subtyping for influenza B may not be available and may not always be reported for influenza A In some health systems, outpatients may not be routinely tested, physician testing bias may skew results Different types of testing to identify cases, with varying accuracy may be used Vaccination verification limited to electronic databases Delays in completing administrative databases [4] Research database analyses Oversampling of specific population subgroups possible Cost and other factors such as distance, could limit sample size More control over data quality and completeness Research results typically batched, resulting in reporting delays Influenza can be completely subtyped with lineages Research staff at risk of infection All enrollees are tested with the same, potentially highquality test(s) Potentially eligible enrollees may decline to participate, could introduce bias Both electronic and manual confirmation of vaccination status is possible Whether residual confounding would differ between the administrative and research databases for other variables is unknown. Without additional sources of vaccination data, the differences in vaccination coverage between databases may be attributed to underreporting to the state registry, especially among adults. Incomplete vaccination data could result in inaccurate VE estimates. However, we attempted to minimize differences between databases by only using state registry-verified vaccinations. Those who self-reported vaccination in the research database that was not verified in PA-SIIS were not included in the analysis. Finally, this study was conducted in one locale and should be repeated in other networks. The selection of the appropriate method for determining influenza vaccine effectiveness depends on a multitude of factors. Among those factors is sample size, which likely obscured significant differences between administrative and research database VE estimates in this study except when limiting the analyses to the emergency departments. In this case, the research estimates were better. Differences in the types of persons enrolled in the databases, suggest that research estimates may be more generalizable. Other advantages of research databases for VE estimates include lack of clinician-related selection bias for testing and less misclassification of vaccination status because multiple sources are used. The advantages of the administrative databases are potentially shorter time to VE results and lower cost. Estimated rates of influenzaassociated outpatient visits during 2001-2010 in 6 US integrated healthcare delivery organizations New method for real time influenza surveillance in primary care: a Wisconsin Research and Education Network (WREN) supported study Influenza vaccine effectiveness in preventing influenza-associated hospitalizations during pregnancy: a multicountry retrospective test negative design study Can routinely collected laboratory and health administrative data be used to assess influenza vaccine effectiveness? assessing the validity of the flu and other respiratory viruses research (forever) cohort Influenza vaccine effectiveness in the 2011-2012 season: protection against each circulating virus and the effect of prior vaccination on estimates Influenza vaccine effectiveness in the United States during 2012-2013: variable protection by age and virus type Influenza vaccine effectiveness against 2009 pandemic influenza A (H1N1) virus differed by vaccine type during 2013-2014 in the United States 2014-2015 influenza vaccine effectiveness in the United States by vaccine type Influenza vaccine effectiveness in the United States-2015/16 Season Estimating influenza vaccine effectiveness in community-dwelling elderly patients using the instrumental variable analysis method Influenza vaccine effectiveness in the elderly based on administrative databases: change in immunization habit as a marker for bias Estimating vaccine effectiveness against hospitalized influenza during pregnancy: multicountry protocol for a retrospective cohort study Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors Caution required in the use of administrative data and general laboratory submissions for influenza vaccine effectiveness estimation Does influenza vaccination status change physician ordering patterns for respiratory viral panels? Inspection for selection bias Field evaluation of vaccine efficacy Royal College of General Practitioners Research and Surveillance Centre (RCGP RSC) sentinel network: a cohort profile Observed power, and what to do if your editor asks for post-hoc power analyses The abuse of power: the pervasive fallacy of power calculations for data analysis Supplementary data to this article can be found online at https://doi.org/10.1016/j.vaccine.2021.01.013.