key: cord-0915768-qp34apct
authors: Ioannou, George N.; Locke, Emily R.; O'Hare, Ann M.; Bohnert, Amy S.B.; Boyko, Edward J.; Hynes, Denise M.; Berry, Kristin
title: COVID-19 Vaccination Effectiveness Against Infection or Death in a National U.S. Health Care System: A Target Trial Emulation Study
date: 2021-12-21
journal: Ann Intern Med
DOI: 10.7326/m21-3256
sha: a8ebe88ff59b446f1ef59e9c509e167923252ec8
doc_id: 915768
cord_uid: qp34apct

BACKGROUND: Little is known about real-world COVID-19 vaccine effectiveness (VE) in racially and ethnically diverse, elderly populations with high comorbidity burden. OBJECTIVE: To determine the effectiveness of messenger RNA COVID-19 vaccines. DESIGN: Target trial emulation study comparing newly vaccinated persons with matched unvaccinated controls. SETTING: U.S. Department of Veterans Affairs health care system. PARTICIPANTS: Among persons receiving care in the Veterans Affairs health care system (n = 5 766 638), those who received at least 1 dose of the Moderna or Pfizer–BioNTech COVID-19 vaccine from 11 December 2020 to 25 March 2021 (n = 2 099 871) were matched to unvaccinated controls in a 1:1 ratio according to demographic, clinical, and geographic characteristics. INTERVENTION: Follow-up for SARS-CoV-2 infection or SARS-CoV-2–related death, defined as death within 30 days of infection, began after the vaccination date or an identical index date for the matched unvaccinated controls and continued until up to 30 June 2021. MEASUREMENTS: Vaccine effectiveness against SARS-CoV-2 infection or SARS-CoV-2–related death. RESULTS: Vaccinated and unvaccinated groups were well matched; both were predominantly male (92.9% vs. 93.4%), had advanced age (mean, 68.7 years in both groups), had diverse racial and ethnic distribution (for example, Black: 17.3% vs. 17.0%, Hispanic: 6.5% vs. 6.1%), and had substantial comorbidity burden. Vaccine effectiveness 7 or more days after the second vaccine dose was 69% (95% CI, 67% to 70%) against SARS-CoV-2 infection and 86% (CI, 82% to 89%) against SARS-CoV-2–related death and was similar when follow-up was extended to 31 March versus 30 June. Vaccine effectiveness against infection decreased with increasing age and comorbidity burden. LIMITATION: Predominantly male population and lack of data on SARS-CoV-2 variants. CONCLUSION: In an elderly, diverse, high-comorbidity population, COVID-19 VE against infection was substantially lower than previously reported, but VE against death was high. Complementary infection mitigation efforts remain important for pandemic control, even with vaccination. PRIMARY FUNDING SOURCE: U.S. Department of Veterans Affairs.

T he U.S. Food and Drug Administration issued emer gency use authorization for COVID-19 vaccines manufactured by Pfizer-BioNTech (BNT162b2) on 11 December 2020, by Moderna (mRNA-1273) on 18 December 2020, and by Janssen (JNJ-78436735) on 27 February 2021. The 2 messenger RNA (mRNA) vaccines, BNT162b2 and mRNA-1273, were shown to have very high efficacy of 95% and 94.1%, respectively, against symptomatic SARS-CoV-2 infection in phase 3 randomized controlled trials (1) (2) (3) .

Real-world studies also suggest that these vaccines are effective. A prospective study of 23 234 adult health care workers from 104 public hospitals in England followed with biweekly SARS-CoV-2 polymerase chain reaction (PCR) testing suggested that BNT162b2 had 85% (95% CI, 74% to 96%) vaccine effectiveness (VE) for preventing symptomatic or asymptomatic infection 7 or more days after the second dose (4) . A study of 3950 health care personnel, first responders, and other frontline workers in 8 U.S. locations tested for SARS-CoV-2 weekly showed 90% effectiveness 14 or more days after the second dose of BNT162b2 (5) . A study of 596 618 persons vaccinated with BNT162b2 in Israel and matched to unvaccinated controls reported that VE at 7 or more days after the second dose was 92% for documented infection (6) . A study of the entire population of Israel aged 16 years or older reported that VE at 7 or more days after the second BNT162b2 dose was 95.3% for asymptomatic infection, 97.0% for symptomatic infection, 97.5% for SARS-CoV-2-related hospitalization, and 96.7% for SARS-CoV-2-related death (7) .

The real-world effectiveness of the 2 mRNA vaccines in ethnically and racially diverse populations across the entire United States is not well characterized, especially in more vulnerable populations, such as elderly persons with high comorbidity burden. The U.S. Department of Veterans Affairs (VA) health care system is the largest national, comprehensive health care system in the United States. It includes a large proportion of older adults with multimorbidity. A recent test-negative, case-control study of VE using VA data reported an extremely high VE See also:

Annals.org Annals of Internal Medicine © American College of Physicians 1

ORIGINAL RESEARCH against infection of 97.1% (CI, 96.6% to 97.5%) 7 or more days after the second vaccine dose (8) . However, this study did not match case patients to controls by date of testing and did not select cases and controls with similar disease severity (9) , which could lead to overestimation of VE. In addition, the case-control study design does not allow estimation of VE against SARS-CoV-2-related death. Accurate estimates of VE in high-risk, real-world populations are critical for guiding policy in this area and determining whether complementary infection mitigation efforts (for example, physical distancing, masking, and screening) remain important for pandemic control even after vaccination. We used a target trial emulation design (10) to estimate the effectiveness of the 2 mRNA vaccines in the VA health care system against both SARS-CoV-2 infection and death.

The VA provides care at 171 medical centers and 1112 outpatient clinics throughout the country. It uses a nationwide electronic health records system, enabling accurate ascertainment of relevant baseline characteristics and potential confounders. We used data from the VA's Corporate Data Warehouse, a relational database of VA enrollees' comprehensive electronic health records, including the VA COVID-19 Shared Data Resource, which includes analytic variables provisioned by the VA Informatics and Computing Infrastructure on all VA enrollees who were tested for or vaccinated against SARS-CoV-2 (11) . We also used linked Medicare data obtained through the VA Information Resource Center (12) to identify any additional VA enrollees diagnosed with or vaccinated against COVID-19 through Medicare-covered services.

The VA Corporate Data Warehouse, COVID-19 Shared Data Resource, and VA Information Resource Center Medicare data were last accessed on 13 September 2021 to allow adequate time for outcomes extending to 31 June 2021 to be electronically recorded. The study was approved by the VA Puget Sound Institutional Review Board (#01886).

We created a cohort of all VA enrollees aged 18 years or older who were alive as of 11 December 2020 (the date of emergency use authorization for BNT162b2) and had an inpatient or outpatient encounter in the VA health care system in the preceding 12 Only 12 620 veterans received the Janssen vaccine. These persons were considered as potential controls, and those who matched were censored at the time they received the Janssen vaccine.

We designed this observational study to emulate a target trial of COVID-19 vaccination versus placebo (10) . Vaccine recipients were matched in a 1:1 ratio to unvaccinated VA enrollees who had the same inclusion criteria (that is, were alive, had received VA care in the prior 12 months, and had no evidence of prior SARS-CoV-2 infection as of their matched vaccinated person's index date). To facilitate identification of matching cohorts of vaccinated and unvaccinated persons anchored around the same calendar date (14, 15) , we divided the observation period into 14 one-week periods. In each week, we identified all persons who were alive, uninfected, and unvaccinated as of the first day of the week. Among this "at-risk" cohort, we identified all treated persons who could potentially be included in our target trial, defined as those who were vaccinated during the week, and all potential controls, defined as those who were not vaccinated at the start of the week. We used the baseline characteristics of all persons at the beginning of the week to identify the single best unvaccinated matched control person for each vaccinated person using a combination of exact and coarsened exact matching and propensity score matching, implemented by STATA's kmatch command (16) (StataCorp). Entropy balancing of means in all matching characteristics was included as a refinement in the matching process. We used exact matching by Veterans Integrated Services Network (the administrative regions of VA) to account for geographic variability in vaccination distribution and risk for infection. We used coarsened exact matching by age (2-year buckets) and Charlson Comorbidity Index (CCI) (2point buckets)-the 2 characteristics most strongly associated with both receipt of vaccination and development of SARS-CoV-2 infection or death in VA enrollees (17) (18) (19) (20) . The characteristics used in the propensity score logistic regression model estimating the probability of undergoing vaccination during the week were determined a priori and are listed in the next section. STATA's kmatch procedure assigned to each vaccinated person 2 unvaccinated persons who had the same Veterans Integrated ORIGINAL RESEARCH COVID-19 Vaccination Effectiveness Against Infection or Death Services Network, age bucket, and CCI bucket and the nearest-neighbor propensity score within a caliper of 0.01. Controls were matched without replacement within the week but were eligible to be matched to another vaccinated person in a subsequent week. Among these 2 persons, we then identified the 1 person with the nearest propensity score to the vaccine recipient who was still alive, unvaccinated, and uninfected as of the actual day of the week their matched vaccine recipient was vaccinated. Each vaccinated-unvaccinated matched pair was then followed from the date of vaccination (or index date for the unvaccinated control) until the date of the earliest of the following events: occurrence of an outcome event, death unrelated to COVID-19, vaccination (for unvaccinated controls), vaccination of the matched control (for vaccine recipients), or the end of the follow-up period (30 June 2021). Newly vaccinated persons were eligible for inclusion in the study even if they had previously been selected as a control.

This entire process was repeated 14 times for each of the 14 one-week periods. Vaccine recipients without a matched unvaccinated control (n = 3919 [0.2%]) were omitted from all analyses.

Covariates included in the propensity score model were known to be associated with vaccination and SARS-CoV-2 infection or SARS-CoV-2-related death in the VA population (17, 18, 20, 21) . These included age; sex; selfreported race/ethnicity; urban or rural residence (based on ZIP codes, using data from the VA Office of Rural Health [22] , which uses the secondary rural-urban commuting area for defining rurality); geographic location (categorized according to the 19 Veterans Integrated Services Networks [23]); and the following comorbid conditions: diabetes, congestive heart failure, chronic obstructive pulmonary disease, and chronic kidney disease, defined by appropriate International Classification of Diseases, 10th Revision, codes ( Table 1) . We calculated the 2-year CCI (24, 25) as a measure of comorbidity burden. We used the Care Assessment Need (CAN) score, a validated measure of 1-year mortality in VA enrollees calculated using sociodemographic characteristics, clinical diagnoses, vital signs, medications, laboratory values, and health care use data from the VA's national electronic health record. The CAN score is an independent predictor of COVID-19-related death (17, 26) , while also capturing health care use. It is calculated only in VA enrollees who have an assigned VA primary care provider; therefore, a "missing" CAN score is informative as an indicator of VA enrollees who did not have a primary care provider. Finally, we used the body mass index, calculated using measured weight and height.

The COVID-19 Shared Data Resource captures all veterans who tested positive for SARS-CoV-2 within the VA system on the basis of PCR tests as well as those with such tests done outside the VA system but documented in VA We used Cox proportional hazards regression to compare vaccinated persons and their matched controls with respect to time to development of SARS-CoV-2 infection or SARS-CoV-2-related death, with follow-up for these outcomes extending to 30 June 2021. We calculated the incidence of each outcome and the hazard ratio, unadjusted and adjusted for all of the covariates listed in the preceding section, comparing vaccinated persons with unvaccinated matched controls during each of the following time periods: 14 to 20 days (for Pfizer-BioNTech) or 14 to 27 days (for Moderna) after the first vaccination dose, 21 to 27 days (Pfizer-BioNTech) or 28 to 35 days (Moderna) after the first dose, and 7 or more days after the second dose. Vaccine effectiveness (%) was calculated as 100 Â (1 À adjusted hazard ratio).

The study's primary outcome was VE 7 or more days after the second vaccine dose, in agreement with prior studies (1, 6, 7) . We did separate analyses with follow-up extending until either 31 March or 30 June 2021 to evaluate whether VE declined over time.

We did the following supplementary analyses: We calculated VE as 100 Â (1 À risk ratio), where cumulative risk was calculated as 1 minus the unadjusted Kaplan-Meier estimate, and we used only documented PCR-positive SARS-CoV-2 tests to define incident infection and excluded the cases diagnosed using International Classification of Diseases, 10th Revision, codes in Medicare data.

The study was supported by the VA Office of Research and Development. The funding sources had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

The absolute standardized mean differences and variance ratios of baseline characteristics between vaccinated and unvaccinated persons were calculated for the raw and matched data and showed balance after matching (Appendix Figure, Both vaccinated and unvaccinated groups were predominantly men (92.9% and 93.4%, respectively), had advanced age (mean, 68.7 years in both groups), had diverse racial and ethnic distribution (for example, Black: 17.3% and 17.0%, Hispanic: 6.5% and 6.1%), had substantial comorbidity burden (mean CCI, 2.8 in both groups), and had similar CAN scores (56.6 and 55.7). Major comorbid conditions, such as diabetes, congestive heart failure, chronic obstructive pulmonary disease, and chronic kidney disease, were common and nearly equally distributed in the 2 groups.

Among persons who received both doses of Moderna or Pfizer-BioNTech vaccines, 94.5% received the second dose within 4 days before or after the recommended date. Among persons who received the first vaccine dose, only Table 2) . Vaccine effectiveness against documented infection 7 days after the second dose seemed to decline with increasing age (75% in persons aged 18 to 64 years, 72% in persons aged 65 to 74 years, and 61% in persons aged ≥75 years) and less so with increasing CCI category (72% in CCI 0 to 1, 69% in CCI 2 to 4, and 65% in CCI ≥5) ( Table 2) .

Vaccine effectiveness against SARS-CoV-2-related death was 55% (CI, 42% to 64%) at 14 to 20 days (for Pfizer-BioNTech) or 14 to 27 days (for Moderna) after the Table 3) . Vaccine effectiveness against SARS-CoV-2-related death 7 days after the second dose seemed to decline with increasing CCI category (94% in CCI 0 to 1, 88% in CCI 2 to 4, and 83% in CCI ≥5), with no obvious trends by age category ( Table 3) . Secondary analyses in which VE was calculated as 100 Â (1 À risk ratio) (Supplement Tables 2 and 3 , available at Annals.org) showed minor numerical differences from those with VE calculated on the basis of adjusted hazard ratios. Calculating VE using only SARS-CoV-2 PCR tests with positive results documented in VA records resulted in slightly greater estimates of VE against infection at 7 or more days after the second dose (71% [CI, 69% to 72%]) and almost identical VE against SARS-CoV-2-related death (86% [CI, 82% to 90%]) (Supplement Tables 4 and 5, available at Annals.org).

Our target trial emulation study in the national VA health care system among 2 099 871 persons vaccinated with mRNA COVID-19 vaccines between 11 December 2020 and 25 March 2021 and matched 1:1 to unvaccinated controls estimated that VE at 7 or more days after the second vaccine dose was 69% (CI, 67% to 70%) for SARS-CoV-2 infection and 86% (CI, 82% to 89%) for (6, 7, 28) ; and a recent testnegative, case-control study that was also done with VA data from December to March 2021 (8) . However, that study did not match cases and controls by the date of testing and did not account for the temporal bias introduced by the fact that SARS-CoV-2 incidence decreased during the observation period (there were 47 357 positive results on SARS-CoV-2 tests in the VA in December 2020 but only 9257 in March 2021) in parallel with a rapid increase in the likelihood of having been vaccinated (Supplement Table 6 , available at Annals.org), creating a spurious association between receipt of vaccination and protection from infection and inflating observed VE. Also, case patients and test-negative controls were not limited to persons who manifested similar degrees of disease severity (for example, symptoms suggestive of COVID-19). Mandatory screening of persons who were not at risk for COVID-19 before medical interventions, procedures, and hospitalizations results in a test-negative population that is not comparable with the test-positive persons.

Adherence to receipt and timing of the second vaccination dose was excellent in our study and, thus, unlikely to have resulted in lower VE. It is likely that the advanced age and high comorbidity burden of the VA population contributed to lower VE. This is supported by the lower VE that we saw in subsets of our population with age 75 years or older or CCI of 5 or greater. Recent studies suggested that only a minority of solid-organ transplant recipients develop antibodies against the spike protein after 2 doses of mRNA vaccines (29, 30) , and blunted humoral responses to COVID-19 vaccines have also been reported in patients receiving hemodialysis (31) and patients with hematologic malignancies (32) .

Vaccine effectiveness against SARS-CoV-2-related death could not be assessed in the phase 3 randomized controlled trials or even in prior observational cohort studies because of the small number of deaths observed. The only prior cohort study we are aware of that reported VE against SARS-CoV-2-related death evaluated the entire population of Israel aged 16 years or older from January to April 2021 and reported an overall effectiveness of 96.5% (7) . However, that study did not attempt to match vaccinated to unvaccinated persons using appropriate target trial emulation methods. The large number of SARS-CoV-2-related deaths that occurred in our cohort enabled relatively precise estimates of VE. Our results confirm the very high effectiveness of mRNA vaccines against SARS-CoV-2-related death, estimated at 86% at least 7 days after the second dose, with follow-up extending to 30 June 2021, even in an elderly population (mean age, 68.7 years) with high comorbidity burden.

We found that VE against infection or death did not decrease when the follow-up was extended from 30 March 2021 (mean follow-up, 36 days) to 30 June 2021 (mean follow-up, 101 days). Our study did not have data on SARS-CoV-2 variants and did not extend beyond 20 June 2021, when the B.1.617.2 (Delta) variant became predominant in the United States (33) . Future VA studies could determine if VE declines with even longer followup since vaccination, including the period after 30 June 2021 when the Delta variant was predominant.

Our study is limited by a predominantly male population and lack of regular SARS-CoV-2 screening during the observation period. Therefore, asymptomatic or minimally symptomatic infections were not routinely captured. If persons who had vaccination were more likely than unvaccinated persons to be tested even for asymptomatic or minimally symptomatic SARS-CoV-2 infection, then this bias would tend to attenuate the observed effectiveness against documented infection. Although we identified infections documented in the VA as well as Medicare records, positive results from tests done elsewhere may not be captured unless they were recorded in VA or Medicare records. We identified all vaccinations done within the VA health care system, vaccinations done outside the VA that were documented in VA records, and vaccinations of VA enrollees recorded in Medicare records. If additional vaccinations were done in our matched unvaccinated group during the observation period, this would tend to attenuate the observed VE. Although not seen in our results, waning of efficacy over time may reflect declining vaccine protection or result from bias due to a depletion of susceptibles, such that risk in the unvaccinated declines over time due to immunity from unrecognized infection or other reasons, thereby reducing differences in risk between vaccinated and unvaccinated.

In conclusion, in an elderly population of U.S. veterans with high comorbidity burden, COVID-19 VE against infection was substantially lower than previously reported but effectiveness against death was very high. and Informatics Program, Center for Quantitative Life Sciences,

C4591001 Clinical Trial Group. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine

An mRNA vaccine against SARS-CoV-2-preliminary report

COVE Study Group. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine

SIREN Study Group. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study

Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers-eight U.S. locations

BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting

Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data

SARS-CoV-2 vaccine effectiveness in a high-risk national population in a real-world setting

The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology

Using big data to emulate a target trial when a randomized trial is not available

Department of Veterans Affairs. Corporate Data Warehouse

Department of Veterans Affairs. VA Information Resource Center (VIReC) Medicare Data

SARS-CoV-2 infection rates of antibody-positive compared with antibodynegative health-care workers in England: a large, multicentre, prospective cohort study (SIREN)

Association between bariatric surgery and long-term survival

Balanced risk set matching

Stata module for multivariate-distance and propensity-score matching, including entropy balancing, inverse probability weighting, (coarsened) exact matching, and regression adjustment

Development of COVIDVax model to estimate the risk of SARS-CoV-2-related death among 7.6 million US veterans for use in vaccination prioritization

Risk factors for hospitalization, mechanical ventilation, or death among 10 131 US veterans with SARS-CoV-2 infection

Development and validation of a 30-day mortality index based on pre-existing medical administrative data from 13,323 COVID-19 patients: the Veterans Health Administration COVID-19 (VACO) Index

Trends over time in the risk of adverse outcomes among patients with SARS-CoV-2 infection

Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases

A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

Automated EHR score to predict COVID-19 outcomes at US Department of Veterans Affairs

Accuracy and completeness of mortality data in the Department of Veterans Affairs

Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on Covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study

Safety and immunogenicity of a third dose of SARS-CoV-2 vaccine in solid organ transplant recipients: a case series

Three doses of an mRNA Covid-19 vaccine in solid-organ transplant recipients

Humoral response after 3 doses of the BNT162b2 mRNA COVID-19 vaccine

Immunogenicity of the BNT162b2 COVID-19 mRNA vaccine and early clinical outcomes in patients with haematological malignancies in Lithuania: a national prospective cohort study

Vaccination Effectiveness Against Infection or Death Author Contributions: Conception and design

Appendix Figure. Distribution of baseline characteristics and propensity scores in vaccinated and unvaccinated persons before and after matching.