key: cord-0952375-x2kqvnqz
authors: Lin, Dan-Yu; Zeng, Donglin; Gilbert, Peter B
title: Evaluating the Long-Term Efficacy of COVID-19 Vaccines
date: 2021-03-10
journal: Clin Infect Dis
DOI: 10.1093/cid/ciab226
sha: d5b155cff22dc7fc73aeb9eeadebecdb88c15df1
doc_id: 952375
cord_uid: x2kqvnqz

Large-scale deployment of safe and durably effective vaccines can curtail the COVID-19 pandemic.1−3 However, the high vaccine efficacy (VE) reported by ongoing phase 3 placebo-controlled clinical trials is based on a median follow-up time of only about two months4−5 and thus does not pertain to long-term efficacy. To evaluate the duration of pro- tection while allowing trial participants timely access to efficacious vaccine, investigators can sequentially cross participants over from the placebo arm to the vaccine arm according to priority groups. Here, we show how to estimate potentially time-varying placebo-controlled VE in this type of staggered vaccination of participants. In addition, we compare the per- formance of blinded and unblinded crossover designs in estimating long-term VE.

A number of studies have been conducted around the world to evaluate the efficacy and safety of investigational vaccines against novel coronavirus disease-2019 . 4 Vaccine effect can wane over time because of declining immunologic memory or changing antigenicity of the pathogen. A vaccination can be followed with booster doses to maintain a protective level of immunity among susceptible individuals, but the nature of the protection over time must be understood so that an effective vaccination and boosting schedule can be determined. Thus, after FDA issues an Emergency Use Authorization (EUA), vaccine sponsors should continue to collect placebo-controlled data on primary endpoints in any ongoing trials for as long as feasible. 15 Although continuing blinded follow-up of the original treatment arms is the ideal way to evaluate long-term efficacy and safety, placebo recipients should be offered the vaccine at some point after an EUA. One strategy is the -rolling crossover‖, which vaccinates placebo recipients around the same time as general population members in the same priority tier.

Under this design, placebo participants are vaccinated at different times, with the timing of vaccination depending on enrollment characteristics that define their priority tier. In this article, we show how to properly assess the durability of VE under staggered enrollment and time-varying community transmission, allowing higher-risk placebo volunteers to get vaccinated earlier than lower-risk ones during the crossover period. Our framework provides unbiased estimation of the entire curve of placebo-controlled VE as a function of time elapsed since vaccination, up to the point where most of the placebo volunteers have been vaccinated. We investigate the bias and precision of our approach in estimating longterm VE under various crossover designs, including blinded crossover, in which participants do not know the order of treatments they receive, and unblinded crossover, in which participants are notified of their randomization assignments at the time of crossover. We also discuss how to perform sensitivity analysis when unblinded follow-up data are used. Figure 1 provides a schematic illustration of the rolling crossover strategy in the context of COVID-19 vaccine trials. In this scenario, participants are screened and randomly assigned to vaccine or placebo over a 4-month period, the vaccine is granted EUA in the 5th month on the basis of interim results, and crossover occurs over the next five months. This design provides information about VE for 10 months. We aim to estimate time-varying VE under any crossover design, up to the point when there are very few placebo participants left.

A c c e p t e d M a n u s c r i p t 4 The endpoint of interest is time to symptomatic COVID-19 disease. We allow the risk of disease to vary over the calendar time and to depend on baseline risk factors, such as age, sex, ethnicity, race, occupation, and underlying health conditions; we allow the effect of vaccine on disease occurrence to depend on the time elapsed since vaccination.

We consider two definitions of time-varying VE: (1) day-t VE is the percentage reduction in the hazard rate or instantaneous risk of disease at day t for those who were vaccinated t days ago compared with those who have not been vaccinated; and (2) t-day VE is the percentage reduction in the attack rate or cumulative incidence of disease over the t-day period for those who were vaccinated at the start of the period compared with those who were unvaccinated throughout the period. We denote these two VE measures by VE h and VE a , where h and a stand for hazard rate and attack rate, respectively. These two definitions are equivalent when the effect of vaccine is constant over time. If the effect of vaccine wanes over time, then VE a is larger than VE h .

In Supplemental Appendix 1, we formulate the above concepts through an adaptation of the well-known Cox regression model, 17−18 in which each participant's time to disease occurrence is measured from a common origin, namely the start of the clinical trial, and the hazard ratio of vaccine versus placebo depends on the time elapsed since vaccination. We formally define VE a as one minus the time-averaged hazard ratio, which is approximately the ratio of the cumulative incidence. We derive the maximum likelihood estimator for VE a as a function of time elapsed since vaccination. We show that the estimator is approximately unbiased and normally distributed, with a variance that can be estimated analytically, enabling one to construct valid confidence intervals for the VE a curve. Finally, we propose a method to estimate VE h by kernel-smoothing the estimate of VE a .

We conducted a set of simulation studies mimicking the BNT162b2 vaccine trial. We considered 40,000 participants, who entered the trial at a constant rate over a 4-month period and were randomly assigned to vaccine or placebo in a 1:1 ratio. The vaccine received an A c c e p t e d M a n u s c r i p t 5 EUA from FDA at the 5th month, by which time there were about 300 COVID-19 cases in the placebo group. To reflect the increase of COVID-19 cases since last summer and the expected downward trend in the spring due to vaccine rollout and other factors, we let the disease risk increase over the first 7 months and decrease afterward. We chose three combinations of 5-month VE and 10-month VE: (a) VE a (5 mos.) = VE a (10 mos.) = 95%;

(b) VE a (5 mos.) = 85%, VE a (10 mos.) = 75%; and (c) VE a (5 mos.) = 70%, VE a (10 mos.) = 50%.

We considered the statistically optimal design of keeping all participants on their original treatment assignments until the end of the trial. We refer to this design as Plan A and regard it as a benchmark. We also considered three blinded crossover designs:

B. Crossover starts at month 6, 7, 8, 9, or 10 for participants with priority tier of 1, 2, 3, 4, or 5, respectively, with each participant's waiting time for the clinic visit following the exponential distribution with mean of 0.5 month.

C. 20% of participants follow Plan A, and the rest follow Plan B.

D. Crossover starts at month 6 for all participants, with the waiting time following the exponential distribution with mean of 0.5 month.

Both B and C are priority tier-dependent rolling crossover designs. The difference is that under Plan B, all placebo recipients cross over to the vaccine arm, whereas under Plan C, 20% of participants choose for altruistic reasons to stay on their original treatment assignments.

Under Plan D, all placebo recipients are vaccinated quickly without any priority tiering.

With blinded crossover, placebo participants receive the vaccine and vaccine participants receive the placebo at the point of crossover; none of the participants are aware of the order of their treatments. All participants are followed until the end of the trial or the time of analysis, which is 10.5 months since trial initiation. The designs of these simulation studies are detailed in Supplemental Appendix 2.

The results for the estimation of VE a based on 10,000 simulated datasets are summarized in Table 1 and Figure 2 . The proposed method yields virtually unbiased estimates of the VE a curves over the 10-month period for Plans A-C in all three scenarios of long-term VE;

A c c e p t e d M a n u s c r i p t 6 it also yields accurate variance estimates, such that the confidence intervals have correct coverage probabilities. When VE is constant over time, the standard errors for the estimates of VE a under Plans B and C are slightly lower than those of Plan A. When VE wanes over time, the standard errors for the estimates of 5-month VE a under Plans B and C are also slightly lower than those of Plan A; however, the standard errors for the estimates of 10month VE a under Plans B and C are higher than those of Plan A, with the standard errors being slightly lower under Plan C than under Plan B. Under Plan D, the estimates of 10month VE a may be slightly biased, with higher standard errors than under Plans A-C; these results are not surprising, because under this plan, the number of unvaccinated participants diminishes rapidly after month 6.

We also evaluated the performance of standard Cox regression, 17 With unblinded crossover, participants are notified of their original treatment assignments at the point of crossover, and placebo recipients are vaccinated soon after. In Plan B', everyone in a given priority-tier group is notified of their original treatment assignment on the same day. In Plan D', all participants are notified of their original treatment assignments on the same day. In Plan D‖, participants are notified of their randomization assignments without priority tiering; the timing of crossover is the same as that of Plan D. Plan D‖ was meant to mimic the crossover that has been occurring in the two mRNA vaccine trials, where participants have been unblinded gradually.

Because vaccine recipients may engage in riskier behavior upon unblinding and placebo recipients may also change their behavior upon unblinding (in a manner that likely differs from vaccine recipients), we discarded the data collected after unblinding for both the vaccine and placebo groups by censoring each participant's time to disease at their time of unblinding.

This strategy avoids any bias due to behavioral confounding.

The results for the estimation of VE a based on 10,000 simulated datasets are summarized in Table 2 and The estimates are more bumpy under unblinded than blinded crossover. Comparing Figures   6 and 7 shows that VE h is much lower than VE a in the presence of waning vaccine effect.

For a preventive COVID-19 vaccine to be administered to millions of people, including healthy individuals, its safety and efficacy must be demonstrated in a clear and compelling manner.

Although preliminary results from ongoing phase 3 clinical trials have revealed A c c e p t e d M a n u s c r i p t 9 higher than expected efficacy of COVID-19 vaccines, 4−6 additional follow-up is required to assess long-term efficacy and safety. Indeed, FDA does not consider issuance of an EUA, in and of itself, as grounds for stopping blinded follow-up in an ongoing clinical trial. 15 We recommend the rolling crossover design, which allows placebo volunteers to be vaccinated in a timely manner while still making it possible to assess long-term vaccine safety and efficacy. As our simulation studies have shown, standard Cox regression with a constant hazard ratio seriously over-estimates long-term VE in the presence of waning vaccine effect.

We have developed a valid and efficient approach to evaluate the effect of a COVID-19 vaccine that potentially wanes over time. The estimated curve of time-varying VE can be used to determine when a booster vaccination is needed to sustain protection; this information is also an important input parameter in mathematical modeling of the population impact of COVID-19 vaccines.

We have considered vaccine effects on cumulative incidence and on instantaneous risk.

It is also of interest to study vaccine effects over successive time periods. We present the method and results for estimating period-specific VE in Supplemental Appendices 1 and 2.

To ensure high-quality follow-up data, crossover should ideally be blinded, with participants not knowing their treatment assignments even after crossover. It is advantageous, when possible, to implement crossover on a rolling basis rather than instantaneously since time-varying VE can be estimated (without adding assumptions) only up to the point where there are still a few placebo recipients under follow-up. Indeed, rolling crossover is even more important than blinding for the express purpose of assessing long-term VE without imposing additional assumptions.

Of course, unblinded crossover has practical benefits over blinded crossover: it reduces operational complexity and trial cost. However, unblinding can lead to differential exposure to SARS-CoV-2 between the original vaccinees and the placebo crossovers, which in turn can bias the estimation of VE. This bias can be avoided by analyzing only the blinded followup data. However, discarding the unblinded follow-up data may substantially reduce the precision in estimating long-term VE. We may estimate VE twice, once with all follow-up A c c e p t e d M a n u s c r i p t 10 data and once with only blinded follow-up data, and compare the two sets of results.

Alternatively, our methods can be applied to all follow-up data, followed by a sensitivity analysis to assess the robustness of the results to potential unmeasured confounding caused by unblinding of trial participants. In Supplemental Appendix 3, we show how to apply a best-practice general methodology in epidemiological research 19−20 to perform this sensitivity analysis. Using this methodology, we can assess how strong unmeasured confounding due to unblinding would need to be in order to fully explain away the observed VE. We can also provide a conservative estimate of VE that accounts for unmeasured confounding. there are only a small number of cases in a subgroup. To alleviate this problem, we may formulate the time-varying hazard ratio through a parametric (e.g., log-linear) function, which is then allowed to interact with subgroups.

We have targeted VE over the first 10 months for several reasons. First, it is unlikely that there will be any placebo volunteers beyond month 10 in the ongoing COVID-19 vaccine trials, and follow-up data in the absence of a placebo arm do not provide direct information about VE durability. Second, estimates of long-term VE will become more uncertain as community transmission decreases.

Although we have framed the discussion in the context of randomized, placebo-controlled We have implemented the methods described in this paper in an R package, which is available at https://dlin.web.unc.edu/software/dove/.

A c c e p t e d M a n u s c r i p t 12 M a n u s c r i p t 15 

COVID-19 vaccine trials should seek worthwhile efficacy

Combination prevention for COVID-19

A strategic approach to COVID-19 vaccine R&D

Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine

Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine

Safety and efficacy of the ChA-dOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK

Effect of an inactivated vaccine against SARS-CoV-2 on safety and immunogenicity outcomes: Interim analysis of 2 randomized clinical trials

An mRNA vaccine against SARS-CoV-2 -Preliminary report

Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial

Phase 1-2 Trial of a SARS-CoV-2 recombinant spike protein nanoparticle vaccine

An international randomised trial of candidate vaccines against COVID-19

Development and Licensure of Vaccines to Prevent COVID-19: Guidance for Industry

WHO target product profiles for COVID-19 vaccines

Emergency Use Authorization for Vaccines to Prevent COVID-19: Guidance for Industry

Evaluating the efficacy of COVID-19 vaccines

Regression models and life-tables

The Statistical Analysis of Failure Time Data

Sensitivity analysis in observational research: introducing the E-value

Sensitivity analysis without assumptions

Assessing durability of vaccine effect following blinded crossover in COVID-19 vaccine efficacy trials

The authors are grateful to Yu Gu and Bridget I. Lin for assistance, to Thomas Fleming, David Harrington, and Ross Prentice for helpful discussions.

This work was supported by the National Institutes of Health grants R01 AI029168, R01 GM124104, P01 CA142538, and UM1 AI068635.None of the authors has any potential conflicts to disclose.

A c c e p t e d M a n u s c r i p t 13