key: cord-1040444-eofuq985
authors: Janes, Holly; Gao, Fei; Luedtke, Alex
title: Discussion on “Estimating vaccine efficacy over time after a randomized study is unblinded” by Anastasios A. Tsiatis and Marie Davidian
date: 2021-09-07
journal: Biometrics
DOI: 10.1111/biom.13542
sha: 5a6d94fdd0d782c58631cbbcbe50c7ebde103976
doc_id: 1040444
cord_uid: eofuq985

nan

A recent publication by Tsiatis and Davidian addresses the challenge of evaluating longer term efficacy of a COVID-19 vaccine in the context of a blinded, placebo-controlled phase 3 vaccine trial when, based on interim efficacy results, participants are unblinded and placebo recipients are offered vaccine mid-study. How can long-term durability of vaccine efficacy be evaluated in the absence of concurrent control data throughout follow-up? Importantly, the challenge being addressed is a consequence of highly successful science and public policy-it is attributable to the high efficacy seen to-date for COVID-19 vaccines and the rapid rollout of effective vaccines under Emergency Use Authorization mechanisms. The authors' proposed approach is informed by the first author's deep engagement with the ongoing US-government-funded phase 3 COVID-19 vaccine trials.

Tsiatis and Davidian, hereafter "TD," focus specifically on unblinded placebo crossover, wherein after interim efficacy results participants are unblinded as to initial randomization, and those randomized to placebo and desiring vaccine are provided it. Participants are also free to ask to be unblinded to randomization assignment at any point during follow-up, for example, to access outside-study vaccination. Follow-up of participants crossing over to vaccine both on-study or outside-study continues to ensure that data are collected to evaluate long-term vaccine efficacy. Consistent with all published phase 3 COVID-19 vac-cine trials to-date, TD target estimation of vaccine efficacy as measured by the reduction in incidence of COVID-19 disease under vaccine versus placebo. This incidence is allowed to depend on the time that has accrued since first vaccination and as realized under a blinded trial design. They propose a potential-outcomes-based framework for making inference about vaccine efficacy and explore bias in standard methods of estimation attributable to the unblinded crossover design feature. The specific issue TD are concerned with is potential bias in estimation due to informative unblinding, which occurs when individuals with different risk characteristics cross over to vaccine at different points in time. This issue is especially critical for unblinding due to outside-study vaccination, where motivation for pursuing vaccination and access to vaccine differs across subpopulations based on risk of COVID-19 disease. The issue may also apply to blinded crossover, whereby original placebo recipients are crossed over to vaccine and original vaccine recipients are given placebo, but, because timing of blinded crossover is typically dictated by design, informative unblinding is much less likely in this scenario.

To address this issue, TD develop a framework that defines the individual-level trial data under unblinded crossover of a phase 3 COVID-19 vaccine efficacy trial. As in prior methodological work on these trials, TD base their analysis on calendar time to appropriately align participants enrolled in a staggered fashion with regard to the secular trends in SARS-CoV-2 incidence. They model the COVID-19 disease process by parameters describing the population prevalence, contact rate parameters measuring population mixing, and COVID-19 acquisition probabilities. An estimating equations approach that leverages inverse probability weighting to adjust for bias due to informative timing of enrollment and unblinding is used for estimation.

TD's approach allows the population prevalence to vary in space and time. The contact rate parameters are allowed to depend on baseline covariates, vaccine receipt, and knowledge thereof, for example, allowing for more cautious behavior before full vaccination than after and before knowledge of vaccine receipt. Probability of acquiring COVID-19 is allowed to depend on calendar time, vaccine receipt, time since first vaccine dose, and baseline covariates; however, the ratio of conditional acquisition probabilities given covariates is assumed to depend only on time since first vaccine dose. As a result, vaccine efficacy is assumed to be independent of covariates and calendar time, thus precluding heterogeneity of vaccine efficacy (HVE) among subgroups or by factors varying with calendar time such as emerging viral variants. This assumption allows post-unblinding data to be linked to vaccine efficacy as realized under a blinded trial design, such that longterm vaccine efficacy can be assessed.

TD use simulations to explore the magnitude of bias of standard Cox proportional hazards estimation of vaccine efficacy that does not address the potential for informative unblinding. They conclude that, for settings tailored to mimic the phase 3 trial of the Moderna mRNA-1273 COVID-19 vaccine , standard methods are likely to have only negligible bias. They speculate that this occurs due to the randomized, double-blind study design, the low vaccine refusal rate, the short time window for procuring outside-study vaccination prior to on-study crossover, and the low COVID-19 event rate. It is worth noting that the latter two of these criteria may not be present in later studies conducted during periods of time when infection rates were higher and outside-study COVID-19 vaccines were much more widely available, leading to high rates of unblinding requests overall and higher rates in "high risk" strata such as among healthcare workers and the elderly. We suspect that another, likely more fundamental, reason that standard methods are nearly unbiased in these simulations is the lack of HVE, which is consistent with the assumption underlying their method. Consider a simple example for illustration. Suppose that older participants are more likely to be unblinded earlier and also that vaccine efficacy is lower in older versus younger individuals. In this case, standard marginal vaccine efficacy estimators, for example, based on Cox models or Kaplan Meier, will be positively biased, resulting in overoptimistic estimates of vaccine efficacy. The problem arises because, for each arm, these estimators assume that the risk set at each time is representative of those individuals who have not experienced the event at that time and have discontinued blinded follow-up. However, if unblinding occurs in an informative way that is related to efficacy-for example, through a covariate such as age-then it will necessarily be the case that at least one of the two arm-specific risk sets will fail to be representative and, as a consequence, the resulting efficacy estimator will be biased. This key assumption of the TD method is apt for the current COVID-19 vaccine trials. Indeed, the available data do not appear to suggest meaningful HVE for the COVID-19 vaccines currently under emergency use authorization in the United States (Polack et al., 2020; Baden et al., 2021; Sadoff et al., 2021) , with the exception of potential variation in efficacy by viral variants-an issue we will discuss below. However, it may not be appropriate for other future vaccine trials or future vaccines. Heterogenetity in vaccine efficacy has been observed in other settings. For example, the CYD-TDV dengue vaccine was shown to reduce the rate of severe disease for the subgroup of individuals previously exposed to dengue, even while it increased the rate of severe disease for the subgroup of previously unexposed individuals (Sridhar et al., 2018) .

This work follows several other related pieces that have addressed challenges with crossover of placebo recipients in COVID-19 vaccine trials. Follmann and colleagues (Finzi and Follmann, 2021; Follmann et al., 2021) focused on trials with blinded crossover and as such did not address the issue of informative crossover time. Lin et al. (2021) did consider unblinded crossover and potentially informative crossover time, but data collected after unblinding were not included in the analysis to avoid bias due to changes in behavior post-unblinding. Through carefully modeling those changes in behavior, TD leverage data collected after (potentially informative) unblinding to infer vaccine efficacy as realized in a blinded trial.

Both the prior frameworks (Finzi and Follmann, 2021; Follmann et al., 2021; Lin et al., 2021) and the framework of TD assume no heterogeneity in vaccine efficacy by subgroups or in calendar time in order to leverage postcrossover information for estimating vaccine efficacy. However, a specific issue in COVID-19 vaccine efficacy trials is the recent emergence of new viral variants that may modify vaccine efficacy. For example, the Alpha variant first discovered in the UK and by March, 2021 the dominant variant in the United States by virtue of its increased transmissibility may induce COVID-19 disease that is harder to prevent with vaccines based on the wild-type SARS-CoV-2 virus Wuhan variant (Karim and de Oliveira, 2021; Rubin, 2021) . As noted by TD, the Alpha variant was not in circulation prior to placebo crossover in the Moderna phase 3 trial, and so it will be fundamentally impossible based on the trial data alone to ascertain whether an apparent dimunition in vaccine efficacy long after first vaccination is attributable to lower efficacy against the Alpha variant or is attributable to waning immunity against all variantsor both. In settings where there is some information on COVID-19 incidence due to a new variant before and after the completion of crossover, variant-specific vaccine efficacy will be an important parameter to evaluate. An analysis based on mark-specific hazards can be employed in order to disentangle waning vaccine efficacy from variantspecific vaccine efficacy, albeit with limited precision.

In summary, we applaud TD for formalizing a critical problem in the evaluation of COVID-19 vaccines-and for providing a means to correct for a potential source of bias in many of the ongoing phase 3 trials.

This work was supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (UM1AI068635, R56AI143418, and DP2 LM013340).

Holly Janes https://orcid.org/0000-0002-3237-984X Fei Gao https://orcid.org/0000-0001-6797-5468

Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine

Assessing vaccine durability in randomized trials following placebo crossover

A deferred-vaccination design to assess durability of covid-19 vaccine effect after the placebo group is vaccinated

New SARS-CoV-2 variantsclinical, public health, and vaccine implications

Evaluating the long-term efficacy of covid-19 vaccines

Safety and efficacy of the BNT162b2 mRNA covid-19 vaccine

COVID-19 vaccines vs variants-determining how much immunity is enough

Safety and efficacy of single-dose ad26. cov2. s vaccine against covid-19

Effect of dengue serostatus on dengue vaccine safety and efficacy