key: cord-0831010-1gb5lw6b authors: Lipsitch, Marc; Kahn, Rebecca title: Interpreting vaccine efficacy trial results for infection and transmission date: 2021-06-12 journal: Vaccine DOI: 10.1016/j.vaccine.2021.06.011 sha: aad1cdd7e001f12382e2fcdab6760244ae414cd5 doc_id: 831010 cord_uid: 1gb5lw6b Randomized controlled trials (RCTs) have shown high efficacy of multiple vaccines against SARS-CoV-2 disease (COVID-19), and recent studies have shown the vaccines are also effective against infection. Evidence for the effect of each of these vaccines on ability to transmit the virus is also beginning to emerge. We describe an approach to estimate these vaccines’ effects on viral positivity, a prevalence measure which under the reasonable assumption that vaccinated individuals who become infected are no more infectious than unvaccinated individuals forms a lower bound on efficacy against transmission. Specifically, we recommend separate analysis of positive tests triggered by symptoms (usually the primary RCT outcome) and cross-sectional prevalence of positive tests obtained regardless of symptoms. The odds ratio of carriage for vaccine vs. placebo provides an unbiased estimate of vaccine effectiveness against viral positivity, under certain assumptions, and we show through simulations that likely departures from these assumptions will only modestly bias this estimate. Applying this approach to published data from the RCT of the Moderna vaccine, we estimate that one dose of vaccine reduces the potential for transmission by at least 61%, possibly considerably more. We describe how these approaches can be translated into observational studies of vaccine effectiveness. Randomized controlled trials (RCTs) have shown high efficacy of multiple vaccines against SARS-CoV-2 disease (COVID-19) [1] [2] [3] [4] [5] [6] , and recent studies have shown the vaccines are also effective against infection [7, 8] . Evidence for the effect of each of these vaccines on ability to transmit the virus is also beginning to emerge [9] [10] [11] . It is important to understand the effect of vaccination on infection, shedding and transmission of the virus [12] . This information can inform personal decisions about resuming contact once one has been vaccinated (or one's contact has), prioritization decisions [13] , models of the effect of vaccination [14] , and policy [15] . Hypothetically, it is possible that the 60-95% protection offered by these vaccines [1] [2] [3] [4] [5] 16] against symptomatic disease could (i) be purely protection against symptoms with no effect on infection or transmission (or increased transmission due to decreased case detection), (ii) be largely or entirely due to protection against infection, suggesting an effect on transmission similar to the efficacy against symptomatic infection; or (iii) be 70-95% protective against infection and moreover reduce the shedding of virus by those who do become infected, in which case protection against transmission could be even greater than that against symptomatic disease. The primary endpoint of RCTs to date, however, sheds little light on the magnitude of protection the vaccines could offer against transmission. The effect of a vaccine on transmission is a composite of its effect on becoming infected (because someone not infected cannot transmit) and its effect on the infectiousness of those who get infected despite vaccination [11] : these components have been called the vaccine efficacy for susceptibility to infection and vaccine efficacy for infectiousness [17] . Under plausible assumptions, the efficacy of a vaccine in preventing transmission can be defined as: where VE S and VE I are the vaccine efficacy against susceptibility (acquiring viral infection) and against infectiousness, respectively [17, 18] . RCTs of the Moderna, Astra-Zeneca, and Janssen vaccines have provided some evidence about vaccine effects on the probability that a trial participant will harbor detectable virus by swabbing participants irrespective of symptoms at one or more time points during the trial and testing the swabs by RT [2, 16, 19] . News reports indicate that those still in placebocontrolled trials will provide ongoing samples that can yield similar data over time [20] . In each case, reduced prevalence of viral positivity in vaccine vs. placebo recipients may be interpreted as a reduction in acquisition or duration or both, with potentially direct relevance to transmission. However, some reports from the original trials present composite measures involving infections detected by screening of non-symptomatic individuals combined with those detected by swabbing of symptomatic individuals [6] . Here we describe the results of simulations of randomized trials that are designed to clarify what information is gained by swabbing individuals for viral infection, how this relates to other measures of vaccine efficacy, and what information is present in measures combining different reasons for sampling (no symptoms vs. symptoms). We first show that the vaccine effect on viral positivity (VE V ) in individuals swabbed at random, regardless of symptoms (i.e. the prevalence odds of those testing positive for viral RNA (equation (4))), is closely approximated by a vaccine efficacy measure previously defined for bacterial carriage, despite several departures from the assumptions underlying the prior work [21] . This measure captures the product of the vaccine's efficacy in reducing acquisition and its effect in shortening infection duration. We describe how these departures affect the estimates under varying trial conditions. We show that under the reasonable assumption that vaccinated individuals who become infected are no more infectious during their duration of viral shedding than unvaccinated individuals, VE V is a lower bound on the vaccine's efficacy against transmission. We recommend that samples taken to assess vaccine effects on viral positivity be taken in a cross-section of the population (or, alternatively, in a cohort of contacts), irrespective of symptoms, and that this outcome be analyzed separately from the outcome of a positive test where the test was triggered by symptoms (the primary endpoint in most RCTs for SARS-CoV-2 vaccines). We simulate follow-up of 100,000 individuals for 300 days. For each person each day, we conduct a Bernoulli trial to determine if they will be infected that day, with a probability based on an external force of infection. We assume in our baseline simulations that this daily probability remains constant at 0.001 and also examine a higher force of infection of 0.003 in a sensitivity analysis. To examine the effects of deviations from the steady state assumption described in prior work for estimating VE V [21] , we compare scenarios in which individuals immediately become susceptible again after recovery (''SEIS") to scenarios in which prior infection confers full protective immunity for the duration of follow-up (''SEIR"). We vary the proportion of cases that are symptomatic (Table S1) , with symptom onset occurring five days after infection [22] . After a three day latent period [23] , infected individuals shed virus for a period drawn from a uniform distribution of 15-21 days [24, 25] . While in reality test sensitivity varies by day of infection [26] , we make the simplifying assumption in the simulations that individuals will test positive on any day they are shedding virus. On day 100, after the population has reached a pseudo steady state for prevalence, we randomize half of the individuals to receive a two-dose vaccine, with the doses given 28 days apart. We conservatively assume the vaccine confers 50% of its full two-dose efficacy after the first dose and that there is a seven day delay after each dose for immunity to take effect. We model three types of vaccine efficacy (Table S1 ). First, the vaccine multiplies the probability of infection each day by a factor 1 À VE S : Second, the vaccine multiplies the duration of shedding by a factor 1 À VE D . Third, the vaccine multiplies progression to symptoms among those infected by a factor 1 À VE P . We calculate VE P based on the value of VE S and the assumption that the vaccine reduces symptomatic disease by 95% (VE SP ) [1, 2] , using the equation: We then simulate testing and estimation of three measures of vaccine efficacy: We assume all individuals are tested regardless of symptoms on day t. Those who are shedding virus on day t are counted as positive (i.e. perfect test sensitivity and specificity). We then calculate d VE V using the prevalence odds ratio comparing vaccinated to unvaccinated using eq. (6) below. We estimate vaccine efficacy for non-symptomatic infection by calculating the prevalence odds ratio of PCR positivity among individuals who are not symptomatic on dayt in the vaccinated vs. unvaccinated groups. For this measure of vaccine efficacy, we count as positive any individuals who test positive on day t in cross-sectional testing as well as those who were symptomatic and tested positive on or before day t. We then calculate the odds ratio comparing vaccinated to unvaccinated. Code is available: https://github.com/rek160/Interpreting-VaccineEfficacy. Table 1 summarizes the notation for various measures of vaccine efficacy studied here. 3.1. For a vaccine that reduces both incidence and duration of viral carriage, vaccine efficacy against carriage can be interpreted as the product of these two effects. Prior work (concerning a bacterial pathogen, though in this exposition we refer to the pathogen as virus) showed that under certain assumptions, for a vaccine that reduces incidence but not duration of infection, the reduction in incidence rate caused by the vaccine (termed in the original paper the vaccine efficacy against acquisition [21] , but which we call vaccine efficacy against susceptibility to infection, for consistency with most of the literature [17] ) can be defined as where k v is the incidence rate in the vaccinated and k u is the incidence rate in the placebo arm, and can be estimated as where p u and p v are the prevalence of the virus in the placebo and vaccine arm respectively, so the estimator is just one minus the odds ratio for carrying the pathogen for vaccine vs. placebo recipients. As shown by [21] , if the vaccine does reduce duration of detectable infection, the quantity --vaccine efficacy for viral positivity --can be defined as the combined effect of the vaccine on incidence and duration, and can be estimated identically again from the prevalence odds ratio for the vaccine, and now with effects on both duration and incidence, we have the algebraic relationship where VE V is defined in eq. (5), VE S is defined in eq. (3), and Du is the reduction in average duration of viral positivity due to the vaccine. Thus, to generalize from reference [21] , VE V is an upper bound on VE S : VE V ! VE S with equality for the special case where VE D ¼ 0. 3.2. VE V Is a lower bound on the vaccine's efficacy against transmission, assuming that vaccinated individuals who become infected are no more infectious than unvaccinated individuals. Equations (1) and (7) show that VE V and VE T are similar though not identical; in particular, they differ in only one term: the substitution of VE D in the definition of VE V as opposed to VE I in the definition of VE T . If vaccinated and unvaccinated virus-positive individuals contributed equally to the force of infection, then these two terms would be identical, and we would have VE V ¼ VE T : that is, the reduction in transmission thanks to the vaccine would be the combination of reduced probability of infection and reduced duration of shedding in those infected despite vaccination. If we assume that for every day of being virus positive, a vaccinated infected person is on average no more infectious (and perhaps less due to lower viral loads) than an unvaccinated infected person with the same exposure, then we can conclude that Under this plausible assumption,VE T , which cannot be directly estimated from available trial data, is at least as large as VE V , which can. We therefore proceed to discuss how to estimate VE V for SARS-CoV-2 vaccines. 3.3. Simulated trials show that these estimators applied to a single cross-sectional swab approximately recover the simulated effects of a vaccine on viral positivity, incorporating effects on acquisition and duration, with visible downward bias just after and long after vaccines are administered. Fig. 1 shows results of 300-day simulations of a trial of 100,000 participants randomized 1:1 to vaccine or placebo on day 100. These participants have been exposed to a constant incidence of infection since day 0. The different panels represent (left to right) simulations with VE S ¼ 0; 0:3; 0:6:0:9and (top to bottom) VE D ¼ 0; 0:3; 0:6:0:9. We simulate a 2-dose regimen, 28 days apart with the first dose giving half the full efficacy and the effect of each dose starting one week after it is given, that is, on days 107 and 135 of the simulation. Given the five day incubation period, the vaccine's effects on symptomatic infection will be observed beginning 12 days after the vaccine dose. The solid black lines give the dose-1 and dose-2 predicted values for VE V based on eq. (7), while the curves show the estimates d VE V obtained from the simulated data using eq. (6). Fig. 1A shows the situation under the assumption that individuals naturally infected who recover (clear infection) become once again susceptible to reinfection. This is unrealistic for SARS-CoV-2 but follows the assumptions made in the above equations following [21] . Fig. 1B makes the opposite assumption, that individuals naturally infected (whatever their vaccine status) are completely protected against reinfection for the duration of the simulation. In Fig. 1A , there is close agreement between the simulated curves and the predicted ones, after about day 150. During the first-dose period, estimated efficacy is noisy at the beginning but is typically below the predicted level, reflecting holdover of infections that occurred before (randomization + 7 days), which are by assumption therefore unaffected by the vaccine. In contrast, toward the right-hand side of each panel, the agreement is nearly perfect apart from sampling error, because such holdover infections are vanishingly rare and the assumptions underlying eqs. (6) and (7) are met. Fig. 1B shows a similar pattern, with the important exception that over time, as many individuals in the population are immune, the protection estimated from eq. (6) declines toward the null value. This is because both groups have fewer people at risk as immunity builds up, but when the vaccine has an effect, the placebo group is depleted of susceptible individuals faster than the vaccine group, rendering the two groups more similar and the apparent efficacy lower. This effect, which is a known complexity of randomized [27] [28] [29] [30] and observational [31, 32] studies of vaccine efficacy/effectiveness, is subtle in our primary analysis (Fig. 1B) , but becomes more pronounced when there are longer times of follow up, higher forces of infection (Fig. S1) , and greater heterogeneities in infection risk among the study population. All trials of which we are aware for SARS-CoV-2 vaccines have had a primary endpoint of symptomatic disease, ascertained by asking every participant who experiences a defined profile of symptoms to get tested, and counting the outcome of COVID-19 when such a test is positive. As noted, some trials also test a subset of participants irrespective of symptoms, either at the visit for the second vaccine dose [2] or at defined intervals during follow up [16] . The primary endpoint measures vaccine efficacy against symptomatic infection, which has been called VE SP for vaccine efficacy against susceptibility or progression (that is, protection from 7), while the curves show the estimates obtained from the simulated data using eq. (6) . Panel A shows the situation under the assumption that individuals naturally infected who recover (clear infection) become once again susceptible to reinfection (SEIS). Panel B makes the opposite assumption, that individuals naturally infected (whatever their vaccine status) are completely protected against reinfection for the duration of the simulation (SEIR). Vaccine xxx (xxxx) xxx symptomatic infection that could be preventing infection or preventing symptoms if an individual becomes infected), and is related to VE S and VE P (Table 1 ) by eq. (2) above. Fig. 2 shows simulations similar to those above, but now with a virus assumed to cause symptoms in 1% (red) or 80% (blue) of infected individuals to demonstrate how the relationship between d VE V and d VE combined depends on the proportion symptomatic. In these simulations, all symptomatic individuals are assumed to be tested for the primary outcome on the day of symptom onset, and all asymptomatic individuals are not tested for the primary outcome. In addition, all individuals who have not yet experienced symptoms are tested for viral positivity and the combined VE is estimated. When only 1% of infected individuals are symptomatic, d VE V (solid) and d VE combined (dashed lines) are nearly identical. However, when 80% are symptomatic [33] , d VE combined increases over time but falls below the expectedVE SP: , as it is a combination of VE against asymptomatic and symptomatic disease. If analysis is restricted to only non-symptomatic individuals (Fig. S2) , when there is high VE P (i.e. low VE S for the same VE SP ), d VE nonÀsymptomatic is lower than d VE V . Table 2 shows data from the published RCT of the Moderna vaccine [2] , in which participants returning for their second vaccine dose were tested by RT-PCR for SARS-CoV-2, with 39 and 15 testing positive without symptoms, respectively, in the placebo and vaccine group. If we assume that everyone in the modified intent-to-treat population not infected prior to the second dose was tested (this is not documented in the paper), this corresponds (Table 2) to an estimate of d VE V ¼ 1 À 0:39 ¼ 0:61 95%CI0:31; 0:79 ð Þ , by eq. (6) . Taken at face value, this implies that one dose reduces virus positivity by 61%; our simulations suggest this may be an underestimate for several reasons. First, this estimate only includes asymptomatic individuals. As Figure S2 shows, if only individuals not symptomatic at the time of swabbing were included, as in the Moderna study, there could be additional underestimation of d VE V because vaccinated individuals without symptoms may disproportionately contribute to the non-symptomatic group. To resolve this potential bias, the data could be reanalyzed to also include anyone who tested positive on the day of the second dose and either was symptomatic or later became symptomatic. A mod- 7), while the solid curves show the estimates obtained from the simulated data using eq. (6) (the solid red line is the same as Fig. 1B) . The dashed lines give the dose-1 and dose-2 predicted values for VESP, based on equation (2), while the dashed curves show the estimates of VE combined obtained from the simulated data. When only 1% of infected individuals are symptomatic, the solid red and dashed red lines are nearly identical. However, when 80% are symptomatic [33] , the dashed blue line increases over time but falls below the expected VESP. est underestimate could also occur due to holdover of individuals infected before the first dose took effect and still positive at the time of the second dose. Finally, as noted above, one expects that VE V VE T (eq.8), so we conclude that the Moderna data from the second-dose swab provides evidence of at least a 61% (95% CI 31-79%) reduction in transmissibility due to a single dose of Moderna vaccine. The VE estimate combining cases ascertained by symptoms and those ascertained by this testing protocol in Table S18 of [2] is 89.5% (85.1%-92.8%). As described above, this combined measure is lower than the d VE SP estimated in the study of 94.1% (89.3-96.8%). We have shown that if analyzed correctly, data from randomized trials that test a cross-section of vaccine and control recipients irrespective of symptoms on a given day for virus can estimate the vaccine efficacy against viral positivity. While a complete estimate of VE T would require estimates of both VE V and of the daily infectiousness of a vaccinated, infected individual compared to an unvaccinated, infected one, and their correlation across individuals, it is very likely in practice that VE V is a lower bound on VE T : that is, an estimate from trial data of VE V provides strong evidence that VE T is at least as high. Our main findings are as follows: first, that a single crosssectional comparison of PCR positivity odds between individuals in vaccine vs. control groups provides a relatively accurate estimate, subject to sampling error, of vaccine effectiveness against viral positivity, which is a composite of effects in reducing susceptibility to infection and in reducing duration as described in Eq. (7). This can be shown analytically under certain assumptions. Second, we show by simulation that plausible deviations from these assumptions do not dramatically change results and, when they do, tend to bias toward the null hypothesis of no efficacy. A combined analysis of viral positivity detected due to symptoms and positivity detected by routine screening of non-symptomatic persons will be some combination of efficacy against viral positivity VE V and against symptomatic infection VE SP with no clear interpretation in terms of elementary quantities of interest. Thus separate analysis is recommended. Finally, if the cross-sectional sampling is restricted to those who are not symptomatic, it may underestimate VE V , especially for vaccines which are highly protective against symptoms (high VE P Þ:We therefore recommend that the crosssectional sample include those who are symptomatic. If this is infeasible (for example, if individuals are instructed not to come for a vaccine dose if they are symptomatic, and the testing happens at the vaccine dose), then we recommend that those who are tested because they are symptomatic and test positive on a particular day be included among the positives in the cross-section, constituting a partial exception to our recommendation of separate analyses. Our results have been described in the setting of a randomized trial. These results apply also to observational studies as well insofar as they are designed to mimic a target trial [34] and achieve adequate control of confounding and other sources of bias. In observational studies of vaccine effectiveness to date, cases have often been identified in whoever gets tested, for whatever reason [7, 35, 36] . These probably constitute a mix of (i) those tested because symptomatic, (ii) those tested because they are contacts of a known or suspected case (for example in a contact tracing investigation), and (iii) those tested without either reason, for example those who get tested in a regular program by their employer or those who get tested to comply with a travel restriction that requires a negative test before travel. Those positive in group (i) are approximately equivalent, in the observational setting, to those who meet the primary outcome of confirmed COVID-19 from randomized trials. Those positive in group (iii) are perhaps equivalent, in the observational setting, to those who test positive in the routine follow-up of persons in a randomized trial. Group (ii) does not have a clear equivalent in the randomized trials, which typically do not gather information on contacts. For observational studies, our results therefore imply that it would be ideal to analyze symptomatic cases separately from those routinely tested, and if possible to distinguish those tested due to possible exposure (group ii) from those tested for other reasons, such as for travel clearance (group iii). Those tested because they are symptomatic (group i) should be analyzed analogously to the trials, as the reduction in incidence rate. Those tested for exposure (group ii) are a group in which the efficacy measure is conditioned on exposure, and thus should be analyzed using methods to estimate the secondary attack rate, a risk measure. These recommendations follow standard approaches described in the landmark paper of Halloran et al. 1997 [37] . And those tested for neither reason (group iii) should be analyzed using the odds ratio approach described in this paper, extending others' prior work [21] . We have not considered another approach that has been used in COVID-19 trials [19] to estimate the effect on asymptomatic infections: serologic testing of participants at the middle or end of the trial [30] . This can contribute to an estimate ofVE S and thus provide a lower bound on VE T , but does not address the duration of infectiousness or the viral shedding of the detected asymptomatic infection. Nevertheless, this is an important additional way to obtain evidence relevant to bounding the vaccine's efficacy against transmission. In summary, with careful analysis, data from swabs of individuals in vaccine and comparator arms can yield estimates of a key quantity, the vaccine's efficacy in reducing viral positivity, likely a lower bound on the vaccine's efficacy in reducing transmission. Future work should consider how quantitation of virus in both symptomatic and non-symptomatic individuals who do test positive may further refine these estimates. This work was supported by the Morris-Singer fund by National Cancer Institute of the US National Institutes of Health Seronet cooperative agreement U01CA261277 and by the UK Department of Health and Social Care using UK Aid funding managed by the NIHR.The content is solely the responsibility of the authors and (Table S18 of [2] , minus those who became infected prior to the second dose (Fig 3 of [2] ). Vaccine xxx (xxxx) xxx does not necessarily represent the official views of any government agency. The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Dr. Lipsitch reports consulting/honoraria from Bristol Myers Squibb, Sanofi Pasteur, Janssen, and Merck, as well as a grant through his institution from Pfizer. He has served as an unpaid advisor related to COVID-19 to Pfizer, One Day Sooner, Astra-Zeneca, Janssen, and COVAX (United Biomedical). Dr. Kahn discloses consulting fees from Partners In Health. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19 Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia Covid-19: Novavax vaccine efficacy is 86% against UK variant and 60% against South African variant Single-dose administration and the influence of the timing of the booster dose on immunogenicity and efficacy of ChAdOx1 nCoV-19 (AZD1222) vaccine: a pooled analysis of four randomised trials The effectiveness of the first dose of BNT162 b 2 vaccine in reducing SARS-CoV-2 infection 13-24 days after immunization: real-world evidence Interim Estimates of Vaccine Effectiveness of BNT162b2 and mRNA-1273 COVID-19 Vaccines in Preventing SARS-CoV-2 Infection Among Health Care Personnel, First Responders, and Other Essential and Frontline Workers -Eight U.S. Locations Decreased Infectivity Following BNT162b2 Vaccination 2021 Estimating vaccine efficacy against transmission via effect on viral load Indirect Protection by Reducing Transmission: Ending the Pandemic with SARS-CoV-2 Vaccination. Open Forum Infect Dis Understanding COVID-19 vaccine efficacy Modelinformed COVID-19 vaccine prioritization strategies by age and serostatus Vaccine optimization for COVID-19, who to vaccinate first? CDC. When you've been fully vaccinated 2021 Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK Design and Analysis of Vaccine Studies Leveraging pathogen sequence and contact tracing data to enhance vaccine trials in emerging epidemics Vaccines and Related Biological Products Advisory Committee Meeting February 26. 2021 FDA Briefing Document Long-Term Studies Of COVID-19 Vaccines Hurt By Placebo Recipients Getting Immunized. National Public Radio Estimation of vaccine efficacy against acquisition of pneumococcal carriage Incubation period of COVID-19: a rapid systematic review and meta-analysis of observational research SARS-CoV-2 (COVID-19) by the numbers SARS-CoV-2, SARS-CoV, and MERS-CoV viral load dynamics, duration of viral shedding, and infectiousness: a systematic review and meta-analysis Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening Quantifying antibody kinetics and RNA detection during early-phase SARS-CoV-2 infection by time since symptom onset Apparent declining efficacy in randomized trials: examples of the Thai RV144 HIV vaccine and South African CAPRISA 004 microbicide trials The hazards of hazard ratios Estimability and interpretation of vaccine efficacy using frailty mixing models Analyzing vaccine trials in epidemics with mild and asymptomatic infection Depletion-of-susceptibles bias in influenza vaccine waning studies: how to ensure robust results Measurement of Vaccine Direct Effects under the Test-Negative Design Asymptomatic SARS-CoV-2 infections: a living systematic review and meta-analysis Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting Study designs for evaluating different efficacy and effectiveness aspects of vaccines We thank Dr. Lee Kennedy-Shaffer for helpful discussion.