key: cord-0267184-oalocjde authors: Graziani, C. title: A Simplified Bayesian Analysis Method for Vaccine Efficacy date: 2020-12-11 journal: nan DOI: 10.1101/2020.12.07.20244954 sha: bd30ecec50659572423685063502981bbb05d58f doc_id: 267184 cord_uid: oalocjde We describe a simplified Bayesian analysis of vaccine trial data, in which a reparametrization of the Poisson likelihood leads to a factorization in which the protective vaccine efficacy VES and the nuisance parameter appear in different factors. As a consequence the posterior density acquires a factorized form, and marginalization over the nuisance parameter is trivial. Estimates of VES accordingly become a matter of simple manipulations of one-dimensional posterior probability densities. We demonstrate the method using the publically-released final Phase III data from the Pfizer/BioNTech and Moderna mRNA vaccines and for the interim data released for the Sputnik V adenovirus vaccine, for SARS-CoV-2. placebo group -results in a factorization of the likelihood into a factor depending only on V E S and a second factor depending only on λ. As a result, one obtains a simple, intuitive result for the reduced likelihood L e (V E S ), which in turn permits very simple inferences about credible regions, that are better insulated against variations due to prior choice. We describe the method in §2. In §3 we exhibit it's use in a study that compares the efficacy of three recently-announced vaccines agains SARS-CoV-2: The Pfizer/BioNTech mRNA vaccine [11] , the Moderna mRNA vaccine [12] , and the Sputnik V adenovirus-based vaccine [13] . We supply some final discussion in §4. We assume that infections in the placebo and vaccine samples are well-described by Poisson models with different means µ P and µ V , respectively. For the sake of notational compactness, we will denote the protective vaccine efficacy V E S by the symbol e. We introduce a model for e by assuming that where r is the vaccine-to-placebo ratio of person-times at risk -that is, if the vaccine and placebo groups represent P V and P P person-times of exposure to the pathogen respectively, then r ≡ P V /P P . If the infection numbers recorded during the trial are N V and N P respectively for the vaccine and placebo groups, then the likelihood under the model is the product of the Poisson likelihoods for the two groups, that is This form of the likelihood entangles the nuisance parameter µ P with the quantity of interest, the efficacy e, so that after introducing a prior distribution and using Bayes' theorem, the posterior distribution over (e, µ P ) is similarly entangled. Rather than submitting to the necessity of explicitly marginalizing the posterior over µ P , we may introduce a factorized form of the likelihood by means of a change of variables. We introduce the parameter λ, defined as The new parameter λ is the expected number of overall infections, summed over the two groups -this is clear, since the sum of two Poisson random variables is itself a Poisson random variable whose mean is the sum of their respective means. We may use Equation (2.3) to substitute λ for µ P in Equation (2.2) . The resulting likelihood is where we have introduced the reduced likelihood for e, The advantage of the form of the likelihood in Equation (2.4) is that the dependence on e is factorized from the dependence on λ. If we now introduce a factorized prior π λ (λ) × π e (e), then we obtain a factorized posterior distribution. Marginalization over λ is trivial, and we obtain for the marginalized posterior the formula where κ is a normalization constant. From this point of the discussion, we assume, for simplicity, a proper uniform prior π e (e) = 1 on the domain e ∈ [0, 1]. In §3.3 we will re-examine the effect of prior choice on this model. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted December 11, 2020. ; One may easily show that if N V 0 then the posterior in Equation (2.6) has a maximum at e MP ≡ 1− N V /r N P = 1− N V /P V N P /P P , which accords with intuition -the relative incidence 1 − e MP is given by the ratio of the person-time adjusted infection rates. On the other hand, when N V = 0 the posterior increases monotonically, attaining a domain maximum at e MP = 1. This also accords with intuition: if no vaccinated subjects become infected, the most probable efficacy is 1.0 -perfect efficacy. Of course, in this case the distribution may still have substantial mass at much lower efficacies, depending on the value of N P . The normalization constant κ may be calculated by integrating Equation (2.6) with respect to e, in the interval [0, 1]. This may be done by the change of variables e → = 1 + (1 −e)r , expanding the resulting integrand using the binomial theorem. The result is The usage of the formula for the posterior in Equation (2.6) is very straightforward. One may simply bin the interval [0, 1] into N Samp equally-spaced sample points, compute the factorized likelihood at each point (multiplying by a non-uniform prior if so desired), and compute the normalization κ either by Equation (2.7) (for a uniform prior) or by a straightforward numerical quadrature. Then, after normalization, various desired interval estimates (such as credible regions, or upper/lower bounds of prescribed probability) can be obtained by sorting the array containing the 1-dimensional discretized posterior over e, and processing it as appropriate. We now demonstrate the use of these procedures using public data from SARS-Cov-2 vaccine trials. At the time of this writing (early December 2020), the final data from the Phase III trials of the mRNA vaccines against SARS-CoV-2 from Pfizer/BioNTech [11] and from Moderna [12] are available from public information, although not yet from scientific journal articles. The two data releases include results for entire trials, and for subsamples of infections leading to "severe" COVID-19 symptoms. In addition, preliminary data from the Sputnik V vaccine trials are also available [13] . The data are summarized in Table 1 . The results of applying the reduced likelihood Bayesian analysis (with a uniform prior) to these samples and subsamples are summarized in Table 2 , and displayed in the panels of . These are all obtained by straightforward application of the method described in §2, implemented in a short Python script that makes use only of the Numpy [14] and Matplotlib [15] modules. The code is available for public download under an MIT license at https: //github.com/CarloGraziani/BayesVaccineEfficacy. We now discuss these results in more detail. From the left-hand side panels of Figure 3 .1, we may note first that, as expected, the preliminary data released for the Sputnik V vaccine is much less constraining of the efficacy than is the final Phase III data for the Pfizer/BioNTech 3 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint These results are also consistent with the announced efficacy estimates from the (as of this date unpublished) statistical analyses carried out by Pfizer/BioNTech (estimated efficacy: 95% [11] ) and by Moderna (estimated efficacy: 94.1% [12] ). The Bayesian estimate for the efficacy of the Sputnik V vaccine is also consistent with the announced estimate from preliminary data (91.4% [13] ), although as pointed out above this data is less constraining of the final result than is the data for the other two trials. An analysis of the "severe" case data suggests that the recruitment programs for the Pfizer/BioNTech and the Moderna trials were significantly different, and that the Moderna trial recruited populations at higher risk of severe COVID-19 symptoms. According to the published data [12] the Moderna trial featured 30,000 subjects, divided equally into placebo control and vaccine groups. Of the 15,000 control subjects, 30 developed "severe" symptoms. On the other hand, the Pfizer/BioNTech trial [11] featured 43,661 participants, half of whom were in the control group, and only 9 of these developed "severe" symptoms. On this data, it seems clear that the populations in the two trials must have been selected quite differently, since the control group incidence of severe COVID-19 appears to have been about 4.5 times higher for the Moderna trial than for the Pfizer/BioNTech trial. This is possibly explained by the released information about the Moderna trial [12] , which asserts "The COVE study includes more than 7,000 Americans over the age of 65. It also includes more than 5,000 Americans who are under the age of 65 but have high-risk chronic diseases that put them at increased risk of severe COVID-19, such as diabetes, severe obesity and cardiac disease. These medically high-risk groups represent 42% of the total participants in the Phase 3 COVE study." The Pfizer/BioNTech information release makes less precise assertions about co-morbidities and other risk factors, to the effect that "Approximately 42% of global participants and 30% of U.S. participants have racially and ethnically diverse backgrounds, and 41% of global and 45% of U.S. participants are 56-85 years of age." Other factors, such as geographic distribution of the trial subjects, may also explain the discrepancy. From a comparison of the posteriors and credible regions for the two trials, it would appear that Moderna's trial was better adapted to the purpose of assessing protective efficacy against severe COVID-19 than was the case with the Pfizer/BioNTech trial. The efficacy is much better-constrained in the Moderna trial than is the case for the Pfizer/BioNTech trial -Moderna's 90% credible region is e ∈ [0.917, 1.0] in contrast to Pfizer/BioNTech's 90% credible region of e ∈ [0.452, 0.993]. Note that the fact that Moderna's severe efficacy posterior peak is at e = 1.0 whereas Pfizer/BioNTech's is not is merely due to the accident that N V = 0 for the Moderna trial, as opposed to N V = 1 for the Pfizer trial. As should be clear from the form of the reduced likelihood (Equation 2.5), the happenstance of observing 0 rather than 1 produces this posterior shape, which should not be overinterpreted. The most reasonable 1 Given two posteriors π 1 (e 1 ), π 2 (e 2 ) over the efficacies e 1 and e 2 of two vaccines it would be a simple matter to compare the efficacies by computing a posterior density over the random variable ∆e ≡ e 2 −e 1 , by implementing a discretized quadrature of the integral π (∆e) = ∫ de 1 π 1 (e 1 )×π 2 (e 1 +∆e). We do not carry out such a comparison in this work. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted December 11, 2020. ; https://doi.org/10.1101/2020. 12 Table 3 : Summary of Bayesian efficacy estimation for SARS-CoV-2 vaccine trials assuming the "show-me" prior. takeaway from this comparison is that the Pfizer severe COVID -19 efficacy is quite poorly constrained, but the overlap in the respective 90% credible regions again suggests that it is entirely possible on present data that the severe COVID-19 efficacies of the two vaccines are the same. To test robustness against prior choice, we select a prior that is somewhat informative compared to a uniform prior: We refer to this prior as the "show-me" prior, because it embodies skepticism about values of e near 1.0, and insistence on very informative data before conceding that such values are plausible. With the "show-me" prior, the results from the Bayesian analysis are changed as reflected in Table 3 . A comparison of the results in Tables 2 and 3 shows that the final results for the Pfizer/BioNTech and Moderna trials are barely affected by the change -the modifications are all in the third significant figure of the confidence regions and the posterior maxima. The modifications to the Sputnik V estimates are somewhat larger, but still modest -all in the second significant digit. Again, this is to be expected, in consequence of the lower numbers and broader likelihood for this preliminary trial data. The Moderna Severe results also vary modestly, in the second significant digit. Here the biggest change is that the "show-me" prior drives the posterior to 0 at e = 1.0, so the max-posterior point is moved to 0.966 instead. For these kinds of trial numbers, it is clear that even a rather extreme prior such as the "show-me" prior cannot affect the results of the analysis very substantially. The largest change is in the Pfizer/BioNTech Severe results, where the changes are in the most significant digit. This is hardly surprising, since the likelihood shape (which is also the posterior shape in the top-right panel of Figure 3 .1) is not even arguably well-concentrated in comparison to the "show-me" prior. For this kind of small-number data, more careful consideration of prior choice is clearly necessary. As illustrated in §3, the reduced likelihood approach leads to an easily-implemented 1-dimensional posterior density on the unit interval for the vaccine efficacy. Simple and intuitive results are easily extractable from such posteriors, and with the numbers characteristic of vaccine trials such as the three considered above, the overall efficacy estimates are quite robust to prior choice. The most cogent take-away conclusion from the Bayesian analysis in §3 appears to be that on present data, the three vaccines for SARS-CoV-2 cannot be said to differ greatly in their efficacies. The differences are largely in the degree to which those efficacies are constrained by their respective trial data. More thought needs to be given to prior choice in the event where subsamples have low counts. In the Pfizer/BioNTech Severe case, where there were 10 total infections, use of a skeptical prior such as the "show-me" prior can at least provide some conservatism to estimates. In such cases it is possible that more principled prior choice approaches, such as the reference prior approach advocated in [8] could lead to better-constrained, but still reasonable results. This is of course true irrespective of whether one adopts the reduced likelihood approach advocated here or another standard Bayesian nuisance parameterization. 6 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted December 11, 2020. ; Some generalization of the approach is possible: to test time-varying immunogenesis (whether short-term ramping up of immune response, or long-term decline of same, as in e.g. [16] ), it is straightforward to introduce an empirical parametrized time dependence to the efficacy, i.e. e = (t; θ ), where θ are parameters. If the data is in time bins, then the time-dependence (t; θ ) could be integrated with respect to t over each time bin, and the likelihood computed as the product of the individual bin reduced likelihoods, which are now functions of θ . The resulting analysis no longer leads to simple manipulations of 1-dimensional posterior distributions on the unit interval, and MCMC methods may be advisable, depending on the dimensionality of θ . However, one benefit is that the nuisance parameter is still banished from the analysis, so that the parameter space has its dimensionality reduced by 1, and is limited to the empirical parameters of interest. Design and analysis of vaccine studies Bias reduction for risk ratio and vaccine effect estimators Obtaining confidence intervals for the risk ratio in cohort studies On sample sizes to estimate the protective efficacy of a vaccine Approximate is better than "exact" for interval estimation of binomial proportions Confidence intervals for a binomial proportion Bayesian estimation of vaccine efficacy A bayesian framework for the ratio of two poisson rates in the context of vaccine efficacy trials Reference posterior distributions for bayesian inference Bayesian data analysis Pfizer and BioNTech Conclude Phase 3 Study of COVID-19 Vaccine Candidate, Meeting All Primary Efficacy Endpoints Moderna Announces Primary Efficacy Analysis in Phase 3 COVE Study for Its COVID-19 Vaccine Candidate and Filing Today with U.S. FDA for Emergency Use Authorization Second interim analysis of clinical trial data Array programming with NumPy Matplotlib: A 2d graphics environment Long-term persistence of zoster vaccine efficacy The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.