key: cord-0826866-ha2ecjca
authors: Diament, Sean M.; Kaya, Ayse; Magenheim, Ellen B.
title: Frames that matter: Increasing the willingness to get the Covid-19 vaccines
date: 2021-11-12
journal: Soc Sci Med
DOI: 10.1016/j.socscimed.2021.114562
sha: 171dae45b2653bf12abc23411645def4387265a0
doc_id: 826866
cord_uid: ha2ecjca

Herd immunity against Covid-19 demands a high rate of vaccination, which may be challenging to meet given vaccine hesitancy in the U.S. How can Americans' willingness to get vaccinated be increased? Using a survey experiment, we apply seven framing treatments to a representative sample of 1642 U.S. residents that test ways to increase willingness: expert and political figure endorsements, demonstrations of receiving the vaccine, information about the vaccine's approval process, and information underscoring the pandemic's devastating economic impact. We find the approval process and the economy treatments increase the odds of higher vaccination willingness by 42% (p = 0.068) and 46% (p = 0.060), respectively. Additionally, we find suggestive evidence that the effectiveness of a message depends on whether a respondent finds the message/messenger credible. The study advances the understanding of vaccine hesitancy by demonstrating effective public health messaging strategy can facilitate greater acceptance of vaccination.

The toll that the Covid-19 pandemic has taken on countries around the world is staggering. In the United States alone, the number of deaths surpassed 500 thousand by March 2021. Fortunately, the rapid development of highly effective vaccines (Polack et al. 2020; Baden et al. 2021 ) offers hope for achieving herd immunity and containing the disease's spread (Haas et al. 2021; Tande et al. 2021 ). Yet, herd immunity can be achieved only if a large majority of a country's population gets vaccinated; for example, Kwok et al. (2020, e32) estimate that figure at 69.6% for the U.S. However, surveys taken before and after the commencement of the distribution of vaccines (see Funk and Tyson 2020; Saad 2021; Kaiser Family Foundation 2021) suggest that a sizeable portion of the eligible American population is vaccine hesitant, which "refers to [a] delay in acceptance or refusal of vaccination despite availability of vaccination services" (MacDonald 2015, 4161) .

A key question for American public policy is how to boost willingness to get vaccinated, so that herd immunity becomes achievable more quickly-or at all. What approaches could potentially boost an individual's positive inclination toward the vaccine? We start with the hypothesis that intentions to be vaccinated can be increased through a selection of messengers and messaging, which creatively reorient how individuals evaluate the situation, invoke trust in vaccination, and emphasize the salience of the issue, thus increasing the intention to take the vaccine in a timely manner (Wood and Schulman 2021) .

To better understand vaccination intention, we conducted a survey experiment on a representative sample of 1,642 American residents in February of 2021. The survey included seven treatments designed to measure the relationship between different messengers/messaging and the intention to receive a vaccine. Survey experiments randomly vary a framing message J o u r n a l P r e -p r o o f across subject groups in order to isolate an independent treatment effect on respondent preferences (Tversky and Kahneman 1981; Druckman 2001) . These frames act as priming tools to illicit a directional response from subjects (Zaller 1992; Chong and Druckman 2007) , including on important factors related to vaccination hesitancy (Nyhan et al. 2014; Chen and Stevens 2017) . Specifically, our treatments consider the pro-vaccination message or the demonstration of vaccination coming from a public figure, knowledge about the scientific process underlying the vaccine's approval, and information about the negative economic repercussions of the pandemic.

Our dependent variable (DV) is the individual's willingness to take a Covid-19 vaccine based on the following question: "If access is not an issue, are you willing to get the coronavirus vaccine?" Answers could be "yes, whenever available", "no, I am not willing", and "maybe, but not immediately". For those that answer "maybe", we subsequently inquire about when they would feel comfortable being vaccinated. Our analysis, then, essentially focuses on the individual's willingness to be vaccinated, which is of course not the same as the individual actually getting vaccinated (see Dai et al. Forthcoming) . Nonetheless, willingness constitutes the necessary first step in convincing individuals to receive the coronavirus vaccine (see Bokemper et al. 2021; Kerr et al. 2021; Kreps and Kiner 2021) . In a novel difference from existing studies, the DV in our study allows us to measure the preferred timing of vaccination across respondents.

In addition, to better understand attitudes about vaccination, we collected a rich array of sociodemographic information, social and political views, and opinions on health matters.

Our findings suggest that, relative to the control group, respondents' willingness to be vaccinated increases most robustly under two different messaging treatments: when individuals are informed about the Food and Drug Administration (FDA)'s vaccine approval process and J o u r n a l P r e -p r o o f when individuals are reminded about the pandemic's economic consequences. The effects of public figures' embrace of vaccination are not as strong, although we find some evidence for their efficacy (see also Bokemper et. al. 2021 on public health experts and Bokemper et. al. 2021 and Kreps and Kriner 2021 on messaging by politicians).

Moreover, these findings, which are robust under multiple specification checks, suggest centering a public health communication strategy on messaging content-which boosts information about vaccine safety, as well as the negative impact of the pandemic-may be more effective than relying on specific public messengers' endorsements of the vaccine. Further, our analysis shows other frames, such as endorsements and demonstrations, also have the potential to positively boost willingness, especially if the respondent correctly perceives the intended message. Finally, beyond known correlates, we demonstrate other variables matter for willingness, such as media consumption (which we think proxies information on the pandemic and vaccines), concern about getting sick, and the rate of mortality from Covid-19 in the individual's locality. Overall, the study illuminates potential pathways to reduce vaccine hesitancy.

Our survey experiment design, which is further detailed in the Appendix, prioritized two goals. First, we aimed to construct a novel temporal outcome measure that could help ascertain when individuals might be willing to take the vaccine (if not immediately) with nuance beyond what extant surveys show (Funk and Tyson 2020; Saad 2021; Kaiser Family Foundation 2021) .

As Table 1 shows, our seven-level outcome variable integrates time: an individual willing to be vaccinated immediately receives a 6, the highest quantitative score for the DV; whereas, an individual never willing to be vaccinated gets a 0 (zero), with all other levels staggered in-J o u r n a l P r e -p r o o f between. While the aforementioned surveys broadly illuminate what percentage of the population may be willing to be vaccinated, our study advances existing knowledge by examining how quickly some of the hesitant populations may be willing to take the vaccine. (Table 1) Second, we wanted to scientifically test a variety of messages that could increase intention for vaccination uptake in the American adult population. As Table 1 shows, the control condition provided respondents with contextual background, including statistics on the prevalence of the pandemic and the development of highly effective vaccines, then asked respondents whether (and when) they intend to get vaccinated. Our average "yes" rate of 57.1% in the control is comparable to other surveys conducted during a similar time period (see Table   2 ) (KFF 2021). The information contained in the control group provides a stringent test of our framing treatments, since the treatments vary the vignettes beyond this baseline of well-known information to better assess how willingness for the vaccination can be further increased.

All seven randomized treatment conditions within the survey were designed to simulate real-world messages that increase the respondent's knowledge by: showing images of different non-partisan medical professionals getting vaccinated (Treatments 1 and 2) or endorsing vaccination (Treatment 5); messages from political leaders across the U.S. political divide (Treatments 3 and 4); substantive framing about the FDA's approval process of the vaccine (Treatment 6); and details about economic conditions during the pandemic (Treatment 7).

These treatments naturally intersect with other comparable studies, but also differ from them in important ways. Like other studies (Bokemper et. al. 2021; , our choice of survey design reckons with the fact that the American population is largely polarized and distrustful of governmental actors, institutions, and policies relative to other advanced J o u r n a l P r e -p r o o f democracies (Citrin and Stoker 2018) . Particularly, T 3 and T 4 aim to assess whether partisanship can be leveraged to increase vaccination propensity by triggering "in-group" sentiment through ideology (Tajfel and Turner 1986) . Throughout, we also take into account factors that relate to political trust, which has been shown to correlate with vaccination sentiment (Lazarus et al. 2021; Kerr et al. 2021) .

Similar to other studies, we also examine public health experts like Dr. Fauci (Bokemper et al. 2021 ), but we also advance upon these analyses. Particularly, T 1 presents Nurse Sandra Lindsay-a Black woman who was the first individual to receive a vaccine in the U.S. outside of clinical trials-getting vaccinated. Our primary goal by including Nurse Lindsay was to have reference to a public health expert who was widely publicized, but not a career federal employee like Dr. Fauci (our T 2 and T 5 ). We recognize that Nurse Lindsay could potentially channel group representation to increase trust in vaccine safety and efficacy within the African American community (Mansbridge 1999; Tate 2003) , but testing this hypothesis was not our primary goal.

Also, by including both a Dr. Fauci demonstration (T 2 ) and an endorsement (T 5 ), we aimed to analyze potential visual versus textual effects. Importantly, two treatments-regarding the FDA approval process and explaining pandemic-related economic consequences-are constructed as valence appeals meant to solicit favorable reactions across different respondents. The approval process treatment is intended to increase trust in the vaccine's safety, while the economy treatment hypothetically crystalizes threats to economic livelihood and cues a desire for normalcy. Given the salience of vaccine safety in encouraging individuals to be vaccinated, studies similar to ours have grappled with how to elicit confidence in the vaccines (Kerr et al. 2021 ), but to our knowledge, we are the first J o u r n a l P r e -p r o o f to examine the specific FDA treatment. We are also not aware of any other study that uses information about the effect of the pandemic on the economy as a treatment.

As emphasized earlier, experimental framing research relies on randomly varying messages across subject groups to isolate an independent treatment effect on preferences.

However, the very mechanism undergirding the enterprise could be its undoing if other survey questions prime respondent reactions to the experimental conditions (McFarland 1981; Iyengar and Kinder 1987; Gaines, Kuklinski, and Quirk 2007) . To minimize this risk, we required the randomized experimental treatment to be the first question a respondent receives, followed by other randomized questions. This approach privileges the validity of the experimental conditions. Furthermore, our key control variables, such as socio-demographic indicators, have low risk, if any, of being susceptible to priming, since they are largely descriptive (age, race, gender) and/or deeply held (partisan affiliation). This method provides the greatest security within the survey experimental design by guaranteeing the treatments are not influenced by other questions (Gaines, Kuklinski, and Quirk 2007) , thereby ensuring we capture respondents' real-world sentiments (Sniderman 2011, 109-110) .

Before fielding the survey, we sought and received approval from [Name Redacted for Review]'s Institutional Review Board. We then contracted with Qualtrics to access their diverse national pool of respondents. Qualtrics fielded the survey from February 9 th to 14 th and yielded a representative sample of 1,642 American adults. To guarantee representativeness, our sample used a quota system on race, gender, partisan affiliation, and age within each of the eight experimental groups. This kind of within-treatment representativeness is a more rigorous J o u r n a l P r e -p r o o f approach to ensure internal validity (between the reference groups) and external validity (for generalizability) than survey-wide representativeness. Table 2 details this within-group balance by comparing our sample to the demographic breakdown of the U.S. adult population. The sample is well-balanced: groups reach near parity on all metrics, except the control group skews younger, more Republican, and less Black. That said, given the random assignment into the experiment's control and treatment groups, our results are reasonably generalizable. Finally, the table also presents existing metrics on vaccination willingness for the U.S. adult population in comparison to the values we observe in our control and treatment groups.

The full codebook and an additional robustness check for representativeness are presented in Appendix Tables A1 and A2, respectively. Using G*Power (Erdfelder et al. 1996) , our study is estimated to capture an effect size of 0.3 (standard deviation) between control (n=205) and treatment group (lowest n=201) at the α=0.05 level (two-tailed; power=0.85).

According to G*Power, this is considered a small effect size, suggesting our study is attuned to both large and acute statistical differences. We used STATA/IC 15.0 in our statistical analyses and the data (as a .csv file) and code (as .dta and .do files) are available at [authors' GitHub link is redacted to maintain anonymity during review].

To analyze treatment effects, Figure 1 presents the t-tests between the control group (x ̅ =4.078, 95% CI=3.738-4.419) and each of the treatment groups. Neither demonstration, by Nurse Lindsay (T 1 : x ̅ =4.412, 95% CI=4.096-4.727, p=0.157) nor by Dr. Fauci (T 2 : x ̅ =4.412, 95%

CI=3.757-4.409, p=0.984), significantly change willingness to be vaccinated. While the Nurse Lindsey treatment mean is larger than that of the Dr. Fauci treatment, it is still not statistically J o u r n a l P r e -p r o o f significantly different from the control mean. Similarly, all three textual endorsement means (T 3 :

x ̅ =4.239, 95% CI=3.907-4.571, p=0.505; T 4 : x ̅ =4.272, 95% CI=3.943-4.601, p=0.420; and T 5 :

x ̅ =3.928, 95% CI=3.587-4.268, p=0.539) are not statistically different from the control mean.

The insignificance of both Fauci treatments (T 2 and T 5 ) means there is inadequate evidence to contrast a show of demonstration versus a message of approval by the same expert here. It is nonetheless telling that the demonstration had no apparent impact on willingness.

In contrast, the FDA approval process (T 6 ) is associated with an increase in the mean vaccination willingness from 4.078 in the control to 4.548 (95% CI=4.254-4.842, p=0.040).

Relative to the control, a description of America's economic woes during the pandemic (T 7 )

shows higher pro-vaccine sentiment with a mean of 4.512 (95% CI=4.197-4.828, p=0.066). T 6 and T 7 likely increase willingness in different ways-the former provides reassuring information that a relatively large group of experts have weighed in and have converged on the decision, while the latter plausibly invokes both empathy and fear. In this light, frames that increase substantive knowledge on salient matters appear to boost vaccination willingness.

We next run a series of ordered logit models, which allow us to examine other correlates of willingness and account for any heterogeneity across treatment groups that our aforementioned quotas may not entirely eliminate. We use ordered logit because our DV is an ordinal variable with interval levels that hierarchically order the data. These results are presented in Table 3 , with the coefficients as odds ratios (OR) and robust standard errors (SE) in

parentheses. An odds ratio of higher than 1 indicates greater odds (for willingness), while less than 1 means lower odds. While the experimental analysis is conducive for causal assessments, J o u r n a l P r e -p r o o f Models 2 through 5 are better understood as providing evidence of correlation between variables and odds of greater vaccination willingness. (Table 3) The results in Model 1 present our primary findings. For respondents in the FDA condition, the odds of having a higher vaccination willingness are 1.42 times-or 42%-higher than that of respondents who were in the control group (OR=1.420, SE=0.273, p=0.068).

Similarly, those in the economy condition are 46% more likely to have a higher willingness Model 3 shows that sociopolitical views are also significantly associated with vaccination willingness. The more one leans Republican on a seven-point Likert scale, the less likely they are J o u r n a l P r e -p r o o f to hold a greater willingness by 19.2% (OR=0.808, SE=0.022, p<0.001). This mirrors findings from similar studies constructed at different points in time (Kreps and Kriner 2021, 3255) . Table   A3 in the Appendix substitutes a measure of ideology for party, showing conservative respondents have lower odds of greater vaccination willingness (OR=0.781, SE=0.027, p<0.001). A non-surprising finding-but formerly insignificant in Model 2-is the lower likelihood of Black individuals, relative to Whites, by 31% toward (higher) vaccine willingness (OR=0.607, SE=0.109, p=0.006). This is consistent with previous research that finds due to unethical experimentation (Washington 2006) , discrimination in the healthcare system (Chapman, Kaatz, and Carnes 2013) , and a general lack of trust in the political system (Woko, Siegel, and Hornik 2020) , this population displays greater hesitancy toward vaccination. The media index in Model 4 is not found in prior studies related to this topic, but we included it to assess exposure to information, which existing work suggests as boosting J o u r n a l P r e -p r o o f inclinations toward taking the flu vaccine (Chen and Stoecker 2020) . While the notion of fake news suggests that more information is not necessarily good, we find the respondents with the maximum media intake across a battery of four news sources-newspaper, radio, television and internet-are over three times as likely to report higher vaccination willingness (OR=3.154, Finally, in Model 6, we examine how the societal impacts of Covid-19 relate to an individual's odds of greater vaccination willingness. One obvious place to start is to test the relationship between local death toll and vaccination sentiment. Since the variable for countylevel Covid-19 deaths produces a skewed distribution, we use the natural log of the variable to handle skewness and outliers. Model 6 indicates that an increase in log deaths is associated with about 8% higher odds that the respondent will have a greater willingness to vaccinate (OR=1.081, SE=0.036, p<0.019). For ease of interpretation, we also computed the predicted probability of deaths in relation to likelihood of greater vaccination willingness (not shown in table) . Holding all other variables constant at their means, going from approximately one standard deviation below the mean value for logged Covid-19 deaths (6.14) to roughly one deviation above the mean (from 4 to 8), on average, increases the share of respondents willing to get vaccinated immediately by 7 points, from 57.17% (95% CI: 51.35%-62.98%) to 64.59% (95% CI: 61.47%-67.71%). Finally, the decreased odds based on incidence rate of Covid-19 in the county are just shy of significance (OR=0.966, SE=0.022, p=0.118). Both of these indicators suggest further research is needed to understand the relationship between Covid-19 impact and individual opinion, such as on vaccination.

Since the experiment is fully randomized, the addition of new variables in Models 2-5 does not change the substantive results from Figure 1 and Model 1. We think some of the loss of treatment significance in Models 4-6 has more to do with the importance of the variables we add J o u r n a l P r e -p r o o f to the estimations than the lack of statistical power in our estimations. As Table A4 in the Appendix shows, when we include media exposure and personal health considerations in separate estimations, these covariates are highly significant and lead the treatments to lose significance. We also explore whether the lack of significance in some variables could be due to our sample sizes: simulations on G*Power (Erdfelder et al. 1996) suggest a sample size of 1,642 and α=0.05 level (two-tailed) can detect odds ratios of 1.2 at a conventional threshold for sufficient power (0.83). While this suggests an adequate sample size for most of the variables in our models-including the initial estimates for the FDA and economy treatments-it is still possible that relatively small but statistically significant effects have failed to materialize in these models.

In cases of heavy multiple hypothesis testing (i.e., many explanatory variables and/or outcomes), it may be important to employ a False Discovery Rate (FDR) approach. FDR techniques probabilistically assume Type I errors (incorrectly rejecting the null hypothesis) grow with the number of variables in a model. But, this approach also increases the chance of encountering a Type II error (failing to reject the null when the null is false). Given the stakes of the public health crisis and to provide a conservative estimate of associations, we employ FDR to account for potential Type 1 errors.

Specifically, we employ the Benjamini, Krieger, and Yekutieli (2006) FDR approach on Model 6 in Table 3 to ensure none of the significant results are false positives. This approach takes the expected proportion of errors at a given alpha and iteratively deduces potential errors by accounting for the number of variables that are confirmed-the remaining less significant variables are more likely to be false positives. The technique produces q-values, which are essentially error-adjusted p-values. The q-values for the 29-variable Model 6 (shown in Table A5 in the Appendix) maintain the same results as before with three differences: the statistical significance of age and "other" racial identity drop from p<0.01 to q<0.05, while one's selfassessment of health status falls just outside of conventional levels of significance (from p=0.076 to q=0.129).

For more fine-grained analysis, in Tables A6 to A12 in the Appendix, we also investigate heterogenous treatment effects. While our study did not intend to analyze specific effects on subgroups, it nonetheless makes sense-within the limitations of our data-to assess whether the treatments affected the hesitant groups differentially. Based on aforementioned existing research and our results, we identify three key hesitant groups: Republicans (differentiated as both

African-Americans, and women. For some of these groups (such as Black respondents), our sample sizes are too small to reach conclusions, as small sample sizes could mean false negatives (in turn, the low-n constraint means the significant findings can generally be trusted). With this drawback in mind, we find the FDA approval process still stands out as the most robust treatment to positively boost willingness, including for African-Americans (Control: n=23, The takeaway from these models is one of caution: while interventions like the ones we present can be effective in boosting willingness, individual exposure to information (our media variable) and health considerations-be they individual or about local circumstances-are shown here to have a greater impact on vaccine willingness.

Finally, we probe our consistent finding that demonstrations and messaging by specific individuals do not affect the intention to get vaccinated. We hypothesize that individuals are more likely to increase their intention to be vaccinated if they are persuaded the messenger genuinely supports vaccination. To explore this, we utilize the manipulation check question we asked immediately following the treatment to assess whether respondents understood the intended message. This form of "internal analysis" allows us to assess whether respondents comprehended the treatment, and subsequently analyze the sub-sample of respondents who did (Wilson, Aronson, and Carlsmith 2010, 66 Due to the nature of the vignette, the economy treatment check asks whether the pandemic positively or negatively impacted the economy, with the correct answer being negative. This particular type of assessment is known as a "factual manipulation check", since there is a correct

and an incorrect answer (Chandler, Mueller, and Paolacci 2014, 121) . Heretofore, we have examined the full sample. While it is imperative for researchers to always focus on the full sample, examining those that passed the manipulation check can provide extensions to the core analysis (Montgomery, Nyhan, and Torres 2018; Aronow, Baron, and Pinson 2019, 572-573) .

Hence, we now turn our focus to only those that passed the manipulation check.

We begin by hypothesizing about why people might fail the manipulation check. Two primary reasons come to mind: one, the respondent did not fully comprehend or pay attention to the treatment; two, the respondent did not believe the message being communicated, either out of personally-held distrust or a perception that the messenger was not genuine. The latter J o u r n a l P r e -p r o o f explanation suggests that for framing effects to work, the messengers need to be perceived as credible endorsers to overcome an individual's existing views (Lodge and Taber 2013) . We examine both options, but it is important to note at the onset that in the absence of another experiment to more directly observe "perceived credibility" of the message, our discussions are suggestive at best.

Our survey includes a measure for the duration (in seconds) each respondent took to answer the entire survey, which we use to examine the correlation between the respondent's speed of going through the survey-which could proxy lack of attention in general-and passing the manipulation check. We find a very weak correlation between duration and passing each manipulation check by experimental group (Pearson's r range across groups=-0.125-0.070) These numbers suggest, as best as we can tell, it is not attention per se that causes failure in the manipulation check.

If it is lack of comprehension (i.e., option 1), our most difficult treatments could reasonably have the highest attrition rate (i.e., failing the manipulation check). Since the demonstrations literally show Nurse Lindsay and Dr. Fauci receiving vaccinations, we would expect relatively low and comparable levels of attrition in these two treatments. However, this turns out not to be the case. As Figure 2 visually depicts, 61 subjects failed the manipulation check (52 "cannot tell" and 9 "disapprove") in the Nurse Lindsay treatment (T 1 ), while that number is only 34 (28 "cannot tell" and 5 "disapprove") in the Fauci demonstration (T 2 ).

Given the demonstrations are easy to understand and the lack of meaningful correlation between the demonstrations and the speed of answering the survey, it is conceivable that the manipulation check proxies how credible/convincing the respondents find the specific J o u r n a l P r e -p r o o f messenger/message. Indeed, out of our groups, T 4 (Trump endorsement) had the largest attrition with 80 (38.8%) of the 206 group members getting the manipulation check wrong (24 "disapprove" and 56 "cannot tell"). This was unsurprising to us because of Trump's oscillation on public health issues and the notable role of anti-vaccination groups within his political constituency (Oliver and Wood 2018) . Further, T 4 consists of 33 Democrats, 13 Republicans, and 34 independents or third-party members, or alternatively, 36 Biden voters, 11 Trump voters, 7 voters in support of other candidates, and 26 non-voters, suggesting a partisan skew in failing this condition. In contrast, our relatively more interpretive treatments-the FDA and economy ones-had lower attrition rates: 37 and 14 manipulation check failures, respectively.

All this said, we recognize that there could be a third option: individuals, who are already inclined to get the vaccine could be more likely to pass the manipulation check. Similarly, it could be that the respondent interprets the message as more credible when they have an a priori inclination for vaccination. And finally, it is possible people understand the direction of the treatment, but ex post facto fail the manipulation check to be consistent with their stance on the issue, thereby minimizing cognitive dissonance. Despite randomization, we are not able to arbitrate these possibilities in the absence of the aforementioned research design centered on the mechanisms of respondent information processing. Nonetheless, it is important to recognize the differences in the sample of individuals who passed the manipulation check to gain a fuller sense of possible treatment effects. which found Dr. Fauci messaging increased willingness (Bokemper et. al. 2021) . In short, message recipients generally respond more favorably when they process the intention behind the message and/or find the message credible.

To better understand the magnitude of the treatment effects on this group, in addition to other explanatory variables, we again run ordered logit models on the manipulation check subsample (full results available in Table A13 While we are unable to definitively pinpoint a specific mechanism, it is clear that respondents that correctly interpreted the directionality of the treatment were, on balance, more willing toward vaccination. We interpret these results to mean that when the messenger is perceived to be sincere, framing effects can be even more powerful. Hence, with some qualification, we consider this a promising finding since it means a wider array of communication approaches, including endorsements, could reduce vaccine hesitancy.

This article contributes to a burgeoning literature on the determinants of vaccine acceptance and how to boost it through messaging and behavioral nudges (Bokemper et al. 2021; Dai et al. Forthcoming, Kerr et al. 2021; Kreps and Kiner 2021; Schwarzinger et al. 2021) . Although our study naturally intersects with existing analyses in certain dimensions, such as how perceptions of vaccine safety affect the willingness to get vaccinated, it offers a unique survey experiment with original results. In addition to our dependent variable capturing the timeline for intention to be vaccinated, our treatments focus on not just public figure endorsements, but also information about the vaccine approval process and the pandemic's effects on the economy. Having multiple studies (conducted at different points in time and in different countries) applying different techniques to decrease vaccine hesitancy is critical for public health efforts, as extant work-including this study-suggests strategies to boost individual willingness to be vaccinated are crucial first steps in vaccine uptake, and thus herd J o u r n a l P r e -p r o o f immunity. We contribute to efforts to understand which strategies are more (or less) likely to be effective, and on whom.

This study shows, first, that an explanation about the FDA's vaccine approval process and an emphasis on the economic toll of the pandemic have the most robust effect on the respondent's willingness to take the vaccine. In particular, the FDA treatment increases the likelihood of greater willingness to be vaccinated by between 42% (full sample) and 100%

(manipulation check sub-sample) compared to the control group. Importantly, the FDA treatment appears to work across different hesitant groups. Moreover, the economy treatment has the most persistent effects across the differently specified ordered logit models.

Second, we find that other frames, such as politician endorsements or demonstration of vaccination, could also work, but they remain contingent upon the individual's processing of the stimulus and/or perception of the message's sincerity. Third, this study uniquely documents how fear of catching Covid-19, the level of media consumption, and Covid-19-related mortality in the individual's general area also matter. As such, public campaigns could be crafted to localities and inform the public about the issues of their specific area.

There are, of course, limitations to this study and those of its kind. As noted at the onset, while we measure willingness to be vaccinated as the initial step in public health campaigns for vaccination, the actual vaccination uptake is what ultimately matters for herd immunity. Indeed, existing literature suggests between one-third and half of individuals that state an intention to receive the seasonal influenza vaccine actually followed through with getting vaccinated (Bronchetti et al. 2015, 277-278) . This said, the present moment may be associated with greater follow-through, given the pervasiveness of Covid-19, the associated mortalities, and the deep socio-economic effects of the pandemic. Future studies would benefit from devising a wholly J o u r n a l P r e -p r o o f contained research design relying on a panel data approach with multiple observations across time per respondent, including verified (as opposed to self-reported) uptake of vaccination. Such a methodological approach allows for monitoring how well intention to vaccinate translates into demonstrated action.

Crucially, motivations-including intentions-are a key precondition for action (Brewer et al. 2017, 158) , especially in a U.S. public health regime that generally relies on voluntary compliance and minimal government enforcement. Therefore, measuring one's disposition toward vaccination and identifying persuasive appeals through framing experiments are necessary scholastic endeavors (ibid, 161; Stevens 2017, 1071) that may contribute to successful governmental public health initiatives (Benartzi et al. 2017) .

To be sure, while persuasive messages may alter intention to vaccinate, it remains unclear how long this intention lasts immediately following the survey. Existing meta-analyses of persistent (or decaying) post-treatment effects through experimental surveys are mixed, but nonetheless find treatments that educate the respondent with new, actionable information are comparatively better at leading to attitudinal or behavioral change than treatments that make preexisting knowledge more accessible or salient (Coppock 2017) . We believe the FDA and economy treatments belong in this knowledge boosting category of treatment.

This study's focus on individual-level intention to vaccinate through an experimental survey yields several successful frames which increase articulated willingness. Informationbased communications, of the type tested in this study, appear to present potentially powerful tools to overcome hesitancy in the race to achieve herd immunity.

J o u r n a l P r e -p r o o f "Within the month" (5) "Between 1 and 3 months" (4) "Over 3 months to 6 months" (3) "Over 6 months to 1 year" (2) "Over 1 year" (1)

"No, I am not willing" (0) J o u r n a l P r e -p r o o f 

A note on dropping experimental subjects who fail a manipulation check

Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine

Should governments invest more in nudging?

Adaptive linear step-up procedures that control the false discovery rate

Timing of COVID-19 vaccine approval and endorsement by public figures

Increasing vaccination: putting psychological science into action

Attention, intentions, and followthrough in preventive health behavior: field experimental evidence on flu vaccination

Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers

Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities

Applying lessons from behavioral economics to increase flu vaccination rates

Mass media coverage and influenza vaccine uptake

Framing theory

Political trust in a cynical age

The persistence of survey experimental treatment effects

Forthcoming). Behavioral nudges increase COVID-19 vaccinations: two randomized controlled trials

Evaluating framing effects

GPOWER: A general power analysis program

Intent to get a COVID-19 vaccine rises to 60% as confidence in research and development process increases

The logic of the survey experiment reexamined

Beyond the first dose -Covid-19 vaccine follow-through and continued protective measures

Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data

News that matters: television and American opinion

KFF Health Tracking Poll/KFF COVID-19 Vaccine Monitor

Effect of information about COVID-19 vaccine effectiveness and side effects on behavioural intentions: two online experiments

Public attitudes toward COVID-19 vaccination: the role of vaccine attributes, incentives, and misinformation

Factors influencing Covid-19 vaccine acceptance across subgroups in the United States: evidence from a conjoint experiment

Herd immunityestimating the level required to halt the COVID-19 epidemics in affected countries

A global survey of potential acceptance of a COVID-19 vaccine

The rationalizing voter

Vaccine hesitancy: definition, scope and determinants

Should Blacks represent Blacks and women represent women? A contingent 'yes

Effects of question order on survey responses

How conditioning on posttreatment variables can ruin your experiment and what to do about it

Effective messages in vaccine promotion: a randomized trial

Enchanted America: how intuition and reason divide our politics

Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine

Predictors of intention to vaccinate against COVID-19: Results of a nationwide survey

U.S. readiness to get COVID-19 vaccine steadies at 65%. Gallup

COVID-19 vaccine hesitancy in a representative working-age population in France: a survey experiment based on vaccine characteristics

Cambridge handbook of experimental political science

Impact of the COVID-19 vaccine on asymptomatic infection among patients undergoing preprocedural COVID-19 molecular screening

The social identity of intergroup behavior

Black faces in the mirror: African Americans and their representatives in the

The framing of decisions and the psychology of choice

Medical apartheid: the dark history of medical experimentation on Black Americans from colonial times to the present

The art of laboratory experimentation

An investigation of low COVID-19 vaccination intentions among Black Americans: the role of behavioral beliefs and trust in COVID-19 information sources

Beyond politics -promoting Covid-19 vaccination in the United States

The nature and origins of mass opinion