key: cord-1031484-3wu6r32l
authors: Lu, Fangcao; Sun, Yanqing
title: COVID-19 vaccine hesitancy: The effects of combining direct and indirect online opinion cues on psychological reactance to health campaigns()
date: 2021-10-22
journal: Comput Human Behav
DOI: 10.1016/j.chb.2021.107057
sha: e14f6562d2ddc27d8b016bcf0e21025ea30a219d
doc_id: 1031484
cord_uid: 3wu6r32l

This study aims to examine whether and how user-generated comments and reaction emojis on COVID-19 vaccine-promoting Facebook posts induce psychological reactance to posts and vaccine hesitancy in audiences of the posts. An online experiment including 465 American adults showed that, compared with COVID-19 vaccine promotion posts accompanied by pro-vaccine comments, those accompanied by anti-vaccine comments provoked greater reactance in audiences through the mediating effects of bandwagon perception and the presumed influence of the posts on others. Greater reactance, in turn, increased audiences' COVID-19 vaccine hesitancy. Additionally, reaction emojis altered the comments' effects such that pro-vaccine comments triggered less reactance than anti-vaccine comments when the pro-vaccine comments were accompanied by agreement emojis (i.e., “like” and “love”); whereas there was no significant difference between pro-vaccine comments and anti-vaccine comments in reactance when the pro-vaccine comments were accompanied by rejection emojis (i.e., “angry” and “sad”). Furthermore, audiences’ pre-existing attitudes did not affect the effects of opinion cues on their` reactance and vaccine hesitancy.

Vaccine hesitancy, which refers to indecision about vaccine acceptance, has been identified as a major threat to global health (World Health Organization [WHO] , 2019). To combat this problem, various educational interventions have been launched to disseminate scientific information advocating vaccination. However, such attempts are often ineffective and may even have opposite effects to what is desired (Nyhan et al., 2014; Nyhan & Reifler, 2015) . For example, COVID-19 vaccine education efforts are one of the largest public education campaigns in history (The Ad Council, 2020) . However, the mass campaigns were accompanied by decreased vaccination intentions. Although 74.1% of Americans said they would likely receive the COVID-19 vaccine in April 2020, only 56.2% still planned to do so by December 2020 (The PostStar, 2021) . Psychological reactance theory (PRT; Brehm, 1966; Brehm & Brehm, 1981 ) explains this resistance. The theory suggests that when individuals feel that persuasive messages interfere with their freedom of choice, they seek to regain their freedom by opposing the messages. This inclination toward opposition has been termed reactance (Brehm, 1966) . Health campaigns encourage adopting healthy behaviors and such persuasion attempts are likely to provoke reactance in audiences-people who read the messages-thereby influencing health outcomes (for a review, see Reynolds-Tylus, 2019) . In particular, audiences' reactance to vaccination campaigns were found to negatively influence their attitudes and decision regarding vaccination (Finkelstein et al., 2020; Richards et al., 2021) . This suggests that vaccination campaigns may intensify audiences' hesitancy by provoking their reactance. Thus, individuals' reactance to vaccination education and its relationship with vaccine hesitancy should be closely examined.

Further, negative public discussion about vaccination may also aggravate vaccine hesitancy "by drawing attention to it" and "legitimize it through familiarity" (Goldstein et al., 2015) . Particularly, on social media, negative public discussions can be represented as anti-J o u r n a l P r e -p r o o f vaccine comments that oppose the vaccination promotion messages. Indeed, anti-vaccine comments are rampant on social media. For example, on Facebook 85% of comments following the introduction of COVID-19 vaccines have been in opposition, describing serious side effects of the vaccines and disseminating conspiracy theories (Wawrzuta et al., 2018) .

According to the Modality-Agency-Interactivity-Navigability (MAIN) model (Sundar, 2008) , opposing comments can trigger a bandwagon heuristic: If others disagree with a message, then I should disapprove of it as well (Waddell, 2017) . As a result, audiences are likely to evaluate the messages negatively, believing that they pose a threat to their freedom and thus generating reactance to the messages (Li & Sundar, 2021) . Moreover, according to the influence of presumed influence (IPI) model (Gunther & Storey, 2003) , presumed influence refers to individuals' perception that media have substantial effects on others, and individuals often adjust their attitudes or behaviors based on their presumed media influence on others. Opposing comments that accompany a message reduce audiences' perceived message influence on others (Lee & Jang, 2010) , thus causing them to follow others' rejection of the campaign. In other words, through the mediating of bandwagon perception and presumed influence on others, anti-vaccine comments are likely to trigger reactance to vaccination education, further exacerbating vaccine hesitancy. Recognizing the process through which the interaction of public education and online discussion affects audiences' reactance to health campaigns helps advance our understanding of how new media affordances and active user engagement influence health communication.

In addition to comments, which are direct opinion cues, social media also presents audiences' opinions in an indirect manner. Audiences do not need to type words, as their actual viewing behavior or the easy clicking of the like or share buttons can imply their attitudes (Lünich et al., 2012; Walther & Jang, 2012) . Social media facilitates a combination of direct and indirect opinion cues, as users can like or vote for comments that other users J o u r n a l P r e -p r o o f have posted. These indirect opinion cues have the potential to change the influence of comments (Lünich et al., 2012) . However, previous studies have failed to identify significant effects of combining direct and indirect opinion cues on audiences' information processing (e.g., Lee et al., 2020; Peter et al., 2014) . This may be due to the fact that the manipulation of indirect opinion cues does not fully represent unfavorable attitudes (Leong & Ho, 2021) .

Facebook's introduction of reaction emojis seems to address this omission. Beyond the simple "like" feature, since 2016 Facebook has incorporated new graphic symbols, which enable users to express both positive (i.e., "like," "love," "wow," and "haha") and negative (i.e., "angry" and "sad") attitudes toward a message (Tian et al. 2017) . As Facebook has solidified its status as the largest social network worldwide, with over 2.7 billion monthly active users (Tankovska, 2021) , exploring how the direct and indirect opinion cues on Facebook manifest public opinion and influence people's psychological reactance is especially relevant.

The present study conducted a 2 (comments to posts: support vs. oppose vaccines) × 2 (reaction emojis to comments: agreement vs. rejection) between-subjects experiment to examine how combining direct and indirect online opinion cues that accompany Facebook posts promoting the COVID-19 vaccine affects audiences' psychological reactance to the posts and how the reactance drives their vaccine hesitancy. The mediating effects of bandwagon perception and presumed influence on others were assessed. We also explored whether and how audiences' pre-existing attitudes impacted the effects of the combined direct and indirect opinion cues. This study reveals the social-psychological process through which the combination of online opinion cues following persuasive health messages affects the psychological reactance typically elicited by the messages. Additionally, it demonstrates how the advancement of digital technology has integrated media and social influence to change established media effects. Practically, this study provides insights that can be used to J o u r n a l P r e -p r o o f enhance the persuasiveness of health campaigns and address vaccine hesitancy by identifying factors that incline individuals to oppose vaccination education.

According to PRT, when people perceive that they are free to engage in certain behaviors or to think and feel in certain ways, they possess some degree of freedom (Brehm, 1966; Brehm & Brehm, 1981) . This is important, as people have a basic need for freedom, such as making decisions regarding their own behavior, thoughts and feelings without being controlled by anyone else (Wicklund, 1974) . Anything that restricts these freedoms or makes them more difficult to exercise constitutes a threat (Brehm, 1966) . When people are faced with a threat to their freedoms, they perceive that their basic need for freedom is not being met, and they feel inclined to act against the threat. This inclination is called psychological reactance (Brehm & Brehm, 1981) .

The original reactance study contended that the concept of reactance cannot be directly measured (Brehm, 1966) . Nevertheless, Dillard and Shen (2005) proposed four viable conceptualizations of psychological reactance: (1) a single cognitive process consisting of negative cognitions (e.g., disagreement with the message; negative evaluation of the message, its advocacy, or its source), (2) a single affective process consisting of anger (e.g., irritation, annoyance, and rage), (3) a dual cognitive-affective process in which negative cognitions and anger are separate, and (4) an intertwined cognitive-affective process in which negative cognitions and anger are combined. These four conceptualizations have been tested in various types of health campaigns, such as flossing (Dillard & Shen, 2005) , binge drinking (Kim et al., 2013) , physical activity (Quick & Considine, 2008) , safe sex behavior (Quick & Stephenson, 2007) , antismoking (LaVoi et al., 2017; Shen, 2011) , and skin cancer J o u r n a l P r e -p r o o f prevention (Shen, 2015) . The results consistently showed that the intertwined model was the best operationalization of reactance (for a review, see Rains, 2013; Reynolds-Tylus, 2019) .

To improve the design of health campaign messages, previous studies have identified various antecedents of reactance. Various message features are thought to influence reactance. The controlling language, lexical concreteness (Miller et al., 2007) , arguments, and personal insults (Kim et al., 2017) in messages can provoke reactance. Conversely, narratives, other-referencing (Gardner & Leshner, 2016) , gain-framing (Cho & Sands, 2011) , and empathy-arousing strategies (Shen, 2015) are likely to attenuate reactance. The attributions of the message source also affect people's perceived reactance. The more that audiences perceive similarity and trustworthiness of the message source, the less likely they are to generate reactance to the message (Silvia, 2005; Song et al., 2018) .

As health campaigns go online, new media affordances provide cues that shape the way people digest campaign messages, thus influencing people's reactance to the campaigns.

According to the MAIN model (Sundar, 2008) , internet users are less likely to encode and deliberate all the information, as there is a vast amount of information in online spaces.

Instead, they tend to scan through information and make quick judgments on the information based on its accompanying interface cues, such as user comments, number of likes, and number of shares (Sundar, 2008) . The presence of interface cues following a media message is assumed to activate different sets of heuristics that bias people's message evaluation (Sundar et al., 2015; Waddell & Sundar, 2020) . The cues related to the opinions of others are believed to alter audiences' bandwagon perception, the perception that others in general support or favor a message (Xu et al., 2013) . This perception, in turn, triggers bandwagon heuristics-the evaluation of a message such that, if others like the content, then it must be good; and if others dislike the content, then it must be bad (Waddell, 2017) .

One opinion cue that has attracted attention in the context of health campaigns is user comments. Opposing comments make audiences see others' disagreement with the message and decrease their bandwagon perception about others' approval of the message. This in turn leads audiences to negatively evaluate the message (Winter et al., 2015) . These negative evaluations lead audiences to believe the health persuasion is an unreasonable intervention that should not interfere with their personal choice. Such perceptions then enhance their perceived threat of the message to their freedom and provoke their reactance to it (Li & Sundar, 2021) . In studies examining the effects of comments, opposing comments have been found to undermine audiences' perceived persuasiveness of health campaigns (Walther et al., 2010) and enhance their anger toward the campaign (Shi et al., 2014) compared with supporting comments. It is expected that, compared with pro-vaccine comments, anti-vaccine comments following COVID-19 vaccine promotion posts reduce bandwagon perception about others' approval of the message and trigger audiences' anger and negative cognitions about the posts, which represents psychological reactance. Accordingly, the following hypotheses are presented: H1: When exposed to anti-vaccine comments accompanying a COVID-19 vaccine promotion post, individuals generate less bandwagon perception about others' approval of the message than when exposed to pro-vaccine comments.

H2: When individuals have less bandwagon perception about others' approval of the message, they generate greater psychological reactance.

In addition to triggering the bandwagon effect, comments also change individuals' perceived post influence on others, thereby affecting the individuals' reactions to the post.

This process can be conceptualized as IPI. IPI originated from Davison's (1983) third-person J o u r n a l P r e -p r o o f perception, which proposes that people tend to believe "the media's greatest impacts are not on me or you but on others-the third persons" (p. 3). Moreover, people take attitudinal or behavioral actions in response to this self-other discrepancy of media effects. This communication phenomenon has received robust support in various media contexts (Perloff, 1999; Sun et al., 2008) . It was later expanded into the IPI model (Gunther & Storey, 2003) , which suggests that individuals often think others are susceptible to media influence and that this perception drives the individuals' attitudinal or behavioral change. Unlike the thirdperson effects that focus on the self-other discrepancy of undesirable media effects, the IPI model includes both desirable and undesirable media influences and focuses on the presumed media influence on others (Gunther & Storey, 2003) .

Comments accompanying a message are likely to modify individuals' perception of the message's influence on others, as comments can serve as exemplars of others' general opinions toward the message. According to exemplification theory (Zillmann, 1999 (Zillmann, , 2006 , exemplars refer to the opinions or experiences of the person involved in an issue. People tend to form their judgments and beliefs about an issue based on concrete and available exemplars because they are vivid and thus easy to process and remember. People may take comments on an online media message as exemplars of the entire audiences' opinion about the message, as comments are available and vividly present the audiences' opinions (Lee & Jang, 2010) .

When individuals encounter anti-vaccine comments on a COVID-19 vaccine promotion post, they may assume that the general audience does not agree with the post and still holds resistance to the COVID-19 vaccine. They may suppose that the promoting post has a limited influence on others' acceptance of the COVID-19 vaccine. In contrast, when people encounter pro-vaccine comments, they assume that others are influenced by the COVID-19 vaccine promotion post and thus accept the COVID-19 vaccine. It is expected that, compared with pro-vaccine comments, anti-vaccine comments following a COVID-19 vaccine J o u r n a l P r e -p r o o f promotion post make people perceive that the post has less influence on others' acceptance of the COVID-19 vaccine. This leads to the following hypothesis: H3: When exposed to anti-vaccine comments accompanying a COVID-19 vaccine promotion post, individuals perceive less influence of the post on others' acceptance of the COVID-19 vaccine than when exposed to pro-vaccine comments.

Perceived media influence affects audiences' reactance because audiences may change their attitudes or behaviors in order to accommodate their perceptions of others' reactions to the message. People often gauge group or social norms based on their perceived media influence, and they tend to accommodate the presumed influence on others to fit the perceived social environment (Chia, 2006) . For example, when adolescent drinkers assume that anti-drinking campaign messages make their peers less likely to drink, such a perception decreases the drinkers' own positive attitudes and intention related to drinking (Ho et al., 2014) . In other words, when people perceive less influence of a COVID-19 vaccine promotion post on others' acceptance of the COVID-19 vaccine, they perceive that they themselves, as others, should be less influenced and persuaded by the post. This perception that the COVID-19 vaccine promotion post should not interfere with their own personal choice makes audiences perceive a greater threat to freedom from the post and have greater reactance to it. Accordingly, the following hypothesis is presented: H4: When individuals perceive less influence of a COVID-19 vaccine promotion post on others' acceptance of the COVID-19 vaccine, they have greater psychological reactance.

Vaccine hesitancy refers to "the delay in acceptance or refusal of vaccination despite availability of vaccination services" (MacDonald & The SAGE Working Group on Vaccine Hesitancy, 2015). It is not only a refusal of vaccination (i.e., the intention or behavior to J o u r n a l P r e -p r o o f reject vaccines) but also unfavorable and hesitant attitudes toward vaccination (Larson et al., 2015) . Such hesitancy toward vaccination may result from people's reactance to vaccination promotional messages. According to PRT, people's reactance to a threat to freedom motivate them to take actions to restore their freedoms (Brehm & Brehm, 1981) . People may attempt to restore their freedoms directly or indirectly. Direct restoration includes rejecting the health recommendation (e.g., refusing to exercise after exposure to a campaign advocating regular exercise; Miller et al., 2007) , whereas indirect restoration includes increasing favorability toward a threatened behavior (e.g., increased liking of smoking after watching antismoking public service announcements; Shen, 2011) . Similarly, greater reactance to a COVID-19 vaccine promotion message may correspond to greater attitudinal and intentional refusal of the COVID-19 vaccine, in other words, COVID-19 vaccine hesitancy. This leads to the following hypothesis:

H5: Individuals' level of psychological reactance to a COVID-19 vaccine promotion post is positively related to their level of COVID-19 vaccine hesitancy.

Aside from direct commentary, social media also provide indirect opinion cues to present audiences' attitudes toward a comment (Walther & Jang, 2012) . One example is the number accompanying a comment that indicates how many people like the comment (Peter et al., 2014) . Another example is the computational determination of the relatively prominent position of displayed content based on audience votes (Lee et al., 2020) . These indirect opinion cues only indicate audiences' positive attitudes toward certain comments and do not represent unfavorable attitudes. In contrast, Facebook's reaction emojis allow users to convey their agreement with a comment by giving positive feedback (e.g., "like" or "love") or their rejection of a comment by giving negative feedback (e.g., "sad" and "angry"). The slanted J o u r n a l P r e -p r o o f emojis, therefore, offer a more comprehensive way to represent audience sentiment than other indirect opinion cues (Leong & Ho, 2021) .

Emojis can reflect whether a comment is representative of public opinions about the original message. People often infer public opinions about a media message based on the comments following the message (Lee & Jang, 2012) , as people usually judge the whole based on some of its parts (Zillmann et al., 1996) . One of the cognitive mechanisms underlying this inference is the representativeness heuristic, which suggests that people usually judge a phenomenon based on its salient features or essential characteristics (Kahneman et al., 1982) . In other words, only when individuals believe that the available cases are typical in a phenomenon do they infer the phenomenon based on the cases.

Likewise, when the reaction emojis following a comment agree with the comment (e.g., "like" or "love"), people may believe that the comment expresses opinions shared by others.

Conversely, emojis showing rejection toward the comment (e.g., "sad" or "angry") may lead people to think that the comment does not represent the typical opinions of others. We intend to compare whether and how outcome variables differ between the combined direct and indirect opinion cues conditions. As there are no previous studies in this area, we propose the following research question: RQ1: Do outcome variables, that is, bandwagon perception, presumed influence, psychological reactance and vaccine hesitancy differ between the combined direct and indirect opinion cues conditions? If so, how?

According to theories of motivated reasoning (Kunda, 1990) , people usually accept information consistent with their pre-existing beliefs and draw conclusions they want to believe. This biased processing results from people's tendency to avoid incongruent information and their self-serving attribution to believe they are correct (for a review, see J o u r n a l P r e -p r o o f Kunda, 1990) . Comments included in health campaign messages present a slant toward the messages, and audiences are more likely to accept comments with a slant that is consistent with their pre-existing attitudes than those with an inconsistent slant. Hence, comments with a consistent slant toward the audiences' pre-existing attitudes have greater influence on the audiences' processing of the accompanying campaign. For example, Shi et al. (2014) reported that anti-smoking comments included in an anti-smoking campaign exerted a stronger influence on audiences' acceptance of anti-smoking information and their intention to stop smoking than pro-smoking comments when audiences were ready to quit smoking.

However, the combined direct and indirect opinion cues following a health campaign message present two slants, which may be congruent or incongruent with each other, toward the message at the same time. First, comments present a slant toward a media message by directly conveying supporting or opposing opinions toward the message. Second, reaction emojis combined with a comment also reveal a slant toward a media message. The reaction emojis can convey support for a message when they show agreement to comments supporting the message or when rejection emojis are used for opposing comments accompanying the message. They can also be against the message, with rejection emojis used in response to supporting comments or agreement emojis used in response to opposing comments. Whether and how audiences' pre-existing attitudes affect the influence of the two simultaneously presented slanted opinion cues remain unexplored. Thus, we intend to examine how audiences' pre-existing attitudes toward COVID-19 vaccines affect the influence of combining direct and indirect cues following a COVID-19 vaccination promotion post. This leads to the following research question: 

We conducted an online experiment by creating a human intelligence task on Amazon's Mechanical Turk (MTurk) in January 2021. MTturk is a feasible platform for data collection because it recruits representative and diverse samples that pay more attention to survey than other convenience samples (Berinsky et al., 2012; Thomas & Clifford, 2017) .

We employed a multi-message design to avoid case-category confounding effect (Jackson, 1992) . We first made four Facebook posts highlighting the benefits of COVID-19 vaccines based on information from major official websites and online news outlets (e.g., the Centers for Disease Control and Prevention, WHO, The New York Times, CNN, and ABC News). Following Dillard and Shen's (2005) study, we created messages based on the standard format of a fear appeal (Rogers, 1983) , which is widely applied in health campaigns, including the COVID-19 pandemic (Stolow et al., 2020) . In line with the standard format of a fear appeal, the four messages all included a threat-to-health component and a Additionally, a total of 40 comments (50% supporting and 50% opposing the COVID-19 vaccine) were collected from social media. For example, a pro-vaccine comment stated that "Got my COVID vaccine today. I totally trust this vaccine and encourage everyone to get it when it's their turn." In contrast, an anti-vaccine comment stated that "Vaccine contains J o u r n a l P r e -p r o o f aborted fetal cells, ethylene glycol, fats, M-RNA, and other poison that is immoral, dangerous and genetically disruptive or altering…pure poison...." Any incivility was removed, and typos were corrected. These comments were slightly modified to have approximately equivalent length (i.e., 30 words).

A pilot test (N=60) was conducted to decide which messages and comments should be selected for the main study. The participants were randomly assigned to two of the four messages and 20 of the 40 comments. They were asked to rate their perceived threat to freedom from each message on a five-point scale (1 = strongly disagree and 5 = strongly agree). Items were adapted from a previous study (Dillard & Shen, 2005) and included the following: The message (1) threatened my freedom to choose, (2) tried to make a decision for me, (3) tried to manipulate me, and (4) tried to pressure me. Moreover, the participants were also asked to evaluate their perceived effectiveness of the campaign message on a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). The measurement was adapted from Walther et al. (2010) and included three items: The message I read (1) was convincing,

(2) was important, and (3) Since most people only read five comments during online media consumption (Lee et al., 2020) , we randomly divided the 10 supporting comments into two supporting comments sets J o u r n a l P r e -p r o o f and the 10 opposing comments into two opposing comments sets, so each of the four comment sets contained five comments. We made the commenters identical across the four comment sets. For each comment set, we attached one agreement emojis set (i.e., "like" and "love") and one rejection emojis set (i.e., "sad" and "angry"). In this way, we generated eight opinion cues sets. Then, we randomly generated five numbers between 0 and 100 as the number of emojis, as the pilot study showed that most of the respondents were familiar with the number of reaction emojis accompanying comments on vaccination promotion posts. The numbers were identical across conditions (please see the appendix for the stimuli).

This study utilized a 2 (comments to posts: support vs. oppose vaccine) × 2 (reaction emojis to comments: agreement vs. rejection) between-subjects design. The experimental setting was set up using online survey software-QuestionPro-and 833 participants provided consent and started the survey. We first asked the respondents to report their attitudes toward COVID-19 vaccines. After attention checks, 686 participants began the experiment. Using the randomizer programming of QuestionPro, we randomly present one of the two COVID-19 vaccination promotion messages and one of the eight opinion cues sets to each participant. After exposure to stimuli, the respondents were asked to report their bandwagon perception, presumed influence, psychological reactance, and vaccine hesitancy.

Next, we asked two questions as experimental manipulations to check whether the experimental manipulation was implemented as intended and then asked for the participants' demographic information. Upon completion of the survey, the participants were debriefed and thanked. Each participant received $0.70 for compensation after completing the questionnaire. Figure 1 describes the flow of participants through each stage of the study.

[ Figure 1 here] 

The respondents were asked to report their pre-existing attitudes toward COVID-19 vaccines, bandwagon perception, presumed influence, psychological reactance, and vaccine hesitancy. Among these, bandwagon perception, presumed influence, psychological reactance, and vaccine hesitancy were latent variables. We ran a confirmatory factor analysis (CFA) and found it to fit the data well, indicating good construct validity for the latent variables (χ2 [df = 98] = 300.50, p = .00, CFI = .98, GFI = .93, NFI = .97, RMSEA = .06).

The standardized regression weights (SRW) of all indicators ranged from 0.76 to 0.98. The composite reliability (CR) ranged from 0.88 to 0.98, and the coefficients of average variation extraction (AVE) ranged from 0.66 to 0.91.

The respondents were asked to evaluate whether getting a COVID-19 vaccine was (1) 

This measurement was adapted from Xu (2013) 

This measurement was adopted from Tsfati et al. (2011) . Using a seven-point scale (1 = strongly disagree and 7 = strongly agree), the participants were asked to indicate whether they agreed that the post made other people supportive of receiving the COVID-19 vaccine (M = 5.06, SD = 1.88). They were also asked to evaluate whether the post positively or negatively affected others' attitudes toward the COVID-19 vaccine using a seven-point scale (1 = in a very negative manner and 7 = in a very positive manner) (M = 5.10, SD = 1.81). The two items were highly correlated (r = .91, p < .001) and formed a reliable measurement of presumed influence (both SRW > 0.95; CR = .95; AVE = .91).

Psychological reactance was operationalized as a composite of anger and negative cognition. The measurement for anger was proposed by Dillard and Shen (2005) . The respondents were asked to indicate the extent to which the post made them feel (1) irritated,

(2) angry, (3) annoyed, and (4) aggravated using a seven-point scale (1 = none of this feeling and 7 = a great deal of this feeling) (α = .97, M = 2.11, SD = 1.64). The measurement for negative cognitions was adopted from Gardner and Leshner (2016) . Further, the respondents were asked to indicate whether they agreed that the message was (1) (1 = never and 7 = almost always). These two items were considered the control variables in the subsequent analysis.

The manipulations of comments and reaction emojis were both successful. We first asked the respondents if they thought the comments were favorable to COVID-19 vaccines using a seven-point scale (1 = very unfavorable; 7 = very favorable). The respondents in the pro-vaccine comments conditions (M = 6.62, SD = .87) perceived the comments as more favorable to the COVID vaccine than those in the anti-vaccine comments (M = 1.63, SD = 1.39) condition (t[536.443] = 54.29, p < .001). The respondents in the pro-vaccine comments condition who perceived the comments as negative (n = 7) and those in the anti-vaccine comments condition who perceived the comments as positive (n = 26) were removed. As for reaction emojis, the participants were provided with two pictures to reflect on the reaction emojis they saw. One picture showed "like" and "love," while the other showed "sad" and "angry." A Chi-square analysis with the reaction emojis as independent variable and emoji J o u r n a l P r e -p r o o f recall as the dependent variable was significant (χ 2 [1] = 139.21, p < .001). The majority of the participants in the agreement emojis (73.3%) and rejection emojis (73.6%) conditions chose the picture consistent with the given condition. Participants who chose the wrong picture (n = 158) were removed.

The experimental randomization was successful. We conducted a series of one-way analysis of variance (ANOVA) to confirm that there were no significant differences in the demographic factors, such as gender (p = .73), education (p = .39), income (p = .49), race (white vs. non-white, p = .85), or age (p = .17) across conditions. A Chi-square analysis was conducted to confirm that there was also no significant difference in the participants' political identification across conditions (p = .80).

The stimulus sampling for the multi-message design was also successful. The two COVID-19 vaccine-promoting messages had no different effects on any of the variables of interest (for bandwagon perception, p = .71; for presumed influence, p = .74; for reactance, p = .24; for vaccine hesitancy, p = .64). Each of the two sets of opinion cues in the anti-vaccine comments with agreement emojis condition, in the anti-vaccine comments with rejection emojis condition, in the pro-vaccine comments with rejection emojis condition, and in the pro-vaccine comments with agreement emojis condition did not have different effects on each variable of interest (p-values ranged between .13 and .89, all > .05). Thus, the data of the two campaign messages with eight sets of opinion cues were collapsed for analysis to increase the statistical power.

To test the research hypotheses that constitute the proposed model (see Figure 2 ), we used AMOS 26.0 to conduct structural equation modeling (SEM). The model included six measurement variables (for a correlation matrix, see Table 1 ). We regressed all the variables Additionally, we calculated the achieved power to assess the fit of SEM based upon RMSEA, AGFI, and GFI (MacCallum et al., 1996, 1997) . The achieved power based on RMSEA, AGFI, and GFI was 0.99, 1.00, and 1.00, respectively, indicating an acceptable power. The results are shown in Figure 3 .

[ Figure 2 here]

[ Table 1 here]

[ Figure 3 here] First, in terms of H1, the respondents reported less bandwagon perception when exposed to anti-vaccine comments than pro-vaccine comments. Thus, H1 was supported.

Second, as stated in H2, when the respondents had less bandwagon perception, they had greater psychological reactance; thus, H2 was supported. Next, the respondents perceived less presumed influence when exposed to anti-vaccine comments than pro-vaccine comments.

Thus, H3 was supported. Then, as stated in H4, less presumed influence corresponded to greater reactance; thus, H4 was supported. Finally, as stated in H5, a higher level of reactance led to greater vaccine hesitancy; thus, H5 was supported. Table 2 provides an overview of the hypotheses and results of the hypotheses tests.

[ Table 2 here] The model accounted for 44% of the respondents' psychological reactance and 28% of their vaccine hesitancy. Sobel tests were performed to validate the mediating effects.

Bandwagon perception (Sobel test = 5.58***, se = .03) and presumed influence (Sobel test = 3.40***, se = .03) mediated the effects of the comments on psychological reactance.

Psychological reactance mediated the effects of bandwagon perception (Sobel test = J o u r n a l P r e -p r o o f −6.03***, se = .05) and presumed influence (Sobel test = −3.41***, se = .03) on vaccine hesitancy.

To concurrently explore RQ1 and RQ2, we conducted a multivariate analysis of variance (MANOVA) to determine the effects of two types of comments (anti-vaccine vs.

pro-vaccine), two types of reaction emojis (agreement vs. rejection), and two types of preexisting attitudes (opponents of vaccine vs. supporters of vaccine) on the five dependent variables, bandwagon perception, presumed influence, anger, negative cognition, and vaccine hesitancy. ANOVAs on the dependent variables were conducted as follow-up tests to the MANOVA (see Table 4 ). To control for Type I error, we tested each ANOVA and the following pairwise comparisons using an alpha level stricter than .05 (i.e., .001 level) (Allen & Bennett, 2008) .

[ Table 4 here]

Regarding RQ1, the results showed that neither the main effect of reaction emojis (p =.64) nor its interaction with comment slant (p =.27) had a significant effect on any dependent variables (see Table 4 ). To explore the possible effects that reaction emojis have on bandwagon perception, presumed influence, anger, negative cognitions, and vaccine hesitancy, we compared these dependent variables across the four opinion cues treatment conditions. A MANOVA was conducted with the opinion cues treatment condition as a fixed factor and bandwagon perception, presumed influence, anger, negative cognitions, and vaccine hesitancy as dependent variables. The results revealed a significant multivariate main effect for the opinion cues treatment conditions (Wilks' λ = .73, F[15, 1256 .46] = 10.23, p < .001, η 2 = .10). The follow-up ANOVA showed significant differences among treatment conditions for bandwagon perception, presumed influence, anger, and negative cognitions but not for vaccine hesitancy (see Table 5 ).

[ Table 5 here]

The follow-up pairwise comparisons are shown in Table 5 . The respondents in the two anti-vaccine comments conditions reported less bandwagon perception and less presumed influence than the respondents in the two pro-vaccine comments conditions. However, we found that although the respondents in the two anti-vaccines comments conditions felt greater anger than the respondents in the two pro-vaccine comment conditions, when pro-vaccine comments were accompanied by rejection emojis rather than agreement emojis, there was no significant difference in anger between anti-vaccine and pro-vaccine comments conditions. Moreover, although the respondents in the two anti-vaccines comments conditions elicited greater negative cognitions than the respondents in the two pro-vaccine comments conditions, when the pro-vaccine comments were accompanied by rejection emojis and the anti-vaccine comments were accompanied by agreement emojis, there was no significant difference in negative cognitions between pro-vaccine and anti-vaccine comments conditions. Finally, no significant difference was observed between the four opinion cues conditions in vaccine hesitancy. In summary, reaction emojis did not affect the effects of comment slant on bandwagon perception, presumed influence, or vaccine hesitancy.

However, the emojis did affect the effects of the comment slant on anger and negative cognitions, that is, reactance.

Concerning RQ2, the first MANOVA that we conducted with two types of comments, two types of reaction emojis and two types of pre-existing attitudes as fixed variables; and with bandwagon perception, presumed influence, anger, negative cognition, and vaccine hesitancy as dependent variables, showed that the multivariate effect of pre-existing attitudes was significant (Wilks' λ = .43, F[5, 451] = 118.50, p < .001, η 2 = .57). As shown in Table 4 13; p < .001) than the COVID-19 vaccine supporters. In contrast, the multivariate effect of the interaction of comments and pre-existing attitudes was nonsignificant (p = .13). The multivariate effect of the three-way interaction among comments, reaction emojis, and pre-existing attitudes was also nonsignificant (p = .55). The follow-up ANOVA showed that neither the interaction of comments and pre-existing attitudes nor the interaction among comments, emojis, and preexisting attitudes had a significant effect on any outcome variables (see Table 4 ).

The main goal of this study was to examine the effects of combining direct and indirect online opinion cues accompanying vaccination-promoting posts on audiences' psychological reactance to the posts and their vaccine hesitancy. We found that, compared with pro-vaccine comments, anti-vaccine comments following COVID-19 vaccine promotion posts intensified people's reactance to the posts and provoked their COVID-19 vaccine hesitancy. The reaction emojis following user comments influenced the effects of the comments on reactance such that pro-vaccine comments triggered less reactance than antivaccine comments only when the pro-vaccine comments were accompanied by agreement emojis.

The study found that, compared with supporting comments following health promotion posts, opposing comments in response to the posts provoked audiences' reactance.

In particular, when audiences encountered opposing comments accompanying health J o u r n a l P r e -p r o o f campaign messaging rather than supporting comments, they thought that others dislike the health campaign. Moreover, they also perceived the health messaging to have a limited influence on others' acceptance of health recommendations. In other words, the slant of comments affects people's perception of the prevalence and impacts of the health promotion among audiences. Under social influence, people would adopt their perceptions of others' attitudes and rejection of the promotion, which, in turn, affect the magnitude of their psychological reactance. In previous studies on health promotion, individuals have been regarded as dispersive media receivers who process media information alone. The message features and source attributes of health campaigns affect the threat to freedom that audiences perceive, thereby influencing their reactance. Meanwhile, in the context of new media, the health campaigns are juxtaposed with user comments to exert social influence on individuals' information processing. The interaction of the health campaign and the accompanying public discussion found in our study demonstrates how the advancement of media technology facilitates social influence in the form of comments to change media effects.

Additionally, comments are often intertwined with indirect opinion cues rather than being solely associated with a media message. This study provides a comprehensive operationalization of indirect opinion cues by utilizing slanted reaction emojis (agreement vs. rejection emojis). The findings reveal that the supporting comments generated less anger than opposing comments only when the supporting comments were accompanied by agreement emojis and not rejection emojis. This finding suggests that rejection emojis harm the influence of supporting comments. Nevertheless, the findings do not necessarily imply that emojis exert an influence on people's perception by providing feedback on comments. We found no significant difference between supporting comments and opposing comments concerning negative cognitions when the opposing comments were accompanied by agreement emojis and supporting comments were accompanied by rejection emojis. This J o u r n a l P r e -p r o o f finding suggests that people may regard emojis following comments as a reaction to the original message.

As Masullo and Kim (2020) stated, there may be two potential ways that emojis accompanying comments affect audiences. Audiences might regard emojis following comments as a reaction to the comments (e.g., when people are "angry" at a comment they are expressing disagreement with the content of the comment) or to the original message (e.g., when people are "angry" at a comment they are signaling that they agree with the commenter that no matter what issue the comment is about, the issue also makes them angry). Masullo and Kim (2020) found that "angry" emoji in uncivil comments could moderate the effects of uncivil comments such that uncivil comments increased individuals' dislike for the political out-group while the same comments lead to decreased dislike for the outgroup when accompanied by "angry", thus partially supporting the first explanation. The findings of our study provide supports for both explanations. This is because comments and the original messages both reveal a slant, and some participants view the emojis as expressing a reaction toward the comments slant while others regard them as reactions to the original messages slant. Future research could examine how specific reactions alter the way people perceive the comments and the original message. There also appears to be a need to find potential factors that moderate the effect of emojis on people's information processing.

Psychological reactance deserves more research attention, as it has been shown to provoke vaccine hesitancy. The dissemination of health information advocating vaccination is widely utilized to combat vaccine hesitancy in online and offline settings (for a review, see Jarrett et al., 2015) . However, our findings indicate that mass vaccination education may adversely affect and even intensify vaccine hesitancy if it provokes reactance in audiences. We also found that audiences' pre-existing attitudes toward the COVID-19 vaccine affected all outcome variables. However, there was no significant moderation effect of preexisting attitudes on the effects of opinion cues on any outcome variables. This suggests that, rather than moderating the effects of online opinion cues on audiences' information processing and vaccine hesitancy, audiences' pre-existing attitudes are likely to directly affect their processing of health information and vaccine hesitancy. This may result from the difference in audiences' perceived similarity with the commenters and message source. The lack of personalized information across online spaces leads to a distorted high-level attitudinal consensus among audiences (Reicher et al., 1995; Walther et al., 2010) . Comments are left by anonymous online peers. Audiences are likely to identify with commenters and accept these comments, regardless of whether or not the comments are consistent with their pre-existing beliefs. In contrast, the source of health promotion messages tends to be public health organizations or advertisers, with which audiences feel less similarity and identification. Audiences are likely to process the health promotion message based on their pre-existing attitudes rather than social influence. The findings reveal the strong detrimental effects of opposing comments: Audiences, regardless of their pre-existing attitudes, are likely to be affected by opposing comments equally. Some may worry that Message 2 in the experimental stimuli contains an inaccurate statement-that COVID-19 vaccination will help protect people by creating an antibody response "without having to experience sickness" (see Message 2 in Appendix)-and they may wonder whether participants perceived untrustworthiness in this statement, viewing the message as misleading. Actually, the phrase "without having to experience sickness" simply J o u r n a l P r e -p r o o f indicates that people can create an antibody response without having to experience COVID-19. Such wording is common in online health campaign messaging (e.g., British Society for Immunology, National Institute of Allergy and Infectious Diseases, Coconino government).

Besides, evidence shows that participants perceived Message 2 the same way as Message 1.

First, in the pilot study, we found no significant difference in the participants' perceptions regarding the persuasiveness of the two messages. Second, in our preliminary analyses, we found no differences in the effects of the two COVID-19 vaccine-promoting messages on any of the variables of interest. Third, in our main study, we measured the participants' counterarguments to both messages. There were no significant differences between the two messages in the intensity of the counterarguments they elicited (t[552] = -.06, n.s.).

Nevertheless, the present study has several limitations. First, the study only included COVID-19 vaccine promotion messages and did not examine the possibility that the effects of online opinion cues could vary across different topics and contexts. Therefore, subsequent studies are needed to test the model across various topics and contexts. Second, we manipulated user comments in terms of either supporting-only or opposing-only and reaction emojis in terms of either agreement-only or rejection-only. The manipulation of opinion cues could be improved by including a condition without any opinion cues or a condition in which comments and reaction emojis are combined rather than uniformly slanted. Third, we employed perceived threat to freedom to justify the comments' effects on reactance but did not empirically test the concept. Future research should thus measure the concept and include it in the SEM to support the justification. Next, the study was conducted in MTurk with a non-representative sample. The convenience sampling approach may compromise the external validity of the findings. However, the demographics of our sample resemble those of a national random sample in some regards (e.g., 52% of the participants were female, while 50.7% of the US population is female; 77.4% of the participant were white, while 78.5% of J o u r n a l P r e -p r o o f the US population is white [Statista, 2021] ). Still, future studies are encouraged to utilize a national representative sample to test the model examined here.

Finally, when making judgments under uncertainty, individuals often rely on heuristic principles to reduce the complexity of assessing probabilities, thereby simplifying their judgment. Such heuristic-based judgments may lead to severe and systematic errors (Kahneman et al., 1982; Tversky & Kahneman, 1974) . Within the framework of psychology research, this study used an online experiment to test hypotheses by exposing participants to stimuli and asking them to make judgments under uncertainty. The participants' judgments may be biased, so it is difficult to make accurate predictions about their vaccine hesitancy.

Interpretation of the research findings should take this drawback, inherited from the general difficulties of psychology research, into account.

Despite these limitations, this study demonstrates how combining direct and indirect opinion cues accompanying online health campaign messages affects audiences' psychological reactance to the campaign and unravels the underlying mechanisms. The findings indicate that to reduce audiences' reactance to vaccination education efforts and combat their vaccine hesitancy, health promoters should present supporting comments on health promotion posts. Moreover, it is important to encourage vaccine supporters to leave supporting comments and click agreement emojis in response to the supporting comments.

Opposing comments may mislead audiences' judgment about the social environment and make people believe that others in general reject vaccination education and remain hesitant about the COVID-19 vaccine. Online intervention programs could inform people about how misleading online opinion cues can be. Theoretically, the current study reveals the effects of combining media and social influence on individuals' information digestion. New media affordances, such as the comments section, enable audiences to view others' evaluation and acceptance of a persuasive campaign message, which in turn affects their reactance. 

H1. When exposed to anti-vaccine comments accompanying a COVID-19 vaccine promotion post, individuals generate less bandwagon perception about others' approval of the message than when exposed to pro-vaccine comments.

H2. When individuals have less bandwagon perception about others' approval of the message, they generate greater psychological reactance.

H3. When exposed to anti-vaccine comments accompanying a COVID-19 vaccine promotion post, individuals perceive less influence of the post on others' acceptance of the COVID-19 vaccine than when exposed to pro-vaccine comments.

H4. When individuals perceive less influence of a COVID-19 vaccine promotion post on others' acceptance of the COVID-19 vaccine, they have greater psychological reactance.

H5. Individuals' level of psychological reactance to a COVID-19 vaccine promotion post is positively related to their level of COVID-19 vaccine hesitancy. SEM Supported Note. Familiarity with user comments and reaction emojis were controlled for the analyses. An alpha level at .001 is used to control for Type I error. ***p <.001. Note. AC=anti-vaccine comments, PC=pro-vaccine comments, AE=agreement emojis, RE=rejection emojis. Table entries are mean values with standard deviations in parentheses. Familiarity with user comments and reaction emojis were controlled for the analyses. An alpha level at .001 is used to control for Type I error. Cell means with different superscripts differ at p <.001 level, ***p < .001.

SPSS for the health & behavioural sciences

Evaluating online labor markets for experimental research: Amazon. Com's Mechanical Turk

A theory of psychological reactance

Psychological reactance: A theory of freedom and control

How peers mediate media influence on adolescents' sexual attitudes and sexual behavior

Gain-and loss-frame sun safety messages and psychological reactance of adolescents

The third-person effect in communication

On the nature of reactance and its role in persuasive health communication

Psychological reactance impacts ratings of pediatrician vaccine-related communication quality, perceived vaccine safety, and vaccination priority among US parents

The role of narrative and other-referencing in attenuating psychological reactance to diabetes self-care messages

Health communication and vaccine hesitancy

The influence of presumed influence

Examining how presumed media influence affects social norms and adolescents' attitudes and drinking behavior intentions in rural Thailand

Message effects research: Principles of design and analysis

Strategies for addressing vaccine hesitancy-A systematic review

Judgment under uncertainty: Heuristics and biases

Comparing separate process and intertwined models for reactance

The intertwined model of reactance for resistance and persuasive boomerang

The case for motivated reasoning

Measuring vaccine hesitancy: The development of a survey tool

Are graphic cigarette warning labels an effective message strategy? A test of psychological reactance theory and source appraisal

What do others' reactions to news on internet portal sites tell us? Effects of presentation format and readers' need for cognition on reality perception

Online bandwagon effects: Quantitative versus qualitative cues in online comments sections

Perceiving online public opinion: The impact of Facebook opinion cues, opinion climate congruency, and source credibility on speaking out

Can interactive media attenuate psychological reactance to health messages? A study of the role played by user commenting and audience metrics in persuasion. Health Communication

Social navigation on the internet: A framework for the analysis of communication processes

Power analysis and determination of sample size for covariance structure modeling

Studying multivariate change using multilevel models and latent curve models

Vaccine hesitancy: Definition, scope and determinants

Exploring "angry" and "like" reactions on uncivil Facebook comments that correct misinformation in the news

Psychological reactance and promotional health messages: The effects of controlling language, lexical concreteness, and the restoration of freedom

Does correcting myths about the flu vaccine work? An experimental evaluation of the effects of corrective information

Effective messages in vaccine promotion: A randomized trial

The third person effect: A critical review and synthesis

Exemplification 2.0

Examining the use of forceful language when designing exercise persuasive messages for adults: A test of conceptualizing reactance arousal as a two-step process

Further evidence that psychological reactance can be modeled as a combination of anger and negative cognitions

The nature of psychological reactance revisited: A meta-analytic review

A social identity model of deindividuation phenomena

Psychological reactance and persuasive health communication: A review of the literature

African American parents' perceived vaccine efficacy moderates the effect of message framing on psychological reactance to HPV vaccine advocacy

The effectiveness of empathy-versus fear-arousing antismoking PSAs

Antecedents to psychological reactance: The impact of threat, message frame, and choice. Health Communication

Effects of online comments on smokers' perception of antismoking public service announcements

Deflecting reactance: The role of similarity in increasing compliance and reducing resistance

Source effects on psychological reactance to regulatory policies: The role of trust and similarity

Demography-statistics & facts

How fear appeal approaches in COVID-19 health communication may be harming the global community

Understanding the third-person perception: Evidence from a meta-analysis

The MAIN model: A heuristic approach to understanding technology effects on credibility

Toward a theory of interactive media effects (TIME)

Number of monthly active Facebook users worldwide as of 4th quarter 2020

Ad council launches $50 million fund for national COVID-19 vaccine education effort

COVID-19 vaccines are on the way, but fewer Americans want to take them

Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments

Facebook sentiment: Reactions and emojis

The influence of presumed media influence on news about science and scientists

Judgment under uncertainty: Heuristics and biases

What does the crowd think? How online comments and popularity metrics affect news credibility and issue importance

Bandwagon effects in social television: How audience metrics related to size and opinion affect the enjoyment of digital media

The influence of online comments on perceptions of antimarijuana public service announcements on YouTube

Communication processes in participatory websites

What arguments against COVID-19 vaccines run on Facebook in Poland: Content analysis of comments. Vaccines

Freedom and reactance

They came, they liked, they commented: Social influence on Facebook news channels

Ten threats to global health in 2019

Social recommendation, source credibility, and recency: Effects of news cues in a social bookmarking website

Exemplification theory: Judging the whole by some of its parts

Exemplification effects in the promotion of safety and health

Effects of exemplification in news reports on the perception of social issues