key: cord-0817491-drycxbuf
authors: Zhang, Menghan; Qi, Xue; Chen, Ze; Liu, Jun
title: Social Bots’ Involvement in the COVID-19 Vaccine Discussions on Twitter
date: 2022-01-31
journal: Int J Environ Res Public Health
DOI: 10.3390/ijerph19031651
sha: 7464b788add5d8b29316de0e592f8463c80acef6
doc_id: 817491
cord_uid: drycxbuf

During the COVID-19 pandemic, social media served as an important channel for the public to obtain health information and disseminate opinions when offline communication was severely hindered. Yet the emergence of social bots influencing social media conversations about public health threats will require researchers and practitioners to develop new communication strategies considering their influence. So far, little is known as to what extent social bots have been involved in COVID-19 vaccine-related discussions and debates on social media. This work selected a period of nearly 9 months after the approval of the first COVID-19 vaccines to detect social bots and performed high-frequency word analysis for both social bot-generated and human-generated tweets, thus working out the extent to which social bots participated in the discussion on the COVID-19 vaccine on Twitter and their participation features. Then, a textual analysis was performed on the content of tweets. The findings revealed that 8.87% of the users were social bots, with 11% of tweets in the corpus. Besides, social bots remained active over three periods. High-frequency words in the discussions of social bots and human users on vaccine topics were similar within the three peaks of discourse.

As a platform promoting public engagement and discussion, social media is an important tool for the public to receive information, make judgments, and form cognition [1] . Intensive global efforts towards physical distancing and isolation to curb the spread of COVID-19 may have intensified the use of social media as individuals try to remain connected while in quarantine [2] . Social media has hence become an important channel for the public to understand and discuss vaccine-related information, acting as a channel tool to reflect and shape public perception [3] .

Since the first COVID-19 vaccines were developed, social media has become one of the prominent spheres for the discussion and debate on vaccines, and information on social media will influence public attitudes towards vaccination [4] . Social media has played a central role in disseminating scientific information and online discussion during the current pandemic [5, 6] . Nevertheless, increasing evidence has shown that the online content, discourse, and sentiment on social media may not be limited to genuine humanrun accounts. In the earlier studies on vaccines, social bots-automated accounts controlled and manipulated by computer algorithms [7] -have been accused of participating in vaccine discussions on social media [8, 9] , including public sentiment manipulation [10, 11] . "Weaponized" social bots can even pretend to be humans, spread false information on a large scale, and harm public health [8] . The normalization of social bots in social media may 2 of 14 affect public perceptions of vaccine efficacy and shape public health knowledge [8] , leading to a complicated opinion climate within social media and affecting public vaccination [12] [13] [14] .

So far, scholars have found that social bots have become even more active in the era of COVID-19 [15, 16] . To what extent social bots are participating in the discussion of COVID-19 vaccines online? Furthermore, the characteristics of their involvement have not been empirically investigated. The work detected social bots participating in the online discussion of the COVID-19 vaccine on Twitter and used textual analysis combined with a Python-based program to specifically analyze social bot-generated and human-generated tweets, thereby finding out the engagement characteristics of social bots.

Social media is defined as web-based and mobile-based Internet applications allowing the creation, access, and exchange of ubiquitously accessible, user-generated content [17] . Information interaction on social media has the characteristics of real-time and anonymity of information dissemination [18] , as well as the potential to reach a large audience and disseminate information rapidly [19, 20] . These characteristics of social media make it an essential driving force for acquiring and disseminating information in different fields, e.g., in crisis management [21] , politics [22] , and dissemination of health information [5] . Analyzing data from social media offers new opportunities to analyze several aspects of health communication [12] . Kata found that at least 80% of Internet users seek health information online, and 16% of these seek online information about vaccination [23] . With the difficulty in communicating offline during the pandemic, more and more people are likely to use social media for information about vaccines and to express their views on social media platforms. Against this background, social media can provide an additional informal source of data that can be used to identify health information not reported to medical officials or health departments and to reveal viewpoints on related topics [18] .

With the spread of the epidemic, people are increasingly relying on different social media platforms to receive information and express opinions [10, 24] . Starting in mid-2020, when the COVID-19 vaccine had not yet been developed, false information about the vaccine has been spread on social media platforms [25] . The U.S. Food and Drug Administration (2020) issued the Pfizer-BioNTech COVID-19 vaccine emergency-use authorization on December 11, 2020, implementing the COVID-19 vaccine as an essential measure to combat the COVID-19 epidemic. Discussions on the vaccine controversy have subsequently become one of the foci of online discussion [26] .

Social media emerged as one of the main spheres for the debate and discussion of information related to COVID-19 and its vaccines [2] . Hussain Their study found that the public is optimistic about vaccine development, effectiveness, and trials, as well as concerns about vaccine safety and economic feasibility [27] . However, some studies show that the spread of conspiracy theories, false information, and various rumors about the COVID-19 vaccine on social media has caused an "infodemic" [28] .

During the COVID19 pandemic, social media has become the main channel for disseminating official, authoritative information [29] , rumor-defying information, and serious news [30] . Many government officials worldwide used Twitter as their primary communication channel to regularly share COVID-19-related policy updates and news [31] .

In recent years, social bots have become increasingly prevalent on social media platforms. Social bots, as mentioned, are a program "that automatically produces content and interacts with humans on social media" [7] . These "automated social agents" [32] can mimic human users [33] . A study of social bots on Twitter shows that, of all English-speaking active users, 9 to 15% exhibit bot-like behaviors [15] . Shao et al. (2018) noted that a single social bot account can forward hundreds or even thousands of times in a short period, which enables social bots to be "super-spreaders" of information [34] .

Scholars have detailed how social bots have been involved in politics [35, 36] , business [37] , health [38] , and other topics. In health-related topics, social bots have been accused of promoting polarized opinions, misleading information, and manipulating public sentiments [10] . For example, social bots can also influence the public's attitudes towards the efficiency of cannabis in dealing with mental and physical problems [39] . In the COVID-19 pandemic, some pilots studies found that social bots spread and amplify false medical information and conspiracy theories [15] .

Scholars have located social bot's involvement in online vaccine discussions. Broniatowski et al. (2018) have identified "influencing bots" in a series of vaccine-related tweets designed to spread vaccine conflicts on Twitter. More specifically, "influencing bots" entail two types of inorganic users: accounts of "bots" generating automated content and those of "trolls" misrepresenting their identities and attempts to instigate conflicts purposefully [8] . Yuan et al. analyzed a retweet network related to the rubella vaccine on Twitter after the 2015 California Disneyland measles outbreak and confirmed the intervention of social bots. Their bot analysis discovered that 1.45% of the corpus users are identified as likely social bots, producing 4.59% of all tweets within the dataset. Moreover, bots may deepen the trend towards highly clustering anti-vaccine movements [9] .

Without denying the contribution of these studies, researchers have failed to address the questions of how to distinguish social bots from human users and which social bots circulate in vaccine-related discourse. Besides, it remains empirically understudied as to whether there are social bots within the online discussion of the COVID-19 vaccine, and what are the characteristics of social bots. This work detected the existence of social bots in COVID-19 vaccine-related discussion on Twitter and characterized social bots' involvement. The research questions are as follows.

RQ1: How many social bots are participating in the COVID-19 vaccine discourse on Twitter?

RQ2: What are the characteristics of social bots' involvement in the question of COVID-19 vaccines?

The following section will first address the methodological issue regarding social bots' detection, text classification, and analysis.

The work took the discussion related to COVID-19 vaccines on Twitter as the content of the analysis. We selected one important time point as a timestamp for data collecting: the U.S. Food and Drug Administration (FDA) officially approved the COVID-19 vaccine produced by Pfizer as the beginning of data collection on 11 December 2020, which was an important beginning for COVID-19 vaccination. The end of data collection was on 31 August 2021.

When people discuss vaccines and vaccination on Twitter, they often use the names of pharmaceutical companies to represent their vaccines. From the 22 vaccines that have been approved for use worldwide, the four vaccines with the widest coverage were selected as keywords, including Pfizer, Moderna, Johnson & Johnson, and AstraZeneca [40] . We observed the hashtags of four pharmaceutical companies and their vaccines on Twitter. Therefore, a list of hashtags related to the COVID-19 vaccines was crafted, including #Pfizer vaccine, #Moderna vaccine, #Johnson & Johnson vaccine, and #AstraZeneca vaccine, which made our search results more comprehensive (# is the representation of hashtags on Twitter).

For example, the relevant tweets of #Pfizer can also be retrieved when the #Pfizer vaccine is used for retrieval.

For selection of hashtags, when #vaccine was used as the retrieved hashtag, the tweets' content included multiple vaccines. In other words, the public attaches the hashtag #vaccine to their tweets when discussing any vaccines, such as HPV vaccines or pet vaccines. The reason for not using #covid-vaccine and #covid vaccine is that the hashtag #covidvaccine does not exist on Twitter. Searching with the hashtag #covid-vaccine shows tweets containing the keywords "covid" and "vaccine." Using #covid vaccine also leads to search results with discussions related to #covid rather than COVID-19 vaccines due to the space between "covid" and "vaccine," deviating from our research direction. Using the specific hashtags of four pharmaceutical companies and their vaccines is thereby more focused on the research topic of this work, and we would like to see public discussions on more specific vaccines. In other words, more general hashtags are not used, which is our limitation. This issue will be studied in the follow-up research.

Then, the official API data interface service provided by Twitter was used to collect tweets, with all the tweets in these hashtags crawled. Twitter users' privacy is respected without any personal information collected and displayed. A total of 314,342 vaccinerelated tweets from 11 December 2020, to 31 August 2021, were obtained. Table 1 reports the complete list of search terms and the total number of tweets containing each hashtag. In the dataset we obtained, the number of tweets in English accounted for 96%, and that of non-English tweets accounted for 4%. Considering that English is the main language in the Twitter platform, we excluded non-English tweets from the dataset. We filtered out the repeated tweets, hyperlinks, and emoji symbols in the tweets. After data cleaning, 225,277 valid tweets were obtained. 

Social bots are identified by Botometer formerly (Indiana University, Bloomington, IN, USA), known as BotOrNot. The program was developed by the Institute of Network Science in Indiana University and uses a specific identification algorithm integrating user behaviors, relationships, and spatiotemporal data. It is currently the most authoritative software for identifying bot accounts [7, 39, 41] . As a machine-learning framework, Botometer extracts more than 1000 features of users' behaviors through user profile data, language, social network structure, temporal activity, and sentiment. Then the program produces a score indicating the likelihood that a Twitter account is a bot. Scores closer to 1 represent a higher chance of being bots, while those closer to 0 are more likely to belong to humans [10] . Based on previous studies [7, 39, 41] , the threshold was set to 0.5 to separate bots from humans in this work. A score above 0.5 was defined as a social bot.

The contents of tweets were analyzed to further explore what social bots and human users said in discussing the COVID-19 vaccines. Firstly, these 255,277 valid tweets were divided into two parts, i.e., posted by human users and by social bots, respectively. Besides, tweets posted by social bots and human users are classified by month.

Firstly, the high-frequency words used were summarized to understand the foci of the discussion between social bots and human users. Cipin, a professional word frequency analysis software based on Python, was applied, with NLTK Library used for word seg-mentation. NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum [42] . Loading American National Corpus, we used NLTK chunk nltk.tokenize for word segmentation. Firstly, nltk.sent_tokenize(text) was used to complete the segmentation between sentences, and nltk.word_tokenize(sentence) used to complete word segmentation.

The tweets of human and social bots were input to the software to discover word frequency statistics. We excluded these hashtags (#Pfizer vaccine, #Moderna vaccine, #Johnson & Johnson vaccine, and #AstraZeneca vaccine) used to retrieve tweets. After obtaining the statistical results, the top five high-frequency words were selected. Tweets containing these high-frequency words were retained and manual text analysis performed.

When detecting a large number of accounts, we used the API calling method provided by Botometer. The Python program was used to call the API, and the detection could obtain a value range of 0-1. The higher the score, the more likely the account was a social bot. Based on the judgment criteria of [7, 10] , the work adopted a conservative threshold to ensure the scientific nature of the research results, i.e., the threshold was set to 0.5 to distinguish between humans and social bots. If the score was higher than 0.5, the account is considered to be a bot.

We entered 255,277 valid tweets from 102,844 Twitter users into Botometer to obtain a score. Figure 1 shows the distribution of scores returned by Botometer. Botometer detects 9122 social bots and 93,722 human users among 102,844 Twitter users, accounting for 8.87% and 91.13% of total users, respectively. Social bots post 28,081 and humans post 227,196, accounting for 11% and 89% of the total tweets, respectively. This finding proves the first question that 8. 

Regarding the number distribution of tweets, the change in the number of soc tweets is consistent with that of humans. Meanwhile, Figure 2 shows that the nu tweets by human users increases by more than 25% in December-January 2020, April 2021, and June-July 2021. In other words, they have the highest discussion these three stages. So what do social bots post when human users are arguing Covid-19 vaccine on social media? These three time periods are divided for inves December 2020 to January 2021 as the first phase of discourse (Stage 1), March 2021 as the second phase of discourse (Stage 2), and June to July 2021 as the thir of discourse (Stage 3). Next, what humans and social bots discuss in these three p specifically analyzed. 

Regarding the number distribution of tweets, the change in the number of social bots' tweets is consistent with that of humans. Meanwhile, Figure 2 shows that the number of tweets by human users increases by more than 25% in December-January 2020, March-April 2021, and June-July 2021. In other words, they have the highest discussion heat at these three stages. So what do social bots post when human users are arguing about a COVID-19 vaccine on social media? These three time periods are divided for investigation: December 2020 to January 2021 as the first phase of discourse (Stage 1), March to April 2021 as the second phase of discourse (Stage 2), and June to July 2021 as the third phase of discourse (Stage 3). Next, what humans and social bots discuss in these three periods is specifically analyzed.

tweets is consistent with that of humans. Meanwhile, Figure 2 shows that the number of tweets by human users increases by more than 25% in December-January 2020, March-April 2021, and June-July 2021. In other words, they have the highest discussion heat at these three stages. So what do social bots post when human users are arguing about a Covid-19 vaccine on social media? These three time periods are divided for investigation: December 2020 to January 2021 as the first phase of discourse (Stage 1), March to April 2021 as the second phase of discourse (Stage 2), and June to July 2021 as the third phase of discourse (Stage 3). Next, what humans and social bots discuss in these three periods is specifically analyzed. 

The tweets released by social bots and humans in these three periods were analyzed to further explore the three periods mentioned above. Table 2 presents high-frequency words of social bots and human users in the three periods. According to the statistics of high-frequency words, the foci of the three concentrated discussions are different. Stage 1: Discourse on the Effect of Vaccines (from December 2020 to January 2021). Word-frequency statistics were used to obtain the high-frequency words in social bots and human tweets. Four of the five most frequent words in social bots and human tweets coincided: Dose, Effective, Trial, and Against. Their discussions focused on whether the COVID-19 vaccines were effective. As shown in Table 3 , tweets posted by humans during the Stage1, human users expressed more concerns about vaccine risks in their tweets, such as concerns about serious side effects of vaccine experiments (example 1), doubts about vaccine effectiveness (example 3) , and concerns about the adequacy of vaccine quantity (example 5). On the contrary, social bots conveyed positive information that a variety of COVID-19 vaccines could be gradually approved, and the number of vaccines could be guaranteed (examples 2 and 4). They explained that the vaccines were highly effective after rigorous scientific trials and vaccination could resist COVID-19 (example 6). Besides, social bots could connect efficient vaccines with various positive words, such as Freedom (example 7), Protective (example 8), Hope (example 9), and others, by emphasizing the immunization rate after vaccination. While emphasizing the effectiveness of vaccines, they could describe life after vaccination very positively. Unlike the previous discussion, from March to April 2021, the focus of the discussion changed to "blood clots" caused by the COVID-19 vaccination. In this period, the words most used by humans are News, Blood Clots, Pause, CDC, and Health, and the words most used by social bots are Blood Clots, Pause, CDC, News, and Biden. Table 4 shows the typical tweets posted by social bots and human users at the Stage 2.

In example 10, some users believed that other vaccines might also cause "blood clots" or other side effects, but this had not yet been confirmed. At stage 2, social bots introduced the vaccine's efficacy as in stage 1 and joined the discussion of "blood clots". In response to the public's concern that "infection with COVID-19 did not kill, but people died after COVID-19 vaccination (examples 12 and 14)", social bots used the number of people who get "blood clots" after vaccination as the support for the argument, adopting the discourse "advantages outweigh disadvantages". For example, they emphasized that "Two in one-million women who have had the COVID-19 vaccine have had blood clots (example 11)". Besides, they believed that the consequences of vaccine side effects were not worth mentioning compared with the severe consequences caused by the failure of the immune barrier (example 13). 

Humans Bots

Example10: #Janssen vaccine also has 'possible link' to blood clots: https://onenewspage.com/video/20210420 /13805259/ (accessed on 14 December 2021) Janssen-vaccine Has possible link to blood clots.htm (accessed on 14 December 2021).

Example 11: 2 in 1 million women who have had the COVID-19 vaccine have had blood clots, 1 in 5000 women taking the pill have blood clots, 1 in 6 people with COVID gets blood clots. Although both social bots and human users had actively participated in discussing "blood clots", there were differences in their discourse. When using #CDC to indicate the information source of tweets, human users sent more CDC notifications about the risk of "blood clots" (example 16), while the social bots skillfully avoided this topic. They repeatedly emphasized that the COVID-19 vaccines had high protection and showed that CDC had unsealed the Johnson & Johnson vaccine under investigation, indicating that it was still safe (example 15). Interestingly, the word "Biden" also appears in a large number of social bot tweets. In example 17, tweets of social bots containing "Biden" described one topic: Biden had purchased 500 million Pfizer vaccines and would make a "historic" donation. These vaccines would be donated to 92 developing countries to alleviate the vaccine gap caused by worldwide, uneven vaccine distribution.

Stage 3: Debate on Delta (from June to July 2021). In July 2021, the discussion on COVID-19 vaccines climbed to a peak again. At this stage, the high-frequency words from social bots are Delta, Against, FDA, Warning, and Effective, while the high-frequency words from human users are Delta, Against, FDA, Health, and Warning. Although the high-frequency words of social bots are highly consistent with human users, there are differences in their description of whether COVID-19 vaccines can effectively resist the Delta variant. Table 5 shows the typical tweets posted by social bots and human users at the Stage 3. 19 ,000 people in the UK found one Pfizer shot was about 36% effective against Delta, but it was highly effective after two shots.

businessinsider.com -Example 23: #Pfizer's COVID-19 vaccine works very well against the Delta variant-but only after 2 doses. The study on 19,000 people in the UK found one Pfizer shot was about 36% effective against Delta, but it was highly effective after two shots.

The public questioned the effectiveness of existing vaccines in dealing with the Delta variant (examples 18 and 20 ). Yet, social bots advocated that COVID-19 vaccines were effective against the Delta variant (example 19). They said that the immune response could last at least eight months after vaccination (example 21). If people wanted to get better protection under the spread of the Delta variant, they could inject reinforcing needles (example 22). Besides, social bots were more inclined to convey such a message to human users without being fully vaccinated. If the existing vaccine was only vaccinated with one shot or not fully vaccinated, the protection rate and practical utility were limited (example 23).

Vaccine discussions on social media are constantly evolving, with trends linked to real-world events [43] . Our research further confirms this view. The three critical discussion periods were closely related to actual events. The three critical discussion periods were closely related to actual events. We show the relationship with Figure 3 . People mainly discussed whether the vaccine was effective and whether the vaccine cost money from December 2020 to January 2021, because the COVID-19 vaccines had just come out. On December 11 and 18, 2020, the FDA issued emergency-use authorizations for Pfizer and Moderna vaccines to prevent COVID-19. COVID-19 mRNA vaccines were officially released to the public as an effective tool to combat the epidemic. However, could vaccines resist COVID-19? How could vaccines be obtained? Were the vaccines free? The public did not know much about the COVID-19 vaccines. They alleviated their anxiety in this information environment full of uncertainty and crisis by communicating and forwarding news on Twitter.

On December 11 and 18, 2020, the FDA issued emergency-use authorizations for Pfizer and Moderna vaccines to prevent COVID-19. COVID-19 mRNA vaccines were officially released to the public as an effective tool to combat the epidemic. However, could vaccines resist COVID-19? How could vaccines be obtained? Were the vaccines free? The public did not know much about the COVID-19 vaccines. They alleviated their anxiety in this information environment full of uncertainty and crisis by communicating and forwarding news on Twitter. On April 16, 2021, AstraZeneca COVID-19 vaccines were reported to have produced a rare "blood clots" event. Meanwhile, the government proposed to suspend the company's COVID-19 vaccine production because six women vaccinated with Johnson & Johnson had rare "blood clots" and one woman died. The "blood clots" event aroused people's concerns and fears about the COVID-19 vaccines, which were discussed fiercely on social media.

In July 2021, the Delta virus was found in at least 98 countries and regions. According to the CNN June 29 report, the number of cases of COVID-19 infection in the United States was increasing. The reported cases almost covered the whole of the United States. The On 16 April 2021, AstraZeneca COVID-19 vaccines were reported to have produced a rare "blood clots" event. Meanwhile, the government proposed to suspend the company's COVID-19 vaccine production because six women vaccinated with Johnson & Johnson had rare "blood clots" and one woman died. The "blood clots" event aroused people's concerns and fears about the COVID-19 vaccines, which were discussed fiercely on social media.

In July 2021, the Delta virus was found in at least 98 countries and regions. According to the CNN June 29 report, the number of cases of COVID-19 infection in the United States was increasing. The reported cases almost covered the whole of the United States. The Delta virus has become the world's leading strain of COVID-19, which has increased the number of new cases and deaths worldwide. At this time, the discussion on social media also turned to the discussion of the delta mutant strain.

In recent years, the rise of social bots has attracted the attention of scholars [7, 41, 44] . Researchers found that these automated agents have become even more active in the era of COVID-19 [10, 17] . Social bots have been found in the discussion on the COVID-19 epidemic on Twitter [10, 16] . The emergence of bots influencing social media conversations about public health threats will require researchers and practitioners to develop new communication strategies that take into account the influence of bots [45] .

The work collected 314,342 tweets from 21 December 2020, to explore whether social bots participated in the discussion of the COVID-19 vaccines on Twitter. Our research was launched when the FDA approved the Pfizer vaccine, the first batch of COVID-19 vaccines in the world, on 30 August 2021. Nine months of observation and analysis were used to track the changes in this topic discussion. The number of tweets on vaccine topics by humans and social bots fluctuated. Data collection over a long period enables us to find these differences and changes between human users and social bots in tweet content and expression while observing discussion on COVID-19 vaccines. Then, Botometer was used to detect social bots. Our findings reveal and thereby enrich current discussions on social bots from the following three aspects.

(1) Our findings evidenced the participation of social bots in the discussion of COVID-19 vaccines on Twitter. We detected 28,081 (11%) tweets published by 9122 (8.87%) social bots in the dataset. (2) The activity level of social bots was consistent with that of human users. The number of tweets increased significantly from December 2020 to January 2021, March to April 2021, and June to July 2021. (3) Social bots and humans remaining active in these three periods were mainly relating their discussion to actual events.

From December to January 2020, human users were still skeptical about the COVID-19 vaccines because they had just come out. However, social bots began to promote the effectiveness of the vaccines actively. From March to April 2021, the "blood clots", caused by AstraZeneca, a side effect after vaccination, led to public panic about the COVID-19 vaccines. Although social bots began to talk about "blood clots" at this stage, their discourses were opposite to those of human users. By citing positive sources from CDC, social bots emphasize that "blood clots" were not a common problem. From June to July 2021, the Delta virus ravaged the world. Human users were worried about whether the existing COVID-19 vaccines could resist the Delta virus, but social bots thought that the vaccines were effective against the Delta virus. Generally, social bots are optimistic towards vaccines, although social bots have not always discussed the effectiveness and importance of vaccines.

To conclude, the work enriches the existing research on social bots and online vaccine discussions and helps to comprehensively understand social bots' characteristics and current dynamics. With the spread of the COVID-19 epidemic, vaccination against COVID-19 has become an important means to control the epidemic. The normalization of social bots in social media may affect public perceptions of vaccine effectiveness and shape public health knowledge, complicating the climate of opinion on social media. Researchers and practitioners need to understand the engagement of social bots in social media to develop new communication strategies. We hope that, through our study, researchers and policymakers can understand how social bots are participating in COVID-19 vaccine discussions.

The work detected the participation of social bots in the online discussion of the COVID-19 vaccine on Twitter. Textual analysis was used to specifically analyze botgenerated and human-generated tweets, thus finding out the engagement characteristics of social bots. However, while bot detection algorithms can automatically label suspicious social bots, the Twitter platform's current mechanism does not reveal the manipulators behind social bots. Therefore, although characterizing the content posted by social bots and their participation in discussions and proposing several possible explanations, the work cannot infer the intentions of social bots based on their expressions. Previous researchers have suggested that social bots can also influence people's emotions about vaccines [9, 10] . Therefore, our subsequent research will focus on the emotional expression of social bots in the discussion of the COVID-19 vaccine and whether they affect the public's sentiments towards the COVID-19 vaccine. Institutional Review Board Statement: The study was conducted in accordance with the China's Personal Information Protection Law, and approved by the National Social Science Fund of China (21CXW028). Ethical review and approval were waived for this retrospective study, while the personal data protection statement is available on the project's website (https://rtgksm.wordpress.com/, in Chinese, accessed on 14 December 2021).

Informed consent was waived for this retrospective study, while the personal data protection statement is available on the project's website (https://rtgksm.wordpress. com/, in Chinese, accessed on 14 December 2021).

The data presented in this study are available on request from the first author. The data are not publicly available due to privacy protection.

Troll factories: Manufacturing specialized disinformation on Twitter

Social media and vaccine hesitancy: New updates for the era of COVID-19 and globalized infectious diseases. Hum. Vaccines Immunother

Vaccine hesitancy, vaccine refusal and the anti-vaccine movement: Influence, impact and implications

Reasons for taking the COVID-19 vaccines by us social media users

Social media can have an impact on how we manage and investigate the COVID-19 pandemic

Social media for research, education and practice in rheumatology

The rise of social bots

Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate

Examining emergent communities and social bots within the polarized online vaccination debate in

Social bots' sentiment engagement in health emergencies: A topic-based analysis of the COVID-19 pandemic discussions on Twitter

Online human-bot interactions: Detection, estimation, and characterization

Social media analytics-Challenges in topic discovery, data collection, and data preparation

The effects of anti-vaccine conspiracy theories on vaccination intentions

The effect of falsely balanced reporting of the autism-vaccine controversy on vaccine safety perceptions and behavioral intentions

Nearly Half of the Twitter Accounts Discussing 'Reopening America' May Be Bots, Researchers Say

Assessing the risks of 'infodemics' in response to COVID-19 epidemics

Users of the world, unite! The challenges and opportunities of social media

Using social media for actionable disease surveillance and outbreak management: A systematic literature review

Opportunities and challenges of web 2.0 for vaccination decisions

Fung, I.C.H. Sentiment, contents, and retweets: A study of two vaccine-related Twitter datasets

Sense-Making in social media during extreme events

Social media and political communication: A social media analytics framework

Anti-vaccine activists, Web 2.0, and the postmodern paradigm-An overview of tactics and tropes used online by the anti-vaccination movement

Twitter discussions and emotions about the COVID-19 pandemic: Machine learning approach

The Anti-Vaxx Industry: How Big Tech Powers and Profits from Anti-Vaccine Misinformation.Centre for Countering Digital Hate

Exploring public perceptions of the COVID-19 vaccine online from a cultural perspective: Semantic network analysis of two social media platforms in the United States and China

Artificial intelligence-enabled analysis of public attitudes on Facebook and Twitter toward COVID-19 vaccines in the United Kingdom and the United States: Observational study

Not just conspiracy theories: Vaccine opponents and proponents add to the COVID-19

Policy response, social media and science journalism for the sustainability of the public health system amid the COVID-19 outbreak: The Vietnam lessons

The causes and consequences of COVID-19 misperceptions: Understanding the role of news and social media

World leaders usage of Twitter in response to the COVID-19 pandemic: A content analysis

Dissecting a social botnet: Growth, content, and influence in Twitter

Do social bots dream of electric sheep? A categorization of social media bot accounts

The spread of low-credibility content by social bots

Automation, algorithms, and politics | political communication, computational propaganda, and autonomous agents-Introduction

The Brexit botnet and user-generated hyperpartisan news

Social media bots and stock markets

Cannabis surveillance with Twitter data: Emerging topics and social bots

Who falls for online political manipulation?

The First Seven COVID-19 Vaccines to Have Been Approved Internationally Received Orders for over 15 Billion Doses Altogether

Evolution of bot and human behavior during elections. First Monday

Social bots distort the 2016 US presidential election online discussion. First Monday

Temporal trends in anti-vaccine discourse on Twitter

Health communication trolls and bots versus public health agencies' trusted voices

The authors declare no conflict of interest.