key: cord-0284638-t7a1cous
authors: Weinzierl, Maxwell; Harabagiu, Sanda
title: Identifying the Adoption or Rejection of Misinformation Targeting COVID-19 Vaccines in Twitter Discourse
date: 2022-02-18
journal: nan
DOI: 10.1145/3485447.3512039
sha: 1835271fa8536a707d14e3b9cff111a2b8ab0444
doc_id: 284638
cord_uid: t7a1cous

Although billions of COVID-19 vaccines have been administered, too many people remain hesitant. Misinformation about the COVID-19 vaccines, propagating on social media, is believed to drive hesitancy towards vaccination. However, exposure to misinformation does not necessarily indicate misinformation adoption. In this paper we describe a novel framework for identifying the stance towards misinformation, relying on attitude consistency and its properties. The interactions between attitude consistency, adoption or rejection of misinformation and the content of microblogs are exploited in a novel neural architecture, where the stance towards misinformation is organized in a knowledge graph. This new neural framework is enabling the identification of stance towards misinformation about COVID-19 vaccines with state-of-the-art results. The experiments are performed on a new dataset of misinformation towards COVID-19 vaccines, called CoVaxLies, collected from recent Twitter discourse. Because CoVaxLies provides a taxonomy of the misinformation about COVID-19 vaccines, we are able to show which type of misinformation is mostly adopted and which is mostly rejected.

Although billions of inoculations against the SARS-CoV-2 virus, the causative agent of COVID-19, have been administered around the world starting with 2020, too many remain hesitant about this vaccine. It is believed that hesitancy is driven by misinformation about the COVID-19 vaccines that is spread on social media. Recent research by Loomba et al. [17] has shown that exposure to online misinformation around COVID-19 vaccines affects intent to vaccinate in order to protect oneself and others. However, exposure to misinformation about the COVID-19 vaccines does not mean that those exposed adopt the misinformation. This is why knowing if misinformation is adopted or rejected when encountered in social media discourse will enable public health experts to perform interventions at the right time and in the right place on social media, addressing vaccine hesitancy successfully.

Misinformation detection on social media platforms, such as Twitter, is performed in two steps: (1) the recognition whether a social media posting contains any misconception, reference to conspiracy theories or faulted reasoning; and (2) the recognition of the stance towards the targeted misinformation. The stance defines the attitude the author of the micro-blog manifests towards the misinformation target, as exemplified in Table 1 . When the misinformation is adopted, an Accept stance is observed, whereas when it is rejected, the Reject stance reflects the attitude towards the targeted misinformation.

Misinformation Target: The COVID vaccine renders pregnancies risky.

Stance: Accept Tweet: <@USER> Chances of a healthy young woman dying of COVID if they even catch it: 0.003% Chances of COVID vaccine causing miscarriage, birth defects, or future infertility: <Data Unavailable> Risk management would say DON'T TAKE THE VACCINE IF YOU'RE PREG-NANT. Stance: Reject Tweet: Vaccinated women who breastfeed can pass #COVID19 protection to their babies. COVID-19 #vaccines aren't considered a risk to infants during pregnancy or from breastfeeding. During the study, none of the women or infants experienced serious adverse events. <URL> Table 1 : Examples of tweets with different stance towards misinformation targeting COVID-19 vaccines.

Although the identification of misinformation about COVID-19 vaccines in the Twitter discourse is fundamental in understanding its impact on vaccine hesitancy, we consider that efforts focusing on this first step of misinformation detection have made important progress recently, generating high-quality results [16, [18] [19] [20] . In this paper we focus on the second step of misinformation detection, namely the identification of the stance towards misinformation, which still needs improvements.

A significant barrier in the identification of stance towards misinformation targeting the COVID-19 vaccines stems from the absence of large Twitter datasets which cover misinformation about these vaccines. To address this limitation, we present in this paper a new Twitter dataset, called CoVaxLies, inspired by the recently released COVIDLies dataset [11] . CoVaxLies consists of (1) multiple known Misinformation Targets (MisTs) towards COVID-19 vaccines; (2) a large set of [tweet, MisT] pairs, indicating when the tweet has the stance of: (a) Accept towards the MisT; (b) Reject towards the MisT; or (c) No Stance towards the MisT. In addition, we provide a taxonomy of the misinformation about COVID-19 vaccines, informed by the MisTs available in CoVaxLies, enabling the interpretation of the adopted or rejected misinformation about COVID-19 vaccines.

As it can be noticed from the examples listed in Table 1 , identifying the stance of a tweet with respect to a given MisT is not a trivial language processing task. The framework for stance identification presented in this paper makes several contributions that address the Twitter discourse referring to misinformation. First, it takes into account the attitude consistency (AC) observed throughout the Twitter discourse between tweet authors that adopt or reject a MisT. AC is informing the equivalence between stance identification and the recognition of agree or disagree relations between pairs of tweets. Second, this stance identification framework captures the interactions between discourse AC, the stance values of tweets towards a MisT, and the language used in the articulation of the MisT and the content of the tweets. Third, it considers that the Twitter discourse about a MisT encapsulates knowledge that can be represented by learning knowledge embeddings. This knowledge contributes, along with the neural representation of the content language of tweets, to the prediction of agreement or disagreement between pairs of tweets referring to the same MisT. Finally, the system implementing this novel stance identification framework has produced in our experiments very promising results on the CoVaxLies dataset.

The remainder of the paper is organized as follows. Section 2 describes the related work while Section 3 details the CoVaxLies dataset. Section 4 describes stance identification informed by attitude consistency (AC). Section 5 presents the experimental result while Section 6 is providing discussions of the results. Section 7 summarizes the conclusions.

In previous work stance identification on Twitter was cast either as (1) a classification problem, learning to predict the stance value of a tweet towards a given target claim; or (2) an inference problem, concluding that a tweet entails, contradicts or does not imply the given target claim.

Stance identification as a classification problem: Several datasets were used in prior work aiming stance classification on Twitter. The PHEME dataset [35] consists of Twitter conversation threads associated with 9 different newsworthy events such as the Ferguson unrest, the shooting at Charlie Hebdo, or Michael Essien contracting Ebola. A conversation thread consists of a tweet making a true and false claim, and a series of replies. There are 6,425 conversation threads in PHEME, 1,067 were annotated as true, 638 were annotated as false and 697 as unverified. A fraction of the PHEME dataset was used in the RumourEval task [8] . The stance labels are 'support', 'deny', 'comment' and 'query'. There are 865 tweets annotated with the 'support' stance label; 325 tweets annotated with the 'deny' stance label; 341 tweets annotated with the 'query' stance label and 2789 tweets annotated with the 'comment' stance label. Several neural classification architectures for stance identification were designed by participants in RumourEval [1, 15, 29] . However, Ghosh et al. [10] have shown that the original pre-trained BERT [9] without any further fine-tuning outperforms all these former state-of-the-art models on the RumourEval dataset, including the model that utilizes both text and user information [7] .

More recently, another dataset containing stance annotations was released, namely the COVIDLies dataset [11] . The starting point was provided by 86 common misconceptions about COVID-19 available from the Wikipedia page dedicated to COVID-19 misinformation, which became Misinformation Targets (MisTs). For each known MisT, a set of tweets were annotated with three possible values for stance towards each misconception: (1) agree, when the tweet adopts the MisT; (2) disagree, when the tweet contradicts/rejects the MisT; and (3) no stance when the tweet is either neutral or is irrelevant to the MisT. Of the 6761 annotated tweets, 5,748 (85.02%) received a label of no stance; 670 (9.91%) received a label of agree and 343 (5.07%) received a label of disagree. Recently, using this dataset, Weinzierl et al. [30] used a neural language processing model that exploits the pre-trained domain-specific language model COVID-Twitter-BERT-v2 [? ] and refined it by stacking several layers of lexico-syntactic, semantic, and emotion Graph Attention Networks (GATs) [28] to learn and all the possible interactions between these different linguistic phenomena, before classifying a tweet as (a) agreeing; (b) disagreeing or (c) having no stance towards a MisT.

Stance identification as an inference problem: When the COVIDLies dataset of stance annotations was released in [11] , stance identification was presented as a natural language inference problem which can benefit from existing textual inference datasets. In fact, Bidirectional LSTM encoders and Sentence-BERT (SBERT) [24] were trained on three common NLI datasets-SNLI [6] , MultiNLI [32] , and MedNLI [25] .

We were intrigued and inspired by the COVIDLies dataset, and believed that we could create a similar dataset containing misinformation about COVID-19 vaccines, which would not only complement the COVIDLies data, but it would also enable the development of novel techniques for identifying the stance towards misinformation targeting COVID-19 vaccines. In order to identify T R , the collection of tweets which potentially contain language relevant to the MisTs from CoVaxLies, we relied on two information retrieval systems: (1) a retrieval system using the BM25 [3] scoring function; and (2) a retrieval system using BERTScore [34] with Domain Adaptation (DA), identical to the one used by Hossain et al. [11] . Both these retrieval systems operated on an index of C T , retrieving tweets by processing CoVaxLies MisTs as queries.

Researchers from the Human Language Technology Research Institute (HLTRI) at the University of Texas at Dallas judged 7,346 tweets to be relevant to the MisTs from CoVaxLies and organized them in [tweet, MisT] pairs, annotated with stance information. There are 3,720 tweets which Accept their MisT, 2,194 tweets which Reject their and 1,238 tweets that have No Stance. We note that CoVaxLies contains an order of magnitude more stance annotations than PHEME [35] , the most popular Twitter dataset containing stance annotations, and therefore it presents clear advantages for neural learning methods. Figure 1 illustrates the taxonomy of misinformation available in CoVaxLies. The themes represent the highest level of abstraction, while the concerns differentiate the various MisTs from CoVaxLies. The taxonomy emerged from discussion between public health experts from the University of California, Irvine School of Medicine and computational linguists from HLTRI. Nine misinformation themes were revealed, all characterizing aspects that impact confidence in the COVID-19 vaccine. Confidence, along with convenience and complacency, are well known universal factors contributing to vaccine hesitancy, according to the 3C model [21] . For each misinformation theme, as shown in Figure 1 , a different number of concerns were revealed: the largest number of concerns pertain to the theme predicating the fact that the COVID-19 vaccines are unsafe (8 concerns) while the smallest number of concerns pertain to the themes claiming that the vaccines are not effective or that information about the vaccines is concealed. Using the information provided by the taxonomy illustrated in Figure 1 , we notice in Figure 2 that the misinformation themes that dominate the tweets from CoVaxLies are those about the ingredients of the COVID-19 vaccines, about the adverse events and the fact that the vaccines are unsafe. Moreover, the dominant misinformation regarding the vaccine ingredients claims that the vaccines contain the virus, while the dominant concerns of the lack of safety of the vaccines indicates risky pregnancies or Bell's palsy. When considering the distribution of tweets that adopt the misinformation, those that reject it and those that are neutral (because of having no stance) for the tweets across all the misinformation themes, we noticed, as illustrated in Figure 3 , that the misinformation that is most adopted has the theme of considering alternatives to the COVID-19 vaccines, immediately followed by misinformation regarding the testing of the vaccines and the ingredients used in the vaccines. Interestingly, most of the misinformation that is rejected has to do with the theme indicating that the COVID-19 vaccines are unnecessary, or that they affect the immune system.

Central to our stance identification framework is the belief that the stance of any tweet towards a particular MisT should not be considered in isolation. Because participates in the Twitter discourse about , its stance should be consistent with the attitude of the other tweet authors towards . We hypothesize that all the authors of tweets that Accept must be agreeing among themselves with regard to . Similarly, all the authors of tweets that Reject must also be agreeing among themselves with regard to . But also, any author of a tweet that has an Accept stance towards must disagree with the author of any tweet that has a Reject stance towards . Therefore, all these tweet authors have Attitude Consistency (AC) towards . AC can be illustrated as in Figure 4 , by linking all the tweets that have the same stance towards a MisT through implicit agree relations, and all tweets that have opposing stances towards with implicit disagree relations. In this way, all the tweets that have an Accept stance towards are organized in a fully connected graph spanned by agree relations and similarly, all the tweets having a Reject stance towards are organized in a fully connected graph spanned also by agree relations. In addition, disagree relations are established between all pairs of tweets that have opposing stance towards . Moreover, all tweets that do not have either an Accept or Reject stance towards are considered to automatically have No Stance towards . Hence. the stance values = { , } are the only ones informing AC.

Stance: ACCEPT Stance: REJECT agree Figure 4 : Stance Misinformation Knowledge Graph As shown in Figure 4 , a Stance Misinformation Knowledge Graph is organized for each , referred to as SMKG( ). For clarity, the SMKG( ) illustrated in Figure 4 shows only several of the agree and disagree relations. For each MisT available in the CoV-axLies dataset, we generate an SMKG( ) when considering only the tweets annotated with Accept or Reject stance information, available from the training set of CoVaxLies. However, there are many other tweets in CoVaxLies with no known stance towards any of the MisTs available in the dataset. We refer to the entire set of such tweets as the Tweets with Unknown Stance towards Misinformation (TUSM).

To identify the stance of tweets from TUSM we assume that AC is preserved. This entails three possible cases when considering in addition to the SMKG( ), tweets from TUSM, e.g. 1 , 2 , 3 or 4 , as illustrated in Figure 5 . All the three cases of AC show that the unknown stance of any tweet ∈ TUSM can be identified as Accept when knowing if (a) an agree relation is predicted between and , a tweet known to have an Accept stance towards ; or (b) a disagree relation is predicted between and , a tweet known to have a Reject stance towards . Similarly, the unknown stance of any tweet ∈ TUSM can be identified as Reject when knowing if (a) a disagree relation is predicted between and , a tweet known to have an Accept stance towards ; or (b) an agree relation is predicted between and , a tweet known to have a Reject stance towards . If none of these relations can be predicted, then the stance of is identified as No Stance towards . To formalize the interaction between the implicit relation types and the values of the stance towards a MisT identified for a pair of tweets and we considered a function that selects the Relation Type that preserves AC (RTAC), defined as:

where the value of the stance of towards is while the value of the stance of is . Moreover, we believe that AC can be further extended to account for an entire chain of agree and disagree relations spanning tweets with unknown stance towards .

Transitive Attitude Consistency extends the interaction between the values of the stance towards a MisT and the binary agree and disagree relations to an entire chain of such implicit relations that may connect a tweet from TUSM to a tweet from SMKG( ), whose stance is known. For example, Figure 6 shows how the identified stance towards of tweets , , and is informed by chains of agree or disagree relations originating either in or , tweets from SMKG( ). It is important to note that this extension has to take into account that every time a new stance towards a MisT is identified for a tweet ∈ TUSM, the confidence that the AC is preserved is computed by an Attitude Consistency Score ( ). depends on , the number of relations in the chain originating at a tweet with known stance, available from SMKG( ) and ending at a tweet ∈ TUSM, with unknown stance:

( , , ). To compute ( , , ) we first need to consider the way in which we can represent the SMKG( ). The knowledge graph of SMKG( ) can be represented in a continuous vector space called the embedding space by learning knowledge embeddings for its nodes and edges. When formalizing the SMKG( )= ( ; ), each node ∈ can be seen as = ( , ), where a tweet is paired with its stance towards ; and each edge ∈ is either an agree or a disagree relation. Knowledge embedding models learn an embedding for each tweet as well as an embedding for the agree relation in SMKG( ) and

an embedding for the disagree relation in SMKG( ). But more importantly, knowledge embedding models use a relation scoring function for assigning a plausibility score to any potential link between two tweets and , given their knowledge embeddings and and the embedding of the relation they share. Because the relation between and must preserve AC between the stance identified for and the stance identified for , the relation between these two tweets is provided by the function ( , ). The embedding of the relation indicated by ( , ) is computed as:

Hence, the scoring function of the relation between the pair of tweets and is defined as ( , ( , , ) , ), where is provided by various knowledge embedding models, such as those that we discuss in Section 4.3, Given the representation of SMKG( ) through knowledge embeddings, we can define ( , , ), starting with the chains of length = 1:

( , , ) for chains of length > 1 is computed by considering that we have defined already = { , } and that we shall take into account all tweets from TUSM when generating chains of agree and/or disagree relations originating in SMKG. We compute ( , , ) as:

To consider the overall * of any tweet with stance towards we average the across all possible chains of relations, of varying lengths, up to a maximum length :

Finally, stance towards of a tweet ∈ TUSM is assigned the value corresponding to the maximum * :

= argmax ∈ * ( , , )

However, Equation 5 shows how we assign stance of value Accept or Reject to tweets with previously unknown stance towards a MisT . To also assign the stance value No Stance, we relied on the development set from CoVaxLies to assign a threshold value ( ) for each MisT , such that when * ( , , ) ≤ ( ), for stance values Accept and Reject, we can finalize the stance of a tweet as having the value No Stance. With all stance values finalized for tweets from TUSM towards any MisT from CoVaxLies, we update SMKG( ) to contain all the tweets from TUSM that have either an Accept or a Reject stance towards .

Knowledge embedding models such as TransE [5] and TransD [12] have had significant success in modeling relations in knowledge graphs. More recently, new knowledge embeddings models capture more complex interactions from the knowledge graph, e.g. TransMS [33] , TuckER [2] , and RotatE [27] . Each knowledge embedding model provides a different method of scoring the likelihood of relations in the knowledge graph SMKG( ), as shown in Table 2 .

The scoring of a relation in each knowledge embedding model relies on , the embedding of a relation that maintains AC with the stance towards a MisT of the tweets connected by the relation, and on the embeddings of these tweets, and .

Scoring Function ( , , ) In Table 2 , we denote || · || as the 1 norm, is the identity matrix, ℎ( ) is the non-linear hyperbolic tangent function and is a real numbered parameter dependent on each MisT. The operator ⊙ represents the Hadamard product, and × indicates the tensor product along the n-th mode. Any of the scoring functions listed in Table 2 measure the likelihood of an agree or disagree relation between a pair of tweets which preserves the AC with the stance of the tweets. However, the content of the tweets, communicated in natural language, with the subtleties and deep connections expressed in language also need to be captured when scoring these relations.

The AC of various tweet authors is expressed through the language they use in their tweets. Therefore. it is imperative to also consider the interaction of the language of tweets with the stance towards misinformation and the attitude consistency of the tweet author's discourse. Because the identification of stance towards misinformation is equivalent to discovering the type of relation, either agree or disagree, shared by a pair of tweets that preserves AC, we designed a neural language architecture which considers (1) the contextual embeddings of each MisT , as well as each pair of tweets and having a stance towards ; and (2) knowledge embeddings learned for the SMKG( ) such that we predict the likelihood of a relation between and to be of type agree or to be of type disagree. This neural architecture for Language-informed Attitude Consistency-preserving Relation scoring (LACRscore) is illustrated in Figure 7 .

Given a MisT , the LACRscore system first performs Word-Piece Tokenization [9] on (a) the textual description of , producing tokens 1 , 2 , ..., , as well as on the text of tweets and , which are then passed through the BERT [9] COVID-19 Language Model COVID-Twitter-BERT-v2 [22] pre-trained on the masked language modeling task [9] for 97 million COVID-19 tweets. This process of further pre-training has been shown to improve performance on downstream tasks in various scientific [4] , biomedical [14] , and social media [23] domains. COVID-Twitter-BERT-v2 produces contextualized embeddings 1 , 2 , ..., +2 for the wordpiece tokens in the MisT along with the [ ] and [ ] tokens. In this way, we encode the language describing the MisT using a contextualized embedding 1 ∈ R 1024 , where 1024 is the contextual embedding size for COVID-Twitter-BERT-V2. Similarly, the language used in the tweets and is represented by contextual embeddings 1 and 1 after being processed through COVID-Twitter-BERT-v2. But, it is important to note, that the scoring function from any of the knowledge embedding models provided in Table 2 , cannot operate directly on the contextual embeddings 1 , 1 or 1 , as they do not have the same dimensions of the knowledge embeddings these models learn. Additionally, we need to produce two knowledge embeddings for the MisT to represent both the agree and disagree relation embeddings. Therefore, in LACRscore we needed to consider two forms of projection encoders, capable to project from the contextualized embedding space into the knowledge embedding space. For this purpose, we have relied on the Misinformation Knowledge Embedding Projection Encoder (M-KEPE), using two separate fully-connected layers, to project from 1 into the necessary knowledge embeddings and from any of the knowledge embedding models considered. Similarly, the Tweet Knowledge Embedding Projection Encoder (T-KEPE) uses a different fully-connected layer than M-KEPE to project from 1 and 1 to and respectively. As shown in Figure 7 , these encoders produce the arguments of the scoring function , provided by some knowledge embedding model. The likelihood of an agree or disagree relation between tweets and with respect to the MisT is computed by ( ,  ,  ) and ( , , ). LACRscore was trained on the SMKG( ) derived from the training collection of CoVaxLies, described in Section 3.1. Relations from each SMKG( ) were used as positive examples, and we performed negative sampling to construct "Attitude Inconsistent" examples. Negative sampling consists of corrupting a relation between tweets with stance and with stance towards MisT , which preserves AC. This corruption process is performed by randomly sampling either (1): a different tweet ( , ) ∈ ( ) with the same relationˆ= , to replace such that ( , ) ≠ , or (2): flipping from an agree relation toˆ=disagree relation, or vice versa. The negative sampling will ensure that AC relations will be scored higher than non-AC relations. Moreover, we optimized the following margin loss to train LACRscore when scoring relations:

where is a training score threshold which represents the differences between the score of AC relations and the non-AC relations. The loss L is minimized with the ADAM [13] optimizer, a variant of gradient descent.

To evaluate the quality of stance identification on the test collection from CoVaxLies we use the Precision (P), Recall (R), and F 1 metrics for detecting the Accept and Reject values of stance. We also compute a Macro averaged Precision, Recall, and F 1 score. The evaluation results are listed in Table 3 . The bolded numbers represent the best results obtained. When evaluating the LACRscore system, we have considered (1) In addition, we have evaluated several baselines. First, we considered the system introduced by Hossain et al. [11] , listed as the Natural Language Inference between Tweet text and MisT text (NLI-Tweet-MisT) system. As a baseline, we have also considered the Domain-Specific Stance Identification (DS-StanceId) [30] system, which utilizes the "[CLS]" embedding from COVID-Twitter-BERT-v2 to directly perform stance classification. In addition, we considered the Lexical, Emotion, and Semantic Graph Attention Network for Stance Identification (LES-GAT-StanceId) [30] system which relies on Lexical, Emotion, and Semantic Graph Attention Networks.

The NLI-Tweet-MisT system produced a Macro F 1 score of 50.2, indicating that stance identification as inference over language is not sufficient. Far superior results were obtained by the DS-StanceId system with a Macro F 1 score of 82.7, showcasing the advantage of fine-tuning stance identification systems. The LES-GAT-StanceId system produced a Macro F 1 score of 83.7, which indicates that integrating Lexical, Emotional, and Semantic Graphs further improves stance identification. The LACRscore system with the TuckER configuration produced a Macro F 1 score of 85.0, indicating that identifying the stance towards misinformation through AC presents performance advantages over previous methods. Unsurprisingly. the LACRscore system with the TransMS + ACS configuration performed best, producing a Macro F 1 score of 87.1, which indicates that the transitive nature of AC should not be ignored. The results also show that detecting misinformation rejection tends to be more difficult than the identification of misinformation adoption. System hyperparameters were selected by maximizing the F 1 score of each system on the development set. The LACRscore system was trained with the following hyperparameters: a linearly decayed learning rate of 1 −4 which was warmed up over the first 10% of the 36 total epochs, an attention drop-out rate of 10%, a batch size of 32, and the tweet and MisT knowledge embedding size was set to 8 for all knowledge embedding models, as we found that to perform best on the development set. The LACRscore system utilized the training set for learning to score AC-preserving relations by optimizing the margin loss, described in Equation 7. The LACRscore system with the ACS configuration utilized a maximum chain length of 32, the length value performing best on the development set. The hyperparameter is set to 4.0 for all knowledge graph embedding models, and we sampled 1 negative corrupted relation for each AC relation in the SMKG( ). Threshold values ( ) were also automatically selected by maximizing the F 1 score of the LACRscore system on each MisT on the development set.

Because the LACRscore system produced the best results with the TransMS and ACS configuration, we performed an analysis of the F 1 scores of this system across each of the themes available in the CoVaxLies Misinformation Hierarchy, considering both the adoption and rejection of misinformation, as illustrated in Figure 8 . The identification of adopted misinformation has remarkable performance, across all themes. Moreover, the misinformation rejection is identified quite well too, except for the theme of concealing information about vaccines. This is explained by the observation that this theme is addressed by few tweets in CoVaxLies, as illustrated in Figure 2 , and moreover, it has the smallest percentage of rejection stance values, as illustrated in Figure 3 .

In this paper we present a new method for identifying the stance towards misinformation informed by attitude consistency (AC), which accounts for very promising results on CoVaxLies, a new Twitter dataset of misinformation targeting the COVID-19 vaccines. AC proves to be a stronger signal for stance identification than lexical, emotional and semantic knowledge alone. Moreover, AC informs the knowledge encapsulated in the misinformation discourse on Twitter, which explains the promising results produced by this method, both for the adoption and rejection of misinformation about COVID-19 vaccines.

Misinformation Targets (MisTs), which represent common misconceptions about the COVID-19 vaccines or refer to conspiracy theories associated with these vaccines, have two different sources.

In Table 4 , all examples marked with ⋄ correspond to some of the MisTs identified as known misinformation from Wikipedia and other trusted sources, while all examples marked with □ correspond to some of the answers to questions about vaccine confidence, originating from Rossen et al. [26] . 

The CoVaxLies dataset, comprising the Misinformation Targets (MisTs), the misinformation taxonomy, and the [ , ] pairs, which associate a with its evoked along with stance annotations. The CoVaxLies dataset is publicly available at the following GitHub repository: https:// github.com/ Supermaxman/ vaccine-lies/ tree/ master/ covid19

Code needed to reproduce the experiments described in this paper is also publicly available at the following GitHub repository: https:// github.com/ Supermaxman/ covid19-vaccine-nlp

We note that an early version of CoVaxLies was presented in Weinzierl and Harabagiu [31] , but in that version of CoVaxLies only 17 Misinformation Targets (MisTs) were available, namely the MisTs discovered from Wikipedia and other trusted sources, which are available in this later version as well. Moreover, the previous version of CoVaxLies did not contain any stance annotations, and it did not contain the misinformation taxonomy which were made available in the current version.

USFD at SemEval-2016 Task 6: Any-Target Stance Detection on Twitter with Autoencoders

TuckER: Tensor Factorization for Knowledge Graph Completion

Okapi at TREC-5

SciBERT: A Pretrained Language Model for Scientific Text

Translating Embeddings for Modeling Multirelational Data

A large annotated corpus for learning natural language inference

You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP

SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Stance Detection in Web and Social Media: A Comparative Study

COVIDLies: Detecting COVID-19 Misinformation on Social Media

Knowledge Graph Embedding via Dynamic Mapping Matrix

Adam: A Method for Stochastic Optimization

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

IUCL at SemEval-2016 Task 6: An Ensemble Model for Stance Detection in Twitter

Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent and Convolutional Networks

Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA

Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning

Annual Meeting of the Association for Computational Linguistics

Rumor Detection on Twitter with Tree-structured Recursive Neural Networks

Detect Rumors on Twitter by Promoting Information Campaigns with Generative Adversarial Learning

Vaccine hesitancy: Definition, scope and determinants

COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter

BERTweet: A pretrained language model for English Tweets

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Lessons from Natural Language Inference in the Clinical Domain

Accepters, fence sitters, or rejecters: Moral profiles of vaccination attitudes

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

Graph Attention Networks

pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection

Misinformation Adoption or Rejection in the Era of COVID-19

Automatic detection of COVID-19 vaccine misinformation with graph link prediction

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

TransMS: Knowledge Graph Embedding for Complex Relations by Multidirectional Semantics

BERTScore: Evaluating Text Generation with BERT

Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads