key: cord-0444349-5uyh2snu
authors: Singhal, Shivangi; Shah, Rajiv Ratn; Kumaraguru, Ponnurangam
title: Factorization of Fact-Checks for Low Resource Indian Languages
date: 2021-02-23
journal: nan
DOI: nan
sha: 1172c9bdebeb5b61080342f2d38527c154105502
doc_id: 444349
cord_uid: 5uyh2snu

The advancement in technology and accessibility of internet to each individual is revolutionizing the real time information. The liberty to express your thoughts without passing through any credibility check is leading to dissemination of fake content in the ecosystem. It can have disastrous effects on both individuals and society as a whole. The amplification of fake news is becoming rampant in India too. Debunked information often gets republished with a replacement description, claiming it to depict some different incidence. To curb such fabricated stories, it is necessary to investigate such deduplicates and false claims made in public. The majority of studies on automatic fact-checking and fake news detection is restricted to English only. But for a country like India where only 10% of the literate population speak English, role of regional languages in spreading falsity cannot be undermined. In this paper, we introduce FactDRIL: the first large scale multilingual Fact-checking Dataset for Regional Indian Languages. We collect an exhaustive dataset across 7 months covering 11 low-resource languages. Our propose dataset consists of 9,058 samples belonging to English, 5,155 samples to Hindi and remaining 8,222 samples are distributed across various regional languages, i.e. Bangla, Marathi, Malayalam, Telugu, Tamil, Oriya, Assamese, Punjabi, Urdu, Sinhala and Burmese. We also present the detailed characterization of three M's (multi-lingual, multi-media, multi-domain) in the FactDRIL accompanied with the complete list of other varied attributes making it a unique dataset to study. Lastly, we present some potential use cases of the dataset. We expect this dataset will be a valuable resource and serve as a starting point to fight proliferation of fake news in low resource languages.

Fake news is spreading like wildfire. All the major happenings around the globe like 2016 U.S. elections [5] , global pandemic (COVID-19) [1, 3] and 2019 Indian General elections [13] have been heavily affected by the proliferation of infodemic on social and mainstream media. To limit the escalation of fake news, fact-checkers are constantly making an effort to find out the authenticity behind the unverified claims. Fact-checking is generally defined as a process to find out the veracity behind such viral claims. The conclusion drawn is always backed up by evidence and logical reasonings [21] . Fake news is a global concern but majority of the automatic fact-checking [2, 4, 7, 17, 21, 22] and fake news detection [10, 14-16, 23, 24] solutions have been designed for English, a language predominantly used in web-based social networking. Though that is good on a global scale, but when it comes to India, it fails due to following reasons, (i) people speak diversified range of languages, (ii) due to housing a number of languages, the communication in English is bit problematic because out of the 74% literates in the country, only 10% can read English 1 . Recently, Rasmus Kleis Nielsen, director at Reuters Institute for the Study of Journalism, also addresses this issue in his interview 2 stating that, the problems of disinformation in a society like India might be more sophisticated and tougher than they are within the West. We wish to understand the menace of misinformation in Indian territory where false information, manipulative photos and deduplicate [22] is shown. The pink highlighted text shows the investigation steps adopted by fact-checkers. Though the justification (veracity_reasoning) in both the dataset is present but the investigation_reasoning in FactDRIL gives minute details of the fact-checking process. This attribute is exclusive of FactDRIL and is not present in any of the existing fact-checking datasets.

news reappear on the online ecosystem time and again. To the best of our knowledge, there is no existing fact-checking data for Indian languages. This has caused hindrance in devising solutions for automatic fact-checking, which can negatively affect the large group of population.

In this paper, we aim to bridge the gap by proposing FactDRIL: the first large scale multilingual Fact-checking Dataset for Regional Indian Languages. Our contribution can be summarized as follows:

• We curated 22,435 samples from the eleven Indian fact-checking website certified with IFCN ratings. The samples are in the various low-resource languages: Bangla, Marathi, Malayalam, Telugu, Tamil, Oriya, Assamese, Punjabi, Urdu, Sinhala and, Burmese. We release the dataset that can act as a prime resource for building automatic fact-checking solutions for Indian ecosystem. 3 • We introduced an attribute in the feature list termed as investigation_reasoning. This attribute provides an explanation of the intermediate steps performed by fact-checkers to conclude the veracity of the unverified claim. This is important to study because it will help us to dig into the fact-checking mechanism and propose solutions to automate the process. We discuss in detail the use-case of this curated feature and the methodology designed to excavate it from the crawled unstructured data dump.

There have been several datasets released in past that focused on fact-checking. An overview of the core fact-checking datasets is given in Table 1 .

The very first effort towards this direction was made by [21] in 2014. The paper released a publicly available dataset that consist of sentences fact-checked by journalist available online. The statements were picked from the fact-checking blog of Channel 4 and the Truth-O-Meter from PolitiFact. The statements mainly captured issues prevalent in U.S. and U.K. public life. Apart from statements, the meta-data features like, (i) publish date, (ii) speaker, (iii) fine-grained label associated with the verdict and, (iv) URL were also collected.

Another study was done in 2016 by [7] . In this paper, the data was collected from numerous sources including rumor sites and Twitter handles. The news pertaining to the world, U.S. national and technology were captured. For each claim, journalist would search for the articles that are either in support, against or observing towards the claim. The final dataset consist of claims with corresponding summarized headings by the journalist and associated veracity label with the final verdict on the claimed statement.

Both the previously mentioned datasets were quite small in numbers. To overcome this drawback, the LIAR dataset was introduced by [22] in 2017. It consists of around 12.8K short statements curated from Politifact website. It mainly contain samples collected from a variety of sources including, TV interviews, speeches, tweets and debates. The samples cover wide range of issues ranging from the economy, health care, taxes to elections. The samples were annotated for truthfulness, subject, context, speaker, state, party, and prior history. For truthfulness, the dataset was equally distributed into six labels: pants-fire, false, mostly false, half-true, mostly-true, and true. In 2018, Alhindi et al. [2] proposed LIAR-PLUS, an extended version of the LIAR dataset. For each sample, human justification for the claim was automatically extracted from the fact-checking article. It was believed that justification combined with extracted features and meta-data would boost the performance of classification models. Another dataset that came into existence in 2018 was FEVER [18] . It consists of 185,445 claims that were not naturally occurring but were generated by altering sentences extracted from Wikipedia.

Later in 2019, two new functionalities i.e. evidence pages and knowledge graph triples were introduced to fact-checking data that resulted in overall improvement of the accuracy. Augenstein et al. [4] presented the largest dataset on fact-checking that had 34,918 claims collected from 26 fact-checking websites in English listed by Duke Reporters Lab and on fact-checking Wikipedia page. The prime benefit introduced was the 10 relevant evidence pages per claim. Other features included claim, label, URL, reason for label, categories, speaker, fact-checker, tags, article title, publication, date, claim date, and the full text associated with claim. Authors in [17] argue the need of labelled ground truth and other features to perform supervised machine learning methods. and emphasis on the requirement of a huge storage space. To solve the problem, they proposed ClaimsKG, a knowledge graph of fact-checked claims. It enables structured queries for features associate with the claim. It is a semi-automated method that gather data from fact-checking websites and annotate claims and their corresponding entities from DBpedia.

Next, we discuss how our proposed dataset-FactDRIL is different from the existing fact-checking datasets.

The dataset presented in the paper is unique due to following reasons:

• Misinformation in India: Till date, various datasets proposed in the fake news research, consist of news samples pertaining to USA and its neighbouring countries [8, 10, 12, 14-16, 20, 23, 24] . We believe that studying disinformation in India will be more challenging than West due to, (i) low literacy rates that not only make it hard to induce decision-making ability in an individual, but add to an explosion of fake news and divisive propaganda, (ii) Most Indians tend to trust messages from family and friends 4 . This means that content gets forwarded without any checks, further driving misinformation in the social sphere, (iii) majority of population access news primarily through WhatsApp 5 where information rarely gets checked for validity and authenticity. • Multilingual information: The 2011 Census of India 6 shows that the languages by highest number of speakers (in the decreasing order) are as follows: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Urdu, Odia, Malayalam and Punjabi. Whereas, only 10.67% of the total population of India converse in English. Though current datasets are in English but the above statistics explicitly indicates a need to shift the study of fake news from English to other languages too. • Investigation reasoning: With this dataset, we present a detailed explanation of how the investigation was carried out by manual labourers in concluding the truthfulness of the viral news. The proposed attribute is totally different from the veracity reasoning label present in the fact-checking datasets [2, 22] . The primary difference is that the former attribute explains the intermediate steps performed during manual intervention whereas the latter attribute concludes the veracity of unverified claim with a reasoning without emphasising on how they searched for that reasoning. Figure 1 explains the difference between the two. We believe such an information will be helpful in preparing solutions to automate the manual fact-checking efforts. • Multimedia and multi-platform information: Fake news can be published in any form and on any social and mainstream platform. The curated dataset incorporates the information about media (images, text, video, audio or social media post) used in fake news generation and the medium (Twitter, Facebook, WhatsApp and Youtube) used to populate it in the surroundings. • Multi-domain information: Previous fact-checking dataset as shown in Table 1 covers information in certain domain only. For example, Emergent [7] only captures the national, technological and world related happening in U.S.A whereas [2, 22] includes health, economy and election related issues. This is due to the fact that different fact-checking websites focuses on capturing news of specific genre. Since in our proposed dataset, we crawled information from all the fact-checking websites that exist in the country. This gives us a leverage to encapsulate all the happenings around the territory making it a rich dataset to study.

To sum up, in our perception, this is the first dataset introduced in the fake news domain that collects information from Indian territory. The detailed characterization of three M's (multi-lingual, multi-media, multi-domain) in the dataset accompanied with veracity reasoning and other varied attributes makes it a unique dataset to study.

In our opinion, we have curated, the first large scale multilingual Indian fact-checking data. Figure 2 shows the complete data curation process. In this section, we focus on the first step of the curation process i.e. the data collection.

Though fact-checking services play a pivotal role in combating misinformation, little is known about whether users can rely on them or not. To corroborate trust among audience, fact-checking services should endeavour transparency in their processes, as well as in their organizations, and funding sources. With an objective to look out for trusted Indian fact-checking websites, we came across International Fact Checking Network (IFCN). Next we discuss in detail about IFCN, its measuring criteria and sources chosen for preparing the final dataset. 

The International Fact-Checking Network is owned by Poynter Institute of Medical Studies located in St. Petersburg, Florida. It was set in motion on September 2015. The prime objective to establish IFCN was to bring together the fact-checkers present across the globe, under one roof. It also intent to provide a set of guidelines through the fact-checkers code of principles that are mandatory for the fact-checking organizations to follow. The code of principles is designed for agencies that actively work towards broadcasting the the correct investigation against the false claim made either on mainstream or social media platforms.

The organizations that are legally registered with an objective to routinely scrutinize the statements made by public figures and prominent institutions are generally granted the IFCN signatory. The statements can be in the form of text, visual, audio and other formats particularly related to public interest issues. On the other hand, organizations whose opinion look influenced by the state or any other influential identity or a party are generally not admitted the grant.

To be eligible to get an IFCN signatory, the organization is critiqued by independent assessors on 31 criteria. The assessment is then finally reviewed by the IFCN advisory board to ensure fairness and consistency across the network. There are about 82 Verified signatories of the IFCN code of principles among which 11 are based on India. To ensure the authenticity and verifiability of the curated data, we have considered those Indian fact-checking sources that are IFCN rated verified.

Next, we discuss the 11 Indian fact-checking websites considered for our data collection process.

The prime benefit of gathering data from fact-checking websites is that we can read the reasoning behind the veracity of a news sample. The detailed description of the investigation gives an useful insight to the reader about how and why the viral claim was false. With this objective in mind, we decided to collect data from the 11 fact-checking websites that are on a mission to debunk fake information from the Indian ecosystem. An overview of the fact-checking websites considered for data curation is provided in Table 2 . The table highlights the key features of a particular website in the form of, (i) who formed the website, (ii) organization establishment year, (iii) languages debunked by the website and, (iv) domain covered. 

In this section, we discuss the schema of our proposed dataset. This is the second step of the data curation pipeline as shown in Figure 2 . Data Collection We list down all the authentic fact-checking sources that unfolds claims written not only in multilingual languages but lingua franca too. We set-up a data extraction system that make use of a Python library, Beautiful Soup 7 to extract data from web pages. Our system checks the sources for new data once in 24 hours.

In this paper, we present a study on samples curated from a time period of Decemner 2019-June 2020. By the end of the data curation process, we had 22,435 news samples from 11 factchecking websites. Among them, 9,058 samples belong to English, 5,155 samples to Hindi and remaining 8,222 samples were distributed in various regional languages i.e. Bangla, Marathi, Malayalam, Telugu,Tamil, Oriya, Assamese, Punjabi, Urdu, Sinhala and Burmese.

We curated numerous features from the unstructured data. We have then categorized the extracted feature set into various classes like meta features, textual features, author features, media features, social features and event features. A sample showcasing all the attributes is present in Figure 3 .

Meta Features We consider those attributes as meta_features that tells us about the sample, like website_ name, article_link, unique_id, publish_date.

• website_name: Denotes the name of the source from where the following sample is collected.

It also gives an additional information about the language in which the fact-checked article is originally written. • article_link: The attribute gives you the original link of the curated sample.

• unique_id: This attribute acts as the primary key for data storage.

• publish_date: The attribute signifies the date on which the article was published by the fact-checking websites.

Textual Features A fact-checked article is generally segregated into three divisions, title of the article, claim and investigation. All these together form the textual features in our proposed dataset. The crawled data from the website is highly unstructured. The information in the form of claim and investigation is generally present in the content part of the data. This information is extracted from the content attribute using human intervention. This is discussed in detail in Section 6.2.

• title: The title of the article.

• content: This attribute act as the body of the article that consist of information in the form of claim and investigation. • faaltu_content: Many a times, the crawled information from the website also contains the text present in the social media post (say, text present in the tweets or Facebook posts) attached as a reference within the article. Though, such text pieces will be a hindrance while reading the content but they might give useful information if the corresponding social media post is deleted from web. To take into account all such possibilities, we coined an attribute, faaltu_content 8 in our proposed FactDRIL that takes into account the text parts of the social media post attached with the article. The example of the same is shown in Figure 3 . • investigation: This attribute help readers in understanding why the fact-checkers concluded a particular post to be fake. The whole inspection process is discussed in detail with tools and technology used for its exploration. • Other features like claim_title, investigation_title, post_summary are present in some samples.

These attributes give one line summary of the claim, investigation and complete article. Such attributes are useful when the reader intends to take a quick look at the article to find out questions like, What was claimed, who was claimant and how it was investigated to be false.

Author Information This set of attributes showcase information about the people who are involved in fact-checking.

• author_name: The person who wrote the fact-checking article. 

The claim viral on any social media platform or mainstream media have many modality. Similarly, the investigation carried out to conclude the status of any viral news is also backed by a numerous number of supporting claims that can again be in any multimedia form. The set of attributes that are categorized into multimodal features are as follows:

• top_image: The first image that is present at the very beginning of the fact-checking article.

It generally show the picture present in the viral claim. • image_links: The links of all other images that will either belong to the original claimed images group or are presented in support to the viral claim are put under this feature as a list object. • video_links: For those samples where prime media used for fabrication is video, the link to the original video is provided by fact-checkers to backed their investigation. This attribute stores all such links. • audio_links: All the supporting audio links related to the viral claim are presented in this attribute. • links_in_text: To provide complete justification to what was said in the investigation section of the report, authors provide different media links in support of their investigation. All such links are piled in this attribute. But to identify, where a specific link is mentioned in the fact-checked article, an attribute named as, bold_text is used for easy identification and matching of the corresponding text from the article.

The attribute stores the tweet ids present in the sample. The tweet ids can be the post that, (i) needs to be investigated or, (ii) is present in the support of the fake claim. With this attribute, we can extract the complete information from the tweet thread.

Event Features The set of features in this group gives information about the event to which a news sample belong to. These include domain and tags attributes. For example, the Boom article titled: 'False: Chinese Intelligence Officer Reveals Coronavirus Is A Bioweapon' had the following tags (Coronavirus China, COVID-19, Coronavirus outbreak, Bioweapon, Biological warfare, China, Intelligence Officer) associated with it. This kind of information is helpful in identifying the genre of the article.

In this section, we address the three key questions that facilitated the data annotation process. 

The need to do annotations were two fold. First, we want to highlight the key pieces of extracted text to underline its usability. Since the extracted text neither showed a particular pattern for automatic extraction nor had any distinction between the different textual features discussed in Section 5.1, we performed manual annotations to excavate the key pieces from the data dump. Second, there was a need to scrutinize each sample to make sure that the proposed dataset do not contain samples, (i) that were investigated to be true, (ii) articles containing general fact information that news websites usually publish 9 and, (iii) weekly-wrap up articles that increase the chance of deduplication in the dataset.

We chose against crowd-sourcing the annotation process due to non-trivial nature of the sub-tasks. We hired individuals who were well versed with the any of the low-resource languages mentioned earlier. They were in the age group of 19-45. Due to unavailability of language professionals, we performed the annotation process for regional languages with a single entity, whereas two professionals were hired each for lingua franca (i.e. Hindi and English). The annotators were first provided with the annotation guidelines which included instructions about each sub-task, definition of the attributes that need to derived from the text and few examples. They studied this document and worked on a few examples to familiarize themselves with the task. They were given feedback Table 3 . Inter-annotator agreement for the two sub-tasks. The values in bold indicates that Gwet's AC(1) and AC(2) scores were calculated for the samples.

Language on the sample annotations, which helped them to refine their performance on the remaining subset. The annotation process was broken down into two sub-tasks. In the first sub-task, annotators have to segregate the text into two attributes namely, claim and investigation. In the second sub-task, annotators have to make sure to chose samples that were sentenced to be false by fact-checkers.

There were two sub-task given to the annotators. To evaluate the performance on the first task, a customized metric was prepared. For second task, inter-annotator agreement score using Cohen's Kappa and Gwet's AC(1) and AC(2) statistic [9] was considered. Next, we will discuss in detail these evaluation metric.

For the first sub-task, annotators were required to do a close reading of the text to extract meaningful information and place it under correct header (i.e. claim or investigation). To evaluate the performance, we checked for matched ordinal positions in each annotated piece. A counter value is kept to calculate the number of mismatches. Having tested multiple possible values, we chose a threshold value, d=6 and if d goes beyond the chosen value, then that sample is sent in the conflict resolution bin. The final inter agreement score is computed using the percent agreement for two raters [19] . It is calculated by dividing total count of matched sample with the total number of samples in the data.

For the second sub-task, we evaluated inter-annotator agreements using Cohen's Kappa [11] . We observe a mix of moderate and substantial agreement for most of the task. Table 3 summarizes the Cohen's kappa measures for the sub-task. Though Cohen's Kappa performs exceptionally well when dichotomous decision is involved and takes care of chance agreement too but it fails badly when annotators show near to 100% agreement. This phenomenon is termed as 'the paradoxes of Kappa'. During our evaluation, we observe high agreement between annotators for 1000 samples. To solve the 'the paradoxes of Kappa' issue, we used Gwet's AC(1) and AC(2) statistic [9] . It overcomes the paradox of high agreement and low reliability coefficients. Table 3 summarizes the Gwet's score for these samples.

Conflict Resolution We chose a straightforward approach for adjudication in case of disagreements in all the sub-tasks. If the two annotators present a contradictory views for a sample, the annotations for all the disagreed tasks are then adjudicated by verbal discussions among them.

We begin by providing a statistical overview of our proposed dataset. Figure 4 (a) shows the distribution of samples across languages in our proposed FactDRIL. Surprisingly, Regional languages surpasses Hindi by 37%. The diffusion of samples in the regional interface is majorly dominant by Bangla, Malayalam, Urdu and Marathi langauge. Figure 4 (b) represents the number of samples belonging to the eleven fact-checking websites. Among them Fact Crescendo website rules in debunking fake news dissemination in different languages. 

We analyze the topic distribution of fact-checking articles in different languages, i.e. English, Hindi and Regional languages. From Figure 5 (a), (b) , (c), we can conclude that political activity is an important ground for fake news creation. With the onset of 2020, the world has witnessed a global pandemic i.e. Coronavirus. This has not only affected lives of people but has also given rise to infodemic of misinformation. To no surprise, the second popular domain for fake news creation in India was Coronavirus followed by health and religion. 5 . Topic Distribution in English, Hindi and Regional languages (left to right). All the figures clearly shows that the majority of the fake news dissemination across the country is centered towards political domain. Figure 6 (a) shows that the fact-checking trend came to India in 2012, majorly debunking news in English Language. As and when fake news dissemination in English got little popular i.e. 2017, we saw it intruding in the other languages too. This steady shift to other languages was observed quite lately i.e. in 2017 and 2018. We observe sharp peaks and drops in the graph that will be an interseting study to do in future. Figure 6(b) shows the year-wise distribution of samples in the dataset. The graph shows a steady increase in fake news creation over the years with a major peak observed in 2019. For both these observations, the data considered for the year 2020 is till June. 

There are varied threads of misinformation research that can be initiated with the help of FactDRIL. We would like to formally propose some ideas for the same.

• Improve Misinformation Detection System: Till date, various efforts have been made to eliminate misinformation from the ecosystem. The major drawback observed in such methods is two fold. First, the system performs well on trained samples and fails drastically for the real-world data. Secondly, the performance of classifiers varies considerably based on the evaluation archetype and performance metric [6] . We present a dataset that provides a detailed investigation of the fake sample that includes, (i) the modality faked in the news, (ii) 'how' the sample was concluded to be false and, (iii) tools used to draw that conclusion.

We believe that a detailed study in understanding the formation of fake news pattern can improve the performance of classifiers to tackle misinformation in the real world. • Suppressing Fake News Dissemination at an Early Stage: We all know that fake news is not new, probably it is as old as humanity. Despite constant efforts made to eliminate it from the ecosystem, it finds a way to dissolve in our lives. The impact of fake information is devastating. We believe eradicating fake news is bit challenging but creating systems that can suppress its effect is a feasible task. With FactDRIL, we can develop technologies that can be stationed at different social media platforms. Such system can use information from the debunked pieces and stop proliferation of its variants on the platform. • Bias among fact-checkers: Fact-checking is tedious. Different websites aim to debunk news of different genre. There can be websites that aims at exposing a particular kind of information. It will be interesting to look out for biases in the fact-checking pattern and its related effects.

In this paper, we presented FactDRIL: a Fact-checking Dataset for Regional Indian Languages.

To our knowledge, this is first large scale multi-lingual Indian fact-checking data that provide fact-checked claims for low resource languages. The dataset can be accessed on the request via link. 10 We believe such datasets will allow researchers to explore the fake news spread in regional languages. Additionally, researchers could also look out for dissemination of fake content across the different language silos. The dataset comprises of 22,435 samples crawled from the 11 Indian fact-checking websites with IFCN certification. A vast range of low-resource languages has been covered including, Bangla, Marathi, Malayalam, Telugu, Tamil, Oriya, Assamese, Punjabi, Urdu, Sinhala and Burmese. The features curated from the data dump is further grouped under meta, textual, media, social, event and author features. We also present a new attribute to the feature list i.e. investigation reasoning and explain its relevance and need in the current fact-checking mechanism. Currently, this feature is extracted via manual intervention. In future, we plan to automate the attribute extraction from the text. We would also like to organize challenges around this data to instigate researchers in asking interesting questions, find limitations and propose any improvements or novel computational techniques in detecting fake news in low-resource languages.

Somya Sagarika, and Joyojeet Pal. 2020. Temporal patterns in COVID-19 related digital misinformation in India

Where is Your Evidence: Improving Fact-checking by Justification Modeling

Fake news and COVID-19: modelling the predictors of fake news sharing among social media users

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Influence of fake news in Twitter during the 2016 US presidential election

Toward a Better Performance Evaluation Framework for Fake News Classification

Emergent: a novel data-set for stance classification

Fake News vs Satire: A Dataset and Analysis

Computing inter-rater reliability and its variance in the presence of high agreement

MVAE: Multimodal Variational Autoencoder for Fake News Detection

Interrater reliability: The kappa statistic. Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB 22

BREAKING! Presenting Fake News Corpus for Automated Fact Checking

A Dataset of Fact-Checked Images Shared on WhatsApp During the Brazilian and Indian Elections

FakeNewsNet: A Data Repository with News Content, Social Context and Dynamic Information for Studying Fake News on Social Media

SpotFake+: A Multimodal Framework for Fake News Detection via Transfer Learning (Student Abstract

SpotFake: A Multi-modal Framework for Fake News Detection

ClaimsKG: A Knowledge Graph of Fact-Checked Claims

FEVER: a Large-scale Dataset for Fact Extraction and VERification

Three estimates of interrater reliability for nominal data

The Data Challenge in Misinformation Detection: Source Reputation vs. Content Veracity

Fact Checking: Task definition and dataset construction

Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection

EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection

SAFE: Similarity-Aware Multi-Modal Fake News Detection