key: cord-0498528-qmfr60yb
authors: Ali, Ayesha; Qazi, Ihsan Ayyub
title: Countering Misinformation on Social Media Through Educational Interventions: Evidence from a Randomized Experiment in Pakistan
date: 2021-07-06
journal: nan
DOI: nan
sha: 98b42d596696a20cb0c10c2b7c581395cf1eabea
doc_id: 498528
cord_uid: qmfr60yb

Fake news is a growing problem in developing countries with potentially far-reaching consequences. We conduct a randomized experiment in urban Pakistan to evaluate the effectiveness of two educational interventions to counter misinformation among low-digital literacy populations. We do not find a significant effect of video-based general educational messages about misinformation. However, when such messages are augmented with personalized feedback based on individuals' past engagement with fake news, we find an improvement of 0.14 standard deviations in identifying fake news. We also find negative but insignificant effects on identifying true news, driven by female respondents. Our results suggest that educational interventions can enable information discernment but their effectiveness critically depends on how well their features and delivery are customized for the population of interest.

Misinformation is a growing concern in developing countries where low-cost smartphones and mobile Internet access has led to increasing use of social media platforms as sources of information and a place for social and political activity. 1 This trend has brought many new users online, including those with limited digital literacy, which is the ability to access, understand and use information online (Hargittai 2005) . According to the International Telecommunications Union (ITU 2019), less than half of the population in 40 developing and emerging countries, possesses basic computer skills such as copying a file or sending an email with an attachment. These relatively inexperienced technology users are joining social media in a landscape in which misinformation is a common phenomenon. This can have far-reaching consequences for individuals and society. 2 Can simple educational interventions improve the ability to identify misinformation among low digital literacy populations? Educating users about misinformation and the digital space can offer a promising pathway to reduce the prevalence and impact of fake news. 3 Although the benefits of educating people on countering misinformation seem plausible, there is little direct evidence about the effectiveness of educational interventions to counter misinformation in developing countries.

In this paper, we evaluate the effectiveness of two educational interventions for countering misinformation using a clustered randomized control design with 750 participants drawn from urban Pakistan. A cluster in our study is a small geographic area of 100 m x 100 m, covering low and middle income areas in the city of Lahore. To avoid spillovers, we assign 50 clusters each to control and the two treatment arms. Within each cluster, we draw five individuals from five distinct households. A unique feature of our study is that we recruit our sample from the field, instead of relying on online survey platforms that are less representative of low 1 According to the 'Digital in 2020' report (We Are Social 2020), social media use in developing countries is increasing with annual growth rates ranging from 12% to 38%. 2 These effects can range from election interference (Allcott and Gentzkow 2017; Grinberg et al. 2019 ), polarization (Iyengar et al. 2019; Boxell, Gentzkow, and Shapiro 2020) , lack of adherence to preventive measures during the COVID-19 pandemic (Allcott et al. 2020; Bursztyn et al. 2020) , and to even conflict (Zhuravskaya et al. 2020; Samuels 2020; Müller and Schwarz 2020; Conger, Isaac, and Frenkel 2021) . 3 While platform providers, such as Facebook, can filter and flag potentially fake content, it is challenging to remove it completely due to the nuanced nature of misinformation that varies based on context, demographics, and culture (Pennycook and Rand 2019a; Lee 2019; Walter et al. 2020; Oeldorf-Hirsch et al. 2020) . In messaging services, such as WhatsApp, that provide end-to-end encryption of messages, directly filtering or flagging content may not even be possible. digital literacy users. 4 In the baseline, we measure the extent to which individuals are able to correctly identify fake news using a list of actual true and fake news stories circulated on social media. This is followed by the delivery of educational interventions in the treatment arms. In the first treatment (treatment 1), individuals are shown an informational video in the national language that educates them about common features of misinformation. In the second treatment (treatment 2), individuals are first shown the video and then given personalized feedback about their own responses to fake news stories shown at baseline. The feedback highlights the features of each fake news item that can enable the individuals to identify the news as fake.

We measure the effectiveness of our interventions through two endline surveys, by asking individuals about their beliefs regarding two additional sets of news stories circulated on social media. We also measure different ways users engage with the news including why they label a news item as true or fake, the degree of emotional arousal, intent to verify and share on social media. The endlines surveys were conducted after one week and then 4 to 6 weeks after the treatment was delivered. We were unable to directly observe online behaviors due to the high usage of WhatsApp an end-to-end encrypted platform (96%) and low usage of open platforms like Twitter (11%) in our sample. Tracking individuals on Facebook (46% usage in our sample) is also difficult due to privacy and safety concerns especially among female users (Younas, Naseem, and Mustafa 2020) . To assess the longevity and external validity of our treatments, we conducted a follow-up phone survey fifteen months after the interventions on assessing the accuracy of beliefs about news related to the COVID-19 pandemic as well as information-seeking and preventative behaviors during the pandemic.

Our results show that treatment 1 does not have any significant impact. However, we find that those who received treatment 2 were 0.14 standard deviations more likely to correctly identify fake news relative to the control group. There is no significant effect on correctly identifying true news. As a result, the overall effect of treatment 2 on all news is 0.076 standard deviations, which points to improved discernment of news. Those who received treatment 2 are 0.13 standard deviations less likely to report non-neutral (positive or negative) emotional reactions, relative to the control group. Furthermore, individuals who received treatment 2 are less likely to mention prior beliefs and more likely to mention features of the news, such as unreliable source, as the reason why they consider a news story to be fake or true.

We show through various robustness checks that experimenter demand effects are unlikely to explain the effect of treatment 2. For example, we do not find any effect on placebo news items (constructed fake news but with some features of true news) which suggests that respondents are not simply terming news as fake to please the experimenter. We are also able to reject the explanation that the treatment effect comes from randomly guessing more news items as fake. Furthermore, we do not find evidence of a larger impact in a sub-group who might identify news as fake in order to conform to the socially desirable trait of not falling for fake news.

More detailed results show that individuals respond heterogeneously to treatment 2. In particular, the effect of treatment 2 is driven by those with higher digital literacy and by the male respondents in our sample. Moreover, our results also show that there is limited impact on female respondents. We find that they are likely to label both fake and true news as fake which suggests no improvement in information discernment and increased skepticism even towards true news on social media.

Our phone survey on COVID-19 misinformation, which took place after the first wave of the pandemic in Pakistan, shows a high level of general awareness and accuracy of beliefs about COVID-19 related fake news in the entire sample. We find that general awareness masks any potential treatment effect. However, we find a 0.09 standard deviation improvement in the accuracy of beliefs in the treated individuals who have never seen or heard the news before. We also find evidence of more conservative information-seeking and sharing behaviors on social media and greater adoption of preventative health behaviors such as social distancing and mask ownership among the treated group relative to the control group. We interpret these findings to be indicative of long-lasting benefits of educational interventions.

To explain our findings, we present a simple conceptual framework of how individuals form beliefs about the accuracy of news encountered on social media. We model individuals as motivated reasoners, i.e., they like news conforming to their prior beliefs. 5 However, they are also able to apply analytical reasoning to form beliefs about the news, rather than relying on "gut feelings" or intuitions. 6 In our model, educational interventions decrease the cost of applying analytical reasoning by providing simple decision rules to evaluate the credibility of sources as well as indicators of problematic content without expending significant effort. 5 There is a large body of literature that models individuals as seeking to confirm prior beliefs to minimize psychological costs in the tradition of Lord, Ross, and Lepper (1979) , for example (Mullainathan and Shleifer 2005) , (Gentzkow and Shapiro 2006) . Empirical evidence on misinformation also shows that people are more likely to believe and share fake news that conforms to their prior beliefs (Allcott and Gentzkow 2017; Pogorelskiy and Shum 2019) . 6 According to dual-process theories, human cognition can be characterized by a distinction between intuitive, autonomous (System 1) processes and deliberative, analytic (System 2) processes (Kahneman 2011; Evans and Stanovich 2013; Pennycook, Fugelsang, and Koehler 2015) . In the misinformation literature, Pennycook and Rand (2019b) show that analytical reasoning improves accuracy of beliefs about the news.

Thus, as a result of educational interventions, individuals will be more likely to form correct beliefs about fake news, even if it conforms to their prior beliefs.

Our work is related to emerging literature in economics, psychology, political science, and related fields that seeks to understand the effectiveness of ways to reduce the impact and distribution of misinformation. Most of the literature has examined the effectiveness of behavioral interventions such as nudges, which are coupled with the presented news. For example, attaching warnings to news stories disputed by third-party fact-checkers (Pennycook, Bear, and Collins 2019) and using crowd-sourcing to generate trust ratings to differentiate reliability of sources have been found effective in increasing the accuracy of beliefs about misinformation (Pennycook and Rand 2019a) . Moreover, a general warning about misleading information on social media or attaching "Disputed" or "Rated false" tags to specific headlines was also found to be effective in reducing perceived accuracy of false headlines (Clayton et al. 2020) . However, an unintended consequence of these approaches is an increase in skepticism towards true news.

Despite this emerging literature, there is little evidence on the effectiveness of educational interventions that focus on teaching social media users how to recognize fake news, without nudges or attaching labels to the news itself, and especially in the context of low digital literacy users. We are aware of two contemporaneous media and information literacy evaluation programs carried out in developing countries. Badrinathan (2020) evaluates the effectiveness of in-person media literacy training on two tools for identifying misinformation: navigating a fact-checking website and reverse image searching using a low digital literacy sample from India. The paper finds no significant effect on the respondents' ability to identify misinformation. Similarly, evaluated the effectiveness of a social media literacy intervention in which individuals in United States and India were provided with tips to spot fake news. Their India sample consisted of high digital literacy users recruited online and a rural sample with very low familiarity with social media. The authors find a positive effect of giving tips online to highly educated social media users which declines over time. There is no effect in their rural sample for whom the tips are given verbally by enumerators.

Our paper adds to the literature on countering misinformation in developing countries among individuals with low digital literacy. Moreover, by evaluating two treatments we also provide evidence on which features of such programs are likely to be important in improving the ability to identify fake news. Our results show that despite their popularity, general educational messages do not produce any significant impact. Rather educational programs involving personalized feedback and guidance are more effective and longer-lasting for an average social media user in populations where digital literacy is lower relative to developed countries. Our findings also highlight the importance of customizing interventions from the gender perspective, as we show that females in our sample do not experience improved discernment between fake and true news. While we are not able to directly observe the online or offline behaviors of these respondents, the results from our phone survey on COVID-19 is suggestive of positive long-term effects on real-world behaviors in the context of the pandemic.

The remainder of the paper is structured as follows: section 2 outlines the context and describes the experiment, data, and empirical methodology; section 3 presents the results; section 4 presents a model to explain belief formation and impact of educational interventions; section 5 discusses our results and their implications; and section 6 concludes.

We choose to focus our study on social media users in Pakistan, a large developing country where approximately 18% of the population or 37 million people use social media. According to the 'Digital 2019: Pakistan' report (DataReportal 2019), social media penetration has grown from just 4% in 2013 to 18% in 2019. Many of these users are newly connected to the Internet through their smartphones. Facebook.com and whatsapp.com are among the websites with the largest average monthly traffic in the country (SimilarWeb 2019). Furthermore, opinion polls conducted with a nationally representative sample of Internet users show that 40% visit Facebook and YouTube to get news (Gallup 2019) .

Our sample is drawn from low and middle income areas of the city of Lahore; the second largest city of Pakistan with a population of around 11 million. We focus on low and middle income areas as we expect that digital literacy will be lower for these strata of the urban population. We recruit our sample from the field, instead of relying on online survey platforms that are less representative of low digital literacy users (Guess and Munger 2020; Hargittai and Shaw 2020) . We examine the effectiveness of two educational interventions using a cluster randomized design, where individuals within a small geographic cluster (of 100 m x 100 m) are assigned to either control, treatment 1 or treatment 2 status. In order to minimize the potential of spillovers within this small area, we carry out randomization at the cluster rather than the individual level. Our final sample consists of 150 geographic clusters, with 5 individuals per cluster, summing to a total of 750 individuals divided equally into control, treatment 1, or treatment 2. Online Appendix A contains details of the sampling, survey procedures, and randomization.

In the first visit, we carry out the baseline survey which includes demographic and social media use questions, followed by questions to measure digital literacy. We capture digital literacy using two types of questions (i) basic skills such as the ability to use social media without assistance, connect to WiFi and/or mobile data, use Google search and read English on social media, and (ii) specific skills related to social media applications measured by asking whether the individual is aware of and uses common application features. We ask about sixteen common WhatsApp and Facebook features (eight for each application). 7 These questions are consistent with prior work, which validated survey measures of digital literacy through participant observation, and found that composite variables of survey questions that measure users' knowledge of computer-and Internet-related terms and functions are good predictors of actual digital literacy (Hargittai 2005; Hargittai 2009 ).

After gathering these baseline characteristics, we measure beliefs and perceptions about news on social media. To do so, we ask a series of questions about a set of actual true and fake news stories drawn from social media. All individuals view the same set of stories, however, we randomize the order in which they are shown. In the baseline we show three true and fake news stories each, covering current events and general interest topics. The news stories are shown in the form of screenshots of messages, posts, or tweets, similar to how users would typically receive news on social media. Some of the news stories are in English only, while others are completely or partially in the national language, Urdu. It is fairly common for people to receive news in English as a large fraction of content on social media is in English. In order to make sure that the respondents understand the content, they are first asked to carefully view a printed version of the screenshot of the news and then asked to listen to an audio recording of the news. If the news was in English the audio translates it into Urdu. After viewing and hearing the news we ask a series of questions to ask whether they recalled seeing the news before, their beliefs about whether the news is true or not, their reactions, and their level of engagement with the news. Our main outcome of interest is the response to the question, "Right now do you think the news is true or not?" We use this question to create our primary outcome, which is an indicator equal to 1 if they correctly identify news, either fake or true.

In addition to the fake and true news stories, we also show one placebo (or constructed) news story. This story is similar in content and format to our list of actual fake and true news stories, but was not drawn from social media news. The responses on the baseline placebo news item are used to measure and control for false recall, that is, if an individual is more or less likely to say that they recall seeing the news before, just because they are part of a study or due to any design feature of our study. False recall might also reflect pure recall error on the part of the individual which would happen even outside the study setting. At the end of the baseline survey, individuals in the treatment arms receive one of the educational interventions described below. The baseline survey and delivery of treatments in the treatment arms is carried out by a pair of male and female enumerators.

After the baseline survey, individuals in the first treatment group watch a three-minute video about fake news in Urdu language. The video explains briefly and simply what is fake news, why it is generated, and the negative consequences of fake news. The remaining part of the video is dedicated to explaining the following three tips to identify fake news: (i) untrustworthy or missing source, for example, unknown author, unverified account, or absence of an authentic link supporting the news, (ii) poor quality of news, that can be spotted visually, for example, altered images or videos, informal or incorrect language, or excessive use of hashtags and emojis, and (iii) content or language that reflects bias, and tries to provoke extreme reactions and emotions. These tips are consistent with the Facebook's "Tips to Spot False News", which were developed in collaboration with the nonprofit First Draft and subsequently promoted at the top of users' news feeds in 14 countries in April 2017. A variant of these tips was later distributed by WhatsApp in newspaper advertisements in Pakistan and India in 2018 (Constine 2020; Guess, Lerner, Lyons, Montgomery, Nyhan, Reifler, and Sircar 2020) .

Individuals in the second treatment group watch the video and then receive personalized feedback based on how successful they were in identifying each of the three fake news items shown at baseline. The personalized feedback was delivered verbally by the enumerators using pre-designed and printed feedback cards. The feedback consisted of two components: (i) a score of 0 or 1, based on whether the individual correctly identified the fake news item as fake and (ii) drawing attention to the features of the fake news item, that is source, quality, or bias, which could be used to identify the news as fake. The feedback cards were designed to be encouraging and informational, ensuring that the individuals understand why they got something right or wrong. The feedback was delivered regardless of whether they identified the fake news item to be fake or true.

We conducted our baseline survey and the two endline surveys between May 2019 to July 2019. The first endline survey comprised six fake and three true news items, which were shown to the participant in the same manner as in the baseline. For each news item, we ask the same set of questions as in the baseline. In the first endline, we also include two placebo news items to assess experimenter demand effects (or Hawthorne effects) of our treatments (i.e., do the treatments make individuals act in a manner that they perceive is desired by the experimenter, which in this case is identifying news as fake). The second endline was carried out 4 to 6 weeks after the first endline. It consisted of five fake and three true news stories presented in a similar manner as the baseline and first endline.

We were able to reach 731 of the RCT participants in the first endline (response rate of 97.5%). Online Appendix Table A .1 shows that there are no systematic differences in the attrition rate across the three groups. 92% of the first endline surveys were completed by the fifth day after the baseline and the remaining were completed by the ninth day. We were able to reach 744 RCT participants (response rate of 99.2%). Overall, we find that attrition was random and not correlated with the treatment status.

In addition to these two endline surveys that were conducted in the same format as the baseline, we also conducted a follow-up phone survey in September 2020. We were able to successfully reach 82% of the participants who were part of our RCT. The response rate was balanced across the three groups (online Appendix Table A .1). The phone survey exclusively focused on news statements related to the COVID-19 pandemic. In this survey, we included seven fake news items and three true news items. For each news item we asked two questions (i) whether the individual had heard the news before, and (ii) to what extent would they agree with the news statement (ranging from agree, disagree to not sure). In this survey, our goal was to assess the extent to which individuals were affected by COVID-19 related misinformation, and whether our treatments had any effect in making them less susceptible to fake news.

In order to find the average treatment effect of each intervention on the probability of correctly identifying a news item as true or fake, we estimate the following equation:

where Y ik is the outcome of individual i for news items shown post treatment, X i is a vector of baseline controls (described later). T 1i is an indicator for treatment 1 and T 2i is an indicator for treatment 2. β 1 and β 2 are the average treatment effects for treatment 1 and treatment 2, respectively. The outcome variable is equal to 1 if the individual correctly identifies fake news as fake or true news as true, and 0 otherwise. We standardize this variable by using the mean and standard deviation of the control group so that treatment effect is estimated in terms of standard deviation changes. We also examine the effect of our interventions on how individuals engage with the news items. Specifically, we measure the effect on reporting extreme emotions. This variable is zero if the respondent reports feeling neutral, 1 if they report feeling positive or negative and 2 if they report feeling extremely positive or negative. We also study the effect on each of the following binary variables: whether the individuals say they will discuss the news with friends or family, search online about the news, and share news on social media, or do nothing. Since our interventions are designed to make individuals aware of common features of fake news, we also evaluate the effect on the reasons mentioned by the respondents for why they believe a news item to be true or fake. Specifically, we construct binary variables for whether an individual mentions prior political or religious beliefs, source problem, quality problem, or biased content, as reasons for why they consider a news item to be true or fake. We also standardize these outcomes by the mean and standard deviation of the control group. In addition, we examine the existence of heterogeneous treatment effects by the following baseline moderators: digital literacy, gender, age, and education.

We now describe some important characteristics of our sample. In terms of household characteristics, we find that the median household monthly expenditure was Rs. 30000 (approximately USD 200). 8 Households have 6 to 7 members with 2 to 3 members who are social media users. 9 Roughly half of our sample is female, the mean age is 29 years and the most common educational qualification is Intermediate (Grade 12).

In terms of social media use, 96% of our sample has a WhatsApp account, 46% has a Facebook account, 11% has a Twitter account, and spend approximately 3 hours per day on social media. The popularity of end-to-end encrypted messaging services like WhatsApp and low penetration of open platforms (e.g., Twitter) in our sample makes it challenging to 8 According to the Household Integrated Economic Survey 2018-19 conducted by the Pakistan Bureau of Statistics, the average monthly household expenditure in urban areas was Rs. 47362. The mean expenditures by quintile were Rs. 23515, Rs. 29130, Rs. 32931, Rs. 38689, and Rs. 64681 (HIES 2019) . 9 We define such a user as someone of age 18 or above having a WhatsApp or Facebook account.

directly observe online behaviors. Tracking individuals on Facebook is also difficult due to privacy and safety concerns especially among female users (Younas, Naseem, and Mustafa 2020) . In addition to using social media to view content, about two-thirds say they use social media to share content and 40% say they use social media to create content. When looking at digital literacy, we find that approximately 85% report being able to use social media without assistance, 83% say they can connect to WiFi or mobile data, followed by 78% reporting the ability to use Google search, and 63% report being able to read English text on social media. In our empirical work, we take the sum of these four binary indicators (standardized by the mean and standard deviation of the control group) as our measure of basic digital literacy skills, referred to as the basic digital literacy score. In addition, we also create indices using data on knowledge and usage of common WhatsApp and Facebook features. While the basic digital literacy score is effective at separating those at the low end and middle of the digital literacy spectrum in our sample, the latter indices are better at separating individuals at the high end. This is consistent with prior work that recommends having different measures for capturing variations at the low end and the high end of digital literacy (Guess and Munger 2020) . For our empirical work, we use principal component analysis to define separate scores for knowledge and usage of WhatsApp and Facebook features, standardizing these measures using the mean and standard deviation of the control group. We refer to these as the WhatsApp Score and Facebook Score.

In online Appendix Table A .2, columns 1 to 3, we report the summary statistics of our RCT sample by treatment status. In columns 4 to 6, we report the p-value for the pairwise difference in means across the groups, that is treatment 1 versus control (column 4) and treatment 2 versus control (column 5), and treatment 1 versus treatment 2 (column 6). Standard errors are clustered at the sample grid level. Of the twenty-four baseline characteristics reported, we find significant differences across the groups for only four characteristics. Participants from the control group have a greater likelihood of owning a smartphone (relative to both treatment groups). They are also more likely to report knowing more WhatsApp features (relative to both treatment groups). Participants in the second treatment group are more likely to falsely recall placebo news shown at baseline (relative to the control group), and likely to use fewer Facebook features (relative to the first treatment group). All other characteristics are not significantly different across the three groups.

Therefore, we conclude that our randomization worked and that the three groups are similar in terms of baseline socio-economic, demographic, digital literacy, and social media use variables. Table 1 shows the average treatment effect of our interventions on correctly identifying news. While we do not find any significant effect of treatment 1, treatment 2 improves the probability of correctly identifying fake news by 0.14 standard deviations relative to the control group (statistically significant at the 5 percent level) when the endlines are pooled. At endline 1 and endline 2, the effects were 0.12 and 0.15 standard deviations, respectively. The effect is larger but less precisely estimated in the second endline. The precision is improved by adding baseline controls to the regressions. In Section 3.2 we perform various checks to examine the extent to which these effects are driven by experimenter demand effects, that is respondents are labeling news as fake since as they are trying to please the experimenter.

Columns 3 and 4 in Table 1 show the average treatment effect on the sample of true news items only. We do not find any significant effect of either treatment on correctly identify true news. One plausible explanation for no effect on true news could be that individuals were generally aware of the true news in both the treatment and control groups and that explains why we do not find any significant effect of the treatment. However, across both the treatment and control groups, the majority of individuals do not recall seeing the true news items before (65% in the control group and 70% in the treatment 2 group) suggesting general awareness about true news does not drive our results. However, the negative coefficient of the treatment variable for true news (although insignificant) might be due to experimenter demand effects or increased skepticism towards social media news. We explore these explanations of the results in Section 3.2 and 3.4.

In columns 5 and 6 we report the effect on all news items which captures the overall effect of the treatment in improving discernment between true news and fake news. This can be regarded as a stronger test of whether our treatment causes individuals to be more discerning social media users. In the results with both endlines pooled together and baseline controls, we find a positive effect of 0.076 standard deviations (significant effect at 10%) of treatment 2. This effect is dominated by improvements in correctly identifying fake news, which points to improved discernment. The results are similar in magnitude (although less precisely estimated) without controls and separately across the endlines.

The effects of treatment 2 on identifying fake news found in Table 1 , compare favorably with previous literature on similar educational interventions. For example, ) find an effect size of 0.11 in their online convenience sample in India and no effect on a rural sample that received tips to spot fake news in person. Similarly, Badrinathan (2020) does not find any significant effect. The effect in this study is also larger than the effect size of 0.08 found in (Clayton et al. 2020) , as a result of giving general warnings about misinformation on social media users in the US recruited from MTurk. Table 2 shows the effect of the treatments on secondary outcomes measuring engagement with the news items. Those who received treatment 2 are 0.13 standard deviations less likely to report non-neutral (positive or negative) emotional reactions (statistically significant at the 5 percent level), relative to the control group. This suggests that treatment 2 may have had a mitigating effect on initial reactions to news due to increased awareness about the nature of fabricated news on social media. They are also 0.14 standard deviations more likely to say that they will do nothing after hearing the news (significant at the 10 percent level), 0.16 standard deviations less likely to say that they will discuss the news with friends and family (significant at the 5 percent level). We do not find any effect on searching online for the news or intent of sharing it on social media. Table 2 also includes the sharpened q-values using the two-stage procedure described in (Anderson 2008) , to adjust p-values for multiple hypothesis testing. All outcomes that have p-values below 0.05 or between 0.05 and 0.1 have sharpened q-values below 0.10. In additional results available upon request, we find that there are similar effects for each of these outcome variables when looking at fake and true news separately. This suggests that after receiving the treatment, individuals are less likely to feel strong emotions, less likely to discuss the news with others, but there is no change in behavior related to verification or sharing of the news.

Since our educational interventions focused on teaching individuals specific ways to identify fake news, in Table 3 , we estimate the effect of the treatments on the reasons stated for labeling news as true or false. We find that individuals are 0.11 standard deviation less likely to mention prior beliefs (statistically significant at the 5 percent level) and 0.14 standard deviation more likely to mention a source problem as the reason for their answer (statistically significant at the 10 percent level). We also find that for outcomes with p-values less than 0.1, the sharpened q-values were 0.107. This suggests that treatment 2 helps individuals in engaging more analytically with news (e.g., by helping them examine the characteristics of the presented news such as its source). If we separate the effect by fake and true news, we find that the effect on prior beliefs and source are driven by fake news items, which further suggests that individuals are learning and applying what was taught in the second intervention.

In this section, we examine the extent to which the effect of treatment 2 might be driven by experimenter demand effects. Since our intervention explains to participants the negative consequences of fake news, it might leave an impression that the researchers are expecting participants to identify news items as fake. Thus, in the endline surveys, participants are likely to identify news as fake, in order to satisfy the researchers. However, outside the experimental setting, individual behavior may not necessarily change. Such changes in behavior among the treatment group are known as experimenter demand effects (or Hawthorne Effects) (Duflo, Glennerster, and Kremer 2006; Zizzo 2010) . We examine the plausibility of experimenter demand effects in explaining the observed treatment effect.

Our first robustness check involves adopting a modified version of the approach used in (Dhar, Jain, and Jayachandran 2018) , to zoom into a sub-sample of our experimental subjects in which social desirability trait may be dominant. Specifically using our baseline survey data, we identify individuals who answered 'no' when asked if they have ever fallen for fake news. If the participants sense that the experiment is about correctly identifying fake news, and they would like to please the experimenter, they might be more likely to respond 'no' to this question. These individuals should also make a greater effort to identify news as fake in the endline, and the treatment effect should be driven by this sub-sample. 44% of our respondents fall into this category. Table 4 shows that the additional treatment effect for this sub-sample is small and insignificant as compared to the treatment effect on the remaining sample for fake news items. For true news we would expect a larger negative coefficient for the social desirability sample if respondents were identifying true news as fake due to experimenter demand effects. We do not observe any evidence for such effects. Overall, we do not find any evidence of a significant differential response to the treatment in the social desirability sample when pooling together all news items.

As a second robustness check, we carry out a falsification test to ascertain whether treatment 2 just increased the probability of labeling more news items as fake without making the treated individuals' judgements more accurate. In this test, we simulate a treatment group in which participants randomly identify news items as fake. Note that the base rate of calling any news item fake was 59% in treatment 2 compared to 54% in the control group. We create three simulated dependent variables in which the responses of control and treatment 1 are kept intact, whereas the responses of treatment 2 individuals are replaced by the simulated responses with the rate of saying fake fixed at 59%, 75%, and 90%, respectively. We standardize these dependent variables by the mean and standard deviation of the control group.

We find that if participants in treatment 2 had been randomly guessing 59% of the news items as fake, the treatment effect on fake news would have been significantly negative, as respondents would have correctly identified fewer items as fake than the control group. If we raise the rate of saying fake in the treatment group to 75%, then we observe a treatment effect on fake news comparable to the actual treatment effect. However, in this case, the discernment of true news is significantly lower than that in the control group, as the simulated treatment individuals are also labeling more true news as fake. Finally, if we raise the rate of saying false in the treatment group to 90%, there is a large positive effect on fake news but a significantly larger negative effect on true news. Thus, the results in Table 5 support the interpretation that treated individuals are not randomly guessing about the accuracy of news items, in response to being primed about fake news or trying to act in conformance with the experimenters objective.

Finally, we also analyze the treatment effect on placebo news items. These are synthetically constructed fake news items, which were never circulated on social media or any other media. There are two placebo news items shown to respondents in the first endline survey, as part of the list of news fake and true stories. Each placebo news item contained highly implausible news but also included an (invalid) Web link highlighting the source of news, which could only be checked for accuracy by visiting the link. If respondents were simply terming more news items as fake, then we would expect a positive treatment effect on placebo news, that is respondent identifies news as fake more often than the control group. While around 90% of respondents (in both control and treatment groups) termed placebo news items as fake, we find a small negative effect (though not significant) on correctly identifying placebo news as fake news (see online Appendix Table A .3). We do not find any effect on secondary outcomes such as engagement with the news or reasons for terming news as fake. This suggests that experimenter demand effects are unlikely to explain treatment 2 effects.

Overall, these three robustness checks provide evidence against the explanation that experimenter demand effects are driving the treatment effect.

We now investigate the existence of heterogeneous treatment effects by four baseline characteristics: digital literacy, gender, age, and education. Figure 1 displays the heterogeneous effects, for each sub-group defined by these characteristics, separately for fake, true, and all news pooled together. We discuss these results below.

Our primary measure of digital literacy is the basic digital literacy score, which is an aggregation of the four questions used to capture Web-use, technical skills as well as the ability to understand and evaluate information online (as described in Section 2). We standardize the basic digital literacy score by the mean and standard deviation of the control group. Figure 1 shows the treatment effect estimates for individuals with high and low digital literacy (defined using one standard deviation above and below the mean respectively). Notes: The figure shows the effects of treatment 2 by digital literacy score, gender, age, and education. High and low digital literacy is defined as one standard deviation above or below mean digital literacy score. Median age is 27 years and median education level is Intermediate/Grade 12. All regressions include standard controls. Error bars show 95% confidence interval, with clustered standard errors.

The effect on the sub-group of low digital literacy individuals is positive at 0.07 standard deviations but not statistically significant. On the other hand, the effect on the sub-group of high digital literacy individuals is almost twice as large as the average treatment effect.

There is no effect on true news for both sub-groups. Overall, the treatment effect is positive and significant for high digital literacy individuals when we pool all news together, which suggests improved discernment between true news and fake news. In online Appendix Table A .4, we further assess the heterogeneity of treatment effect by two additional measures of digital literacy, namely the WhatsApp and Facebook scores which capture application-specific knowledge and usage. These indices are effective at capturing the variations across individuals with high basic digital literacy scores. We find that treatment 2 has a positive and significant effect on individuals with higher WhatsApp scores. The interactions of treatment 2 (and even treatment 1) indicators with the score are positive and statistically significant. We do not find any significant differential effect by Facebook score.

Facebook users only comprise half of our sample and they are likely to be more educated and more experienced social media users, which can explain why the effects are no different for those with higher Facebook scores.

Separating the treatment effect by gender shows that males benefit more from the treatment, both in terms of recognizing fake news and distinguishing between fake news and true news. The effect on fake news for females is smaller relative to males and statistically insignificant, while the effect on recognizing true news is negative, suggesting greater skepticism in females regarding the accuracy of news (see Figure 1 and online Appendix Table  A .5).

One reason for the lower take-up can be that females also had lower levels of digital literacy. 10 While females have the same years of formal education as males and 69% of them report being able to read English on social media versus 57% of the males, on other dimensions of digital literacy they lag behind males. For example, 75% report being able to connect to WiFi and mobile data (versus 91% of males), 74% report being able to use online search (versus 83% of males), and 82% report being able to use social media without assistance (versus 88% of the males). Thus females are likely to have less experience in navigating social media and assessing the credibility of information encountered online.

To check whether low digital literacy could be mitigating the treatment effect for females, we check for heterogeneous effects by digital literacy for females. Online Appendix Table  A .6 shows that the treatment effect using the interaction between treatment and digital literacy score for the female sample only. We find positive interactions of both treatment indicators for fake news items, suggesting greater ability of high digital literacy females to recognize fake news. For true news, we observe that the second treatment had a negative and significant effect which is an average effect. There is no significant difference for females by digital literacy. Taken together, these additional results suggest that although higher digital literacy females are more likely to correctly identify fake news, on average there is also increased skepticism towards true news by females relative to the control group. We further confirm this finding in 3.4.

To study the average treatment effect by age and education, we divide our sample using the median age and education which is 27 years and Intermediate/Grade 12, respectively. We find suggestive evidence of larger effects on fake news for younger and more educated sub-groups, no significant effect on true news, and no significant improvement in the overall ability to distinguish true and fake news (see Figure 1 and online Appendix Table A .5) . 11 Taken together, these results suggest that there is scope for designing customized educational interventions that are specifically targeted towards users at the lowest end in our sample in terms of digital literacy and females. For these sub-populations, the existing interventions may be complex and ineffective. As a result, designing interventions that require even lower cognitive effort and prior training in analytical reasoning can help them discern fake news better.

While there does not seem to be evidence for experimenter demand effects driving the treatment effect, respondents may still be terming news as fake without improvement in discernment and simply due to increased skepticism towards news on social media. To assess this, we separate the treatment effect due to the overall change in the probability of labeling a news item as fake from the effects due to the change in correctly identifying fake news items as fake. Specifically, we run a regression in which the dependent variable is equal to one if a respondent labels a news item as fake and the treatment indicator interacted with the news item being actually fake.

In column 1 of Table 6 where we pool all fake and true news items from both endlines, we find that on average the second treatment effect does raise the likelihood of labeling news as fake. However, the effect is not significant. The differential effect on fake news is 1.5 times larger than the average effect but also not precisely estimated. The total effect on fake news, is thus due to both an average increase in probability of terming a news item as fake and a differential effect. This is consistent with our previous findings in Table 1 , where we find suggestive evidence of negative effects on true news and Figure 1 , where we show that females are driving the negative and significant effect on true news. Therefore, in columns two and three of Table 6 , we separate the treatment effect on males and females. We find that males are not just labeling more news as fake as a result of the treatment. The entire effect of the treatment on males is driven by improved discernment of fake news. However, for females the opposite is true. There is no additional effect on correctly identifying a fake news item as fake. This lends support to the interpretation that male respondents became more discerning about fake news as a result of receiving the second treatment. Females do not experience any improved discernment and are just as likely to label true news items as fake. Moreover, we do not find any differential effect for females exhibiting social desirability trait (see online Appendix table A.7). These results are suggestive of increased skepticism about news on social media amongst females after receiving the treatment, rather than purely experimenter demand effects.

We now analyze the effect of our interventions by the characteristics of the news items shown in the endline surveys. We classify the news items along four dimensions related to the content and format of the news. To study the effect of content, we create an indicator equal to one if the news had political content. If individuals already hold prior beliefs about the topic, then we should expect the effect of our treatment to be mitigated for those individuals whose prior beliefs conform with the fake news. We also create a second indicator which is equal to one if the news is related to personal health, safety, and food. We group these topics as they convey information that can lead to immediate action by the individual. For these types of news items, we expect that if the individual takes the wrong action based upon the misinformation then they can incur substantial physical or psychological costs. We term these high-cost news items. We formalize these costs in our conceptual model discussed in Section 4. To study the effect of news format, we create indicators for whether the news was conveyed in Urdu (as opposed to English), and whether the news contained only text (relative to a combination of text and image). Table 7 shows that the treatment effect is indeed smaller for political news, suggesting that prior beliefs do have an offsetting effect. The treatment effect is larger for news items which we term as the high-cost news items. Furthermore, we also observe that individuals in the control group are more likely to fall for such types of fake news as compared to all other types of fake news shown in the endlines. 12 This is a novel insight as it shows that people are more likely to fall for misinformation on matters that touch upon their everyday lives and behavior, but there is also greater scope for teaching people how to correctly identify such misinformation. Table 7 also shows a smaller treatment effect for news that was presented in Urdu and news that was presented in text-only format. The latter finding suggests that non-textual elements of news can help individuals discern the accuracy of the news.

To assess the longevity of our treatment effect and effects on behaviors outside the experimental setting, we conducted a follow-up phone survey focusing on assessing beliefs and behaviors related to the COVID-19. The phone survey was conducted in September 2020, fifteen months after the interventions. We were able to reach 89.3% of the respondents in our original RCT sample. Online Appendix Table A .1 shows that attrition was random and unrelated to the treatment status of the individual. In this survey we assessed the accuracy of beliefs about news related to COVID-19. We asked people about seven popular fake news items and three true news items related to COVID-19. For each news item, individuals were first asked whether they had heard the news before. Then they were asked to state to what extent they agreed with the statement. They had the option to agree, disagree, or say they were unsure. In addition, respondents were also asked questions about information-seeking and preventative behaviors during the pandemic.

The survey was conducted in September 2020, when the infection rates had declined considerably in Pakistan from the peak daily rate of around 7000 cases to 500 daily cases (COVID-19 Pakistan 2020). We found that all three true news items (related to handwashing, wearing a mask, and physical distancing) were universally known and everyone surveyed had heard them before. Moreover, 98% of the individuals had accurate beliefs about the true news items. For the seven fake news items, we found that 81% recalled having heard the news before, and 79% had accurate beliefs, that is they disagreed with the fake news. Therefore, it appears that even though misinformation related to COVID-19 had been circulating widely, by the time we conducted our phone survey, a large fraction of individuals had formed accurate beliefs about such news, potentially leaving little room for a treatment effect to be observed if one exists. 13 In Table 8 we estimate the effect of our treatments on the accuracy of beliefs about COVID-19 related news. Accuracy is equal to 1 if the individual correctly disagrees with a fake news item, 0.5 if they say they are not sure and 0 otherwise. Similarly, for true news, accuracy is equal to 1 if the individual correctly agrees with a true news item, 0.5 if they say they are not sure and 0 otherwise. As before, we standardize the dependent variable using the mean and standard deviation of the control group. Columns 1, 3, and 5 show the 13 Several large-scale COVID-19 awareness campaigns were launched by the Government of Pakistan in collaboration with various organizations using print, electronic, radio and social media across the country (NHSRC-Pakistan 2020; UNICEF 2020). From March 19 and May 7 in 2020, 1028.5 million COVID-19 awareness messages were sent in Urdu, English, and regional languages to mobile users across Pakistan. Among these included using Ring Back Tones (RBTs) to spread awareness messages, which were activated on 131.7 million subscribers' mobile devices (Khurram Ali Mehran 2020). According to a nationally representative survey conducted by Gallup Pakistan, ringtone messages had a positive impact on knowledge, perception, and behaviors related to COVID-19 in Pakistan (Javeria Masood and Umar Taj 2020) treatment effect on fake, true, and all news items respectively using our main specification with baseline controls. We do not find any significant effect of the treatment on the accuracy of beliefs. Given that our follow-up was conducted after the end of the first COVID-19 wave, was our sample (in control and treatment) generally aware of the accuracy of news related to COVID-19, and therefore, there is no effect of our treatment? For true news, all news items were recalled by everyone whereas 81% of individuals had heard about the fake news items, signifying the general prevalence of public health information related to the pandemic. A related explanation for this prevalence is that the high personal cost (e.g., harmful health effects) associated with not having accurate beliefs about COVID-19 led individuals in both treatment and control groups to exert greater effort to form accurate beliefs when they heard such news. Another possibility is that the treatment effect simply faded over time.

To help differentiate between the above two explanations, and assess whether general awareness is playing a role for fake news, we estimate the treatment effect separately for two sub-groups: those who recall and do not recall hearing the news before. Since recall is a post-treatment variable, we first examine whether recall rates of COVID-19 news items are affected by the treatment in online Appendix Table A .8. We do not find any significant effect of either treatment on the probability of an individual recalling a COVID-19 related fake news. This indicates that awareness of COVID-19 news was generally prevalent in our sample and the interventions did not have a differential effect on recall, thereby making it balanced across the groups. This gives us confidence that we can study the treatment effects on COVID-19 fake news by recall. The results are reported in column 2 and 5 of Table 8 for fake news and all news respectively.

The coefficient on the recall variable is negative, which indicates that in the control group individuals who had previously heard a fake news item had less accurate beliefs relative to those who do not recall seeing a news item before. Among the individuals who recalled having heard the fake news before, it is likely that some of them had formed correct beliefs, but there were others who had formed inaccurate beliefs which were not updated over time. These might be individuals who have strong prior beliefs which are consistent with the fake news and therefore, they are less likely to form correct beliefs (see Section 4) . There was no significant difference across treatment and control groups within this sub-group. This suggests that effects due to general awareness about fake news had permeated both groups, but there were some individuals who did not update their beliefs possibly due to strongly held prior beliefs.

Furthermore, we also find that among those who did not recall hearing the fake news item before, treatment 2 improved their accuracy of beliefs by 0.09 standard deviations relative to the control group (significant at the 10 percent level). This suggests that the effect of treatment 2 on those who did not recall seeing a fake news item before, or in the absence of general awareness, though smaller, likely persisted over time.

Next, we assess whether the treatments had any effect on self-reported informationseeking and sharing behaviors of respondents related to COVID-19 (see Figure 2 ). We construct an information behavior index using a list of five survey questions that measure the frequency of use of formal news sources (World Health Organization and government sources) and social media news sources (WhatsApp, Facebook, Twitter, and YouTube) for obtaining information about COVID-19, trust in these sources for providing reliable COVID-19 related information, and frequency of sharing COVID-19 information on social media. We code these variables such that the index takes increasing values the higher is the frequency of use of formal sources, trust in formal sources increases, the lower is the frequency of use of social media sources and trust in social media, and lower is the frequency of sharing COVID-19 related information on social media. Each of these variables is standardized using the mean and standard deviation of the control group. We then take the sum of the standardized variables to create the index. The questions corresponding to each of these measures and their coding is explained in online D. We find that respondents in the treatment 2 group use formal sources more, have greater trust in formal news sources, and are less likely to share COVID-19 information on social media relative to the control group. There is no difference in frequency of use and trust in social media sources. When combined as an index, we find that the difference in the means of the index for treatment 2 and control is positive (0.26 standard deviations) and significant at the 10 percent level. As shown in 2, we find that the distribution of the information behavior index is shifted rightward for the treatment 2 group relative to the control group. This reflects greater usage and trust in formal sources to obtain COVID-19 related information and a lower frequency of sharing COVID-19 related information on social media.

Finally, we construct a preventative behavior index to measure respondents' willingness to observe (self-reported) COVID-19 preventative behaviors. We use a list of four survey ques-tions that measure behaviors related to mask ownership and social distancing. Specifically, we use questions asking about ownership of different types of masks, the market price of different types of masks, interacting with people outside the house (in close proximity within 2 arm's length), willingness to send children to school if they reopen. Each of these variables is coded such that it takes increasing value the more preventative is the behavior. For price, we take higher prices as corresponding to higher willingness to pay and thus greater inclination towards preventative behaviors. The questions corresponding to each of these measures and their coding are reported in online D. We standardize individual variables by the mean and the standard deviation of the control group and aggregate the standardized variables to create a preventative behaviors index.

We find that individuals in the second treatment group are likely to own masks, observe social distancing, and have a lower willingness to send children to school. However, their reported market price of masks is lower than that of the control group. When combined as an index, we find that the distribution of preventative behaviors is rightward shifted and the mean of the index is higher for the treatment 2 group relative to the control group although the difference in means is not statistically significant ( Figure 3 ). Overall, we find suggestive evidence that individuals in the treatment 2 group are more likely to engage in preventative behaviors. Therefore, we conclude that while it is challenging to make a definitive conclusion about the effectiveness of treatment 2 on the perceived accuracy of COVID-19 related fake news, there is evidence to suggest that treatment 2 had an impact on information-seeking and preventative behaviors related to COVID-19.

In this section, we propose a conceptual framework to explain how educational interventions change an individual's accuracy of beliefs regarding news seen on social media. Our conceptual framework combines motivated reasoning and analytical reasoning, which are commonly used to explain how individuals assess information and form beliefs. We provide evidence consistent with the model's predictions using the data from our experiment.

We develop a conceptual framework to model factors that determine how individuals form beliefs about the accuracy of news encountered on social media. Two broadly accepted influences on the beliefs individuals develop about the accuracy of news include motivated reasoning and analytical reasoning. The effects of motivated reasoning on different forms of judgement have been widely documented and supported by prior studies (Mercier and Sperber 2011; Kahan 2013; LaFollette and Woodruff 2015; Kahan, Peters, Dawson, and Slovic 2017; Bavel and Pereira 2018) . For example, individuals are more likely to uncritically accept claims that support their political ideology compared to evidence that is inconsistent with their prior belief (Strickland, Taber, and Lodge 2011) . Thus, motivated reasoning can at least partly explain why individuals may be susceptible to fake news, by accepting stories that conform to their prior beliefs. Another growing body of work in the tradition of dual-process theories of human cognition argue that deliberative and analytical reasoning can lead to sound judgement (Kohlberg 1969; Evans 2003; Stanovich 2004; Shtulman and McCallum 2014; Pennycook, Fugelsang, and Koehler 2015; Pennycook and Rand 2019b) . Thus, applying analytical reasoning can increase the accuracy of beliefs about fake news and true news.

Our framework combines these views by modelling individuals as motivated reasoners, i.e., they like news conforming to their prior beliefs but are also able to apply analytical reasoning to form beliefs about the news. As a result, we show that an individual can correctly identify fake news even if it perfectly conforms to her prior belief if the cost of not finding the 'truth' is high or the cost of analytical reasoning required to discern the truth is low. For example, an individual will expend more time and effort verifying a fake news item related to COVID-19 (which may have serious health implications for the individual) compared to a fabricated story about the chances of rain during the week. 14 Similarly, individuals having higher levels of digital literacy, who can engage in fact-checking, will pay a lower cost for applying analytical reasoning as they are likely to arrive at the truth with less time and effort. In both cases, individuals will form correct beliefs about the fake news even if it conforms to their prior beliefs.

A key insight of our model is that by raising digital literacy or lowering the cost of analytical reasoning, an individual's accuracy of beliefs about news can be improved. Therefore, in our model educational treatments work by lowering the cost of analytical reasoning by providing individuals with simple decision rules to evaluate the credibility of news without expending significant time and effort.

We build upon (Mullainathan and Shleifer 2005) to develop a simple model to explain how individuals determine beliefs about a news story. Individuals receive a news item, N , about a topic of interest (e.g., state of the economy, COVID-19), which can be true or fake, and individuals form beliefs about the accuracy of the news item. We assume that N is drawn from the distribution N (0, v). For simplicity, we assume that both true and fake news items are drawn from the same distribution. Individuals will have accurate beliefs about N whenever they can correctly identify a fake news item as fake and a true news item as true. We also assume that individuals' prior beliefs, B, about the underlying topic follow the distribution N (b, v) , where b = 0 captures the expected bias in their prior beliefs. The individual gets a baseline utilityū from reading the news item, N . Individuals like to read news that confirms their beliefs, therefore, they incur a disutility when N = B. The disutility increases with the square of the distance between the news and the individual's beliefs with an elasticity of φ. Thus, the direct cost of reading news inconsistent with one's beliefs is φ(N − B) 2 .

Individuals can exert analytical reasoning to determine the accuracy of the presented news. Analytical reasoning depends on cognitive effort and productivity, which captures how good the individual is at reasoning given the effort exerted. Individuals with high digital literacy are expected to have higher productivity and thus achieve a high level of reasoning with a low amount of effort. In order to identify fake news correctly, individual needs to exert a high level of analytical reasoning. Conversely, if the individual exerts a low level of analytical reasoning, they form inaccurate beliefs about the news. The level of reasoning required to form correct beliefs will vary across fake and true news and within each category also vary with news characteristics. We denote cognitive effort by a i j , where i ∈ {H, L} and j ∈ {T, F } corresponds to true news (T ) and fake news (F ). Effort is costly and the individual incurs a total cost ξ(a i j ) 2 , for exerting a i j units of effort to determine the accuracy of the news. Note that the marginal cost of effort increases with ξ, while the productivity of effort declines with ξ.

We assume that individuals care about knowing the truth. So when they form correct beliefs by applying a high level of analytical reasoning, they receive the benefit of knowing the truth. Benefit has a fixed component and a variable component, which depends on the degree of inconsistency between the news and prior belief. Specifically, we assume that the benefit of correctly identifying news is δ (1 + (N − B) 2 ) if the news is fake and δ (1+ (N −B) 2 ) if the news is true. These terms reflect that individuals derive the maximum benefit if their prior belief is confirmed after reasoning. Therefore, for fake news, the benefit is lowest if N = B (i.e., when prior beliefs are consistent with fake news but the news was correctly identified as fake). For true news, the benefit is maximized if N = B (i.e., when prior beliefs are consistent with the true news item that was identified as true).

If the individual has incorrect beliefs, they incur the cost of not knowing the truth. This can be a psychological or physical cost that arises due to inaccurate beliefs such as harmful actions taken due to believing and acting upon misinformation. For fake news, the cost is modeled as χ(1 + (N − B) 2 ), which is increasing in the distance of news from prior belief. For true news, we assume that this cost is χ (1+(N −B) 2 ) , which is decreasing in the distance of news from prior beliefs.

With this setup, let U i j be the utility of concluding that the news item is i ∈ {T, F }, when the underlying state is j ∈ {T, F }.

The utility of concluding that a fake news item is fake is given by:

And the utility of incorrectly concluding that a fake news item is true is given by:

Note that in this case, the individual incurs the cost of not knowing the truth, which increases with the distance between and N and B. There is no additional cost of overcoming bias because the reader believes the fake news in this case. Similarly, the utility of concluding a true news item as true is given by:

Note that when the news is true and it aligns with the individual's prior belief, no additional cost is incurred. Moreover, the derived benefit is decreasing in the distance between the news and the prior belief. And finally, the utility of concluding a true news item as fake is given by:

Let p be the probability that a user will correctly identify a fake news item as fake. This is the probability that utility of concluding that a fake news item is fake exceeds the utility of concluding that a fake news item is true.

shows that p, the probability of correctly identifying fake news as fake, is increasing in A and decreasing in both δ and χ. Online Appendix B shows that using the underlying distributions of N and B, the probability of correctly identifying a fake news can be written as:

Next, we discuss the predictions of the model regarding the probability of correctly identifying fake news, as news characteristics and analytical reasoning are varied. We also examine the consistency of the model predictions with the experimental data. 15

We examine how the probability of correctly identifying fake news changes with three different characteristics of the news which are captured in our model, (i) the expected distance between news and the bias of the reader (ii) cost of not knowing the truth, and (iii) benefit of knowing the truth. Our results are stated in Propositions 1 to 3. See online Appendix B for all proofs. In Table 7 , we examined how the treatment effect varies with news characteristics. Here we draw the link between features of the model and our findings. First, we found a relatively small effect for political news. We expect that if individuals have strongly held prior beliefs 15 We do not analyze P (U T T > U F T ) but it can be solved in the same manner as P (U F F > U F T ).

in terms of their political leaning, this is likely to shift the mean of the belief distribution farther away from the mean of the news distribution. In the model, this translates into an increase in the bias in prior belief, which decreases the probability of correctly discerning political fake news. Second, we also found that the treatment effect is larger for news items related to health, safety, and food. We expect that believing fake news on such topics can have immediate negative effects if the individual takes the wrong action. Thus, such news items are likely to have a higher χ and therefore we observe individuals are more likely to identify these types of fake news.

Proposition 4.4. As ξ decreases the probability of correctly identifying fake news increases.

Proposition 4.5. As the difference between (a H F ) 2 and (a L F ) 2 increases, the probability of correctly identifying fake news decreases.

In our model, ξ is inversely proportional to the productivity of effort applied for reasoning. Our educational intervention improves the productivity of effort and decreases the marginal cost of applying analytical reasoning by providing simple decision rules to evaluate the credibility of sources as well as indicators of problematic content. Moreover, by raising productivity, we also expect that the amount of effort required to correctly identify fake news should also decrease. This implies that the difference between (a H F ) 2 and (a L F ) 2 decreases, thereby increasing p. This is the short-run effect of our interventions. In the long run, educational interventions can also make people less biased and cause them to change their prior beliefs.

In order to understand how effort changes as a result of our intervention, we use response time as a proxy for how much effort is devoted to answering the questions related to the beliefs about news items. We expect that if the intervention decreased ξ and a H F , then the response time for each of the news items should fall. Data on response time by news item are available from time audit sensor data collected by the survey application during the interaction with the respondents. We are able to match all of the respondents in the first endline and 714 of the 744 respondents in the second endline to time audit data files available from the survey application. To exclude outliers, we trim the top 1% observations and exclude negative response time observations. For the remaining sample, we check whether the treatment had any significant effect on the time taken by an individual to respond to the questions asked about each news item.

In Table 9 , we investigate the effect of our treatment on response time for questions related to the news items shown at both the endline surveys. The mean response time for all news items in our sample is 93.7 seconds and the standard deviation is 83.6 seconds. In Panel A we report the effect for fake, true, and all news items separately. We find that treatment 2 reduced the response time by 0.073 standard deviations relative to the control group (statistically significant at the 10 percent level) and this effect is larger (although not significant a the 10 percent level) for fake news items. This suggests that treatment 2 works by increasing the productivity (or lowering the cost) of analytical reasoning by giving individuals simple decision rules to help them discern between fake news and true news. As the cost of analytical reasoning declines, individuals are more likely to have correct beliefs. For example, Rubenstein (2013) argues that when presented with difficult or complex tasks, invoking simple decision rules can reduce response time as well as mistakes especially when such rules are correlated with making fewer mistakes, e.g., simple rules for spotting fake news. However, when the notion of a mistake is clear cut (e.g., counting the letter F in a paragraph) then short response times can lead to greater mistakes.

Next, we separate the effect of treatment on response time separately for correct and incorrect responses. Panel B shows the effect of treatment on those who correctly identified a news item. Again we find suggestive evidence of a reduction in the time spent by the second treatment group (although not significant). It is important to note that individuals in the control group who are correctly identifying fake news are likely those with high digital literacy or whose prior beliefs are inconsistent with the fake news. Therefore, they are likely to spend less time answering and also more likely to correctly answer. Despite the composition heterogeneity, we find suggestive evidence that on average, individuals in the second treatment group are spending less time when they correctly identify fake news.

In Panel C, we study the sample with incorrect responses. We find that treatment 2 had a negative effect on the time spent answering fake and true news related questions (significant for true news). Again there are likely to be composition differences in the type of individuals in the control and treatment groups who answer incorrectly. Individuals in the treatment group who answer incorrectly are likely to be those for whom the news conforms to their prior beliefs. Or this group may also comprise very low digital literacy individuals who experience smaller benefits of the treatment. In both cases, individuals in the second treatment group spend less time analyzing the news (relative to the control group) and form incorrect beliefs.

In this section, we discuss the differences between our two educational interventions, the scaling up and external validity of the second treatment.

Our results show that treatment 2 has a positive effect on improving individuals' ability to identify fake news. However, educating users about misinformation through general educational messages does not have a significant effect on improving their beliefs about the accuracy of fake news. There are two features of treatment 2 that distinguish it from treatment 1: (i) personalized feedback, which depends on an individual's own performance in the baseline test, and (ii) guidance provided by the enumerators who point out different features of misinformation.

Personalized feedback has been shown to be more effective in drawing attention than general informational messages (Moon 2002; Thaler and Benartzi 2004; DellaVigna and Gentzkow 2010; Ayres, Raseman, and Shih 2012; Doss, Fahle, Loeb, and York 2018) . Moreover, corrective feedback can facilitate learning and boost long-term memory (Hattie and Timperley 2007) and its timing can play an important role in enhancing learning (Brashier, Pennycook, Berinsky, and Rand 2021) . In order to assess the importance of feedback, we estimate the effect of treatment 2 by baseline scores. Table 10 shows that as baseline score increases, the effect of treatment 2 decreases. This suggests that feedback was more valuable for those with lower baseline scores, while those with higher baseline scores experienced smaller gains.

Another feature of treatment 2 that can partly explain its effectiveness is the guidance provided by enumerators while delivering feedback (e.g., pointing out different features of misinformation). Such guidance (or instruction) becomes more effective when the complexity of the presented material increases (Kirschner, Sweller, and Clark 2006; Jung, Kim, Yoon, Park, and Oakley 2018) . As the median education level in our sample is Grade 12, some users may have found the video message complex or difficult to understand. In fact, our sub-group analysis shows a larger effect on more digitally literate users, which is consistent with the view of complementarity between prior knowledge and new learning.

An important consideration for the delivery of educational interventions is the ease with which they can be delivered at scale. Treatment 2 can potentially be delivered at scale on social media platforms like Facebook and Twitter. First, rather than showing fake and true news items to users, social media platforms can provide retrospective feedback to users based on their engagement with news items that later turn out to be fake. Social media platforms regularly use fact-checking services on their platforms which can aid in this regard (WhatsApp 2021). Second, a video can be shown to users on their news-feed in their own language. Finally, guidance can be provided by showing users the features found by factchecking services (e.g., unverified source, altered images), which can be scalably converted into audio messages using text-to-speech and speech translation systems.

An important question about the external validity of our educational interventions is regarding the extent to which the effectiveness of treatment 2 can be generalized to a different setting. For example, individuals may learn how to identify fake news only when they are presented exactly the way we taught them (i.e., our interventions are "teaching to the test"). To mitigate such concerns, we include news on a range of topics and in a variety of formats that are different than the three baseline fake news items shown to participants. We also conduct two endlines post-treatment and a phone survey fifteen months after the intervention. Our results suggest that the effect of treatment 2 was present across a range of news items presented in different formats and on behaviors likely to be affected by belief in COVID-19 related misinformation. The effect seems to have persisted even after a significant time had elapsed after the treatment was delivered.

However, we acknowledge that our randomized evaluation compares the difference between treatment and control group in a specific geographical area, population, and context, and our unit of observation is not large enough to pick up any general equilibrium effects. Therefore, we do not know the effectiveness of our interventions if implemented in a different context and at a larger scale (Meager 2019) . Moreover, we do not observe real behaviors on social media, as our focus in this paper was low digital literacy users who are not well represented on platforms such as Twitter where it is easier to observe sharing and engagement with news. Therefore, going forward, it would be useful to replicate such a study at a larger scale, and with outcome measures that are reflective of online and offline behavior in various settings where people receive and process information (e.g., politics, health, economics, etc).

Finally, in this work, we do not capture the welfare effects of our interventions such as implications of being more skeptical about news in general, impact on personal well-being and mental health, and the effect on social or political outcomes such as trust, coordination, and polarization. Measuring the welfare effects of such programs is an important area for future work.

As digital spaces become an important avenue for news consumption, it is important to understand how we can minimize the impact of misinformation on individuals and society. This is an even more challenging problem in contexts where individuals with limited digital literacy skills are joining online platforms in increasing numbers. Educational interventions focusing on increasing the ability of individuals to identify fake news offer a promising approach to combat misinformation, though evidence on their effectiveness is limited.

In this paper, we designed and evaluated the effectiveness of two educational interventions to counter the spread of misinformation in a randomized control setting. Our findings show that educating individuals about how to recognize misinformation through a general video-based message does not have any significant effect. However, showing the educational video and providing personalized feedback about individuals' past engagement with misinformation, and guidance on how to recognize fake news, significantly increases the probability of identifying fake news by 0.14 standard deviations and both fake and true news by 0.076 standard deviations. We find that while there is no effect on identifying true news for males, females are likely to become skeptical even about true news. We conduct several robustness checks to show that experimenter demand effects are not driving our results.

Our findings have important policy implications for combating misinformation in the digital space among low digital literacy populations. We show that while educational programs hold promise, the design and delivery of such programs matter. While general educational messages delivered once do not produce any significant effect, personalized educational messages with feedback and guidance are successful in helping individuals learn how to identify misinformation. However, we do find evidence of lower take-up among females and less digitally literate individuals, which highlights the importance of designing programs that appeal to these segments and help them learn effectively.

Despite these contributions, we acknowledge that our experiment is carried out in a particular setting, and we do not observe actual online and offline behavior. We are also cognizant that there are other dimensions on which educational programs can be customized for effective learning, for example, perceived trustworthiness of who delivers the message, interactivity, and repetition. These remain important areas for future work. Finally, while educational efforts to improve digital literacy are certainly not a panacea to the misinformation problem, they can prove to be a vital way to reduce vulnerability to fake news and thereby improve the informational well-being of individuals and society at large.

Ethics and IRB Review. Our study was approved by the IRB at Lahore University of Management Sciences with protocol number LUMS IRB/03252019).

Funding. We gratefully acknowledge support by the Facebook Integrity Foundational Research Award 2019 from Facebook Research in the form of an unrestricted gift and the Lahore University of Management Sciences (LUMS) Faculty Initiative Fund grant (FIF 597-2020-ECO). https://www.pk.undp.org/content/pakistan/en/home/blog/2020/ innovative-ringtone-messages-positively-impacts-knowledge--perce.html. Notes: Sample is all fake and true news items shown post-treatment in the two endlines. All regressions include baseline controls and endline wave dummy. Standard errors clustered by sample grid in parentheses. * p<0.1, ** p<0.05,*** p<0.01. P-value is the unadjusted p-value and q-value is the sharpened False Discovery Rate-adjusted two-stage q-value. Notes: Sample is all fake and true news items shown post-treatment in the two endlines. All regressions include baseline controls and endline wave dummy. Standard errors clustered by sample grid in parentheses. * p<0.1, ** p<0.05,*** p<0.01. P-value is the unadjusted p-value and q-value is the sharpened False Discovery Rate-adjusted two-stage q-value. Notes: Sample is all news items shown in two endline surveys. All regressions include baseline controls and endline dummy. Standard errors clustered by sample grid in parentheses.* p<0.1, ** p<0.05,*** p<0.01. 

Panel A: Label 59% of news items as fake Notes: Sample is all news items shown in two endlines. Dependent variable is equal to one if respondent labels a news item as fake, standardized by mean and standard deviation of control group. All regressions include baseline controls and endline dummy. Total Effect is the sum of coefficient on treatment indicator and interaction between treatment and fake news. The indicator for treatment 1 and interactions with fake news are not significant. Standard errors clustered by sample grid in parentheses.* p<0.1, ** p<0.05,*** p<0.01. Notes: Sample is news items in the COVID-19 phone survey. See text for details about the construction of the dependent variable. Recall is an indicator equal to 1 if recalled seeing or hearing news before. Columns 2 and 5 estimated with interactions of treatment variables with recall. All respondents report recalling true news, and interactions are omitted. All regressions include standard controls. Standard errors clustered by sample grid in parentheses. * p<0.1, ** p<0.05,*** p<0.01. Notes: Dependent variable is standardized to the the mean and standard deviation of control group. All regressions include baseline controls and endline dummy. Standard errors clustered by sample grid in parentheses. * p<0.1, ** p<0.05,*** p<0.01. Notes: High baseline score is an indicator for above median baseline score. Sample is all news items shown in first two endlines. All regressions include standard controls. Treatment 1 and its interaction is not significant. Standard errors clustered by sample grid in parentheses. * p<0.1, ** p<0.05,*** p<0.01.

We implement our survey and randomized control trial (RCT) in the city of Lahore in Pakistan which has a population of approximately 11 million. In order to draw our sample of low and middle income households, we rely on population density data. The pattern of urban development in Lahore is horizontal, and dwelling size is inversely correlated with income levels, therefore, we use population density as a proxy for income levels. Specifically, we use the (AsiaPop 2013) satellite data which provides population counts at a spatial resolution of 100m by 100m, to identify the low and middle income areas of Lahore from where we draw our sample. The selected the areas of Lahore account for 35% of the city's total population and cover seven of the fourteen national assembly constituencies (see online Appendix Figure  A .1). These areas cover the older parts of the city where the median population density is 109 persons per 100m by 100m grid. 16 As a comparison, the median population density in the areas of Lahore not covered by our sample is 28 persons per 100m by 100m grid.

We draw a random sample of 200 grids from the selected areas and survey five households per grid, covering a total of 1000 households in the baseline survey. In order to initiate the data collection within the chosen grids, we randomly dropped a point (x and y coordinate) within the grid. The enumerators arrived at the point and used the left hand rule to survey, within each grid, five households where at least one social media user is present. The definition of social media user for our study is that the respondent must be at least 18 years of age and use either Facebook or WhatsApp.

The RCT sample consists of 150 grids drawn randomly from the baseline grid sample. We randomly assigned 50 grids each to control, treatment 1, and treatment 2 status. In order to ensure gender balance, we also pre-specified the number of male and female participants to be drawn from each grid for the RCT (either 2 or 3 out of a total of 5 individuals per grid point). At the start of the baseline survey, the enumerators were informed (by the survey application), whether a male or female respondent was to be selected for the experiment. If the required gender was not present, the household was replaced. This gives us a sample of 250 social media users per treatment arm, and 750 social media users altogether drawn from 750 unique households, who become part of our experiment and are visited again at endline. Online Appendix Figure A 

The baseline survey is answered by all social media users present in the household at the time of the survey. The survey consists of the following sections:

1. Household and demographic characteristics including household size, monthly expenditures, number of social media users, age, gender, and highest education level. Age is measured as a continuous variable, while education is categorized by highest level completed (0=no education, 1=primary, 2= middle, 3= secondary/grade 8, 4=matric/grade 10, 5=intermediate/grade 12, 6=bachelors, 7=masters, 8=above masters).

2. Social media use including phone ownership and sharing, social media applications used, number of hours spent per day using Facebook and WhatsApp, size and composition of the social media network, top three news sources, and sharing of news on social media. We also ask if they have heard of the term "fake news" and if they ever believed a news that later turned out to be false.

3. Digital literacy measured by self-reported ability to turn on mobile data, WiFi, use Google search, use social media without assistance, read text on social media, and read English and Urdu on social media. We also ask about knowledge and use of common features of WhatsApp and Facebook.

After the end of the baseline survey, we show each social media user a series of news items. We compiled a list of seven recent popular true and fake news stories, covering current events and general interest topics. Out of these three are true news stories and three are fake news stories. We also created one placebo news item which is used to measure and control for false recall. The news stories are shown in the form of screenshots of messages, posts, or tweets, similar to how users would typically receive news on social media. Some of the news stories are in English only, while others are completely or partially in Urdu. The respondents are first asked to carefully view a printed version of the screenshot and then asked to listen to an audio recording of the news in Urdu. After having viewed and heard the news, we ask the following questions:

• Have you seen this news before? (Yes/No)

• If yes then did you think this news was true the first time you saw it? (Yes/No)

• Right now do you think that this statement is true? (Yes/No)

• When you saw this news how did you feel? (Very positive, positive, neutral, negative, very Negative).

• When you saw this news what did you want to do? (Nothing, ask friends and family about it, search online, other)

• Why do you consider the news to be true or fake? (unprompted and select all relevant options from: aligns with my political beliefs, aligns with my religious beliefs, it has the right source (link/report/video/article), it uses professional language, the picture is real, it is not biased, other)

• Will you share the news on social media? (Yes/No)

The follow-up survey was carried out in September 2020 over the phone. Participants were paid Rs. 100 as mobile credit for completing the survey. The survey includes questions about sources of information about news related to COVID, beliefs about a set of popular COVID related news statements, and questions on preventative behaviors. In order to measure their beliefs about COVID related news, we create a set of ten news statements (seven fake and three true). The statements are read out by the enumerator on the phone and the participant answers two questions about each statement,

• Have you seen or heard this statement before? (Yes/No) • To what an extent do you agree with the statement? (Agree/Disagree/Not Sure) Our dependent variable, which is belief about the accuracy of the news statement, takes the value 1 if the individual has correct beliefs about the news statement (that is agrees with true news and disagrees with fake news), 0 if they have incorrect beliefs, and 0.5 if they answer not sure.

Proof of Proposition 1:

Proof. To show that p increases with A = (a H F ) 2 − (a L F ) 2 , we take the partial derivative of p with respect to a, which gives the following expression:

where D = b+ ξA δ+χ −1 √ 2v

, and φ(.) is the probability density function of the standard normal random variable. ∂p ∂A < 0 because φ(.) > 0 and we assume D > 0. Thus p is decreasing in A.

Proof. To show that p increases with χ, we take the partial derivative of p with respect to χ, which gives the following expression 17 :

where D = b+ ξA δ+χ −1 √ 2v and φ(.) is the probability density function of the standard normal random variable. ∂p ∂χ > 0 because φ(.) > 0 and we assume D > 0. Thus p is increasing in χ.

Proof. To show that p decreases with the expected bias b, we take the partial derivative of p with respect to b, which gives the following expression:

17 We obtain the same partial derivative with respect to δ, the benefit of knowing the truth.

where D = b+ ξA δ+χ −1 √ 2v

, and φ(.) is the probability density function of the standard normal random variable. ∂p ∂b < 0 because φ(.) > 0 and we assume D > 0. Thus p is decreasing in b.

Proof. To show that p increases by decreasing ξ, we take the partial derivative of p with respect to ξ, which gives the following expression: The map obtained from the Pakistan Bureau of Statistics (PBS) shows the National Assembly constituencies in the city of Lahore. The constituencies from which the sample is drawn are highlighted in red.

The map shows the grid points (geographic clusters) included in the RCT sample, coded by treatment status. Notes: Mean and standard deviation (in parentheses) of the response rates by endline and treatment status is shown in columns 1-3. Difference in means and p-value of the difference (in parentheses) are shown in columns 4-6. Standard errors are clustered by sample grid. * p<0.1, ** p<0.05, *** p<0.01. Notes: Columns 1 to 3 show the mean and standard deviation of the variable by control, treatment 1, and treatment 2. Columns 4 to 6 report the p-value of the difference in means of the three groups compared pairwise. Education levels are measured on a categorical scale from 1-9 corresponding to no education, primary, middle, matric (grade 10), intermediate (grade 2), bachelors, masters, above masters, diploma. WhatsApp (Facebook) features aware and features used is the the fraction of features that the respondent is aware of and fraction of features used. Baseline false recall is equal to 1 if the respondent says yes to recalling placebo (constructed) news item. Notes: Sample is only placebo news items shown in first endline. All regressions include baseline controls. Standard errors clustered by sample grid in parentheses.* p<0.1, ** p<0.05,*** p<0.01. Notes: Sample is all news items shown in the two endlines. Standard errors clustered by sample grid in parentheses. See text for details about construction of Basic, Whatsapp and Facebook scores. Treatment effect on high digital literacy group is the total effect of treatments one standard deviation above the mean score. Out of a total of 750 individuals, the number of individuals who use WhatsApp and Facebook are 720 and 348 respectively.* p<0.1, ** p<0.05, *** p<0.01. Notes: Sample is all news items shown in the two endlines. Standard errors clustered by sample grid in parentheses. Moderators are indicator for female, above median age (27 years), and above median education (Intermediate/Grade 12). The total treatment effect for subgroups female, above median age and above median education is reported below the interaction effects. * p<0.1, ** p<0.05, *** p<0.01. Notes: Sample is only females and news items shown in the two endlines. All regressions include standard controls. Digital literacy score is standardized by the mean and standard deviation of the control group. The total effect of each treatment on the subgroup is reported. Standard errors clustered by sample grid in parentheses. Notes: All regressions include baseline controls and endline dummy. Standard errors clustered by sample grid in parentheses.* p<0.1, ** p<0.05,*** p<0.01. Notes: Dependent variable is equal to 1 if the individual recalls seeing or hearing the news from before. Column 1 includes all COVID-19 fake news included in the phone survey. COVID-19 true news is omitted as recall is 100%.. In columns 2-4, the sample is fake,true, and all news items shown posttreatment in the two endline survey. * p<0.1, ** p<0.05, *** p<0.01. Table A .9 Survey questions corresponding to the information behavior index and the preventative behavior index.

Polarization and public health: Partisan differences in social distancing during the coronavirus pandemic

Social media and fake news in the 2016 election

Multiple inference and gender differences in the effects of early intervention: A reevaluation of the abecedarian, perry preschool, and early training projects

University of Southampton). Pakistan 100m Population. Alpha version 2010 and 2015 estimates of numbers of people per grid square, with national totals adjusted to match UN population division estimates

Evidence from Two Large Field Experiments that Peer Comparison Feedback Can Reduce Residential Energy Usage

Educative interventions to combat misinformation: Evidence from a field experiment in india

The partisan brain: An identity-based model of political belief

Cross-country trends in affective polarization

Timing matters when correcting fake news

More than just a nudge: Supporting kindergarten parents with differentiated and personalized text-messages

Real solutions for fake news? measuring the effectiveness of general warnings and fact-check tags in reducing belief in false stories on social media

The New York Times. Twitter and Facebook Lock Trump's Accounts After Violence on Capitol Hill

Facebook puts link to 10 tips for spotting 'false news' atop feed

COVID-19 Health Advisory Platform by Ministry of National Health Services Regulations and Coordination

Persuasion: Empirical Evidence

Reshaping Adolescents' Gender Attitudes: Evidence from a School-Based Experiment in India. NBER Working Papers 25331

More than just a nudge: Supporting kindergarten parents with differentiated and personalized text-messages

Using randomization in development economics research: A toolkit

In two minds: dual-process accounts of reasoning

Dual-process theories of higher cognition: Advancing the debate

Gallup and Gillani Pakistan Opinion Poll

Media bias and reputation

Fake news on twitter during the 2016 u.s. presidential election

A digital media literacy intervention increases discernment between mainstream and false news in the united states and india

Digital literacy and online political behavior

Survey measures of web-oriented digital literacy

An update on survey measures of web-oriented digital literacy

Comparing internet experiences and prosociality in amazon mechanical turk and population-based survey samples

The power of feedback

Household Integrated Economic Survey (HIES

The influence of instructional design on learner control, sense of achievement, and perceived effectiveness in a supersize mooc course

Ideology, motivated reasoning, and cognitive reflection

Motivated numeracy and enlightened selfgovernment

Thinking, fast and slow

PTA'S Contributing Efforts to Tackle COVID-19

Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching

Stage and sequence: The cognitive-developmental approach to socialization

The righteous mind: Why good people are divided by politics and religion

BBC. Matter of fact-checkers: Is Facebook winning the fake news war?

Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence

Understanding the Average Impact of Microcredit Expansions: A Bayesian Hierarchical Analysis of Seven Randomized Experiments

Why do humans reason? arguments for an argumentative theory

Personalization and personality: Some effects of customizing message style based on consumer personality

The market for news

Fanning the Flames of Hate: Social Media and Hate Crime

National Action Plan for Corona virus disease (COVID-19) Pakistan

The ineffectiveness of factchecking labels on news memes and articles

The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings

What makes us think? a three-stage dual-process model of analytic engagement

Fighting misinformation on social media using crowdsourced judgments of news source quality

Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning

News we like to share: How news sharing on social networks influences voting outcomes

Response time and decision making: An experimental study

The Washington Post. How misinformation on WhatsApp led to a mob killing in India

Cognitive reflection predicts science understanding

Top Websites Ranking

The robot's rebellion: Finding meaning in the age of darwin

Motivated reasoning and public opinion

Save more tomorrow: Using behavioral economics to increase employee saving

Partner to Create Awareness around COVID-19

Fact-checking: A meta-analysis of what works and for whom

Digital in 2020

IFCN Fact Checking Organizations on WhatsApp

Patriarchy and social media: Women only facebook groups as safe spaces for support seeking in pakistan

Political effects of the internet and social media

Experimenter demand effects in economic experiments

Information behavior index (1) How frequently are you getting information from each of the following sources about the coronavirus through any medium (including reading online, watching on TV, etc.)? [For each source: The World Health Organization (WHO) and Government Sources

How much trust and confidence do you have in each of the following sources when it comes to reporting about the coronavirus? [For each platform: The World Health Organization (WHO) and Government Sources (e.g., covid.gov.pk, press briefings, etc); Responses: A lot

How frequently do you use the following social media platforms for receiving/sharing news about the coronavirus?

How much trust and confidence do you have in news about the coronavirus received through the following social media platforms?

Preventative behavior index (1) On average how many people do you interact with outside your house

On a scale of 1-5, to what extent do you agree or disagree with the following statement: 'I will send my children back to school tomorrow if they re-open

What type of mask do you have? (can select multiple options)

In your opinion, what is the market price of each of the following types of masks?