key: cord-0639286-vry590rx authors: Zheng, Chengbo; Ma, Xiaojuan title: Evaluating the Effect of Enhanced Text-Visualization Integration on Combating Misinformation in Data Story date: 2022-05-24 journal: nan DOI: nan sha: eeb40ac80ad9761d14c2d6a3d6d01e2957c386e7 doc_id: 639286 cord_uid: vry590rx Misinformation has disruptive effects on our lives. Many researchers have looked into means to identify and combat misinformation in text or data visualization. However, there is still a lack of understanding of how misinformation can be introduced when text and visualization are combined to tell data stories, not to mention how to improve the lay public's awareness of possible misperceptions about facts in narrative visualization. In this paper, we first analyze where misinformation could possibly be injected into the production-consumption process of data stories through a literature survey. Then, as a first step towards combating misinformation in data stories, we explore possible defensive design methods to enhance the reader's awareness of information misalignment when data facts are scripted and visualized. More specifically, we conduct a between-subjects crowdsourcing study to investigate the impact of two design methods enhancing text-visualization integration, i.e., explanatory annotation and interactive linking, on users' awareness of misinformation in data stories. The study results show that although most participants still can not find misinformation, the two design methods can significantly lower the perceived credibility of the text or visualizations. Our work informs the possibility of fighting an infodemic through defensive design methods. In recent years, the proliferation of misinformation on online media has become a paramount public concern. During the COVID-19 pandemic, misinformation, such as ingesting bleach as a coronavirus treatment, is a serious threat to public health [49] . While it is widely known that false or inaccurate information can be disseminated through text and/or images, visualizations appear to be another increasingly popular vehicle for misinformation [13, 34, 45] . Due to its persuasive power [44] , visualization may be an even more dangerous yet harder to detect tool to manipulate people's opinion. A typical example of misinformation in visualizations is the inverted axis, which breaks the conventional direction of the axis and makes readers receive a reversal message to the facts [45] . Prior research has investigated misinformation existed in standalone visualization, e.g., bar chart [31, 45] , thematic maps [12] , etc. However, few studies are concerned with how misinformation can be introduced in narrative visualization. One popular form of narrative visualization as presented in Fig. 1 consists of rich text providing contextual information and some key piece(s) of fact expressed by data visualization [33] . It is commonly known as illustrated text [55] (also magazine style [51] ). While existing research efforts on narrative visualizations mainly concern facilitation of its creation [52] , Figure 1 : (A) [40] : Narrative visualizations combining rich text and visualizations in online mass media. (B) [1] : Combine text and visualization to tell stories on social media there is a pressing demand for in-depth understanding of how problematic design can mislead readers and bring misinformation. Moreover, there lacks effective methods that can help lay public stay aware of misinformation in narrative visualizations. Previous works have explored how to uncover visualization mirage in visual analytics process [37] and how to reveal and repair chart errors [7] . Nevertheless, the interplay between text and visualization in narrative visualizations is rarely considered. A common approach to combat misinformation is performing automatic or manual factchecking [56] . While the number of fact-checking services and algorithms for textual information is growing rapidly [56] , not many of them could efficiently and cost-effectively detect misinformation in narrative visualizations. Facing the aforementioned challenges, in this paper, we begin by asking, "How might misinformation be introduced into narrative visualization?". To this end, we first conducted a literature survey and derived a pipeline that describes the data storytelling production-consumption process. Then, we analyzed every stage of this data storytelling pipeline to identify possible causes of misinformation and then grouped these causes into three categories: author-induced reader-undetectable misinformation (AIRUM), author-induced reader-detectable misinformation (AIRDM) and reader-induced misinformation (RIM). Next, we proceed to explore possible methods to combat misinformation. Countering AIRUM and RIM might involve moderating the behavior of authors and readers of data stories [15, 37] , whereas mitigating AIRDM could be achieved by better scrutinizing the presented data stories. We thus focus on AIRDM in this paper. Besides, as the first step towards combating misinformation in data stories, we focus on the magazine-style of narrative visualization, which is popular in journalism [57] . We target two typical issues for the magazine style under AIRDM : (1) deceptive visualization [45] , and (2) text-visualization misalignment [27] . As discussed above, there are still technical barriers to detecting misinformation in data stories. As an alternative, we are interested in exploring whether there exist some defensive design methods to enhance people's awareness of misinformation, which can prevent readers from falling for problematic data stories in the first place. [48] : (A1) Highlighted word, (A2)Visualization element corresponding to (A1). When reader hover (A1), (A2) will be highlighted by some visual cues. (B) Explanatory annotation [42] : (B1) Visualization elements of interest, (B2) Text that explain (B1). This idea is inspired by the fact that different framing of the narrative visualization can result in different interpretations towards the same data story [22] . Previous studies show that relying on one-sided information to understand data stories may partially result in the unawareness of deceptive visualizations and text-visualization misalignment [27, 31] . Motivated by this observation, in our study, we considered candidate design methods that can enhance the integration of text and visualization, thus balancing readers' interaction with textual and visual information. To this end, We choose two commonly used and easy-to-promote design methods: interactive linking and explanatory annotation [29, 30, 57] . Interactive linking (denoted as Linking in the rest of the paper) highlights the corresponding explanatory visual elements when selecting specific sentences or words in the text (e.g., Fig. 2(A) ). Explanatory annotation (denoted as Annotation) positions interpretative notes extracted from the text close to the corresponding visual elements [5] (e.g., Fig. 2(B) ). Both candidates bridge text and visualization, two critical components to data stories, either statically (Annotation) or dynamically (Linking) and guide readers' attention to regions of visualizations that are central to the storytelling. To evaluate the effectiveness of enhanced text-visualization integration, we designed and conducted a crowdsourcing study on Amazon Mechanical Turk with 222 participants to measure whether Linking and Annotation can affect people's awareness of misinformation and perceived credibility of text and visualization. We carefully selected three data stories on popular topics that are flooded with misinformation as study materials. For each story, we edited the original material to inject one piece of misinformation using one of the three methods identified in previous research: truncated axis (a case of deceptive visualization), inverted axis (another instance of deceptive visualization), and text-visualization contradiction (a kind of text-visualization misalignment) [27, 45] . Our experimental results showed that Linking and Annotation do enhance readers' awareness of misinformation and significantly lower the perceived credibility of the text or visualizations in the corresponding stories. However, we also noticed most participants are still unaware of misinformation. For participants who did not find misinformation in the given stories, we analyzed their subjective feedback on possible reasons and proposed design opportunities to combat misinformation. Misinformation is commonly defined as any false or inaccurate information [56] . Interpretation of "information" in this definition, however, may change in different scenarios, as it could mean facts provided or facts learned [53] . Kong et al. remove this ambiguity by thinking from the readers' perspective and define misinformation as "information that leads to misperceptions about the facts" [27] . In this paper, we follow the definition by Kong et al. and consider incorrect or misleading information readers receive and consume. Despite the different research angles, there is a general consensus on the negative impact of misinformation [56] . Therefore, methods to combat misinformation in the digital world are gaining increasing attention from the research community and the general public. The most straightforward anti-misinformation approach is designing algorithms to automatically detect false or inaccurate facts disseminated online. Researcher have developed algorithms that utilize content data (e.g., text, image), context data (e.g., published time or location), and propagation data (e.g., retweet and likes on social media) to detect misinformation [56] . Still, there are concerns about the generality of current detection methods [20] , and few existing algorithms take visualizations as input into their analysis. In the visualization area, researchers have studied misinformation in the forms of cognitive bias, visualization mirage, deceptive visualizations, etc. [37, 45, 54] . Several works have experimented with means to combat such misinformation in ordinary visualizations. For example, McNutt et al. [37] propose a metamorphic testing method to surface visualization mirage in visual analytics process. Different from previous research, our work focuses on misinformation occurring in the production-consumption process of data stories and explore design alternatives to combat misinformation that come from information misalignment. Visualizations have been widely used to tell stories about data. Segal and Heer [51] introduced the term "narrative visualization" to cover such type of visualizations. Despite the varieties in genres, narrative visualizations generally combines (textual) narratives with interactive graphics [51] . Prior research around narrative visualization can be divided into three categories. The first type of research focuses on exploring the design space of a particular genre or aspect of narrative visualization. For example, Bach et. al [2] conclude design patterns for data comics. The second type of research seeks to develop algorithms and systems to facilitate the authoring process of narrative visualizations. Examples include Calliope [52] which automatically generates data stories from spreadsheets. Kong et al. [28] also proposed a crowdsourcing-based method to create interactive articles. The third type of research investigates factors that influence readers' experience to motivate better design of narrative visualizations. For instance, McKenna et. al [36] look into how the reading flow of data stories affects readers' engagement. Our work is close to the last type of research and aims to investigate factors that might influence readers' awareness of misinformation in data stories. We are particularly interested in factors concerning narrative visualization design. Given the considerably large design space, Hullman and Diakopoulos [22] have proposed a framework to analyze what can affect users' interpretation of narrative visualization, which consists of four editorial layers, i.e. data, visual representation, textual annotation and interactivity. Among them, design techniques applied on the annotation and the interactivity layers can promote reader's digestion of the story content and assist in awareness building by enhancing the connection between text and visualizations within a data story [47] . For example, Kwon et al. [29] propose a technique that dynamically links text with relevant visualizations. Such a technique can effectively guide readers through the author's intended narratives. Zhi et al. [57] investigate two forms of text-visualizationintegration methods -layout and linking. They found slideshow layout with linking can improve readers' recall of and engagement with a narrative visualization. Wang et al. [55] compared different narrative visualization types and found increasing text-visualization integration can promote reader's understanding. Our study focuses on two design methods to connect textual and visualization components in data stories, i.e., Annotation and Linking. Both of them are commonly used in designing magazine-style narrative visualization. Moreover, there exist fast algorithms [28, 30] to implement these two design methods, which make them easy to promote. McNutt et al. propose a pipeline of visual analytics to investigate how errors generated in each phase may undermine messages conveyed in the final visualization [37] . We are inspired to adopt a similar approach to obtaining a comprehensive view of the relationships between critical narrative visualization creation steps and the types of misinformation that can be introduced to mislead readers. To this end, we first construct a pipeline that dissects the productionconsumption process of data storytelling (covered in subsection 3.1) and then conduct a literature survey to identify possible sources of misinformation (following the definition in subsection 2.1) at each stage of the pipeline (detailed in subsection 3.2). Our pipeline is mainly adapted from Lee et al.'s visual storytelling process [33] while also motivated by previous works about visual analytics [8, 37] . As we are more concerned with how possible falsified information injected in the production-consumption process and lead to readers' misperceptions of facts, we follow McNutt et al.'s [37] practice and omit complex cycles and nested sub-process [8, 33] to enable easy tracing of the origin of misinformation. Our pipeline describes the path from raw data to the perceived message by five main steps, namely, Analyzing, Visualizing, Scripting, Arranging, and Reading (see Fig. 3 ). We further conceptualize the production process as steps involving authoring actions and the consumption process as steps of which the output will be directly perceived by readers. According to this, Analyzing is only related to the production process and Reading is only associated with the consumption process. Visualizing, Scripting, and Arranging are involved in both the production and the consumption processes. The detailed definitions of these steps are as follows: Analyzing denotes the process that authors analyze the raw data to extract data facts that they intend to convey [8, 33] . In this step authors determine what to tell (and not to tell) in the final data story. Visualizing characterizes the process that authors create charts to visually communicate the complied data facts to readers [37] . Note that the charts here refer to the graphics readers actually see in the final data story, not the ones that are used to analyze the raw data. Scripting indicates the process that authors organize multiple data facts and their corresponding charts into a certain narrative structure, including ordering, designing interaction, and messaging [51] . The goal of this stage is to combine the scattered fragments of facts into a complete story to convey author-driven narratives [22, 34, 51] . Arranging indicates the process that authors lay out multiple pieces of texts and/or charts in a scene and build the visual narrative flow for the readers [8, 36] . Note that Arranging is concerned with the format of the data story but not the content. Reading indicates the process that readers decoding information of the data story including the text and visualizations. The final message that readers derive is a result of the interplay between the presented information of the data story and their inherent knowledge, purpose and beliefs [6, 17] . By searching for related publications in TVCG, CHI, and other relevant venues, we synthesize in detail 17 number of the most relevant literature. We identify possible misinformation causes in every step of the pipeline proposed above from the literature. We present these causes and corresponding literature in Table. 1. Besides, we also consult works from journalism and psychology to survey more possibilities of causes of misinformation. Next, we put the resulting list of possible causes of misinformation in the context of narrative visualization production and consumption and divide them into three categories: author-induced reader-undetectable misinformation (AIRUM), author-induced reader-detectable misinformation (AIRDM) and reader-induced misinformation (RIM). Author-Induced Reader-Undetectable Misinformation By AIRUM, we mean false or inaccurate information injected into the production process through manipulations invisible to readers. AIRUM mainly occurs in the Analyzing step. For instance, authors might only pick those data that support their intention or even falsify data to meet their claims [10] . Readers with external knowledge about the data story and its context might be able to find such AIRUM; however, it cannot be corrected based only on the information provided by the authors. Author-Induced Reader-Detectable Misinformation We define AIRDM as miscommunication rooted in the production process but can be detected by careful readers. Such misinformation might concern Visualizing, Scripting, and Arranging steps as the intersection of the production and consumption processes illustrated in Fig. 3 . Authors' decisions in these steps are directly reflected in the output, and skeptical readers can spot the misleading elements. Deceptive visualization is a well-known cause of AIRDM in the Visualizing step. The author may truncate the axis or take area as the quantity that misleads readers intentionally or unintentionally [45] . Readers can capture the existence of such misinformation by comparing the difference(s) between the deceptive chart and the conventional ones. Text-visualization misalignment is a typical example of AIRDM that occurs in the Scripting step [27] . The author may manipulate the text-chart correspondence to (mis)guide the reader's attention, but a vigilant reader can find misalignment through close inspection. Reader-Induced Misinformation RIM refers to the misperception that happens during the consumption process and depends on how the visual argument(s) are read in real-world situations. This kind of misinformation relates primarily to the Reading step. The value, belief, and the information exposure environment of an individual could all cause misreadings [19, 38] . RIM impact readers differently. Authors' decisions in the production process can change the tendency of a message being more deceptive or not, but cannot fully determine whether the story will mislead readers [49] . Combating AIRUM requires a more rigorous authoring method (e.g., metamorphic testing [37] ). To fight RIM, promoting education on data and visualization is one of the directions currently under consideration by academics [15] . As for mitigating AIRDM, a plausible approach is presenting the data story in a way that allows readers to scrutinize it easier [49] . Therefore, in the scope of this paper, we focus on AIRDM and conduct a crowdsourcing study to explore whether different narrative visualization designs may affect readers' awareness of such misinformation. We designed a crowdsourcing user study to explore the possible effects of two widely-used design methods in narrative visualizations, explanatory annotation (Annotation) and interactive linking (Linking), on readers' awareness of misinformation in data stories. The misinformation we tested in the study is the type of author-induced reader-detectable misinformation. More specifically, two types of deceptive visualization [45] and one type of text-visualization misalignment [27] were used. Table 1 : Possible causes of misinformation in every step of the production-consumption of narrative visualization. This table is not exhaustive but shows possible causes of misinformation. The lists in Analyzing and Visualizing steps are simplified from [37] . Step Analyzing Cherry Picking Filter out data that is not conducive to the author's intended narrative [10] The Simpson's paradox high-level aggregation of data leads to wrong conclusion [21] Visualizing Break Conventions Create unusual charts that mislead people to analyze them with conventions [13, 31, 45] Concealing Uncertainty Conceal the uncertainty in the chart to cover up the low quality of the data [50] Scripting Text-visualization misalignment The message of the text differs from that of the visualization it refers to [26, 27] Text Wording The degree of text intensity can bias readers' memory of graph [43] Illusions of causality The text makes incorrect causal inductions on chart information [32, 35, 35] Obmiting context Omitting the context needed to understand the story [14] Manipulating order Manipulate the reading order through layout, resulting in order bias [16] Arranging Obfuscation Make it difficult for readers to extract visual information through chaotic layout [13] Reading Personal bias Political attitudes, beliefs and other personal factors lead to misperception of facts [19, 38] Figure 3: The production-consumption process of narrative visualization We collected candidate data stories for the study either from wellknown media (e.g., New York Times) or from influential information publisher (e.g., Our World in Data). In the selection process we took several factors into careful consideration. Most importantly, we focused on topics like health and climate changes where misinformation prevail [9, 11, 49] . Then, we chose interesting and relevant stories to motivate reading. Last but not least, we require that the text in a given story must reference some part(s) of the visualization to increase the likelihood that readers would attend to both types of contents. We also need the original data behind the stories to ensure the feasibility of embedding misinformation into the selected narrative visualization. After several rounds of careful screening and discussions, we finally selected three short stories from three articles for our study: • COVID-19 story 1 : it presents the changes of infected cases during the COVID-19 epidemic in Brazil from June 2020 to January 2021 using a line chart. The text refers to the visualization to explain public policies, like reopening tourist attractions, adopted by the Brazil government. • Obesity story 2 : it stresses that obesity has become a critical risk factor for increasing mortality, by comparing the significance of obesity as a cause of death to other risk factors like smoking in a bar chart. • Carbon Emission story 3 : it argues that developing countries shouldn't rely on fossil fuels for development as what the developed countries have done in the past. A bar chart is used to compare the per person carbon emissions (metric tons CO2) of several countries in 2014. China is highlighted to illustrate the opinion. We did not use the entire article but extracted sections that related to the visualizations. We also slightly edited the sections to make the three stories have similar length (COVID-19 & Obesity: 117 words; Carbon Emission: 109 words). For each selected story, we obtained its data from the source website. We then reproduced the narrative visualizations. As the three stories' original text formats and layouts differ greatly, we unified their styles in our reproduced version by referencing online article styles in New York Times. For visualizations, we only included elements concerned with the text we used in the study to avoid noise. This process resulted in a line chart for the COVID-19 story, a horizontal bar chart for the Obesity story, and a vertical bar chart for the Carbon Emission story as in Fig.4 . Then, building on this basic data visualization, we designed and implemented the three conditions for our experiment. 1. Static Illustrated Text (Static). Text paragraphs are placed on the left side of the page, and the visualization is displayed on the right. Text and the visualization are separated, and for all stories, they take similar space in the experimental website. 2. Explanatory Annotation (Annotation). Text which has no connection to the visualization is put on the left. Each text relevant to a particular part of the visualization is formatted as an annotation enclosed in a textbox. Every annotation locates near the corresponding part in the visualization and has an arrow pointing to it. The layout of annotations and arrows is crafted manually to avoid blocking essential contents of visualization. No hints are implying the reading sequence of annotations. tences that reference the visualization are highlighted. When users hover over the highlighted text, its corresponding part of the visualization will be spotlighted by changing to colour more salient than that of the rest. We concentrate on AIRDM in the scope of this work. For each story in our study, we select one type of misinformation to inject into its production-consumption pipeline as described in the previous section. In particular, we choose to introduce misinformation caused by deceptive visualization and text-visualization misalignment as they have received wide discussions in the research community and have already demonstrated negative impact on the society [27, 31, 45] . To be more specific, we instill Contradictory Slant -a form of text-visualization misalignment -into the COVID-19 story by adding a sentence stating that the number of infections is declining attributed to the Brazilian government while the number shown in the line chart is actually rising. The contradictory slant is an easy-tospot text-visualization misalignment; still, many people fail to find such misinformation [27] . We apply two forms of deceptive visualization to the Obesity story and the Carbon Emission story, respectively, namely Truncated Axis and Inverted Axis. Both of them are popular deceptive techniques for visualization and have been thoroughly discussed in previous works [45] . In the Obesity story, by truncating the X-axis of the horizontal bar chart, we make the risk of obesity seem much smaller (in comparison with the real number) than that of smoking to go with the claim in the text. In the Carbon Emission story, we make China's per capital carbon emissions appear the highest in the bar chart while its value is actually the smallest by inverted the Y-axis. We compared China with Japan and Germany in their carbon emission control performance in the text. Note that in narrative visualization, Truncated Axis and Inverted Axis can be considered as special forms of Contradictory Slant, as the text in all three types of misinformation is not aligned with the visualization. However, the deceptive visual elements in Truncated Axis and Inverted Axis make it more challenging for readers to find the misinformation. As mentioned above, the text component of each data story contains a single sentence that explicitly states the misinformation. Although the misleading sentence is only part of a complete data story, it is central to the conveyed message, either to convince readers that the epidemic is under controlled (COVID-19 story), or to lower the dangerous of obesity (Obesity story), or to exaggerate China's negative impact (Carbon Emission story). Besides, we assign each misinformation to a specific story to avoid finding it too easy. For example, suppose we inject Inverted Axis to the Obesity story and rank the number of death caused by air pollution higher than that by high blood pressure. In that case, readers can find the misinformation easily by common knowledge. We want to observe the impact of the design methods used and lower the impact of other factors. We conducted the between-subjects experiment on Amazon Mechanical Turk (MTurk), a popular crowdsourcing platform [3] with lay public. We set the following inclusion criteria for selecting participants to ensure the quality of the study: 1) number of Human Intelligence Tasks (HIT)' approved greater than 1000; 2) 97% or above HIT approval rate, and 3) self-reported to be in the United States. In total we received the responses from 313 unique participants. Based on the filtering criterion stated in Sec. 4.6, we filtered out 91 responses. The resulting dataset includes 222 responses, with 98 females and 124 males. Each condition was tested by 74 participants. All participants signed a consent form that present the procedure of the study and data we would collect prior to the study. The mean age of participants was 40 years old (20 to 65). Majority of the participants (72.072%) have a bachelor's degree or above. Each HIT took approximately 20 minutes to complete, and the participants received a compensation of US $2.5. Participants went through our study following the steps as presented in Fig. 5 . Each of them read the three data stories under one randomassigned condition (i.e., Static, Linking or Annotation). They com- Figure 5 : The procedure of the study. Note that each participant went through stage 2 three times as they read three stories in total. pleted the tasks by accessing our study website that was posted on MTurk. The participants were provided with an online consent form after the study website is loaded. After they agreed on the terms, we asked questions regarding the demographic information, self-reported visualization literacy level, and the attitudes towards the three topics covered in the stories (as shown in step 1 in Fig. 5) . Then, they proceeded to complete the tasks in the following phases: Story reading phase As shown in step 2 in Fig.5 , the participants were required to read the three data stories and answer the followup questions after reading each story. The order of stories and the assignment of conditions were counter-balanced among participants. The participants could spend as much time on each story but no shorter than 30 seconds (mandatory minimum viewing time). Note that after reading each story, we did not ask participants to report detected misinformation because we didn't want participants to look for misinformation on purpose in subsequent stories. Such questions were asked in the later phases. Story revisiting phase After participants had read all stories and answered corresponding questions, we presented the three stories again and asked them to report if they had found any misinformation (denoted as Story Revisit questions). If participants claim to have detected misinformation, we asked them to specify the sentence(s) that contain the identified misinformation (step 3 in Fig. 5 ). In the instructions of this phase, we emphasized no need to re-read the stories to complete these questions. Fact checking phase After revisiting all three stories, participants were introduced to the facts and the methods to manipulate them (denoted as Fact-Checking questions). They then were asked to reflect on whether such misinformation is easy to spot (step 4 in Fig. 5 ). If they do not find the misinformation during their previous reading, we invited them to write down the reasons. We collected four groups of data from the online study: • pre-study questionnaire; • participant perceptions to the reading experience of each story; • participant-claimed detected misinformation; • qualitative feedback after the fact-checking phase. We measured the participant perceptions to the reading experience via questions covering the four aspects: (1) attention checking. As suggested by Borgo et al. [3] , it is important to check if the crowd workers pay attention to the tasks. Hence, for each story, we asked two multiple-choice recall questions and an open-ended story-summairzation question to test if the participant read the story carefully. We only accepted submissions from participants who answer at least three (out of six from all three stories) multiple-choice questions correctly and provide a informative story summary with no fewer than five words; (2) visualization usefulness. We asked participants to self-evaluate the usefulness of the visualization of the story they just read via a five-points likert scale. (3) perceived credibility. We adapted the credibility scales from Kong et al. [27] to measure the perceived credibility to the text and visualization. More specifically, the participants rated the text and the visualization of the just-read story from five perspective: accuracy, fairness, trust-worthiness, bias, and completeness. We toke average of the rates as the perceived credibility in the subsequent analysis. (4) knowledge consistency. The prior knowledge and belief can influence the reading experience of pariticipants [24, 27] . Thus, we asked participants the level of consistency between the content of the story and their prior knowledge in seven-points likert scale to analyze the influence of prior knowledge to the results. To investigate whether participants were aware of the misinformation in the given data stories, we coded participants' responses to the questions in the story revisiting and fact-checking phases (see subsection 4.5) and labeled their level of awareness based on the following criteria: 1. Participants who stated finding the misinformation in the story revisiting phase and also gave the right sentence that contains that misinformation were labelled as "fully aware". 2. Some participants did not imply in the story revisiting phase that they found misinformation but in the fact-checking phase reported to have noticed the misinformation. For example, "I did see that inconsistency between the text and the visualization, but did not compute that it was an error, which was wrong of me". We labeled such participants as "somewhat aware", if there exists evidence indicating that they hold reasonable suspicion. Such evidence includes but is not limited to doubting (certain part of) the chart in their replies to the story-summarization questions, giving low credibility ratings (≤ 4) in the perceived credibility questions, etc. 3. For any cases other than 1 and 2, we label them as "unaware". One author and one PhD student with psychology background coded the data. After coding the first 100 responses, they met to compare the labeling results and resolve any conflicts through discussion. Then they coded the rest data. An inter-rater reliability analysis using the Kappa statistic shows high reliability (κ = 0.84). Fig. 6 shows the distribution of participants' awareness levels under different conditions in each story with a specific form of misinformation injected. The majority of participants (62.162 − 86.486%) in all conditions failed to detect the misinformation. Despite that, we found a significant marginal impact of the conditions on participants' awareness of misinformation (χ 2 (2) = 5.53, p=0.063) across the three misinformation types using the Kruskal-Wallis test. We further employed Dunn's post-hoc tests with Holm correction for pairwise comparisons. The results show that participants' awareness of misinformation under the Linking condition is significantly higher than that under the Static condition (Z = -2.29, p = 0.033). But we did not find significant effects of Annotation over the awareness. Considering the three stories with different forms of misinformation separately, we compared the awareness under the Annotation and Linking conditions with the one under the Static condition by running Mann-Whitney's U test. We found that in COVID-19 story with contradictory slant, Annotation has a significant effect on participants' awareness of misinformation (U=2303.5, p=0.039, r=0.159). No other statistical significance was found. The above results show that although the effects are limited, Annotation and Linking did have promotions to participants' awareness of misinformation. Linking is generally more effective than Annotation in improving participants' awareness. We can see from Fig. 6 that in all three types of misinformation, the number of participants who were fully aware of misinformation under Linking is higher than that under Static. But for Annotation, although it had an advance in the COVID-19 story with contradictory slant, in Obesity story, it is even worse than Static. After the participants read each data story, they scored their perceived credibility to the text and visualization. In this section, we present our findings deriving from analysing participants feedback. We perform MANOVA tests to investigate the impact of condition (Static, Linking, and Annotation) on the two sets of credibility scores (for text and visualization respectively as mentioned in subsection 4.6, also see Table. 3). Across different forms of misinformation, we could see that the condition has a significant effect on the credibility scores (Approx. F=4.658, Wilks' λ =0.972, p=0.001). ANOVAs revealed that the condition has significant effect on the text and visualization credibility scores (p≤0.001 and p=0.022, correspondingly). A post hoc Tukey test showed that the text credibility scores in the Annotation and Linking conditions are significantly lower than that in the Static condition (p ≤0.001, d = -0.367; p =0.024, d = -0.321); the visualization credibility scores in the Annotation condition is significantly lower than that in the Static condition (p = 0.016, d = -0.261). No other significant differences were found. We further investigated the effects of the conditions under the three types of misinformation separately. The results are shown in Table 2 . We found that Annotation significantly differs to Static on the text credibility scores of the stories with Contradictory Slant and Inverted Axis, and on the chart credibility score of the story with Inverted Axis. Besides, with Inverted Axis, Linking also had a marginal significant effect on the text credibility compared to Static. The above results show that Annotation and Linking can lower the perceived credibility of the data stories that contain misinformation. Worth mentioning, under all three conditions, the average text and visualization credibility scores across different misinformation lean to be positive (above 4). This result is consistent with Sec. 5.2.1, where most participants did not report any (valid) misinformation. We compared the credibility scores of text and visualization using Wilcoxon signed-rank tests for each type of misinformation. Kong et al. [27] found that for visualizations with title-visualization misalignment, people consider the visualization to be more credible than the text. For contradictory slant, which was also used by Kong et al. [27] , our results repeat their findings: The text credibility We are also interested in whether other factors like age, education, visualization literacy, attitudes, visualization usefulness and knowledge consistency will affect the perceived credibility. Hence, we computed Spearman's rank correlations between these factors and the two credibility scores. The results have been shown in Fig. 7 . We found little correlation between age, education, self-reported visualization literacy ("Vis. Familarity" in Fig.7) , attitude and the credibility scores. These results are partially consistent with previous research [31, 45] . The exception is attitude, which was found having effects on perceived bias by Kong et al. [27] . We believe this may result from the differences of stories used the two studies, which is worthy future study. We found a positive correlation between the visualization usefulness and the two credibility scores, especially the visualization credibility score (p≤0.001). Understandably, the participants who thought the text and visualizations were less credible rated the visualization as less useful. However, it is surprising that the participants who found the visualization useful cannot find it less credible. A possible reason is that the power of the misleading text induces the participants to ignore the errors in the visualization. Besides, there is a weak positive correlation between the knowledge consistency and the credibility scores, consisting with previous research [24] . For participants who do not discover misinformation, we collected their subjective feedback on possible reasons and grouped them into seven categories using thematic analysis [4] (see Table 4 ). The most frequently mentioned reason is that their attention is not on elements related to misinformation (45.21%). For example, in the Obesity story with truncated axis, some participants ignored the numbers tagged on each bar: "I had not looked at the values for each of the bars at the initial viewing, as I was primarily focusing on the heights of the bars and comparing it to one another". Also, in the COVID-19 story, some participants paid less attention to information near the end of the story: "It was not easy [to find the misinformation] because it was at the end of the paragraph. My mind was not fully focused on the information knowing that it was like a closing sentence". Participants also identified text as an important factor that affects their understanding of the data story ("Excessive trust in text" in Table. 4 ). This finding is consistent with the claim of Kong et al. [27] that text has a strong power in manipulating the receiving information of data stories. Even though some participants sensed something might go wrong with the chart, they did not consider it as misinformation: "I saw that the text and chart didn't match, but I assumed the text was correct and the chart had been printed wrong". Misinterpretation of the visualizations is another reason. For example, some participants failed to decode one dimension of the data points: "The problem here is seeing where the line points match the date along the x-axis. It is not easy to see how the dates align with the line drawn". As stated in Section 4.2, the visualizations used in the study are closed to what has been used on public online news. Hence, this reason shows the importance of media improving the comprehensibility of visualizations in their publishing stories. Participants also reported Other reasons, include lacking prior knowledge (e.g, "I think I was fooled by [my prior impression of] Canada and somehow thinking of it as being full of empty space and clean air compared to the others"), over-reliance on conventions of visualization (e.g, "I don't know how one would be able to know that the y-axis is inverted without being told so"), misunderstanding the reference relationship (e.g, "I felt like the text was focused on the early January reaction to the December spike"), and lack of intention to find the misinformation (e.g, "This is not easy to spot and I was not really looking for misinformation"). We discuss the design implications from these reasons in Section 6. In this paper, we first explore the possible ways that narrative visualization can bring misinformation in its production and consumption process. As suggested by Correll and Heer [13] , "we should enumerate the ways that visualizations can deceive or mislead.". Our work is a first step towards analyzing the space of misinformation in illustrated data stories. Some forms of misinformation we have summarized, e.g., deceptive visualization [31, 45] and text-visualization While there exist studies aiming at understanding the influence of layout or sequence on narrative visualization [2, 23] , few of them concern misinformation. Hence, more works are needed to deepen our understanding of misinformation in data storytelling. The results of our crowdsourcing study suggest that the two common-used design methods, Annotation and Linking, can raise participants' awareness of misinformation. We also found these two design methods can affect the perceived credibility of text and visualizations. However, the effects observed in our study look limited as most participants cannot find misinformation in the given data stories. But, we believe the two methods, Annotation and Linking, have the potential to collaborate with other methods to form a defensive design to protect lay people from misinformation. Annotation and Linking serve for bridging the perceptual gap between the text and visualization, which induce readers to pay attention to data-related narratives. By adopting other attention-guiding methods to nudge readers reading data stories comprehensively and skeptically (e.g., explicit warning [32] ), we can have solutions to "Inattention to elements related misinformation", the most important cause identified in our study for being unaware of misinformation. Worth mentioning, the two defensive design methods we chose as conditions in the experiment (Annotation and Linking) can be automatically applied to many data stories using existing technologies [28, 30] . With that said, we envision a possible future to defend misinformation: story or news release platforms, such as New York Times, can automatically transform the uploaded data stories from authors into a form that contains some defensive design methods and then release the stories to the public. In that sense, the platform can establish a protective barrier in content dissemination from authors to readers. Even if advanced algorithms cannot detect misinformation, readers will still have the opportunity to be aware of misinformation. In the crowdsourcing study, our qualitative analysis of participants' feedback provides insights into several design implications to mitigate misinformation in narrative visualization, as presented below. Visual Design to Guide Attention Most participants in our experiment attribute their failure to find misinformation to their inattention (Table 4 ). Pennycook et al. [46] postulate that directing people's attention to the accuracy of news can be a promising way to combat misinformation. Consistent with this insights, our results indicate that a promising design direction is to lead readers to attend to the part(s) of stories where misinformation is prone to appear. Such designs could potentially be a cure to "overlooking violation of visualization conventions" and "misinterpreting the visualization". Data-Friendly Text Text is an important component of narrative visualization, and researchers should rethink how its presentation can be improved to mitigate misinformation. Our qualitative analysis reveals that text might cause misperception in readers, thereby leaving problematic messages in the stories unnoticed (e.g., "excessive trust in text" and "misunderstanding the reference relationship" in Table 4 ). Empirical evidence implies that whether textual description is closely linked to the data can affect reader understanding of the data story [39] . A data-oriented narrative style of text, such as emphasizing the data source, clearly expressing the analysis method of the data, and more accurately specifying the visual elements of the discussion, may facilitate readers' judgment of the facts [41] . Knowledge-Augmented Visualization Some participants believe that the reason why they do not discover misinformation is a lack of understanding of visualization or the background of the story ("Lacking Prior Knowledge" in Table 4 ). A feasible way to address this problem is to augment the original narrative visualization with external knowledge. Similar ideas have been applied to improve awareness of diverse social opinions [18] . Our study bears several limitations. First, we conducted experiments on three types of misinformation covering only two of the many causes of misinformation we identified in Table 1 (Break Conventions and Text-visualization misalignment). Our study material is only concerned with two visualization types (i.e., bar chart and line chart). It is worth conducting experiments to understand what defensive designs can be effective to other causes of misinformation such as Manipulating order and Obfuscation as well as narrative visualizations with more diverse visualization types in the future. Second, we did not randomize the type of misinformation for each story in the experiment to prevent the misinformation from being too obvious to see from the common knowledge. Such design, however, can affect the effectiveness of our analysis in comparing different kinds of misinformation but not the conclusion about the two defensive designs, which is the focus of this paper. Third, our measurements to participants' awareness of misinformation can be improved. We analyzed their awareness based on their reflection after reading stories. More accurate measurements can be derived from other data collecting methods such as eye-tracking. Last but not least, we found Annotation and Linking do have effects, but we do not gain insights into how they work from our data. We hypothesize that they work by affecting readers' attention, which requires more investigation. In the future, we will experiment with adopting instruments such as implicit measurement methods in psychology [25] to obtain more accurate assessment of readers' awareness of misinformation. We will further test readers' recall of the story and their potential attitude shift to discern the extent of impact of awareness. We introduce how misinformation can be injected into narrative visualization and explore possible defensive design methods to combat misinformation. Misinformation has been ever dangerous to human society with the ability of online social media to spread information quickly. Combating misinformation is an essential but also challenging task. In this paper, we choose two commonly used methods in narrative visualization as candidates of defensive designs. Our study results show the potential of these two and other similar methods to be a part of the force to combat misinformation. Also, the number of people who do not find misinformation in our study again let us notice the challenge of fighting an infodemic. Comparing the gov't covid19 maps for end of september and start of october, it looks as if things are getting better.until you notice they've changed the numbers corresponding to each colour. had they used the same ones, most of the country would be red or dark red Design patterns for data comics Information visualization evaluation using crowdsourcing Thematic analysis Temporal summary images: An approach to narrative visualization via interactive annotation generation and placement Effects of knowledge and display design on comprehension of complex graphics. Learning and Instruction Vizlinter: A linter and fixer framework for data visualization Supporting story synthesis: Bridging the gap between visual analytics and storytelling Addressing health-related misinformation on social media Hark no more: on the preregistration of chi experiments Deconstructing climate misinformation to identify reasoning errors Surprise! bayesian weighting for de-biasing thematic maps Black hat visualization Misrepresenting covid-19: Lying with charts during the second golden age of data design Correcting misinformation-a challenge for education and cognitive science It's good to be first: Order bias in reading and citing nber working papers Making sense of graphs: Critical factors influencing comprehension and instructional implications Burst your bubble! an intelligent system for improving awareness of diverse social opinions Implications of pro-and counterattitudinal information exposure for affective polarization The future of false information detection on social media: New perspectives and trends What you see is not what you get! detecting simpson's paradoxes during data exploration Visualization rhetoric: Framing effects in narrative visualization A deeper understanding of sequence in narrative visualization Prior knowledge and reading comprehension test bias Attitudes and the implicit association test Frames and slants in titles of visualizations on controversial topics Trust and recall of information across varying degrees of title-visualization misalignment Extracting references between text and charts via crowdsourcing Visjockey: Enriching data stories through orchestrated interactive visualization Automatic annotation synchronizing with textual description for visualization How people are influenced by deceptive tactics in everyday charts and graphs Causal perception in question-answering systems More than telling a story: Transforming data into visually shared stories Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online Illusions of causality: how they bias our everyday thinking and how they could be reduced Visual narrative flow: Exploring factors shaping data visualization story reading experiences Surfacing visualization mirages Motivated misperception? party, education, partisan news, and belief in "death panels Assessing the effect of visualizations on bayesian reasoning through crowdsourcing Global surface temperature Writing a quantitative research thesis What's going on in this graph? -immigration shifts Effects of title wording on memory of trends in line graphs The persuasive power of data visualization How deceptive are deceptive visualizations? an empirical analysis of common distortion techniques Shifting attention to accuracy can reduce misinformation online Communication and persuasion: Central and peripheral routes to attitude change 75 years of major refugee crises around the world Susceptibility to misinformation about covid-19 around the world The role of uncertainty, awareness, and trust in visual analytics Narrative visualization: Telling stories with data Calliope: Automatic visual data story generation from a spreadsheet Oxford dictionary of English Warning, bias may occur: A proposed approach to detecting cognitive bias in interactive visual analytics Comparing effectiveness and engagement of data comics and infographics Misinformation in social media: definition, manipulation, and detection Linking and layout: Exploring the integration of text and visualization in storytelling We thank Yujie Zheng for valuable input. We also thank the anonymous reviewers for their feedback and comments that helped improve this paper.