key: cord-0917952-mlz4pbwa authors: Trivedi, Neha; Lowry, Mark; Gaysynsky, Anna; Chou, Wen-Ying Sylvia title: Factors Associated with Cancer Message Believability: a Mixed Methods Study on Simulated Facebook Posts date: 2021-06-19 journal: J Cancer Educ DOI: 10.1007/s13187-021-02054-7 sha: f7ac73428816e53a00ec139da083226cc61d0e1b doc_id: 917952 cord_uid: mlz4pbwa The ability to share and obtain health information on social media (SM) places higher burden on individuals to evaluate the believability of such health messages given the growing nature of misinformation circulating on SM. Message features (i.e., format, veracity), message source, and an individual’s health literacy all play significant roles in how a person evaluates health messages on SM. This study assesses how message features and SM users’ health literacy predict assessment of message believability and time spent looking at simulated Facebook messages. SM users (N = 53) participated in a mixed methods experimental study, using eye-tracking technology, to measure relative time and message believability. Measures included individual health literacy, message format (narrative/non-narrative), and information veracity (evidence-based/non-evidence-based). Results showed individuals with adequate health literacy rated evidence-based posts as more believable than non-evidence-based posts. Additionally, individuals with limited health literacy spent more relative time on the source compared to individuals with adequate health literacy. Public health and health communication efforts should focus on addressing myths and misinformation found on SM. Additionally, the source of message may be equally important when evaluating messages on SM, and strategies should identify reliable sources to prevent limited health literate individuals from falling prey to misinformation. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s13187-021-02054-7. In recent years, social media (SM) platforms have enabled individuals to search, obtain, and share health information more readily. SM platforms are commonly used to disseminate health-related information and connect communities of users with similar interests or experiences [1, 2] . However, given the ubiquity of information online, the quality of health information on SM is mixed, generating concern over the legitimacy and reliability of health information found on these platforms [2, 3] . As many individuals are inundated with health information on SM, the burden of critically evaluating content and assessing their credibility largely falls on users [4] . Limited ability to discern the credibility of health information could have concerning real-world consequences if user-generated content on these platforms leads SM users to endorse or spread uncorroborated health claims [5] . For example, current research regarding the COVID-19 pandemic has demonstrated that widespread misinformation on SM could be associated with problematic real-world behaviors, such as non-compliance with wearing a mask and social distancing guidelines [6] . In order to mitigate the negative effects of health misinformation, it is imperative to first understand factors, such as user characteristics, associated with how people assess the quality and believability of a health message. First, a person's health literacy, which broadly refers to the "ability to obtain, process, understand, and communicate about health-related information needed to make informed health decisions," plays a critical role in information processing and evaluation [7] . When confronted with health information on SM platforms, users must utilize literacy skills to discern facts from falsehoods. Indeed, with increasingly more equitable access to health information for people across health literacy levels [8] , users with limited health literacy may be unable to accurately assess information credibility [4] . Depending on the health topic, source, message veracity, and message format, individuals' health literacy may play different roles in how people process and determine validity of a health message. Second, message features, such as veracity, content, and format, all play a significant role in the way SM messages are evaluated. Research suggests much of the health information disseminated on SM platforms tends to be misinformation rather than evidence-based information [9, 10] . This requires individuals to engage their cognitive skills to determine the veracity or legitimacy of a health message on SM [11] . In addition, features like information complexity affect believability of a message [12] . Information complexity includes how message content (i.e., specific elements of the body of the text) is created, which in turn influences message believability [13] . Further, use of syntax, technical jargon, and general choice of terminology can alter perception of the credibility of a message [14] . Marketing research has shown that how a message is presented, in addition to the text of the message, can influence how it is perceived [15] . Lastly, the format, such as the use of narratives and storytelling to convey a message, has been found to influence perceived believability given the emotional engagement associated with a health topic, making people vulnerable to spreading misinformation [16] . Conversational and testimonial formats as well as personal experience narratives have been shown to increase perceived message authenticity compared to didactic formats [17] [18] [19] . As established, one's health literacy skills and the features of a message play a role in an individual's evaluation of message believability. Subsequently, it becomes important to investigate whether those factors impact attention to a message. Assessing an individual's actual cognitive processing provides an opportunity to uncover the way one views certain aspects of a message and whether attention to different components of a message influence their assessments of message believability. "Elaboration," using the term from the elaboration likelihood model (ELM), on a message entails their motivation to pay attention as well as their cognitive ability to engage in the information being presented [20, 21] . One's ability to engage in and scrutinize a message directly relates to a person's health literacy and whether he/she is persuaded by a message [22, 23] . ELM posits those with varying health literacy levels process information through one of two "routes": (1) central route, which operates when one has high elaboration and cognitive ability to process message quality, or (2) peripheral route, which operates when one elaborates through low-effort processes and utilizes heuristics cues to understand a message [24] . Researchers can measure visual attention of SM users' in these "routes" of persuasion through eye-tracking methodology, which gives insight into users' attention to various components of a message [25] . For example, one study measured attention through total fixation duration (a common eye-tracking metric defined as a stable gaze of more than 80 ms) and highlighted that participants paid attention to the source when looking at a news post and used this information as a criterion for the decision to read or to skip the news post [26] . Although reliance on source cues has been conceptualized as a lower-effort mode of information processing when it comes to message persuasion [23] , source credibility plays a prominent role in message assessment when other information that may be used to judge the actual quality of a text is limited [27] . Eye-tracking studies have suggested that SM users do not have unlimited cognitive capacity to process health messages, and therefore attend to only salient factors of a message [28, 29] . The source of a message is one such salient factor, commonly viewed as a heuristic cue, that people often use to evaluate information presented to them when the content is difficult to comprehend [30] [31] [32] [33] [34] [35] [36] . A previous eye-tracking study examined the time spent on simulated cancer-related Facebook messages, and found that participants tend to spend more time on health messages from lay people compared to health organizations [28] . Research has also established that source trustworthiness (an information provider's perceived intention to tell the truth or give unbiased information) is directly associated with message credibility [37] . An additional analysis from another eyetracking study indicated that participants who report having a high level of trust in a message source tend to rate a health message from that source as more believable [38] . As highlighted, attention to the source of a message plays a vital role in message credibility and may help explain participants' perceived message believability. Expanding upon prior work using a combination of eyetracking and survey methodologies on SM message processing [28, 38] , we investigated how people with varying health literacy view and assess simulated Facebook health messages. Specifically, this mixed methods study sought to assess how message features and participants' health literacy predict assessment of message believability and time spent looking at the messages. As illustrated in Fig. 1 , three research aims guided our conceptual model. First, we explored how message features and participant's health literacy were associated with perceived message believability (Aim 1). Second, complementing Aim 1 and relying on eye-tracking metrics, we assessed the association of attention to specific message components on perceived message believability (Aim 2). Lastly, we explored how message features and health literacy were associated with attention to message components (Aim 3). The goal of Aim 3 was to understand the implications of the predictors on attention to the text of the message to examine if message content was impacted by individual characteristics and message features. However, to fully understand the implications of the effects of health literacy and message features, we also explored how these predictors were associated with attention to source and image of the message. Eighty participants from the metropolitan Washington DC-Maryland-Virginia area were recruited by a research recruiting firm, which screened participants to determine eligibility. Eligible participants had to be at least 18 years old and use SM regularly-defined as logging into at least one SM account (e.g., Facebook, Twitter, Instagram, or Pinterest) daily. Efforts were made to recruit respondents from diverse demographic groups. A total of 27 participants were excluded from the final sample, for example, due to cancellations, technical issues, or poor eye calibration and low gaze samples. The final analytic sample included 53 participants (see Fig. 2 highlighting the participant flowchart). Data were collected June-October 2018. A total of 16 target posts were developed for the study. Each post focused on one of two prominent cancer prevention topics frequently discussed on SM: human papillomavirus (HPV) vaccination and sunscreen safety [39, 40] . Target posts varied based on the following manipulated features: (1) message format (whether the text of the message was a narrative/story or a factual statement without a narrative), (2) message source (lay individual or a health organization), and (3) message veracity (evidence-based or non-evidence-based information given current scientific evidence on the topic). The study employed a mixed methods approach, integrating eye-tracking and survey questionnaires. After obtaining informed consent, participants were guided through a standardized onscreen calibration exercise for accuracy check using a Tobii T120 Eye Tracker [41] , which was used to track eye movements as participants completed the study. A Dell computer with a 27-in. monitor and with a screen aspect ratio of 16:9 was used for the experiment. Participants sat approximately 24 in. from the monitor. After calibration was completed, participants were directed to view . Although the format for target posts approximated the look of an authentic Facebook post, the posts themselves were not made interactive (i.e., hyperlinks were not active and the "like" button could not be clicked) in order to facilitate interpretation of eye-tracking data. Distractor and target posts were generally comparable in size, length of text, and use of imagery. All participants received a random sequence of stimuli to limit order effects. After viewing the three feeds, participants completed a series of message assessment surveys asking them to evaluate each of the three target posts they viewed in the feeds, as well as three new target posts they were not previously exposed to. Each message was displayed onscreen while individuals answered the survey questions; participants continued to have their eye movements tracked while completing the surveys. Eye-tracking data collected while participants were filling out the message assessment surveys were utilized for the purposes of this analysis. At the end of the session, participants were debriefed with scientifically accurate information about the HPV vaccine and sun safety and received $75 in compensation for their time. The protocol was reviewed and deemed exempt by the Ethics Committee and IRB at the authors' institution. Perceived message believability was assessed with one survey item that asked, "Please tell us what you think about this post: This post is…" Response options ranged from 1 (Not Believable) to 7 (Believable). Health literacy was calculated by using The Newest Vital Sign Assessment Tool [42] . A score of 0-2 suggests an individual has limited health literacy, while a score of 3-4 suggests adequate health literacy. Areas of interest (AOIs) were created to capture the time spent on a specific area of the simulated post (namely, the text, source, or image portion of the post). Defining AOI allows researchers to measure the distribution of attention between specific regions on a stimulus [43] . An AOI was allocated a specific shape on the stimulus to delineate the amount of time spent in that specific region. For the purposes of this study, attention was measured as relative time within an AOI (a commonly used eye-tracking metric) and was defined as "time spent on each AOI divided by total duration of time spent on post." To account for the variability in the amount of visual information on the survey pages (e.g., text length, image size), the analyses were pixel size adjusted. The primary eye-tracking metric outcome of interest was relative time on text AOI, measuring the time spent on the content of the message. In addition, as exploratory analyses, our secondary outcomes of interest included relative time on source AOI and relative time on image AOI. Given that attention is defined as relative time spent on specific areas of the stimuli and are dependent on each other (e.g., spending 50% of time on text leaves only 50% of total time to spend on source and image), our primary analysis focuses on analyzing text only. However, we do report results on source for reference. Participants' believability ratings were measured on an interval scale (1-7) for each post. The data were entered into an ordered-probit Bayesian hierarchical model (BHM) [44] ; an ordered-probit model is specifically designed for interval data. For interval data (e.g., Likert scales), BHM provides more accurate estimates than models that assume the data (e.g., participant believability ratings) come from a continuous distribution [45] . Predictor variables for Aim 1 included health literacy at the participant level (Adequate, Limited), message format (Narrative, Non-narrative), and message veracity (Evidence-based, Non-evidence-based). Predictor variables for Aim 2, relative time on AOIs (text, source, image), were input as continuous variables. Along with analyzing the main effects of each predictor variable, interaction effects were assessed to understand the joint effect of the predictors on perceived message believability. Posterior distributions were used to estimate mean differences between conditions for main effects and interactions of the predictor variables. For each trial, the model accounted for variance associated with each participant (1-53), the simulated Facebook post (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) , and post topic (i.e., HPV vaccine, sunscreen safety). In Bayesian statistics, an analysis (e.g., mean difference, interaction effects) is considered "statistically credible" if 95% of the highest density interval (HDI) of its posterior distribution does not contain zero. It is analogous to confidence intervals in traditional statistical methods. Relative time spent on the AOI for each post was entered separately into a BHM. This model assumed that a given proportion comes from a beta distribution with a mean of theta (ϴ). The primary analysis included text AOI as the main dependent variable, while secondary analyses included source AOI and image AOI. Similarly, to the perceived believability analysis, for Aim 3, health literacy, message veracity, and message format were input as predictor variables separately, to assess main effects, and then combined to examine joint interaction effects. For each trial, the model controlled for pixel size, each participant (1-53), the simulated Facebook post (1-16), and post topic (i.e., HPV vaccine, sunscreen safety). The model estimates the mean differences between conditions in the experiment, which are reported here as percentages along with 95% HDIs. The final sample of 53 participants (Table 1) included more females (n = 40, 76%) than males. Participants' reported race was nearly equally divided between Black/African American and other races. Most of the participants (72%) reported having a college or graduate degree and most (71%) had adequate health literacy skills based on the Newest Vital Sign score. We describe the key findings by study aims below. Aim 1 examined whether one's health literacy level, message veracity, and message format were associated with perceived message believability. Results from Aim 1 highlighted a main effect of message veracity as well as interaction effects for health literacy and message veracity. The main effect showed evidence-based messages were deemed as more believable compared to non-evidence-based posts. There was no main effect of health literacy nor message format on perceived message believability. Mean comparisons are given in Table 2 . Importantly, the main effect of message veracity was qualified by health literacy. Individuals with adequate health literacy rated evidence-based posts as being more believable than non-evidence-based posts (mean difference = 2.3%, 95% HDI [0.74, 3.81]). In contrast, individuals with limited health literacy did not rate evidence-based posts as being more believable than non-evidence-based posts. The first graph in the Appendix highlights the interaction between health literacy and message veracity. No other interactions were statistically credible at 95% HDI. Subsequently, Aim 2 examined the association of attention to text of message on message believability, based on ELM's premise individuals engage in high elaboration through central route processing. There was no statistical relationship between relative time spent on the text AOI and message believability (β = 0.42%, 95% HDI [− 0.85, 1.86]). After observing no relationship between attention to text and message believability, it became necessary to explore if source AOI and image AOI were associated with message believability to understand whether peripheral route processing differed from other aspects of a message; neither of which were significant. The goal of the exploratory Aim 3 was to assess whether message features and one's health literacy were associated with relative time on text AOI. Results showed a main effect that participants spent more time on non-evidencebased text compared to evidence-based text (mean difference = 4.94%, 95% HDI [0. 47, 9.64] ). In addition, there were no other main effects for health literacy nor message format on relative time spent on text. Table 3 highlights results for the mean comparisons. Interestingly, the main effect of message veracity was qualified by message format. Participants spent more relative time on the text of non-evidence-based narratives compared to evidence-based narratives (mean difference = 15.11%, 95% HDI [7.21, 21.76] ). However, for non-narrative posts, there was no difference in the amount spent on the text of the post, regardless of whether it was evidence-based or non-evidence-based. The second graph in the Appendix shows the interaction effect of message veracity and message format. To fully understand participants' attention to all components of a message, additional exploratory analyses were conducted to assess relative time on source AOI. As confirmed by ELM, there was a main effect of health literacy on time spent on source of the message. Participants with limited health literacy spent more relative time on the source compared to participants with adequate health literacy (mean difference = 2.86%, 95% HDI [0.38, 5.99]). There were no main effects for message veracity nor message format on relative time spent on source AOI. Table 4 highlights results for the mean comparisons and the third graph in the Appendix shows interaction effects. There were no main effects for health literacy nor message format on relative time spent on the image AOI. As noted, attention is defined as relative time on specific areas of the message making up 100% of time spent on the stimulus; the mean comparisons and graph of the interactions for image AOI can be found in the Appendix. This study examined how message features and SM users' health literacy predict their message believability assessment and how much attention they pay to various components of simulated Facebook posts. Several key findings emerged that have important implications for communication research methodologies, such as eye-tracking and public health efforts. First, those with adequate health literacy correctly rated evidence-based posts as being more believable than non-evidence-based posts; however, there was no statistically significant difference between believability ratings of evidence-based posts vs. non-evidence-based posts among participants with limited health literacy. This suggests that health literacy, as measured in this study, may play a critical role in the way people discern accurate information from false information. This finding provides additional support for prior research suggesting that online health information seekers who have the ability to obtain, process, and understand information are more likely to identify accurate information from the "noise" found on the internet and possess the ability to evaluate health information found online [46, 47] . More generally, people with adequate health literacy are observed to be more suspicious about online health information than those with limited health literacy [46, 48] . Consequently, they are more likely to search and verify information presented online, and place greater significance on the reliability of that information once it's been verified [49] , whereas individuals with limited health literacy may be susceptible to believing myths and misconceptions [50] and, therefore, less likely to believe evidence-based information. While SM platforms facilitate access to health information generally, individuals with limited health literacy, who are at greater risk of being misled by inaccurate information, may be unable to discern information credibility and therefore may be disproportionately harmed by effects of misinformation exposure. As a result, disparities may be exacerbated when individuals with limited health literacy cannot locate quality information and when exposed to information of mixed quality, are unable to distinguish between evidencebased and non-evidence-based posts. A second notable finding was that participants spend more time on the text of non-evidence-based narrative posts compared to evidence-based narrative posts. This may suggest that participants in this study are attempting to scrutinize those posts in an effort to evaluate the credibility of the message [51, 52] . It is possible that this occurred because the non-evidence-based narrative posts in our study were perceived more vivid or shocking because they graphically described stories about the harms of sunscreen and the HPV vaccine. Novel and emotional rhetoric, such as fear appeals, may influence SM users into sharing such information, suggesting evocative falsehoods tend to spread faster than true information [53] . Additionally, participants may be spending more time on non-evidence-based narratives and anecdotal messages as the power of other individuals' experiences, such as deleterious and questionable consequences of HPV immunization, becomes evidence for one's own actions to refuse vaccines due to "warnings" from others [54] . In fact, the power of storytelling has been leveraged by anti-vaccine activists to spread fear and doubt among parents [55] , and subsequently, personal experience anecdotes have generated endorsement and sharing of false information about vaccine risks and harms. Future communication efforts could seek to leverage the power of storytelling towards science-based, accurate information for positive effects on public health issues. Lastly, through our exploratory analysis, results showed individuals with limited health literacy spent more relative time on the source of the message compared to individuals with adequate health literacy. This finding may be in part explained by ELM. According to ELM, when someone lacks the ability and motivation to elaborate on a message, he/she may utilize peripheral cues to evaluate information [56] . This finding suggests that participants were engaging in peripheral processing and relying on source as a heuristic cue to evaluate the credibility of the post. It is possible that those with limited literacy skills rely more heavily on the message source for assessing message credibility because they find the text of the message difficult to evaluate [57] . Source is often utilized as a heuristic for the evaluation of information credibility [31] . For example, SM users tend to believe in false news if the message comes from trusted sources on SM, suggesting that reliance on heuristics and peripheral processing might make users more vulnerable to misinformation [58] , and our study suggests this problem may be more acute among SM users with limited health literacy. It might be helpful for SM platforms to vet accounts, encourage fact checking groups to debunk misleading health claims and support responsible verification efforts for health information, and potentially engage in correcting misinformation on SM platforms. These strategies may assist SM users (especially those with limited health literacy) in identifying expert sources and avoiding less reliable sources in situations where they find it difficult to distinguish accurate information from misleading information based on message content alone. Collaborative efforts between public health leaders and SM companies to verify sources of SM posts would help alleviate some of the burden placed on individuals to distinguish credible health messages from inaccurate messages online. This study was not without limitations. First, the simulated Facebook posts for this study were static and did not allow for fully dynamic features (e.g., liking, sharing, commenting), which means the experiment did not perfectly recreate the experience participants usually have on Facebook. Future research utilizing non-laboratory settings and observing individuals interacting with their personal Facebook pages would be needed to confirm the findings of this study in more naturalistic contexts. Additionally, the measurement used to assess health literacy in this study encompassed both literacy and numeracy, which may have hindered the accuracy of measuring only health literacy. Future health communication studies focusing on SM messages should include a measure of eHealth literacy. Furthermore, in addition to message features, participants' sociodemographic characteristics (e.g., age, gender) may potentially impact the time spent examining cancer messages believability on social media. Further research is needed to understand if relative time on various message components is impacted by such participants' characteristics. Lastly, this study was focused on a single SM platform, Facebook. Whether our findings apply to other SM platforms is uncertain. Misinformation is also prevalent on other SM outlets; therefore, future studies should examine messages on a wider array of platforms. Web 2.0 for health promotion: reviewing the current evidence A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication Many Americans say madeup news is a critical problem that needs to be fixed The role and impact of health literacy on peer-to-peer health communication A false image of health: how fake news and pseudo-facts spread in the health and beauty industry The causes and consequences of covid-19 misperceptions: understanding the role of news and social media Health literacy: what is it? Social media and obesity in adults: a review of recent research and future directions COVID-19 infodemic: more retweets for science-based information on coronavirus than for false information Systematic literature review on the spread of health-related misinformation on social media Is science failing to pass its message to people? Reasons and risks behind conspiracy theories and myths regarding COVID-19. Reasons and risks behind conspiracy theories and myths regarding COVID-19 Information complexity and medical communication: the effects of technical language and amount of information in a medical message The influence of structural and message features on Web site credibility Making sense of credibility in complex information environments: the role of message sidedness, information source, and thinking styles in credibility evaluation online What you feel, is what you like influence of message appeals on customer engagement on Instagram The integrative model of behavioral prediction as a tool for designing health messages. Health communication message design: Theory and practice Telling stories, saving lives: creating narrative health messages Narrative communication in cancer prevention and control: a framework to guide research and application A test of conversational and testimonial messages versus didactic presentations of nutrition information The elaboration likelihood model of persuasion Eye tracking methodology: theory and practice Enhancing smoking risk communications: the influence of health literacy and message content The elaboration likelihood model of persuasion. In: Communication and persuasion Who can you trust? Credibility assessment in online health forums Selective attention in the news feed: an eye-tracking study on the perception and selection of political news posts on Facebook Selecting science information in Web 2.0: how source cues, message sidedness, and need for cognition influence users' exposure to blog posts How do social media users process cancer prevention messages on Facebook? An eye-tracking study Using the limited capacity model of motivated mediated message processing to design effective cancer communication messages The importance of 'likes': the interplay of message framing, source, and social endorsement on credibility perceptions of health information on Facebook Developing a unifying framework of credibility assessment: construct, heuristics, and interaction in context Social and heuristic approaches to credibility evaluation online Technology and credibility: cognitive heuristics cued by modality, agency, interactivity and navigability. Digital media, youth, and credibility Heuristics for decision and choice Health literacy and self-efficacy for participating in colorectal cancer screening: the role of information processing Social media as information source: recency of updates and credibility of information* Credibility and trust of information in online environments: the use of cognitive heuristics Well, the message is from the institute of something": exploring source trust of cancer-related messages on simulated Facebook posts The HPV vaccine and the media: how has the topic been covered and what are the effects on knowledge about the virus and cervical cancer? Social internet sites as a source of public health information Timing guide for Tobii Eye-trackers and Eye-trackers software Quick assessment of literacy in primary care: the newest vital sign Visualization of eye tracking data: a taxonomy and survey rjags: Bayesian graphical models using MCMC Analyzing ordinal data with metric models: what could possibly go wrong? Low health literacy and evaluation of online health information: a systematic review of the literature Exploring the role of health literacy in the evaluation of online health information: insights from a mixedmethods study Health literacy and use and trust in health information Quantifying health literacy and eHealth literacy using existing instruments and browser-based software for tracking online health information seeking behavior Tacit understandings of health literacy: interview and survey research with health journalists Making the truth stick & the myths fade: lessons from cognitive psychology Making sense of credibility on the Web: models for evaluating online information and recommendations for future research The spread of true and false news online Story and science Vaccine hesitancy, vaccine refusal and the anti-vaccine movement: influence, impact and implications Health literacy and its outcomes: application and extension of elaboration likelihood model Information literacy on the Web: how college students use visual and textual cues to assess credibility on health websites Factors associated with belief or disbelief in false news: from the perspective of elaboration likelihood and moderating effect model Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations We would like to thank our colleague from Fors Marsh Group, Userworks-recruiting firm, and National Cancer Institute Human-System Integration Lab for their assistance with various aspects of the project. The opinions expressed by the authors are their own and this material should not be interpreted as representing the official viewpoint of the US Department of Health and Human Services, the National Institutes of Health, or the National Cancer Institute. Code Availability Not applicable. The authors declare no competing interests. The online version contains supplementary material available at https:// doi. org/ 10. 1007/ s13187-021-02054-7.