key: cord-0858881-85k68r5h
authors: Solnick, Rachel E.; Chao, Grace; Ross, Ryan; Kraft‐Todd, Gordon T.; Kocher, Keith E.
title: Emergency Physicians and Personal Narratives Improve the Perceived Effectiveness of COVID‐19 Public Health Recommendations on Social Media: A Randomized Experiment
date: 2020-12-02
journal: Acad Emerg Med
DOI: 10.1111/acem.14188
sha: efe603c7cfd386bb2877f763e32f4ecb162f3b36
doc_id: 858881
cord_uid: 85k68r5h

BACKGROUND: Containment of the coronavirus disease 2019 (COVID‐19) pandemic requires the public to change behavior under social distancing mandates. Social media are important information dissemination platforms that can augment traditional channels communicating public health recommendations. The objective of the study is to assess the effectiveness of COVID‐19 public health messaging on Twitter when delivered by emergency physicians and containing personal narratives. METHODS: On April 30, 2020, we randomly assigned 2007 U.S. adults to an online survey using a 2x2 factorial design. Participants rated 1 of 4 simulated Twitter posts varied by messenger type (emergency physician vs federal official) and content (personal narrative vs impersonal guidance). Main outcomes were: perceived message effectiveness (35‐point scale); perceived attitude effectiveness (15‐point scale); likelihood to share Tweets (7‐point scale); and writing a letter to their governor to continue COVID‐19 restrictions (write letter or none). RESULTS: The physician/personal message had the strongest effect and significantly improved all main messaging outcomes except for letter‐writing. Unadjusted mean differences between physician/personal and federal/impersonal were: perceived messaging effectiveness (3.2 [95%CI, 2.4‐4.0]); perceived attitude effectiveness (1.3 [95%CI, 0.8‐1.7]); likelihood to share (0.4 [95%CI, 0.15‐0.7]). For letter‐writing, physician/ personal made no significant impact compared to federal/ impersonal (odds ratio 1.14 [95%CI, 0.89‐1.46]). CONCLUSIONS: Emergency physicians sharing personal narratives on Twitter are perceived to be more effective at communicating COVID‐19 health recommendations compared to federal officials sharing impersonal guidance.

The coronavirus disease 2019 (COVID-19) crisis has exposed the critical need for clearly and consistently communicating public health guidelines anchored in the best available evidence. Yet, many voices are competing with public health officials, particularly given that social media outlets frequently supplant traditional news sources. 1 Amid this backdrop, the U.S. has had higher COVID-19-associated deaths and excess all-cause mortality compared to most peer countries. 2

This article is protected by copyright. All rights reserved Despite the alarming rate of viral transmission, the public has not had full compliance with pandemic guidelines. 3, 4 Policymakers and public health officials therefore must be strategic in communicating pandemic-related messages to the public.

Emergency physicians can play a key role in disseminating and amplifying public health recommendations especially during a crisis. 5, 6 Emergency departments experienced the severity of the initial COVID-19 viral surge and were challenged by a rapid response to the influx of ED patients. [7] [8] [9] Serving at the front lines of the epidemic, emergency physicians have played a prominent role as a trusted source in communicating COVID-19 updates and urging the public to stay home. 6, 10, 11 The effectiveness of public messaging can be influenced by the credibility of the messenger 12, 13 and the content of the message. 14 However, there is little experimental data measuring the effectiveness of public health communication through personal narrative or by physicians, which has been commonly seen in social media posts during the COVID-19 pandemic.

Therefore, the goal of this study was to evaluate the effectiveness of a physician versus federal official and personal versus impersonal content in delivering COVID-19 public health recommendations on Twitter, a popular social media platform. We tested the following hypotheses: 1) Emergency physicians deliver a more effective message than federal officials; 2) Personal appeals are more effective than impersonal ones; and 3) The interaction of a physician messenger with a personal message is synergistic.

We conducted a preregistered randomized experiment using simulated Twitter accounts and posts that randomly manipulated messenger type and message content in a 2 × 2 between-subject factorial design. We launched the experiment on April 30, 2020, the day the White House-issued public restriction guidelines were set to expire, transferring decision-making responsibility on restrictions to state governments. This trial was approved by the institutional review board at the University of Michigan.

Written informed consent was obtained from participants before participation. This trial followed the Consolidated Standards of Reporting Trials (CONSORT) 15 guideline with suggested amendments for reporting nonpharmacologial treatments and factorial trials. 16 

This article is protected by copyright. All rights reserved

We recruited U.S.-based adult participants from Lucid Theorem, a nationally representative crowdsourced online subject pool that is quota-sampled to match census demographics on age, gender, race/ethnicity, and region. 17 Participants were eligible if ≥18 years old. We included responses for analysis if ≥80% of study questions were complete. We assessed the impact of weighting the sample based on demographic characteristics of U.S. adults with Internet access as reported by the 2017 U.S. Census. 18 (eTable 1 and eTable 2) Participants in Lucid were compensated at a rate comparable to $1 per study. Median time to complete the study was 11 minutes.

Participants accessed the online survey (Qualtrics, Provo UT) through their personal electronic devices and gave consent blinded to the study objectives. They first underwent a pre-treatment attention assessment with the correct answer embedded in the instruction stem. 19 We randomized participants to 1 of 4 treatment arms with simulated Twitter posts and they answered a series of questions to measure primary outcomes. This was followed by a second attention check to recall the messenger's occupation which was a means of assessing that participants read the post and had received the intervention. Lastly, participants were invited to take a stay-at-home pledge, write a letter to their governor, and to answer additional covariate questions.

We created images of a Twitter account and message for experimental exposures. We used the same male actor for the emergency physician (dressed in scrubs and a surgical cap) and the nonphysician federal official (business clothes). The background photo was a building selected to plausibly appear as either a federal building or hospital. We took other Twitter metrics (date joined, number of accounts followed and followers) from an exemplar emergency physician Twitter account which were the same across conditions.

For message content, we compared the effect of a personal versus impersonal message.

The personal message was based on "the identifiable victim effect", that having more identifiable information about a victim increases caring. 20 In contrast, the language for the impersonal message

This article is protected by copyright. All rights reserved was used directly from a mass federal communication mailed on postcards to 130 million U.S. households 21 as part of the "President's Coronavirus Guidelines for America" and from the White House "Opening up America Again" guidelines. 22, 23 The two messages had approximately the same number of words (personal:61, impersonal:55) and delivered a similar three-part message: (1) young people are at risk; (2) public activity restrictions should continue; and (3) continuing restrictions would reduce the risk of viral resurgence. (Figure 1 ).

Simple random assignment was accomplished via the randomizer tool in Qualtrics. Each participant was assigned to 1 of 4 possible treatment arms with equal probability: 498 to physician/personal (PP); 505 to physician/impersonal (PI); 505 to federal/personal (FP), and 499 to federal/impersonal (FI).

To evaluate the effect of messages, we measured (1) perceived message effectiveness (PME), (2) perceived attitude effectiveness (PAE), and (3) behavioral outcomes: likelihood to share, write a letter to a governor. The PME scale was intended to measure the message's emotional impact, and was adapted from a scale used in the context of smoking cessation research. 24 Participants evaluated the messages as: memorable, grabbed my attention, powerful, meaningful, and convincing on a 7-point Likert scale "Strongly disagree" to "Strongly agree" 

to the original scale reliability (α=.93) and a general factor that accounted for 82.6% of the variance.

We measured likelihood to share the Tweet as an estimator of the messages' behavioral impact. This was measured on a 7-point Likert scale "Extremely unlikely" to "Extremely likely" (coded 1-7). Self-reported willingness to share social media posts has previously been correlated with increased sharing in reality. 26 Lastly, we asked participants whether they were interested in writing a letter to their state governor (yes/no). Participants who agreed were provided a free-text response box to write to the governor (not a form letter) and were truthfully informed we would send this letter anonymously, which we did via state government online communication forms. Because of the cognitive effort involved, the letter-writing task is less susceptible to desirability bias. 27

As an exploratory outcome, we asked participants to take a pledge (yes/no) to stay inside to fight COVID-19. Pledging has been a popular way in the COVID-19 pandemic for concerned groups to encourage social distancing. 28 Prior research indicates that pledging to engage in prosocial behavior (e.g., voting, environmental protection) has a small but significant effect on increasing the desired outcome. 29

We incorporated additional variables in a covariate-adjusted model and to explore heterogeneous treatment effects using demographic information provided by Lucid (age, education, race/ethnicity, sex, household income, political party, state), which we supplemented with survey questions on overall health, marital status, population density, number in household, employment status, and political ideology. We also collected variables related to health behaviors, policy positions, and messaging receptiveness: anxiety about coronavirus, trust in federal officials and physicians, 30 economy vs public health trade-off, 31 political engagement, 32 consumption of media bias via AllSides rankings, 33 empathy (using the empathic concern subscale of the Brief Interpersonal Reactivity Index 34 ), and news exposure frequency. Finally, we incorporated data on

This article is protected by copyright. All rights reserved the extent of COVID-19 cases and restrictions based on the participant's state of residence (Supplement section 3).

Sample size was determined from a pilot survey with 601 Lucid participants conducted two weeks prior and not included in the final study. We estimated with 438 participants per treatment arm (N =1752), the minimum detectable effect at 80% power using a 2-sided hypothesis test (α = .05) is approximately 0.10 standardized units for a bivariate outcome difference of letterwriting.

The statistical analysis plan was pre-registered prior to data collection through the Open Science Framework (Supplement Section 9). We compared demographic characteristics and outcomes across groups by analysis of variance and T-Test for continuous variables and χ2 test and Z-test of proportions for categorical variables. As recommended for the accurate reporting of factorial studies, 16 we present three major comparisons: (1) 4-level treatment effects; (2) each factor pooled (messenger and message content); and (3) interaction between factors. Assumptions for each statistical test were evaluated using standard diagnostic tests and no major violations were found.

We estimated treatment effects using ordinary least-squares linear regression and logistic regression on the 4-level treatment factor, with federal impersonal as the omitted reference category. Regression models were covariate-adjusted to maximize the precision of estimated treatment effects. Covariates were selected by items expected to be associated with social distancing, then manually backward selected for inclusion based on the strength of the association with the outcome and Akaike information criterion (AIC) of the model fit: race/ethnicity, marital status, political party, gender, COVID-19 anxiety, news frequency, and economy vs public health trade-off. All models were assessed for violations of basic assumptions and no major violations were found. Participants with missing value for a variable were included with a missing data indicator for that variable.

We also examined whether subgroups of participants were affected differently by treatments using generalized random forest, a machine learning algorithm that estimates treatment effect heterogeneity as a function of each participant's covariate profile by nonparametric statistical estimation based on random forests. 35 Understanding how demographics may contribute to different responses to messaging can help in creating tailored content for specific groups at

This article is protected by copyright. All rights reserved higher-risk for COVID-19. 4 Identifying these groups would create opportunities for audience segmentation -varying messaging strategies to address different groups -as demonstrated in climate science communication literature. 36 We assessed the effect heterogeneity specifically for PME because as an emotion-based rapid cognition, we hypothesized it would be more likely to be influenced by demographic profiles. 37 R version 3.5.2 (R Foundation for Statistical Computing) was used for statistical analyses, and the grf package was used for Causal Forests. 38

Of 2090 participants who entered the survey, 2007 consented, were randomized, and completed the survey with ≥80% data (eFigure 1). All participants that were randomized were included in the analysis. Participants' mean age was 45 years (SD 16.7 years), 51% (n=1034) were female, 10 .6% (n=214) were Black, and 11.6% (n=234) were Hispanic. Baseline characteristics and covariates were well-balanced across the four treatment arms (Table 1, eTable 3) .

For the 4-level treatment results, participants rated PME, PAE and likelihood to share significantly higher in the physician/personal (PP) condition compared with the federal/impersonal (FI) condition, with largest effect on PME (Figure 2 

This article is protected by copyright. All rights reserved

The average effects of the messenger and message are presented in eTable 7. The pooled treatment effect of both personal content and physician messenger had a statistically significant impact on both PME and PAE. Cohen's D, a standardized measure of effect size, is presented here to facilitate comparing across different scales--0.2 is considered a small effect and 0.5 a medium effect. 39 The average personal content had a stronger effect compared to physician messenger for PME (0. 40 Conversely, personal content did not significantly increase likelihood to share, while the physician messenger retained a positive effect (0.17 [95%CI, 0.05 to 0.30] p=0.006). We found a negative interaction for PME such that physicians had an incrementally increased score compared to federal officials when presenting for the impersonal context, but less so for the personal narrative (-1.18

[95% CI, -2.35 to -0.02]; P=0.045). No significant interactions were found for the other primary outcomes.

We presented participants with two attention checks. Most participants passed the post-outcome measured manipulation check, correctly selecting the occupation in the Twitter profile (81.1%, n=1628). Far fewer passed the pre-exposure check in which the correct answer was hidden within the instruction paragraph (52.1%, n=1046). The groups were similar in treatment effects but had slightly stronger effects in the groups with higher levels of attention checks. (eTable 8, eFigure 2)

We did not find significant heterogeneity in causal forest-estimated treatment effects of the personal message on PME. Causal forest was trained on many key variables, and test set predictions and CIs were assessed (Figure 3) . While some patterns visually emerged among the variables specifically selected for graphical illustration based on hypothesised effect heterogeneity-political ideology, health status, age, and race/ ethnicity-all individual confidence intervals overlapped, coinciding with the null global test.

This article is protected by copyright. All rights reserved To our knowledge, this is the first large-scale, nationally representative, pre-registered, randomized experiment to directly estimate the effect of a physician versus federal official messenger and message content of simulated social media posts on individual perceptions, attitudes, and behavior. We found that public health messages delivered by physicians and personal messages elicited stronger emotions, greater changes in attitudes and an increased willingness to disseminate the message than when federal officials delivered impersonal messages.

We did not observe differences in a stay-at-home pledge (which was near ceiling), nor in willingness to write a letter to the governor to continue restrictions. These findings suggest that to emergency physicians sharing personal stories on social media may be more effective in increasing general adherence to public health guidelines than federal officials sharing impersonal messages. Complementary communication campaigns are still needed to augment these recommendations in order to change pandemic related individual behavior.

Our study adds important findings of source effects and messaging content on a nontraditional communication platform during this public health crisis. We demonstrate that trusted messengers can alter opinions on contentious public policy issues consistent with prior experiments finding a medical scientist and physician increased support for antimicrobial resistance policy 12 and comparative effectiveness research, 13 respectively. The framing of health messages also matters. Similar to identifiable victim effect findings, we found enhanced emotional and attitudinal impact when the message was to help a single, identifiable person (i.e. the COVID-19 victim who was a friend) compared to the concept of helping the many, unidentifiable others. 20, 40 Moreover, findings of increased public health messaging effectiveness from personal narratives is also supported by organ donation literature, which has shown that when viewers are more emotionally involved in a television narrative they were more likely to become organ donors if the show encouraged donation. 41 We also assessed heterogeneous treatment effects to determine if there were distinct subpopulations which were impacted by the intervention differently, a finding which would be helpful for tailoring messaging for different groups. Despite a rigorous investigation harnessing machine learning tools, we found no significant impact of any participant characteristic, on the extent or direction of the message's impact, specifically examining political ideology, health status, age, and race/ ethnicity. Although we did not observe a differential impact of the

This article is protected by copyright. All rights reserved emergency physician or federal official on lower income or minority participants, underserved populations may have lower trust in physicians than those included in our study, 42 and may interact with messages differently from our participants. Future research should examine how to most effectively communicate with underserved minority populations hardest hit by the pandemic.

Our results add to a growing body of research investigating the impact of social media platforms for public health communication. The majority of Twitter users cite it as a news source, 1 presenting an opportunity for health professionals to capitalize on this channel as an adjunct for reaching a broader segment of the public. Physicians, scientists, and health providers have played an increasing role on Twitter, using it to share personal communications 43 and engage with the public on health issues. 44 Relevant to a pandemic, Twitter has been identified as a tool for efficient information dissemination during emergency events 5 and in public health crises to communicate recommendations. 45 Our findings support the increased use of Twitter by healthcare professionals as a platform to communicate directly to the public.

While government mandated public activity restrictions and social distancing recommendations play a key role in preventing the spread of COVID-19, these interventions will be ineffective if the public is not willing to adhere to them. Social media based public messaging may help to improve the public's perception of these measures and thus adherence to health guidelines. However, during the pandemic, several U.S. healthcare institutions urged physicians not to make public appeals. [46] [47] [48] [49] Our findings bolster policies that protect social media use by scientists and health providers to share public health communications directly to the public.

This study has several limitations. First, the experimental design used a simulated Twitter message in the context of an online survey. Federal officials may be restricted on what they can communicate on social media using their official titles, but pilot data for this experiment showed most participants found the Twitter stimuli believable. It is possible that participants would react differently if they encountered these messages on the actual social media platform. However, participant likelihood to share a post has been shown to correlate highly with action in real life. 26 Furthermore, while the effects of user comments on social media were beyond the scope of this study, prior research has shown that user comments may have an additive effect on messaging

This article is protected by copyright. All rights reserved impact, 50,51 though whether it will change reader behavior is unknown. Although we observed an increased willingness to share certain messages, we did not find differences in pledging to stay home nor writing a letter to the governor to maintain restrictions. It remains unclear if the impact of the messages would translate into real-life changes in compliance with social distancing measures. Second, though the participant pool matches U.S. demographics in most regards, our participants had higher educational attainment and lower proportion of Hispanic origin (approximately 15.4% of U.S. population with access to internet versus 11% in our study) 18 We weighted our sample to account for educational differences and still did not observe an appreciable impact on treatment effects (eTable 3). Further supporting generalizability, Lucid participants have exhibited behavioral experimental results similar to U.S. national probability samples. 17 Third, the high levels of reported anxiety created a likely ceiling effect for our outcomes. For PME, almost half of participants rated the message at 6 or above on a 7-point scale. Ceiling effects may have reduced sensitivity to determining differences by treatment, biasing results towards null.

Lastly, we selected white males for the physician and federal official in the study, the most common demographic for both groups. It is possible that other race and genders of the Twitter messenger could have influenced subpopulations of this study differently than white males, however prior patient satisfaction simulation studies did not find differences by physician race or gender. 52

Using a rigorous randomized experiment of a simulated Twitter message, we found that an emergency physician's Twitter message of a personal story and recommendation related to COVID-19 increased the attitudinal, emotional and willingness to share measures of impact compared to a federal official sharing impersonal guidance. These results underscore the advocacy role for physicians on social media in promoting public health recommendations. We did not find an impact on letter writing to their governor to support COVID-19 restrictions nor pledging to stay home. Future directions should explore the real-world impact of emergency physician public health tweets on measures of behavior change. 

This article is protected by copyright. All rights reserved 

This article is protected by copyright. All rights reserved 

The Evolving Role of News on Twitter and Facebook

Pew Research Center's Journalism Project

COVID-19 and Excess All-Cause Mortality in the US

Comparison Countries

COVID-19 Stress, Coping, and Adherence to CDC Guidelines

African American Adherence to COVID-19 Public Health Recommendations. Health Lit Res Pract

Twitter adoption and use in mass convergence and emergency events

COVID-19: Emergency Medicine Physician Empowered to Shape Perspectives on This Public Health Crisis

13 Deaths in a Day: An "Apocalyptic" Coronavirus Surge at an N.Y.C. Hospital [Internet]. The New York Times

A snapshot of emergency department volumes in the "epicenter of the epicenter" of the COVID-19 pandemic

Accepted Article This article is protected by copyright. All rights reserved 2020

Redesigning emergency department operations amidst a viral pandemic

Emerging Lessons From COVID-19 Response in New York City

50 Experts to Trust in a Pandemic

Enlisting the support of trusted sources to tackle policy problems: The case of antimicrobial resistance

Doctor knows best: physician endorsements, public opinion, and the politics of comparative effectiveness research. J Health Polit Policy Law

Perceived Effectiveness of Antismoking Ads and Association with Quit Attempts Among Smokers: Evidence from the Tips From Former Smokers Campaign. Health Commun

Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration

Accepted Article This article is protected by copyright. All rights reserved

Analysis and reporting of factorial trials: a systematic review

Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents

Broadband Adoption and Computer Use by year, state, demographic characteristics -Data.gov

Using screeners to measure respondent attention on self-administered surveys: Which items and how many? Political Science Research and Methods

Helping a Victim or Helping the Victim: Altruism and Identifiability

US households are being mailed "President Trump"s Coronavirus Guidelines for America

Opening Up America Again | The White House

Available from: Accepted Article This article is protected by copyright

Perceived effectiveness of cessation advertisements: the importance of audience reactions and practical implications for media campaign planning

UNC Perceived Message Effectiveness: Validation of a Brief Scale

Self-reported willingness to share political news articles in online surveys correlates with actual sharing on Twitter

A new scale of social desirability independent of psychopathology

Sign the Petition

Walking the walk? Experiments on the effect of pledging to vote on youth turnout

Nurses Continue to Rate Highest in Honesty

All rights reserved 31

Appendix A: Measures and scales

AllSides Media Bias Ratings

Development of a Brief Form of the Interpersonal Reactivity Index (B-IRI)

Generalized random forests

Identifying like-minded audiences for global warming public engagement campaigns: an audience segmentation analysis and tool development

The affect heuristic

Statistical Power Analysis for the Behavioral Sciences

Accepted Article This article is protected by copyright. All rights reserved

Explaining the Identifiable Victim Effect

The Power of Narratives: The Effect of Entertainment Television Organ Donation Storylines on the Attitudes, Knowledge, and Behaviors of Donors and Nondonors

Overcoming Lower-Income Patients' Concerns About Trust And Respect From Providers

Twitter as a tool for communication and knowledge exchange in academic medicine: A guide for skeptics and novices

When Scientists Tweet for Social Changes: Dialogic Communication and Collective Mobilization Strategies by Flint Water Study Scientists on Twitter

Hospitals Muzzle Doctors and Nurses on PPE, COVID-19 Cases

Hospitals Tell Doctors They'll Be Fired If They Speak Out About Lack of Gear

Hospitals Must Let Doctors and Nurses Speak Out. The Atlantic [Internet] Accepted Article This article is protected by copyright. All rights reserved 2020

One Doctor Says Of Her Emergency Room

How Web Comments Affect Perceptions of Political Interviews and Journalistic Control: Effects of Web News Attributions

Interactivity between Candidates and Citizens on a Social Networking Site: Effects on Perceptions and Vote Intentions

Effect of Physician Gender and Race on Simulated Patients' Ratings and Confidence in Their Physicians: A Randomized Trial

Generic machine learning inference on heterogenous treatment effects in randomized experiments