key: cord-0897494-lfsohuir
authors: Bartels, Sophia M.; Gora Combs, Katherine; Lazard, Allison J.; Shelus, Victoria; Davis, C. Hunter; Rothschild, Allison; Drewry, Maura; Carpenter, Kathryn; Newman, Emily; Goldblatt, Allison; Dasgupta, Nabarun; Hill, Lauren M.; Ribisl, Kurt M.
title: Development and Application of an Interdisciplinary Rapid Message Testing Model for COVID-19 in North Carolina
date: 2021-05-19
journal: Public Health Rep
DOI: 10.1177/00333549211018676
sha: 0b3800425c9625aeeb654e5f1eca3eb50150a7fa
doc_id: 897494
cord_uid: lfsohuir

INTRODUCTION: From the onset of the COVID-19 pandemic, public health officials have sought to develop evidence-based messages to reduce COVID-19 transmission by communicating key information to media outlets and the public. We describe the development of an interdisciplinary rapid message testing model to quickly create, test, and share messages with public health officials for use in health campaigns and policy briefings. METHODS: An interdisciplinary research team from the University of North Carolina at Chapel Hill assembled in March 2020 to assist the state health department in developing evidence-based messages to influence social distancing behaviors in the state. We developed and iteratively executed a rapid message testing model; the components of the 4-step model were message creation, survey development, survey administration, and analysis and presentation to health department officials. The model was executed 4 times, each during a 7-day period in April and May, and each subsequent survey included new phrasing and/or messaging informed by the previous week’s survey. A total of 917 adults from North Carolina participated in the 4 surveys. RESULTS: Survey participants rated messages focused on protecting oneself and others higher than messages focused on norms and fear-based approaches. Pairing behaviors with motivations increased participants’ desire to social distance across all themes and subgroups. For example, adding “Protect your grandmother, your neighbor with cancer, and your best friend with asthma,” to messaging received a 0.9-point higher score than the base message, “Stay 6 feet apart from others when out in public.” PRACTICE IMPLICATIONS: Our model to promote social distancing in North Carolina during the COVID-19 pandemic can be used for rapid, iterative message testing during public health emergencies.

demonstrates the necessity of social distancing to reduce the incidence of disease and prevent the health care system from being overwhelmed. 5 The Centers for Disease Control and Prevention (CDC) lists social distancing as the most effective way to reduce the spread of COVID-19, because the route of COVID-19 transmission is mainly among people who are within 6 feet of each other for an extended period. 4 At the start of the COVID-19 pandemic, the need for evidence-based messaging for preventive behaviors, especially social distancing and staying home when possible, among North Carolina residents was clear. Because of the rapidly escalating number of cases, these messages also needed to be expedited. Building partnerships with state health departments and university research teams is one way to accelerate the translation of messaging research to practice. We describe a rapid message testing model centered on a partnership between the North Carolina Department of Health and Human Services (NCDHHS) and the University of North Carolina (UNC) at Chapel Hill. This model allowed our team to develop messages with a broad impact in North Carolina (ie, common denominator communication approach) and identify whether messages resonated among certain groups (eg, rural populations). 6, 7 Our approach could be applied in other states to develop messaging to prevent the spread of COVID-19 and other emerging threats.

An interdisciplinary research team from UNC with expertise in health communication, epidemiology, health behavior, and public policy quickly assembled to address NCDHHS's needs. The interdisciplinary nature of this team was vital because it allowed us to develop messages in real time that were informed by the combined expertise and theories of the 20 or so team members. We aimed to develop messages that resonated with all North Carolina residents, including several key populations: rural residents, populations that are disproportionately affected by COVID-19 (eg, Black/African American people), and people perceived as less likely to social distance (eg, young adults and Republicans).

Although numerous messaging processes exist, 8-10 they do not incorporate perspectives from multiple disciplines in real time, are not agile enough to rapidly develop and test messages in emergency situations, or are unable to respond quickly to changing environments and health department needs. Other messaging approaches have incorporated one of these elements, but our approach is uniquely applicable to the context of emergency situations. We describe the creation and implementation of an interdisciplinary rapid message testing model, which was implemented during the course of 4 surveys, in the context of the COVID-19 pandemic and how it could be used to guide research teams in their work with health departments to develop messages during a public health emergency.

An interdisciplinary team of undergraduate and graduate research assistants, faculty, and staff members from UNC assembled in March 2020 to assist NCDHHS in developing evidence-based messages to influence social distancing behaviors in North Carolina. Team members were from the UNC Hussman School of Journalism and Media, the Gillings School of Global Public Health, and the Department of Political Science. The team of volunteers quickly assembled without institutional constraints (eg, hiring freezes) during the rapidly escalating pandemic. Our interdisciplinary team approached the social distancing messaging from various perspectives. For example, communication experts provided guidance on strategic message development, and health behavior experts contributed their understanding of health behavior change. Team members from epidemiology provided insight on public healthinformed guidelines for social distancing behaviors, and political science experts provided evidence for population-based values and sociodemographic nuances predictive of social distancing.

The research team was split into workgroups for tasks associated with each iterative step of the rapid message testing model-a common process for health communication development. 11 Each workgroup consisted of enough members to allow flexible scheduling and substitution if someone was unavailable (eg, if 2 members were essential, then the group had 3 or 4 members). At each step, faculty in the research group mentored the research assistants. The UNC Institutional Review Board deemed this research as exempt.

Our rapid message testing model was iteratively executed during a 7-day period using an expedited common health communication development process. 11 Step 1 was message creation. On days 1 and 2, two research assistants developed messages to reach all residents in North Carolina. Messages were developed based on recommended behaviors from NCDHHS and public health faculty, social distancing messages from other states, and key theoretical predictors for health behavior change (eg, attitudes, risk perceptions). [12] [13] [14] [15] [16] [17] For example, people may want to socially distance because they want to protect themselves or people at high risk of contracting COVID-19 (perceived severity), feel pressure from family (social norms), or are able and feel a responsibility to do so (self-efficacy). We chose to focus on motivations, or reasons for social distancing, rather than barriers to compliance, because barriers were not well-studied or understood in North Carolina initially. Assessing how much a message motivates-or discourages-action is strongly correlated with actual behavior in other health contexts and is a promising approach for novel health behavior decisions. 18 Step 2 in our rapid message testing model was the development of the week's social distancing message evaluation survey, which tested phrases or messages to encourage social distancing among North Carolina residents. One research assistant used the evaluation goals to create a codebook with programming directions and measures, and a second research assistant programmed the codebook in Qualtrics (Qualtrics Intl), an online survey software. The codebook included new messages from the message development team, items to evaluate these messages, and sociodemographic questions. The codebook was approved by the leadership team before being programmed. Once programmed, the codebook developer and a member of the leadership team tested and finalized the codebook. The codebook was created on days 3 and 4 and programmed and tested on the evening of day 4.

Step 3 was administering the survey to North Carolina residents aged ≥18. On day 5, the survey went live on Amazon Mechanical Turk (MTurk), a crowdsourcing platform often used for communication and social science research 19, 20 and ran until the morning of day 6, or until we had reached about 200 participants. Using MTurk-an established, accessible survey platform-allowed our team to quickly reach participants remotely, given that in-person message testing was not viable because of the pandemic, and filter for participants who lived in North Carolina. We selected a sample size of about 200 participants because of the feasibility of recruitment within our rapid time frame and to have a large enough sample to enhance our ability to detect within-subject differences. 21 A total of 917 adults from North Carolina participated in the 4 surveys. Participants were aged 18-72, with a mean (SD) age of 41 (12.8) in survey 1, 37 (12.9) in survey 2, 37 (12.5) in survey 3, and 37 (13.0) in survey 4. Most participants were White, female, and from suburban areas ( Table 1 ). All participants provided consent electronically before participating in each survey.

Step 4 was analyzing survey results, interpreting findings, and presenting insights to key members of NCDHHS. A research assistant analyzed the survey results on the evening of day 6, using SPSS version 26 (IBM Corp). Key findings, summary descriptive statistics (eg, means), message rankings, and differences in message ranking by subgroup breakdowns were shared for interpretation and development of the week's message framework. In survey 4, we also examined differences in message theme-comparing a baseline behavioral message with other messages with an added motivation-using repeated-measures analyses of variance. Results were packaged succinctly to be directly usable by NCDHHS. On day 7, project leadership shared findings and suggested message strategies. In this meeting, the team heard from NCDHHS how priorities had or had not changed and received guidance for selecting messages for further testing. After the meeting, the entire team discussed findings, incorporated new guidelines, and strategized next steps. This feedback then restarted the rapid testing cycle.

Messaging strategies shared each week with government officials were directly and immediately incorporated into public COVID-19 briefings for North Carolina residents and informed messages delivered throughout the state on retail signage and social media.

Each anonymous weekly survey executed during the rapid message testing model included items for new phrasing and/or messages. Surveys 1-4 were launched on April 9, 15, and 22 and on May 6, 2020, respectively. Insights from each survey informed subsequent surveys in this interdependent process. In survey 1, we focused on phrase salience and understanding motivations. Participants rated how much reasons for social distancing made them want to comply with recommendations and preferred phrasing, on a scale ranging from 1 (not at all) to 5 (a great deal). In survey 2, we focused messaging on 4 behaviors: (1) minimizing in-person socializing with people outside the household, (2) not having anyone from outside the household in one's home, (3) leaving one's home only for essential items and exercise, and (4) staying at least 6 feet away from others when leaving one's home. The key motivations and phrases from survey 1 were incorporated into these messages. In survey 3, we developed 5 themes from the high-scoring messages from survey 2: (1) save lives, (2) protect people you love, (3) keep your loved ones safe, (4) a personalized promise ("For my [friend or family member]-I promise to stay inside for you"), and (5) change is tough. Theme 5 was added based on the literature and on team member recommendations showing the importance of challenging a resistance-to-change mindset. 23 In survey 4, we retained well-performing themes and systematically paired enduring social distancing behaviors with motivations. As the pandemic response was evolving, we shifted our focus from messages centered on not leaving one's home, based on concerns of stigmatizing people who were unable to stay home (ie, essential workers). In addition, new CDC recommendations encouraged wearing face coverings in public, and focus shifted to staying home if sick rather than staying home in general, 24 as states began to ease restrictions.

In survey 1 (content identification), participants were most motivated to keep others (eg, family members, people at high risk for contracting or themselves safe and were not motivated by protecting their image (eg, not wanting to be blamed for spreading the virus). Results were similar among subgroups. High-scoring motivations and phrases from survey 1 were incorporated into the messages tested in our second survey. From survey 2 (message refinement), we learned that people in North Carolina preferred general messages (eg, stay home to save lives) rather than specific suggestions (eg, not socializing in your driveway). They also preferred general messages rather than highlighting specific behaviors as "valid" or "invalid" (eg, "making a trip to the store just to get your favorite ice cream is an invalid reason to leave home"). Residents also responded well to messages that provided examples of people they are protecting by social distancing.

Using these results, we identified 5 themes, noted in the Methods section, to refine in survey 3. From survey 3 (message refinement), our data further demonstrated that a focus on motivations (eg, "stay home to protect your grandmother") was important to encourage social distancing. We also found that people did not like messages that focused on a change mindset (eg, "Change is tough"). Finally, from survey 4, we found that pairing behaviors with motivations substantially increased desires to social distance across all themes and subgroups. For example, adding "Protect your grandmother, your neighbor with cancer, and your best friend with asthma," on average had a 0.9-point higher score than the base message, "Stay 6 feet apart from others when out in public," a substantial increase (Table 2 ).

One major barrier to evidence-based health communication and policy making is the time it takes for new evidence to be disseminated and used. Although it can take years from research question development to dissemination of findings, in emergency situations policy makers need to quickly know the most promising messages. In addition, the work of policy makers and researchers is often done in isolation, leading to evidence that is not useful to the policy maker and a lack of evidence-based policies. [25] [26] [27] Our interdisciplinary model for rapid message testing, in this implementation context, reduces the time needed for evidence-based messages to reach policy makers. This approach increases the relevance of research for policy makers and public health officials. By basing message development on feedback from state agencies, researchers can respond to policy makers' changing needs on a weekly basis, a necessity in public health crises such as the COVID-19 pandemic. This approach draws on elements of lean and agile principles, which are centered on eliminating waste to reach an "ideal process" and incorporating knowledge into practice through rapidly developing usable prototypes based on testing, user feedback, and validation. 28, 29 Agile designs have been used in other areas of COVID-19 research, such as the development of applications to improve well-being during social distancing. 30 As the threat of global disasters and pandemics increases with climate change and population growth, these innovative models fill the need for evidencebased messages, products, and interventions that are developed quickly and respond to changing conditions. 31, 32 The rapid message testing model has 4 strengths. The first 2 strengths are its speed and responsiveness. The model allows evidence-based messaging to be quickly available to policy makers and can be used by public health departments to communicate with the public about health behaviors. The model's ability to be iterative encourages stronger messages with each survey and allows for changes to be made to the . Messages were ranked based on mean score within a theme or category; the denominator for "rank" is the number of messages within each theme or category. b Messages were scored on a Likert-type scale (1 = not at all, 2 = a little, 3 = somewhat, 4 = quite a bit, 5 = a great deal).

messages with the discovery of new information. We found that general messages to protect others and oneself were the most motivating across all participants and subgroups, whereas fear-based messages, which other research has found to be ineffective in promoting behavior change, were not motivating. 33 This model would also allow teams to detect whether messages resonate differently among subgroups for targeted communication (eg, unique messages could be designed for people at various levels of readiness to adopt behaviors). The third strength is that an interdisciplinary team created the model. By including faculty, staff members, and students from multiple departments, message development and evaluation benefited from the combined strength of theories and methodologies. Public health challenges, such as global pandemics, that are multidimensional and complex can be addressed only through the merging of multiple perspectives and disciplines. 34 The final strength of the model is its cost-effectiveness. Leveraging accessible consumer panels (eg, MTurk) and working in real-time collaborations reduces overhead associated with traditional research methods. As opposed to studies that require substantial investment with no guarantee that the results will reach policy makers or be implemented, our model is designed to provide public health officials with evidencebased messages. 35 This model also has several limitations. First, rapidly changing knowledge about COVID-19 transmission resulted in modifications to the key behaviors across each survey (ie, from social distancing only to social distancing and wearing face coverings), similar to campaign strategies that have shifted with emerging evidence for how to best prevent infection. [36] [37] [38] This change in key behaviors required alterations to previously successful messages, which then had to be retested and iterated with input from NCDHHS. However, this limitation was specific to the volatility of the pandemic and would likely be less of a problem for better-understood health behaviors and disasters where messaging can be tracked during longer periods. 39 Second, the rapid time frame of this model, along with the use of a crowdsourcing platform, made it difficult to achieve a representative sample in each survey. Specifically, systematic barriers, such as internet connectivity, reduce the ability of platforms, such as MTurk, to reach diverse populations. In addition, the platform does not allow for the recruitment of a probabilistic sample. These characteristics limit the generalizability of findings to the target audience. To improve targeting of messages for groups at high risk of contracting COVID-19, additional data collection methods, such as participatory research techniques, should be considered. 26 Third, the suggested time frame can make it difficult to generate new messages each week while also amending messaging in response to changing knowledge about COVID-19. This challenge can be addressed by ensuring an appropriate-sized team (which may vary with the scope of the project) with the ability to produce the expected deliverables, although quickly forming an interdisciplinary team that is not voluntary may introduce additional challenges. We recommend having enough team members to rotate through tasks to allow for roles to be covered if someone is not available. Finally, our team used a single item to assess the perceived effectiveness of the message, whereas a multi-item scale may have produced scores that better incorporated all dimensions of message salience.

The proposed rapid message testing model provides a process for researchers to place evidence-based messages in the hands of public health departments and policy makers in real time. This model may be especially helpful during times of national disasters and public health crises. Further research should examine the feasibility of this model for other public health challenges outside of COVID-19. Research should also determine if the utility of this model holds with a range of health areas and if replicated across states. Researchers should also examine if the inclusion of qualitative and participatory research techniques increases positive reception and reach of the messages produced. 40 

North Carolina Department of Health and Human Services. COVID-19 North Carolina dashboard

North Carolina flagged as "red zone": for COVID-19, the Triangle hit the worst

North Carolina Department of Health and Human Services

North Carolina COVID-19 modeling shows social distancing necessary to slow the spread and preserve hospital capacity to save lives

Racial/ethnic disparities and segmentation in communication campaigns

Segmentation on a shoestring: health audience segmentation in limited-budget and local social marketing interventions

Building a health communication brand for University of Texas System tobacco control

Website designs for communicating about chemicals in cigarette smoke

Interdisciplinary health promotion: a call for theory-based interventions drawing on the skills of multiple disciplines

Making Health Communication Programs Work: A Planner's Guide. National Cancer Institute

Historical origins of the Health Belief Model

Executive order no. 121: stay at home order and strategic directions for North Carolina in response to increasing COVID-19 cases

Social distancing

A look at the math behind social distancing amid coronavirus. Global News

Review of predicting and changing behavior: the reasoned action approach

Factors influencing behavior and behavior change

Incremental criterion validity of message perceptions and effects perceptions in the context of anti-smoking messages

Similarities and differences in tobacco control research findings from convenience and probability samples

Crowdsourcing research: data collection with Amazon's Mechanical Turk

How Many Subjects? Statistical Power Analysis in Research

American Community Survey 5-year estimates

Challenging "resistance to change

Centers for Disease Control and Prevention. Guidance for businesses and employers responding to coronavirus disease 2019

Understanding evidence-based public health policy

Power of information: closing the gap between research and policy

A review of model public health laws

Evaluating lean in healthcare

Adopting agile principles in health care. Health Aff blog

Agile requirements engineering and software planning for a digital health platform to engage the effects of isolation caused by social distancing: case study

Global Increase in Climate-Related Disasters. 92. ADB Economics Working Paper Series

Towards systematic evaluation of epidemic responses during humanitarian crises: a scoping review of existing public health evaluation frameworks

Sixty years of fear appeal research: current state of the evidence

Interdisciplinary research in public health: the "why" and the "how

Understanding relevance of health research: considerations in the context of research impact assessment

Harnessing behavioural science in public health campaigns to maintain "social distancing" in response to the COVID-19 pandemic: key principles

Motivations and barriers for the use of face coverings during the COVID-19 pandemic: messaging insights from focus groups

Leveraging media and health communication strategies to overcome the COVID-19 infodemic

Changes in effectiveness of cigarette health warnings over time in Canada and the United States

Visual and participatory research methods for the development of health messages for underserved populations

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr Hill was partially supported by the National Institute of Mental Health (K01MH121186).

Sophia M. Bartels, MSPH https:// orcid. org/ 0000-0002-9431-8735 Katherine Gora Combs, MPH https:// orcid. org/ 0000-0001-8409-5569 Allison J. Lazard, PhD https:// orcid. org/ 0000-0002-2502-2850