key: cord-1048778-2w3uoau1
authors: Southwick, Lauren; Guntuku, Sharath C.; Klinger, Elissa V.; Seltzer, Emily; McCalpin, Haley J.; Merchant, Raina M.
title: Characterizing COVID-19 Content Posted to TikTok: Public Sentiment and Response During the First Phase of the COVID-19 Pandemic
date: 2021-06-22
journal: J Adolesc Health
DOI: 10.1016/j.jadohealth.2021.05.010
sha: e5945a3ee8ac8a7e537c7a40de3fbb071b94ff64
doc_id: 1048778
cord_uid: 2w3uoau1

PURPOSE: The purpose of this study was to characterize COVID-19 content posted by users and disseminated via TikTok, a social media platform that has become known largely as an entertainment platform for viral video-sharing. We sought to capture how TikTok videos posted during the initial months of the COVID pandemic changed over time as cases accelerated. METHODS: This study is an observational analysis of sequential TikTok videos with #coronavirus from January to March 2020. Videos were independently coded to assess content (e.g., health relatedness, humor, fear, empathy), misinformation, and public sentiment. To assess engagement, we also codified how often videos were shared relative to their content. RESULTS: We coded 750 videos and approximately one in four videos tagged with #coronavirus featured health-related content such as featuring objects such as face masks, hand sanitizer, and other cleaning products. Most videos evoked “humor/parody,” whereas 15% and 6% evoked “fear” and “empathy”, respectively. TikTok videos posted in March 2020 had the largest number of shares and comments compared with January and February 2020. The proportion of shares and comments for “misleading and incorrect information” featured in videos was lower in March than in January and February 2020. There was no statistical difference between the share and comment counts of videos coded as "incorrect/incomplete" and "correct" over the entire time period. CONCLUSIONS: Analyzing readily available social media platforms, such as TikTok provides real-time insights into public views, frequency and types of misinformation, and norms toward COVID-19. Analyzing TikTok videos has the potential to be used to inform public health messaging and public health mitigation strategies.

While the COVID-19 pandemic continues, the ways in which users both generate and interact with content on simultaneously evolving platforms such as TikTok may play increasing roles in how individuals make sense of, receive information about, and ultimately adhere to important public health messages.

With social distancing measures and staggered re-opening phases unfolding simultaneously across the globe, individuals are increasingly seeking information via digital and online platforms, by as much as 40%e70% compared with prepandemic times [1, 2] . Previous research found that non-educational screen time among young people has greatly increased [3] . Social media platforms, such as Facebook, Twitter, Instagram, TikTok, and Snapchat, have played a large role in "youth resocialization in a pandemic society" and COVID-19 information dissemination [4] . Nearly all U.S. teens, aged 13 to 17 years, have access to a smartphone, while almost half are "almost constantly" on the Internet [5] . However, increased Internet and social media activity does not translate into increase in validated and reliable information. In 2020, the World Health Organization coined a new term, infodemic, defined as, "an overabundance of informationdsome accurate and some notdthat makes it hard for people to find trustworthy sources and reliable guidance when they need it" [6] . Young people, especially, have the propensity to believe a false claim about COVID-19. A U.S. survey found that nearly one-third of respondents younger than the age of 25 years (28%) believe that "humans originally contracted COVID-19 by eating bats" [7] . Additional research is warranted to explore how young people's perception of the COVID-19 pandemic is reflected on social media platforms [8] .

Previous research demonstrates that social media content, such as Instagram, Twitter, and Facebook, can be analyzed by public health practitioners to identify anonymous real-time public health insights [9, 10] and used during public health emergencies, such as the Ebola and Zika outbreaks [11] . Odlum and Yoon (2018) [11] conducted a content analysis of tweets to assess health information needs surrounding the Ebola outbreak. They analyzed tweets at distinct epidemic time points and observed public fear, frustration, and health information seeking regarding Ebola-related information. Similarly, Seltzer et al. (2017) [9] explored Instagram to characterize public sentiment regarding the Zika outbreak. They reported that more than half of the Instagram posts included misleading, incomplete, or unclear information about the Zika virus. They found that image-based social media platforms are effective at disseminating factual health information during a public health crisis. Both studies signal how user-generated posts on social media platforms can be used as a tool to gauge and track not only public sentiment but also general knowledge during a public health emergency. Because health information, regardless of accuracy, is increasingly being disseminated and consumed through social media [12, 13] , it is therefore critical to better understand and characterize the landscape of misleading or misinformative user-generated social media posts.

In addition to gleaning public health insights, social media and the broader realm of mHealth also offer a digital platform for interventions promoting healthy habits and preventive behaviors, especially among adolescents [14] . There is a growing body of promising evidence suggesting that such interventions are highly accepted among adolescents as a method of augmented health care and are not only accessible but also uniquely scalable as digital technologies, and their users continue to proliferate. A primary concern in this research is maintaining costeffectiveness in developing and implementing mHealth interventions. Using social media platforms as a means of engaging and educating adolescents on topics of public health is one such low-cost method [15, 16] .

TikTok, a global social media platform where microvideos (60 seconds) are shared and amplified, is one of the fastest growing social media platforms in the world [17] . Founded in 2017, the platform allows users to create and share short videos and has become the most downloaded nongaming app globally. It is available in more than 150 countries and reports to have more than 800 million active users [18, 19] . TikTok videos can last from several seconds to several minutes, and the platform is known for its user interface design that facilitates easy-to-use editing and music soundtrack functionality. TikTok's feed is unique from other social media platforms; users interact with the platform on the following sections: "For You" and # Discover (by hashtag or audio file), where the TikTok algorithm populates videos that have a high likelihood of user engagement. This engagement with the platform is different from other videobased platforms, such as YouTube, where there is no video length limit and where users have more autonomy in choosing the next video they wish to view. TikTok is primarily used by young people [4] ; 40% percent of TikTok users are between the ages of 16e24 years [17] and 63.5% are younger than 29 years of age [20] . Especially during the COVID-19 pandemic where nearly all communication is virtual, young people especially are turning to TikTok. This population segment also presents a key demographic for targeted health and risk communication to flatten and curb the COVID-19 pandemic.

There is limited research on TikTok and COVID-19 [4, 17] . Ostrovsky and Chen (2020) [4] analyzed the 100 most popular videos with the hashtags, "covid-19," "covid19," and "coronavirus" in July 2020. Among the 100 videos sampled, they received more than four billion, 33 billion, and 93 billion views, respectively. Basch et al. (2020) [21] analyzed TikTok videos with #Coronavirus and found that one in 10 videos mentioned how COVID-19 is transmitted, its symptoms, and/or prevention methods. They also identified that "anxiety" and "quarantine" were the most cited topics in TikTok videos. Most recently, researchers examined TikTok videos marked with the hashtags of "Mask" and "Masks" to document and explore the extent of mask misinformation published on TikTok [22] . A recent scoping review of social media's intended and unintended uses in the COVID-19 pandemic concluded that social media platforms were critical avenues for "surveying public attitudes, identifying infodemics, assessing mental health, detecting or predicting COVID-19 cases, analyzing government responses to the pandemic, and evaluating quality of health information in prevention education videos" [23] . As the pandemic continues, understanding how popular social media platforms, such as TikTok, can reveal prevailing public attitudes is valuable. These insights can allow for targeted public health campaigns aimed at topics of misinformation and stimulate efforts to address the mental health consequences of the pandemic. Given TikTok's virality and potential to disseminate public health messaging [24] , especially among young people, further research is warranted to characterize how COVID-19 is portrayed and discussed on TikTok and how content changes over time. Gleaning from previous research on disease outbreaks, such as Ebola [11] and Zika [9] , being portrayed on social media platforms, we hypothesize that proportion of informational videos will increase over time, that several videos will portray humor or parody during the initial months of the pandemic, and that videos featuring "healthcare workers" will increase over time.

Our analysis sought to advance our understanding of the COVID-19 pandemic on TikTok. The purpose of this exploratory content analysis was to describe and track the content of TikTok videos with the hashtag #coronavirus in the early phase of the COVID-19 pandemic from January to March 2020. Since TikTok videos are publicly available, this analysis did not undergo institutional review board approval.

We conducted a retrospective review of publicly available TikTok videos posted to the platform in January, February, and March 2020. We used #Coronavirus to extract TikTok videos on February 4, March 6, and April 2, 2020, for a total sample of 2,676 videos. We extracted videos on three dates to capture the change in content, public sentiment, and the proportion of misinformation as the pandemic spread and social distancing and quarantine practices evolved. We also captured how TikTok users engaged with the content (i.e., share) over time. Our sample included 250 randomly selected TikTok videos from each month, January, February, and March 2020, representing a total sample of 750 videos.

We used the coding approach from previous work [9] ; initial 50 videos from January 2020 were coded to identify themes for the codebook. We then iterated on this approach and recoded additional 50 videos from January 2020 to build and refine the codebook. Our codebook consisted of narrative reliability, misleading information, video production, video sentiment, and video content. Narrative reliability classified types of narrators such as "misleading" or "intended to inform." Misleading information included how to properly wear personal protective equipment, how COVID-19 is spread, or incorrect statistics about spread and mortality. Coders consulted the Centers for Disease Control and Prevention guidelines to determine and confirm "misleading information." Video production captured the types of editing and music functionalities used. Video sentiment captured the video's prevailing mood or attitude. Finally, video content included items and objects seen in the video such as healthcare products or images of other social media or technological platforms such as a Google Home (Table 1 ). All sub-theme coding was mutually exclusive. Using REDCap, a secure Web application for building and managing online surveys and databases, the 750 videos were coded ( Table 1 ). The final code for each video was decided by the agreement between three reviewers (ES, LS, and BB).

We performed descriptive statistics to quantify theme proportion per time period (e.g., January 2020, February 2020, and March 2020). To quantify the degree of change in theme prevalence, we compared relative counts across months (January and February 2020; February and March 2020; January and March 2020) using the two-proportion Z-test. All outcomes with p value < .05 were considered statistically significant. Word clouds were generated to highlight the most used hashtags. The word clouds exclude stop words such as #coronavirus, #fyp,#foryourpage, #foryou, #fy, #corona, and #xyzcba in addition to the default set of stop words used in linguistic analyses [25] . We used the default set of stop words used in conventional linguistic analyses from the Natural Language Toolkit library. In addition, we added 6 words to account for the platform-specific behavior and the keyword used to pull the data [25] . The #fyp, #foryoupage, #foryou, and #fy are common hashtag strategy used to increase the likelihood to be featured on the main "For You" homepage; #xyzcba is used similarly.

Twenty-seven percent of videos were identified as having incorrect, misleading, or incomplete information about COVID-19 ( Table 1 Figure 1. ). Videos posted in January had the highest proportion of misleading information (n ¼ 77, 32%). Of the 750 TikTok videos,17% (n ¼ 115) of the videos were coded as "intended to inform" the general public. For example, select videos discussed global infection rates or the Johns Hopkins University COVID-19 U.S. map, while others highlighted that COVID-19 is spread by mosquito bites and how cleaning products such as Lysol can be used to kill "human coronavirus" (Figure 1 ). Figure 2 depicts how the hashtags in videos coded as "intended to inform" change across months as the pandemic accelerates. January and February videos focused on geographic locations, such as China, and safety with the hashtags of #Wuhan, #asian, #health, #staysafe, while March hashtags shifted toward life and lifestyle adjustment owing to quarantine and shelter-in-place orders, such as #college, #edutok, #playwithlife, and #coronatime.

Most videos included real-time footage classified as action (n ¼ 472, 74%) and included supplemental text or emojis within the video (n ¼ 361, 56%). Nearly half of TikTok videos tagged with #coronavirus were coded as "humor/parody" (n ¼ 316, 49%). On average, roughly 1 in seven videos evoked "fear" and approximately 6% evoked "empathy." Figure 3 depicts the most frequently used hashtags in each month: a blend of COVID-19e related words such as coronavirus, COVID-19, quarantine, and cultural references, such as, "Superbowl," "comedy," and being "happy at home."

Approximately 1 in four videos featured health-related content such as ways to protect against the spread of infection such as use of personal protective equipment, soap, disinfectants, hand sanitizer, and other cleaning supplies. Most health-related content was produced by lay individuals. Only 30 videos (4%) were produced by individuals wearing scrubs or announcing their medical training or credentials on TikTok. A small proportion of videos were classified as an expose (n ¼ 63, 9%), as Figure 1 . Screenshots of "incorrect, misleading" labeled TikTok videos and of "intended to inform" labeled TikTok videos. politically oriented (n ¼ 28, 4%), and included racialized comments (n ¼ 54, 7%).

As COVID-19 information became more mainstream and saturated media outlets between January and March 2020, Tik-Tok video content shifted, significantly (p < .05; two-proportion Z-test). Narrative reliability improved over time; January 2020 had the largest proportion of "misleading" videos compared with February and March. March 2020 had the largest proportion (n ¼ 15, 8%) of "intended to inform" video content compared with January and February (Table 1 ). While the proportion of "humor/parody" videos remained constant across January and February, it increased in March. The number of videos coded as "fear" decreased over time (Table 2) . Furthermore, the number of videos expressing "empathy" increased significantly in March compared with January ( Table 1 ). The content of misleading and incorrect videos predominantly featured expose as video production, which reduced significantly as the pandemic progressed from January to March. Several misleading and incomplete videos also used racial stereotypes and elicited "fear" and "humor" ( Table 2) .

TikTok videos posted in March 2020 had the largest number of shares (mean ¼ 56,573; SD ¼ 75,419) and comments (mean ¼ 7,972; SD ¼ 9,712) compared with January and February. However, the proportion of shares and comments for misleading and incorrect videos was lower in March than in January and February (Table 3 ). There was no statistical difference in the share and comment counts of videos coded as incorrect/incomplete and correct over the entire time period.

Our study described and tracked the content of TikTok videos with the hashtag #coronavirus in the early phase of the COVID-19 pandemic from January to March 2020. Some of our hypotheses were supported. The proportion of videos labeled as "intended to inform" increased over time (i.e., January to March 2020). While the proportion of videos labeled as "humor/parody" stayed constant in January and February (40%) and increased to 68% in March 2020. The proportion of videos featuring "healthcare workers" stayed constant at 4% and 5% from January to March 2020.

Our analysis found that TikTok videos tagged with #coronavirus from January to March 2020 were largely humorous (Table 1 ). Most videos reflected pre-COVID-19 habits and lifestyle modifications in the wake of COVID-19. Videos captured how the user and their social networks coped with their "new normal," which consisted of mask wearing, supply shortages, social distancing measures, closed high schools and colleges, and remote learning and workplace practices. TikTok videos were also used to disseminate information related to the COVID-19 pandemic. However, the information was not always accurate or correct; more than a quarter (27%) of videos were identified as having incorrect, misleading, or incomplete information about COVID-19. These findings differ substantially from those reported by Ostrovsky and Chen (2020), which found that less than 1% provided misleading or inaccurate health advice. Our sample included mostly laypeople's videos, which may account for this difference, whereas their sample includes nearly onefifth of videos created by the World Health Organization.

Interestingly, a small proportion of videos (4%) were produced by individuals wearing scrubs or announcing their medical training or credentials on TikTok. TikTok does not have a means of addressing claims that contributors are credentialed healthcare workers. Within the last six months, there has been an increasing trend of healthcare workers using TikTok for health awareness and communications campaigns [21] . Previous research conducted by Ostrovsky and Chen (2020) [4] found that though a small portion of TikTok videos posted in July 2020 were filmed by healthcare professionals, they were broadly "liked" and shared, signifying the demand and opportunity for healthcare providers to use the platform as a means of public health education. As the country prepares for subsequent waves, further research is needed to explore how healthcare professionals use TikTok and other social media platforms to disseminate timely, accurate health information [26] .

As the pandemic accelerated and spread across the globe, we observed a shift in the sentiment of TikTok videos. January 2020 videos were more likely to feature racialized comments (Table 1 ). In January, COVID-19 was often portrayed as a "Chinese" pandemic rather than a global pandemic. This rhetoric speaks to the concept of "platformed racism," coined by Matamoros-Fernández (2017) [27] to describe a form of racism that stems from social media platforms when they create and amplify prevailing racist discourse. In addition, the proportion of "humor/parody" videos increased in March, while the number of videos coded as "fear" decreased over time. Furthermore, the number of videos expressing empathy increased significantly in March compared with January (Table 1) . Emerging research on public health emergencies and fear expressed on social media found a relationship between psychological distance to an epidemic and public attention and sentiment expressed on Twitter [28] . Specifically, the closer the public health emergency is, the higher the frequency of fear expressed on Twitter. This finding is particularly striking and pertinent for public health messaging to craft emotionally pertinent and scientifically driven content [24] .

Our findings suggest that TikTok is poised to promote realtime COVID-19 updates and interpersonal conversations among their users, primarily young people [29] . Drawing on best practices from the Truth Initiative (formerly known as the American Legacy Foundation), a nonprofit public health organization committed to changing cultural and social tobacco norms and preventing youth from becoming tobacco cigarette smokers, we suggest targeted messaging on COVID-19 safety practices to appeal to young people. For example, a TikTok video featuring a handwashing dance and challenge went viral early in the pandemic. The video was created by collaboration between Vietnamese youth and the Vietnam's National Institute of Occupational and Environmental Health to motivate young people to properly wash their hands. This viral video exemplifies how a best practice can reach billions, especially youth on important public health messages and behaviors [24] . With college campuses hosting in-person classes, we recommend similar work to promote prosocial health behaviors to help curb the spread of COVID-19 [30] and increase public understanding of COVID-19 vaccines. Further research is needed to explore how TikTok videos influence their users' health behaviors. Blackstone and Russie (2020) conducted a study where Instagram users attended a four-week wellness intervention with targeted posts discussing fitness, nutritious eating, and positive self-care behaviors. Similar interventions on social distancing and getting the COVID-19 vaccine should be piloted and evaluated on TikTok users. There is a growing call to action for future research on delivering innovative health care digitally, particularly to children and adolescents [14, 31, 32] . Badawy and Radovic (2020) [32] emphasize adolescence as a critical time in which healthy and preventive behaviors can be encouraged and built and stress the potential for digital approaches to provide cost-effective, user-centered, and equitable care. In addition to healthy habit formation and public health education, there is also a call for digital interventions targeting the psychosocial challenges that COVID-19 has presented to children and adolescents, particularly those with chronic health conditions [14, 31] .

Future research directions include qualitative research on TikTok users to better understand how TikTok videos influence their understanding of the COVID-19 pandemic and piloting TikTok interventions to inform young people's perception of the COVID-19 pandemic and how to promote public health practices such as, mask wearing, contact tracing, social distancing, and getting the COVID-19 vaccine.

There are several strengths to our analysis. Our study sample uniquely captures lifestyle changes early in the pandemic when public knowledge was limited. Second, our sample includes videos across three months to capture how content changes over time. Previous research collected TikTok videos from a single date [4, 21, 22] . Related to our knowledge, our sample represents the largest number of TikTok videos coded for a content analysis. Previously research collected and coded 100 TikTok videos [4, 21, 22] . Third, our codebook captured both video themes and production. TikTok is known for its user-friendly production features to which other social media platforms such as Instagram are now emulating. Finally, our analysis captured public attitude change as schools transitioned to online learning and states issued mandatory shelter-in-place orders.

Nevertheless, there are limitations to our sample and analysis. First, we only identified videos using #coronavirus, and we know that additional hashtags such as #covid, #covid19, #covidpandemic, among other related hashtags, often co-occurring with #coronavirus, might yield subsets of videos with differing content, sentiment, and levels of incorrect content and misinformation. For example, #covidtreatment-tagged videos might be more likely to contain misleading, inaccurate, or even potentially dangerous information. We also had no control of changes in TikTok's ranking algorithm over time. Recent press reports that TikTok's algorithm pushes videos based on the type of content uploaded rather than the user who created/uploaded it [33] . Although we looked at video content over a period of months, we looked at a small window in time (JanuaryeMarch 2020) during which the pandemic was just beginning to accelerate in Europe and the United States. In addition, we cannot determine user or viewer demographic characteristics, such as age, gender, raceethnicity and we can only posit the user's intention, such as humor/parody and reliability. We did not translate non-English videos. Similarly, the videos captured and analyzed represent a segment of the population which may not represent the general public. We coded videos without their associated captions to which we may not have discerned the full "intent" of the video. Finally, our codebook was developed in late January 2020 and did not capture emerging trends and topics in March 2020 such as panic buying and remote schooling. Future research should implement a dynamic codebook that quickly adapts to an evolving environment.

As we work to prepare for the next pandemic or public health crisis, we must be equipped to proactively promote timely, accurate health information and dispel misinformation on social media platforms. Our findings demonstrate that TikTok is actively being used for COVID-19 discussion and reflection and misleading/misinformative videos are disseminated. Our analysis characterized the type of videos posted with #coronavirus, providing a call to action to intervene and promote factual health information and combat misinformation. Future research is needed to explore how TikTok can be used as an interventional tool for young people to promote prosocial and accurate COVID-19 information.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Inside out and outside in: How the COVID-19 pandemic affects self-disclosure on social media

Social media use during social distancing

Available at: https://parents-together. org/survey-shows-parents-alarmed-as-kids-screen-time-skyrockets-durin g-covid-19-crisis/?mod¼article_inline

TikTok and its role in COVID-19 information Propagation

Teens who use cellphones do it to pass time, connect with others, learn new things | Pew Research Center

How to fight an infodemic

The State oF the Nation: A 50-State COVID-19 survey Report #14: Misinformation and vaccine Acceptance USA

Dissemination of mask effectiveness misinformation using TikTok as a Medium

Public sentiment and discourse about Zika virus on Instagram

Social media and emergency Preparedness in response to Novel coronavirus

Health information needs and health seeking behavior during the 2014-2016 ebola outbreak: A twitter content analysis

Social media as a new Vital Sign: Commentary

Americans who get news mostly through social media are least likely to follow coronavirus coverage | Pew Research Center

Technology use for adolescent health and wellness

eHealth and mHealth interventions in pediatric cancer: A systematic review of interventions across the cancer continuum

Patient-centered eHealth interventions for children, adolescents, and adults with sickle cell disease: Systematic review

TikTok statistics that You need to know

How health communication via tik tok makes a difference: A content analysis of tik tok accounts run by Chinese provincial health committees

Getting acquainted with social networks and apps: It is time to talk about TikTok

TikTok users by age 2020 | Statista

COVID-19 on TikTok: Harnessing an emerging social media platform to convey important public health messages

Dissemination of mask effectiveness misinformation using TikTok as a Medium

What social media told us in the time of COVID-19: A scoping review

Reddit and TikTok: A proposal for health authorities to integrate popular social media platforms in contingency planning amid a global pandemic outbreak

word_cloud/stopwords at master $ amueller/word_cloud $ GitHub. Available at

Using Instagram to deliver a theory-based wellness intervention designed to promote behavior change

Platformed racism: The mediation and circulation of an Australian race-based controversy on twitter, Facebook and YouTube

Too far to care? Measuring public attention and fear for ebola using twitter

Harnessing youth and young Adult culture: Improving the reach and engagement of the truthÒ campaign

Preventing and responding to COVID-19 on college campuses

Psychosocial challenges and opportunities for youth with chronic health conditions during the COVID-19 pandemic

Digital approaches to remote pediatric health care delivery during the COVID-19 pandemic: Existing evidence and a call for further research

TikTok explains how its algorithm really works | the Independent | the Independent. Available at