key: cord-0696412-awhq1urh authors: Marwah, Harleen K.; Carlson, Kyle; Rosseau, Natalie A.; Chretien, Katherine C.; Kind, Terry; Jackson, Hope T. title: Videos, Views, and Vaccines: Evaluating the Quality of COVID-19 Communications on YouTube date: 2021-08-31 journal: Disaster Med Public Health Prep DOI: 10.1017/dmp.2021.284 sha: 672ba90b4807621672fb736be89daeec98e1831c doc_id: 696412 cord_uid: awhq1urh OBJECTIVES: As the coronavirus disease 2019 (COVID-19) vaccine is introduced, it is critical to recognize that public opinion on vaccines is largely influenced by health communications, with YouTube being a major source of information and misinformation. This analysis graded the accuracy, quality, and reliability of the most viewed YouTube videos depicting COVID-19 and vaccinations over a 6-mo period. METHODS: We collected hyperlinks for the 150 most viewed YouTube videos discussing COVID-19 from January through June 2020. Closed captioning data were searched for the term “vaccine,” yielding 32 videos. This sample was evaluated for quality, accuracy, and reliability using a rubric that incorporated existing instruments: Global Quality Scale (GQS), JAMA Benchmark Criteria, and DISCERN. RESULTS: These 32 videos had 139,764,188 views at the time of data collection. The majority of videos received low scores, with network news sources receiving the lowest scores overall. CONCLUSIONS: The overall quality of COVID-19 YouTube videos related to vaccines may be low and raises a precautionary alert for the public consuming these videos and for health-care providers working to provide the best information to their patients. Existing scoring tools may not capture the complexities of social media. New tools could allow for a better understanding of the modern landscape of health communications. Since its emergence in December 2019, 1,2 the coronavirus disease 2019 (COVID- 19) has upended global systems, and fundamentally changed the way that humans live and interact around the world. The acuity and rapid spread of SARS-CoV-2, as well as the uncertainty provoked by the COVID-19, led the public to turn to the Internet for information and real-time guidance. Previous studies have shown that 50% of the world population has access to the Internet, 3 and that 70% of American adults use the Internet to find health information. 4 As the Internet is a key tool for delivering health information, understanding the quality of health-related content posted online during the recent global health crisis is particularly pressing and relevant. 5 While sharing accurate, reliable information online is an effective way to reduce public panic, as well as provide updates and direction, circulating misinformation or holding back credible information can have devastating consequences. The importance of this issue was underscored by the Director General of the World Health Organization (WHO) on February 15, 2020, at the Munich Security Conference; he asserted that "[w]e're not just fighting an epidemic; we're fighting an infodemic" because "[f]ake news spreads faster and more easily than this virus, and is just as dangerous. 6 YouTube has more than 2 billion monthly users and is accessed by one-third of people using the Internet globally, making it the second largest social networking website. 7, 8 As such, this platform plays host to a variety of health-related media. The quality of YouTube videos about health topics, however, has been called into question. For example, the quality of information on YouTube regarding gastroesophageal reflux disease was found to be low in a study by Aydin and Aydin. 9 Furthermore, during public health crises, studies of YouTube content during the H1N1, Ebola, and Zika epidemics revealed that up to a quarter of videos about these topics shared misleading information. [10] [11] [12] [13] This phenomenon has also been observed during earlier stages of the coronavirus pandemic. In late March 2020, Li et al. found that 27.5% of COVID-19 videos posted on YouTube contained inaccurate information, and that content from reputable sources was under-viewed. 10 these data are particularly alarming, as it suggests that health misinformation from YouTube is reaching more individuals than during previous public health crises. 10 As the world continues to grapple with the consequences of the COVID-19 pandemic, vaccine development has been touted as an opportunity for long-term prevention and control of this global health crisis. Public opinion on vaccines is largely influenced by health communications on this subject, 14 with YouTube being a major source of information and misinformation. 15 Data has shown that there are more anti-vaccine videos on YouTube compared to provaccine content (65.5% vs 25.3%), and that more than one-third of the most popular videos about vaccines on YouTube did not include any scientific evidence. 15 In a recent study examining 100 widely viewed YouTube videos on COVID-19 vaccinations, nearly three-quarters of videos were uploaded by news sources; however, quality and accuracy of the video content was not assessed. 16 As global attention has turned to COVID-19 vaccine development and distribution 17 (Chen, 2020) and to the global exacerbation of pre-existing health disparities by the COVID-19 crisis, [18] [19] [20] [21] [22] [23] [24] [25] [26] it is important to explore these topics in more depth. In an effort to both build on existing literature, as well as update the academic community's understanding of this issue in parallel with the progression of the pandemic, this study seeks to investigate the most viewed content on YouTube during the first 6 mo of 2020 and assess the quality and accuracy of information regarding COVID-19 content related to vaccines on YouTube. The study performed a YouTube (www.youtube.com) query on June 27, 2020, using the search terms "COVID-19," "Coronavirus," and "Coronavirus pandemic." To avoid user-based video recommendations, the search was run in "incognito mode," which clears tracking data such as cookies and viewing history. Filters were applied, so as to collect only English-language results from the past year. Results were sorted by "views" to obtain the most viewed, and, therefore, most popular videos, at the time of the search. The hyperlinks for the 150 most viewed videos were collected, assigned a unique video identification number, and organized in a spreadsheet. This method has been validated in previous studies across disciplines of YouTube's online library. 10, 11, 27, 28 Exclusion criteria were applied to non-English language videos, live-streams, or content not related to COVID-19. English-language videos that met the search criteria were included in the study. The following video characteristics were extracted for each YouTube video: (1) view count, (2) length of video, (3) number of comments, (4) number of "likes"/"dislikes," and (5) uploading source. Uploading source was defined as the uploading user as listed below the YouTube video. Uploading source was further organized into the categories of: Independent Users/Consumers, Professional Organization, University Channels, Entertainments News, Network News, Internet News, Government, Newspaper, Education, or Medical Advertisements/For-Profit Companies. YouTube offers automatically generated closed caption files, containing the text of what is said in a video. Additionally, YouTube contributors can upload their own closed caption files generated by a third-party transcriber. For the 150 most viewed videos, closed captioning data were downloaded as a text file, and a quantitative analysis was performed to determine the most used terms across the full 6-mo period. To determine how many of the 150 total videos discussed vaccines, closed captioning data for all videos were searched for the following terms: Vaccine/Vaccines; Vaccination/Vaccinations; Immunization/Immunize/Immunizations; Vaccine-preventable; Live-attenuated; Inactivated vaccines; Subunit, recombinant, polysaccharide, and conjugate vaccine(s); Toxoid vaccine(s); and Placebo. This list of terms was created using reliable sources including the Centers for Disease Control and Prevention and the World Health Organization and was vetted by 3 practicing physicians across 3 specialties. 29, 30 Of the 150 total videos, 50 videos included these terms in the closed caption search. These 50 videos were further narrowed to include only videos that included the term "vaccine," yielding 32 videos that went on to be reviewed for quality ( Figure 1 ). The 32 videos including the term "vaccine" in the closed captioning search were further evaluated for quality, accuracy, and reliability using a rubric that incorporated tools from prior studies, including Global Quality Scale (GQS), JAMA Benchmark Criteria, and DISCERN. 10, 11, 28, [31] [32] [33] The "JAMA Benchmark Criteria" was created in 1997 and focuses on authorship, attribution, disclosure, and currency through 4 yes/no questions. The "DISCERN tool," validated by 28 health information providers, was adapted to evaluate reliability through a series of 8 yes/no questions. 34 The "Global Quality Scale" offers a 5-point scale evaluating quality and coverage of appropriate content. At least 3 reviewers from the research team independently reviewed and analyzed all 32 videos according to the rubric (Table 1) . To start, 4 reviewers evaluated the same 4 videos using the rubric and discussed how they interpreted the results. Once the process was standardized, 3 reviewers were randomly assigned to each video. Numeric scores were averaged across reviewers. For yes/no questions, the majority answer was selected. Last, videos were scored according to the criteria described in Table 2 . 11, 28 (1) Useful: Did the video contain scientifically correct information about any aspect of the COVID-19: symptoms, treatment, prevention of disease, or measures being taken to combat the disease? (2) Misleading: Did the video contain scientifically unproven or inaccurate information based on available scientific evidence (eg, unsubstantiated claims about pathogenesis and treatment with unproven therapies or negative portrayal of evidence-based solutions)?* *If "misleading" selected: What were the key aspects in which this video was misleading? Quality 28 (adapted from the GQS) Score the video quality using the scale below: (1) Poor quality, poor flow of video, most information missing, not at all useful for patients. (2) Generally poor quality and poor flow, some information listed but many important topics missing, of very limited use to patients. (3) Moderate quality, suboptimal flow, some important information is adequately discussed but others poorly discussed, somewhat useful for patients. (4) Good quality and generally good flow. Most of the relevant information is listed, but some topics not covered, useful for patients. (1) Yes (2) No Attribution: Is copyright information clearly listed AND are references/ sources for content stated or listed? (1) Yes (2) No Currency: Are updates to content with dates provided? (1) Yes Study data were collected and managed using REDCap electronic data capture tools hosted at George Washington University and Children's National Hospital. REDCap (Research Electronic Data Capture) is a secure, Web-based software platform designed to support data capture for research studies, providing (1) an intuitive interface for validated data capture, (2) audit trails for tracking data manipulation and export procedures, (3) automated export procedures for seamless data downloads to common statistical packages, and (4) procedures for data integration and interoperability with external sources. 35, 36 The study was not deemed human subject research and thus did not require Institutional Review Board review. The 150 most viewed videos between January 1, 2020, and June 27, 2020, had over 618 million views. Of the 150 most viewed videos from January 1, 2020, to June 27, 2020, the largest proportion (40%) were uploaded in March and the lowest proportion were uploaded in January (5%) (Figure 2) . March also had the highest engagement numbers, with nearly 350 million views represented. The average video length across all 150 videos was 11 min and 26 s (Table 3 ). The 2 major uploading sources included Network News (27%) and Entertainment News (25%). Less than 2% of the 150 most viewed videos were from professional organizations. Quality, Accuracy, and Reliability of Videos With the Term "Vaccine" The 32 reviewed videos were uploaded between the dates January 1, 2020, and May 31, 2020, and from a variety of sources ( Figure 3 ). Collectively, these videos received 139,764,188 views at the time of data collection ( Table 4 ). The majority of videos were rated useful by the research review team (n = 29), while several (n = 3) were rated misleading. For videos rated misleading, reviewer comments were captured, and analyzed. Comments reflected the videos' use of anecdotal story and hearsay over the use of evidence-based, scientific information. Aggregate scores for quality, accuracy, and reliability domains were then generated for the 32 videos containing the term "vaccine." An overall mean score was then calculated for each domain of quality, accuracy, and reliability for all videos ( Table 4 ). The average quality score for all 32 videos was 3.63 of 5 (standard deviation [SD] = 0.83), the average accuracy score was 1.28 of 4 (SD = 0.81), and the average reliability score was 3.69 of 5 (SD = 1.12). Additionally, after grouping videos by month, a monthly mean score was calculated for each of these domains for January through May ( Figure 4 ). The scores in each domain demonstrated little fluctuation from month to month, with the exception of May showing a decrease in quality and reliability. Of note, the 3 videos that were rated as misleading by reviewers before aggregate scoring also occurred during May. No particular month's videos revealed a considerable increase in the scoring of any domain, despite increasing and growing scientific knowledge regarding COVID-19. Last, each video's aggregate scores for quality, accuracy, and reliability were compared against that same video's number of views. This was done to assess if higher or lower rated videos received considerable engagement. There was no discernable pattern identified between ratings and viewership. This study and assessment of videos has potential limitations. The ratings were done by members of the study; thus, the ratings scales are subject to human interpretation. We offset this limitation by building our scoring instrument combining multiple scales that have been used in prior studies and also having at least 3 reviewers for each video. Next, we used auto-generated closed captioning to identify videos using the word "vaccine." It is possible that the Calculating points from 1 to 5, with 1 being poorest quality and 5 being highest quality. Accuracy and Reliability 28, 31, 32 (adapted from JAMA Benchmark Criteria) Calculating points from 0 to 4, earning 1 point for each "Yes" and 0 points for each "No." A score of 4 indicates higher source accuracy and reliability, whereas a score of 0 indicates poor source accuracy and reliability. Reliability of Information 10,28,32 (adapted from DISCERN tool) Calculating points from 0 to 5, earning 1 point for each "Yes" and 0 points for each "No." A score of 5 points indicates higher reliability, whereas a score of 0 indicates poor reliability. This analysis graded the accuracy, quality, and reliability of the most viewed YouTube COVID-19 videos that included content on vaccines over a 6-mo period, thereby offering insight on the type of information viewed by the general public. Of note, the majority of videos received low scores based on a scoring instrument developed from 3 widely cited tools, with network news sources receiving some of the lowest scores overall. Other studies examining YouTube communication during the COVID-19 pandemic reviewed videos from 1 distinct time point, with an inability to assess trends over time. 37 This study builds on prior work; our data provide insight into how patterns in uploading source, engagement, and quality have evolved over the course of the pandemic. Additionally, by examining the 150 most viewed videos over the course of 6 mo, this sample size is larger than what is currently reported in the literature. To our knowledge, no other study used closed captioning data to further focus review on quality of information regarding vaccinations. Based on the study analysis, scores from the 32 videos with mention of vaccines were consistently low across all 3 rubrics. In looking at scores over time, scores dropped across all 3 rubrics in May. Additionally, the 3 videos that were rated as "misleading" appeared in the analysis in May. All videos rated as misleading included political commentary or coverage of political events. The decreased scores during this time could reflect a political shift and polarization of scientific information. In reviewing uploading sources, network news notably often scored below the 50% threshold; whereas uploading sources from the education category consistently received the highest scores. As the scoring rubrics were designed by the academic community, they may be best equipped to evaluate academic sources, leading to inaccurate scores among network news. In the age of social media, as a large share of the population gathers information through platforms like YouTube, it will be important to adapt scoring instruments to appropriately evaluate quality, reliability, and accuracy across a variety of media channels. Based on the analysis, the number of views did not correlate with rubric scores. It is important to note there was no relationship associated between lower scoring videos and higher levels of viewer engagement. This contradicts previous evidence that demonstrated that controversy may increase interest and lead to increased engagement. 38 The 3 scoring tools were designed before YouTube and social media evolved as primary sources of information for the general public. Reviewers acknowledged difficulty with the binary (yes/ no) scale, as it did not accommodate for a spectrum of information provided within the videos. For example, the JAMA Benchmark Criteria asks "Is copyright information clearly listed AND are resources/sources for content stated or listed?" Reviewers recognized that some videos offered 1 or the other, but the "AND" indicator in this question prevented analysis from further understanding levels of attribution. In the DISCERN Tool, a question states "Are additional sources of information listed for patient reference?" After discussion, reviewers agreed to answer this question based upon the specific video and comments; however, YouTube provided a blanket disclaimer with a link to the Centers for Disease Control and Prevention updates on any video discussing COVID-19. In further iterations of these scoring instruments, assessing platform versus specific media should be clarified. As the COVID-19 vaccine starts to be administered globally, it is more important now than ever to assess both how the public is receiving health information and the quality of these messages. Furthermore, with an overflow of information, there is a growing need to expand educational efforts so the general public can accurately distinguish reliable sources of information. This study highlights the challenges of applying currently existing evidence-based rubrics (GQS, JAMA, and DISCERN) to evolving health information on YouTube. These tools, while the best currently available, may not capture the complexities of social media; the authors would argue that new tools may allow for better understanding the modern landscape of health communications, and are more appropriate for print media. Suggestions include designing a new tool to assess health information across all media platforms, or developing a unique rubric for each social media platform, so as to more accurately capture the way in which information is shared and how the public interacts with it. In developing a guideline to assess the quality of health communications on YouTube, analysis of categories such as video thumbnails, comments, advertisements, and video currency should be considered. In addition, future rubrics can be improved by avoiding the use of stacked questions and defining key terms, so as to reduce the potential for greater heterogeneity in research outcomes and reduced comparability between studies. The World Bank. Individuals using the Internet Misinformation of COVID-19 on the internet: infodemiology study Munich Security Conference 2020 Political Polarization & Media Habits | Pew Research Center Quality and reliability of information available on YouTube and Google pertaining gastroesophageal reflux disease YouTube as a source of information on COVID-19: a pandemic of misinformation? YouTube as a source of information on the H1N1 influenza pandemic YouTube as a source of information on Ebola virus disease Are internet videos useful sources of information during global public health emergencies? A case study of YouTube videos during the 2015-16 Zika virus pandemic What do popular YouTubeTM videos say about vaccines? Child Care Health Dev YouTube coverage of COVID-19 vaccine development: implications for awareness and uptake Promise and challenges in the development of COVID-19 vaccines COVID-19 pandemic: disparate health impact on the Hispanic/Latinx population in the United States Impact of COVID-19 pandemic on sexual minority populations in Brazil: an analysis of social/racial disparities in maintaining social distancing and a description of sexual behavior Assessment of communitylevel disparities in coronavirus disease 2019 (COVID-19) infections and deaths in large US metropolitan areas COVID-19: magnifying the effect of health disparities Health equity and COVID-19: global perspectives Examining Social Determinants of Health, Stigma, and COVID-19 Disparities Racial health disparities and Covid-19-caution and context Socioeconomic gradient in health and the covid-19 outbreak Combating COVID-19: health equity matters Quality of online video resources concerning patient education for the meniscus: a YouTube-based quality-control study YouTube for information on rheumatoid arthritis-a wakeup call? Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor-Let the reader and viewer beware DISCERN: an instrument for judging the quality of written consumer health information on treatment choices YouTube as a source of chronic obstructive pulmonary disease patient education: a social media content analysis Orthopaedic patient information on the World Wide Web: an essential review The REDCap consortium: Building an international community of software platform partners Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support Mental health problems and social media exposure during COVID-19 outbreak Healthcare information on YouTube: a systematic review Disaster Medicine and Public Health Preparedness 7 Author Contributions. Harleen Marwah (Study Lead þ Author þ Analysis); Kyle Carlson (Corresponding Author þ Study Design þ Analysis), Natalie A. Rosseau (Author þ Literature Review); Katherine C. Chretien (Author þ Editor); Terry Kind (Author þ Editor), and Hope T. Jackson (Study Design þ Senior Author þ Editor).