key: cord-0802986-1v4ht9uh
authors: Li, Heidi Oi-Yee; Pastukhova, Elena; Brandts-Longtin, Olivier; Tan, Marcus G; Kirchhof, Mark G
title: YouTube as a source of misinformation on COVID-19 vaccination: a systematic analysis
date: 2022-03-09
journal: BMJ Glob Health
DOI: 10.1136/bmjgh-2021-008334
sha: 9d4045ed1f5a96b8626d1e7b584c4a668d81a8e5
doc_id: 802986
cord_uid: 1v4ht9uh

INTRODUCTION: Vaccines for SARS-CoV-2 have been accessible to the public since December 2020. However, only 58.3% of Americans are fully vaccinated as of 5 November 2021. Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic. Misinformation regarding the safety and efficacy of COVID-19 vaccines has negatively impacted vaccination intent. To date, the literature lacks a systematic evaluation of YouTube’s content on COVID-19 vaccination using validated scoring tools. The objective of this study was to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination. METHODS: A search on YouTube was performed on 21 July 2021, using keywords ‘COVID-19 vaccine’ on a cleared-cache web browser. Search results were sorted by ‘views’, and the top 150 most-viewed videos were collected and analysed. Duplicate, non-English, non-audiovisual, exceeding 1-hour duration, or videos unrelated to COVID-19 vaccine were excluded. The primary outcome was usability and reliability of videos, analysed using the modified DISCERN (mDISCERN) score, the modified Journal of the American Medical Association (mJAMA) score and the COVID-19 Vaccine Score (CVS). RESULTS: CONCLUSION: As the COVID-19 pandemic evolves, widespread adoption of vaccination is essential in reducing morbidity, mortality, and returning to some semblance of normalcy. Providing high-quality and engaging health information from reputable sources is essential in addressing vaccine hesitancy.

Heidi Oi-Yee Li , 1,2 Elena Pastukhova, 2 Olivier Brandts-Longtin, 2 Marcus G Tan, 1,2 Mark G Kirchhof 1, 2 Original research INTRODUCTION COVID-19, caused by the novel SARS-CoV-2, was first noted in December 2019 and has evolved into a global pandemic with over 246 million cases worldwide and nearly 5 million deaths to date. 1 Vaccines against SARS-CoV-2 became accessible to the public since December 2020, after being granted emergency use authorisation status by the Federal Drug Administration. 2 Despite this, as of 5 November 2021, only 58.3% of Americans are fully vaccinated. 3 This is concerning as public trust in vaccination programmes is paramount in reducing morbidity and mortality, limiting the spread of highly transmissible SARS-CoV-2 variants, and putting an end to this pandemic. 4 

What is already known? ► Throughout the COVID-19 pandemic, videos on social media questioning the efficacy and safety of COVID- 19 What do the new findings imply? ► Despite efforts by YouTube in combatting health misinformation, a substantial portion of highly viewed content contains misinformation regarding COVID-19 vaccines continues to be hosted by the second-largest social media platform. ► Public health officials and clinicians should recognise the impact of YouTube and other social media platforms as sources of health information and misinformation for patients and leverage these platforms to disseminate high-quality content to combat misinformation given poor vaccination uptake.

In the digital age, more individuals are relying on the internet and social media as sources of healthcare-related information. [5] [6] [7] [8] In particular, YouTube is one of the most predominant social media platforms with billions of views daily and is the second most used search engine after Google. 9 This readily accessible platform offers multiple genres of content, including entertainment, news, consumers, and education, and allows its users to interact with each other in real time.

Numerous studies have supported YouTube as a source of both reliable and misleading information during the COVID-19 pandemic. [10] [11] [12] A previous study demonstrated that 27.5% of the most viewed YouTube videos on COVID-19 contained misinformation. 11 As vaccine misinformation has been shown to negatively impact vaccination intent, 12 propagation of misinformation regarding the safety and efficacy of COVID-19 vaccines on large platforms, such as YouTube, has potential devastating consequences on public health.

To date, the literature lacks a systematic evaluation of YouTube as a source of information on COVID-19 vaccination using validated scoring tools. The objective of this study is to evaluate the accuracy, usability and quality of the most widely viewed YouTube videos on COVID-19 vaccination.

A YouTube search was performed on 21 July 2021, using keywords 'COVID-19 vaccine' on a cleared-cache web browser. Search results were sorted by 'views', and 150 videos with the greatest number of views were collected and analysed.

Three reviewers (HO-YL, EP and OB-L) reviewed and analysed the 150 videos. Videos that were duplicate, non-English, non-audio, non-visual or unrelated to COVID-19 vaccine were excluded. Discrepancies were resolved by discussion for consensus. 13 Video descriptive characteristics including title, hyperlink, number of views, number of likes and dislikes, upload date, duration and publishing category, were collected.

The reliability and quality of video content were assessed using the modified DISCERN (mDISCERN) score (online supplemental table S1) and the modified Journal of the American Medical Association (mJAMA) criteria (online supplemental table S2). 14 15 A novel fivepoint COVID-19 Vaccine Score (CVS), developed based on previously published work, was used to evaluate the usefulness of video content for the average viewer (online supplemental table S3). 16 

No patients or public were involved in any way in this study.

Following data extraction, two independent reviewers (EP and OB-L) categorised all videos into videos containing factual or non-factual information, and into their video publishing category, in duplicate, with a third reviewer (HO-YL) consultation for consensus. Video sources were categorised into healthcare professionals, network news, education, internet news, consumers, entertainment, government sources and newspapers according to previously published work (online supplemental table S4). 17 Videos were considered non-factual if they contained one or more non-factual statements, defined as contradictory to guidelines published by public health agencies (ie, WHO or Centers for Disease Control and Prevention) at the time of video publication. 18 19 The CVS score was calculated as follows: if a CVS criterion was discussed, it was given a score of 1 if factually correct, a score of 0.5 if ambiguous and 0 if non-factual (online supplemental table S3). The total score was then divided by the number of items discussed (up to a maximum of 5) and expressed as a proportion. mDISCERN and mJAMA scores were calculated with one point given for each criterion correctly addressed (online supplemental tables S1 and S2).

Descriptive statistics including two tailed t-tests and Mann-Whitney U tests compared factual videos to nonfactual videos, with a significance level of p<0.01 to offset multiple hypothesis testing. Percentages, means and ratios compared videos from different sources. All analyses were performed using Microsoft Excel and Mathworks Matlab software (V.R2019b).

Of 150 videos screened, 122 videos (81%) were included. A total of 28 (19%) videos were excluded: 14 for not discussing any CVS criteria, 13 for non-English and one for non-audio (online supplemental figure S1).

Videos were classified into eight categories (figure 1). The total viewership at time of data collection was 169 446 382 views. 24.1% of views were from videos produced by healthcare professionals, 20.9% from network news, 16.9% from education, 10.2% from internet news, 8.4% from consumers, 8.1% from entertainment, 7.3% from government sources and 4.2% from newspapers.

Government videos had the highest average CVS score (0.98) and number of CVS items discussed per minute of video (1.89 items/min). Internet news videos had the lowest CVS score (0.83), and consumer videos had the lowest CVS items discussed per minute (0.20 items/ min). Internet news also had the lowest mDISCERN and mJAMA scores, whereas government videos had the highest mJAMA scores and educational videos had the highest mDISCERN scores (table 1) .

Government videos had the shortest average time since publication of 65 days, compared with 310 days for newspaper videos. Entertainment videos on average had 14.4 times more likes than dislikes, whereas government videos averaged more dislikes than likes, with 3.1 dislikes to every like.

Thirteen videos (10.7%) contained non-factual information and accounted for 11.0% of total viewership, while 109 videos (89.3%) contained only factual information. Of the 13 non-factual videos, 4 were from network news, three from healthcare professionals, two each from education and internet news, and one each from newspaper and consumer sources. Two of the 13 videos (15.4%) containing non-factual information did not contain a YouTube information icon prompting the viewer to seek additional information on COVID-19 vaccination.

When comparing the two groups, there were no significant differences in views per video (p=0.94), likes per video (p=0.40), dislikes per video (p=0.36), duration (p=0.98) and mean days since publication (p=0.56) (table 2) . Videos with non-factual information on average had a greater median ratio of likes to dislikes, compared with videos with factual information (13.89 vs 2.33, p<0.01).

Videos with non-factual information were found to have lower CVS (p<0.001), mDISCERN (p<0.001) and mJAMA (p<0.01) scores, compared with videos with factual information. Classification and examples of nonfactual statements are shown in table 3.

This study highlights that approximately 11% of YouTube's most viewed videos on COVID-19 vaccines contradicted the reference standard, 18 19 reaching over 18 million viewers. Our findings were significantly lower than other studies reporting 27.5% and 45% of videos containing non-factual information. 11 16 This discrepancy is likely due to increased awareness of the impact of social media on disseminating misinformation that has resulted in YouTube's implementation of a COVID-19 medical misinformation policy in May 2020. 20 Furthermore, government sources are increasingly leveraging YouTube as a valuable opportunity to disseminate vaccine information, with an over twofold increase in the percentage of government videos in this study compared with a study published earlier in the pandemic on COVID-19 BMJ Global Health misinformation. 11 Despite producing the most reliable content, videos from government sources had over three times the ratio of dislikes to likes. In comparison, entertainment videos had a ratio of over 14 times the number of likes to dislikes. Similarly, the median ratio of likes to dislikes for videos containing non-factual 

information was almost six times higher than that of videos containing factual information. This suggests the presence of an active community of users who are sceptical of the COVID-19 vaccine and government sources. It is important to note that this may also represent a selection bias, as a previous study demonstrated that participants who relied on private sources for COVID-19 information, such as network news and social media, were less knowledgeable about COVID-19 and less likely to adhere to social distancing guidelines. 21 Additionally, another study demonstrated that YouTube users were less likely to receive the COVID-19 vaccine compared with nonusers. 22 Developing timely strategies to monitor and optimise the quality of information on YouTube during the COVID-19 pandemic is crucial for two reasons. First, it may be difficult for the YouTube algorithm to discern the accuracy and reliability of information conveyed. For example, the infamous COVID-19 film, Plandemic, which contained significant misinformation and conspiracy theories, was only removed from YouTube after public outcry, but not before it had already garnered millions of views at the time of removal, with the potential for serious consequences. 23 Second, the YouTube algorithm allegedly makes recommendations for videos with a higher number of views and a greater proportion of likes. 24 25 Moreover, the algorithm also allegedly adjusts video recommendations based on the user's history of videos watched. 25 Thus, it is plausible that individuals who encounter an initial video containing misinformation, may continue to receive recommendations for other videos containing similar misinformation. To address these concerns, in 2020, YouTube implemented an automated flagging system to demonetise and remove videos containing COVID-19 misinformation. 26 However, this system's reliance on automation has resulted in enforcement errors, such as mistakenly removing videos which sought to refute COVID-19 misinformation. 27 Potential improvements could include increasing viewership of reputable government sources, as they have been shown to be the most reliable, by suggesting these videos to users. Furthermore, given that over 85% of those aged 13-17 years old use YouTube, 28 algorithms suggesting youthfriendly government videos on COVID-19 vaccination to this audience may help increase vaccine compliance in the younger population, which has been reported to be much lower than among adults. 29 

As the COVID-19 pandemic evolves, widespread vaccination is essential in reducing morbidity, mortality and returning to some semblance of normalcy. Providing high-quality health information from reputable sources that is engaging to a variety of viewers will be essential in addressing vaccine hesitancy and improving the public's understanding and adherence to public health measures. Policy-makers should consider implementing mandates surrounding medical misinformation on social media platforms. Public health officials and clinicians should recognise the impact of YouTube and other similar social media sites as sources of health information for patients from all demographic backgrounds and leverage these platforms to disseminate high-quality content to combat misinformation. Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient consent for publication Not applicable.

Ethics approval Ethical approval was not required for this study as it does not involve human or animal subjects.

Provenance and peer review Not commissioned; externally peer reviewed.

Data availability statement Data sharing not applicable as no datasets generated and/or analysed for this study.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

World Health Organization. COVID-19 Weekly epidemiological update

FDA approves first COVID-19 vaccine

Centers for Disease Control and Prevention. CDC COVID data tracker

Social media use in healthcare: a systematic review of effects on patients and on their relationship with healthcare professionals

Consumer health information seeking in social media: a literature review

Internet health information seeking and the patient-physician relationship: a systematic review

Impact of the Internet on patient-physician communication

Meet the 7 most popular search Engines in the world

Youtube videos and informed decision-making about COVID-19 vaccination: successive sampling study

Youtube as a source of information on COVID-19: a pandemic of misinformation?

Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA

Youtube as a source of patient information for Mohs micrographic surgery: a systematic analysis

YouTube for information on rheumatoid arthritis--a wakeup call

Youtube as a source of patient information for abdominal aortic aneurysms

Youtube as a source of information on immunization: a content analysis

Coverage of the Ebola virus disease epidemic on YouTube

World Health Organization. COVID-19 vaccines

Vaccines for COVID-19

COVID-19 medical misinformation policy -YouTube Help

Association between public knowledge about COVID-19, trust in information sources, and adherence to social distancing: cross-sectional survey

Lack of trust, conspiracy beliefs, and social media use predict COVID-19 vaccine hesitancy

Facebook and YouTube are rushing to delete "Plandemic", a conspiracy-laden video

Deep neural networks for YouTube recommendations

How does the YouTube algorithm work in 2021? the complete guide. Hootsuite

Advertiser-friendly content guidelines -YouTube Help

YouTubers' channels and videos are being mistakenly deleted for debunking COVID-19 conspiracy theories

Teens, social media & technology

Willingness of children and adolescents to have a COVID-19 vaccination: results of a large whole schools survey in England