key: cord-0700186-rhgg65rz authors: Szmuda, Tomasz; Özdemir, Cathrine; Ali, Shan; Singh, Akshita; Syed, Mohammad Talha; Słoniewski, Paweł title: Readability of Online Patient Education Material for the Novel Coronavirus Disease (COVID-19): A Cross-sectional Health Literacy Study date: 2020-05-30 journal: Public Health DOI: 10.1016/j.puhe.2020.05.041 sha: cc4fef6c2b272baddb10bed50b3729c89755f3fc doc_id: 700186 cord_uid: rhgg65rz Abstract Objectives The internet has become one of the most important resources for the general population when searching for health care information. However, the information available is not always suitable for all readers due to its difficult readability. We sought to assess the readability of online information regarding the novel Coronavirus disease (COVID-19) and establish whether they follow the patient educational information reading level recommendations. Study design This is a cross-sectional study. Methods We searched five key terms on Google and the first 30 results from each of the searches were considered for analysis. Five validated readability tests were utilized in order to establish the reading level for each article. Results Out of the 150 gathered articles, 61 met the inclusion criteria and were evaluated. None (0%) of the articles met the recommended 5th to 6th grade reading level (of a 11-12-year-old). The mean readability scores were: Flesch Reading Ease 44.14, Flesch-Kincaid Grade Level 12.04, Gunning-Fog Index 14.27, Simple Measure of Gobbledygook SMOG Index 10.71, and Coleman-Liau Index 12.69. Conclusions Online educational articles on COVID-19 provide information too difficult to read for the general population. The readability of articles regarding COVID-19 and other diseases needs to improve so that the general population may understand health information better and may respond adequately to protect themselves and limit the spread of infection. The internet has grown to become one of the most popular resources for people in finding health information 1 . In the United States, up to 80% of all adult internet users have looked online to find information about several health issues 2 . 70% of health seekers express that the information that they have found has influenced them in their decision-making regarding the treatment of their disease 3 . While the information is readily accessible, it's utility may be variable depending on the readability of the information. The mean reading level of adults in the United States has been estimated to be equivalent to that of a 13 to 14 year old 4 However, numerous studies have shown that the reading difficulty of health articles online is much higher than recommended [6] [7] [8] . To date, no readability analysis has been done regarding the novel Coronavirus disease (COVID-19). COVID-19 has resulted in thousands of deaths worldwide and has resulted in more fatalities than the previous two coronavirus epidemics combined (i.e. SARS and MERS). As of March 11, 2020, the outbreak was recognized by the World Health Organization (WHO) as a pandemic 9 . Thus, proper public education becomes critical so that patients may prevent and contain the infection. Since the internet is often first source of information regarding healthcare for patients, it is critical to evaluate the readabilty level of the information 10 . In this study, we sought to assess the readability of online information regarding COVID-19. On March 13, 2020 the Google search engine was used to search for the following five key terms: "Coronavirus," "COVID-2019," "SARS CoV-2," "2019-nCoV" and "What is the coronavirus". 7.39 x 10⁹, 4.59 x 10⁹, 1.43 x 10⁸, 1.02 x 10⁸ and 5.08 x 10⁹ search results were identified for each search, respectively. The first 30 results per key term were evaluated since 90% of internet users do not look past this number 11 . A total of 150 search results were considered for the readability analysis. Articles not in English, duplicates, newspaper articles (since they contained mostly political updates), biomedical journal papers, non-open access articles (behind a paywall) and statistical websites were excluded. All articles were restructured into plain text and all irrelevant material was deleted, such as Gunning-Fog Index uses the average sentence length and the number of words containing three or more syllables for the calculations. Poly-syllabic words are excluded, including: proper nouns, a combination of easy words (including hyphenated words), and poly-syllable verbs whose third syllable is "es" or "ed" 13 . SMOG Index is calculated by counting every poly-syllabic word (containing three or more syllables) in sections containing 10 sentences each placed in the beginning, in the middle, and at the end of the text in question 14 . In contrast to the other readability formulas, Coleman-Liau Index does not take the number of syllables into account. Instead, it makes the assessment based on the average number of letters and sentences per 100 words 15 . The FRE score determines the reading ease based on a scale from 0 to 100, where a lower score indicates a higher difficulty (0-30 is very difficult, 30-50 is difficult, 50-60 is fairly difficult, 60-70 is standard, 70-80 is fairly easy, 80-90 is easy, 90-100 is very easy) 12 Out of the 150 articles, 61 were analyzed as they met the inclusion criteria. The mean reading level of the articles was equivalent to a high school senior/college freshman (17) (18) year old) (12.4 ± 2.1). The minimum score was 8.8, while the maximum score was 20.1. According to the mean FRE score, the articles are considered difficult to read (44 +-11.5). In Table 1 , the mean scores of the readability formulas are presented. Table 2 shows the distribution of the scores. (Table 1) ( Table 2) All articles were of at least a high school sophomore grade level (15) (16) year old). According to the FRE, 91.8% of the articles were found difficult to read. On the FKGL scale, 78.68% of the articles had a readability index of a high school senior grade (17) (18) year old). Similarly, on GFI, SMOG, and CLI, the percentage of articles that were above the readability index of a high school senior (17) (18) year old) were 98.36%, 62.29%, and 88.52%, respectively. Table 3 displays that the information provided from websites related to governments, hospitals and health organizations (such as WHO) are also not following the recommendations for educational material. All of the medical articles were written beyond the recommended 5th-to 6th grade level (11) (12) year old) 16 . (Table 3 ) Figure 1 shows a comparison between the overall readability, government health organization readability and hospital readability. ( Figure 1) We found that online information about COVID-19 is too difficult for the general population to read and comprehend. None of the articles met the 5th-to 6th-grade reading level (11) (12) year old) recommended by the AMA and the USDHHS 5, 16 . Most articles (84%) were disignated as too difficult to read, which makes it tough for the public to acquire understandable information regarding COVID-19. To reduce the spread of infection and thus, reduce the burden on a country's healthcare system, a country's population need access to understabdable information online. We urge prominent orgnizations such as WHO and the Centers for Disease Control and Prevention (CDC) to make their online information friendlier to the general public. Easy readability of relevant medical information empowers individuals to take the right steps to protect themselves. This may reduce panic and anxiety especially in the midst of a pandemic. Our findings are novel since our paper is the first to do a readabilty analysis on COVID-19. Morever, we conducted our readability analysis on COVID-19 during the pandemic itself to provide a contemporary and relevant view of the probelm. Medical terminology is an essential factor affecting the readability of a text. Even if the person's educational level is high, long sentences and unfamiliar words can make the text challenging to read. If they are not accustomed to medical literature, misunderstanding and misinformation may occur and the patient may stop researching basic medical care. Therefore, information aiming to educate patients should be clear and understandable. A 2018 systamatic review analyzing 157 readability studies found that the readability level of online health articles is incomprehensible for the public 8 . This study is limited by the constraints of a cross-sectional study. Since the material available on the internet is constantly increasing, the results of our study will only reflect what information was present for the public at this point in time. However, our study brings awareness to the problem; this may influence hospitals, and government organizations to reduce the reading difficulty on their websites. Morevoer, this readability study is based on text only. Infographics and videos-which may often enhance the understanding of a textwere removed to perform the analysis. This was another limitation of this study. Previous studies have shown that the quality of health information online is biased, misleading and poor 17, 18 . Thus, while we assessed the readabiliity of COVID-19 health articles, the quality of these articles still needs to be evaluated. Additionally, since YouTube has become a major source of patient information, the quality of information on this platform also needs to be assessed [19] [20] [21] [22] . The role and effectiveness of online medical resources concerning telemedicine may also be explored 23 . We encourage websites to display COVID-19 infographics and videos as they may be a more friendly way of providing information to the public. Moreover, infographics are easy to potentially share on social media and may help spread the health information about the disease. Considering Europe is now the epicenter of COVID19 pandemic, it becomes critical that government sites such as ecdc.europa.eu (European Centre for Disease Prevention and Control) publish advisories in simple understandable terms. Studies show that these articles play a critical role in recommending health safety guidelines, reduce the burden on the healthcare system and help healthcare workers prioritise managing the disease effectively [24] [25] [26] [27] . An anlaysis of the 1918 influenza pandemic showed that early implementation of certain interventions (e.g. isolation policies, mask ordinances and bans on public gatherings) reduced influenza transmission 26 . We assert that for these interventions to be effective, the public needs access to understandable online health information so that personal measures to contain and prevent the disease may be taken (e.g. by washing heads frequently, practicing respiratory hygiene and seeking medical care early). Low health literacy has been associated with nonadherence to treatment plans and medical regimens, poor patient self-care, high health-care costs, and increased risk of hospitalisation and mortality 28 . Thus we hope that this paper serves as a "call to action" for health authorities to provide more comprehensible reading material online. Online information regarding COVID-19 is too difficult to read and understand as designated by the AMA and the USDHHS 5 . This includes websites run by governments and health institutions such as the WHO. Since the internet now is one of our most popular sources of information, it is critical that people are provided with understandable information. Health articles too difficult to understand may cause misinformation to spread, public panic due to a lack of accessible information and a greater burden on a country's healthcare system. 6.0 AUTHOR STATEMENTS 6.1 Ethical approval: No ethical approval was required as all the data analyzed were publicly available. 6.2 Funding: none reported. Table 3 : Mean readability score of websites related to governments, hospitals/clinics or health institutions/organizations. • Online health information regarding COVID-19 is too difficult to read and understand. • Based on past research, health articles too difficult to understand may cause misinformation to spread and public panic. • The readability of COVID-19 needs to improve so that the general population may respond adequately to protect themselves and limit the spread of infection. Untangling the Web-The impact of Internet use on health care and the physician-patient relationship Doctor patient relationship: Changing dynamics in the information age The Online Health Care Revolution | Pew Research Center The Health Literacy of America's Adults: Results From the 2003 National Assessment of Adult Literacy Evaluation of literacy level of patient education pages in health-related journals Quality and readability of online information about type 2 diabetes and nutrition Online Patient Education Materials for Orthognathic Surgery Fail to Meet Readability and Quality Standards Readability of Online Health Information: A Meta-Narrative Systematic Review World Health Organization. Coronavirus (COVID-19) events as they happen Surveys of physicians and electronic health information iProspect Search Engine User Behaviour Study A new readability yardstick Guilford College Writing Manual: Gunning's Fog Index Computerized versus hand-scored health literacy tools: A comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials Health literacy and patient safety: Help patients understand Evaluating the quality of health information in a changing digital ecosystem Consumer evaluation of the quality of online health information: Systematic literature review of relevant criteria and indicators Letter to the Editor Regarding "A Quality Analysis of Disk Herniation Videos on YouTube World Neurosurgery YouTube as a Source of Patient Information for Hydrocephalus: A Content-Quality and Optimization Analysis Relative quality of internet-derived gastrointestinal cancer information YouTube as a source of information for narcolepsy: A content-quality and optimization analysis Telemedicine in neurosurgery during the novel coronavirus (COVID-19) pandemic Preventive health information on the Internet. Qualitative study of consumers' perspectives Impact of internet use on health-related behaviors and the patient-physician relationship: A survey-based study and review Public health interventions and epidemic intensity during the 1918 influenza pandemic Straining the System: Novel Coronavirus (COVID-19) and Preparedness for Concomitant Disasters Poor health literacy: A "hidden" risk factor Coleman-Liau Index (CLI) Below 6 0 6-10 7