key: cord-0916480-3aoh8xcp authors: Valizadeh-Haghi, Saeideh; Khazaal, Yasser; Rahmatizadeh, Shahabedin title: Health websites on COVID-19: are they readable and credible enough to help public self-care? date: 2021-01-01 journal: Journal of the Medical Library Association : JMLA DOI: 10.5195/jmla.2021.1020 sha: 8df776b5e4e71db5da69c087d752111754a738d9 doc_id: 916480 cord_uid: 3aoh8xcp OBJECTIVE: There are concerns about nonscientific and/or unclear information on the coronavirus disease 2019 (COVID-19) that is available on the Internet. Furthermore, people's ability to understand health information varies and depends on their skills in reading and interpreting information. This study aims to evaluate the readability and creditability of websites with COVID-19-related information. METHODS: The search terms “coronavirus,” “COVID,” and “COVID-19” were input into Google. The websites of the first thirty results for each search term were evaluated in terms of their credibility and readability using the Health On the Net Foundation code of conduct (HONcode) and Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog, and Flesch Reading Ease Score (FRE) scales, respectively. RESULTS: The readability of COVID-19-related health information on websites was suitable for high school graduates or college students and, thus, was far above the recommended readability level. Most websites that were examined (87.2%) had not been officially certified by HONcode. There was no significant difference in the readability scores of websites with and without HONcode certification. CONCLUSION: These results suggest that organizations should improve the readability of their websites and provide information that more people can understand. This could lead to greater health literacy, less health anxiety, and the provision of better preventive information about the disease. As a large family of viruses, coronaviruses are responsible for multiple diseases. In humans, these include the common cold alongside severe diseases such as Middle East respiratory syndrome (MERS-CoV) and severe acute respiratory syndrome (SARS-CoV) [1] . Coronavirus disease 2019 (COVID- 19) was first reported in Wuhan, China, on December 31, 2019 [2] . According to a report published by the World Health Organization (WHO) in May 2020, COVID-19 spread across 216 countries very rapidly [3] . As a vaccine for this virus has yet to be developed, nonpharmacological interventions, such as increased hygiene, are crucial for controlling the virus and reducing the risk of infection [4, 5] . Many people consider the Internet to be a useful and important source of health information [6] [7] [8] [9] that can encourage the use of preventive strategies and consultation with physicians [10] . On March 13, 2020 , Google Trends reported that the term "coronavirus" was searched five million times (supplemental appendix), underlining the importance of the Internet as a source for health information. However, multiple studies demonstrate that health websites may not be credible and may contain inaccurate or misleading information [11] [12] [13] [14] [15] [16] . Therefore, the information found on health websites could increase uncertainty and anxiety and, thereby, serve to harm people's health [17] [18] [19] . Trust in online health information has recently been of great concern [20] due to deficiencies in people's ability to judge the quality of this information [21, 22] . The current COVID-19 pandemic further raises concerns about nonscientific, misleading, and unclear information disseminated via the Internet or other media [23, 24] , which can See end of article for supplemental content. increase anxiety among the population. This pandemic may be particularly problematic for people with cyberchondria, defined as compulsive searching for health information online [25] , because the existence of inaccurate information on the Internet could serve to increase this behavior. Apart from individuals with health anxiety, the pandemic's high mortality rate causes widespread fear, anxiety, and stress for people around the globe [26] . Although different motivations can contribute to online health information-seeking, searching for information about COVID-19 online can be understood as one way to cope with the stress associated with the pandemic [27] . For instance, people may search for information about COVID-19 to gain clarity about their own or a relative's symptoms as an attempt to alleviate uncertainty [28] . However, the fear and anxiety caused by pandemics such as COVID-19 can serve as an additional barrier to understanding information correctly [29] and may lead to further anxiety, inappropriate use of information, and unnecessary referrals to medical centers. The proper use of health information depends on people's ability to understand, interpret, and comprehend that information [30] and can be influenced by many factors, including its readability [31] . To be considered readable, a text should be easy to read and contain concepts that are easy to understand [32] . Organizations such as the American Medical Association (AMA) and National Institutes of Health (NIH) advise that the readability of health information should not exceed a sixth-grade level [33] and should be understandable to eleven years olds [34] . Moreover, it should not use medical jargon [35] . However, online health information is often written in at a level that is not easily understood by many people [36] [37] [38] [39] [40] [41] . Considering the increased use of the Internet to obtain health information on COVID-19 [42] , it is of great importance to understand the readability and credibility of health websites. The objective of this study was to evaluate the readability and credibility of publicly accessible health websites containing COVID-19-related information. The authors searched for "coronavirus," "COVID," and "COVID-19" using Google, the most frequently used Internet search engine [43, 44] , on February 26, 2020. The search was performed using Google Chrome, and the browser history, cached images, and cookies were removed before the searches were carried out. As 90% of all search engine users browse only the first 3 pages of results pages (i.e., the first 30 results) [45] , the first 30 websites returned by Google for each of the 3 keywords were examined, resulting in a total of 90 websites. We excluded non-English-language websites, irrelevant websites, scientific papers, duplicate websites, inaccessible websites, primarily non-textbased websites (e.g., YouTube), and advertisementsponsored links. Based on these criteria, forty-three websites were excluded, and forty-seven unique websites were selected for evaluation ( Figure 1 ). These websites were then divided into the following five categories according to their statement of affiliation: commercial, news, educational, governmental, and organizational. Four readability scales, each of which uses different techniques, were used to evaluate the readability of the websites: Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog, and Flesch Reading Ease Score (FRE). These scales have been used in many readability evaluation studies of health websites on various topics and are reliable [46] [47] [48] [49] [50] [51] [52] [53] , and NIH recommends using FKGL, Gunning Fog, and SMOG for readability evaluations [54] . The FRE scale produces a score between 0 and 100, with higher scores indicating higher readability levels: scores of 90-100, 60-70, and 0-30 indicate that text can be understood by fifth-grade students, eighth-or ninth-grade students, and university graduates, respectively. The Gunning Fog scale produces scores of 5, 10, 15, or 20, which indicate that text is easy to read, hard to read, difficult to read, or very difficult to read, respectively. The SMOG and FKGL scales estimate the years of education a person needs to have completed to understand a written text; for example, a score of 7.4 indicates that a seventh-grader can understand the text. To apply these scales, a free online readability checker (Readability Formulas) was utilized [55] . This web-based tool has been used in readability evaluations of a wide variety of health-related websites [36, 46, 50, [56] [57] [58] [59] . Various tools-including JAMA [60] , DISCERN [61] , and Health On the Net Foundation code of conduct (HONcode) [62] -are available for evaluating the credibility of health websites. The HONcode consists of nine criteria: authority, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy. The nonprofit HON Foundation, which is officially related to WHO, checks the credibility of websites at the request of the institutions hosting the websites. If they meet the criteria, the HON logo is placed on the website, which indicates that the website has been officially certified and is a reliable source of health information. The HON Foundation also provides a toolbar extension compatible with Chrome and Firefox browsers that helps people easily identify HONcode-certified websites while browsing [63] . HONcode is the oldest and most-used ethical and trustworthiness code for medical and health-related information available on the Internet [64] . It is reliable and has been used in multiple studies to assess the credibility of health websites [11, 12, [14] [15] [16] [65] [66] [67] . Therefore, we used the HONcode toolbar to identify certified websites containing information on COVID-19. We tested whether HONcode-certified websites were more readable than non-HONcode-certified websites using independent t-tests. We also tested whether readability scores differed among website categories and depended on the website position on the search results pages using one-way analysis of variance (ANOVA). Statistical analysis was conducted using SPSS 18. We analyzed forty-seven unique websites containing information on COVID-19 retrieved via Google. The retrieved websites appearing on the first page of the search results included more HONcode-certified websites than those on the second and third pages (Table 1) . However, even on the first page of results, most websites were not officially approved. In total, only six websites were HONcode-certified, and all were commercial and/or organizational. ANOVA showed no significant effect of website category on readability scores (Table 2 ). There was also no significant effect of search results page number on readability scores (Table 3) . T-tests showed no significant differences in readability scores between HONcode-certified and non-HONcode-certified websites (Table 4 ). Among prominent international and national organizations, the readability of website content published by the National Health Service (NHS) and Centers for Disease Control and Prevention (CDC) scored highest and lowest, respectively ( Table 5 ). The readability level of content available through the WHO and NIH websites was "difficult to read." Low literacy levels are a barrier to health knowledge. One solution to this problem is to provide content in plain language that is easy to read [68] . Using plain language can help convey information to a wider population [69] and allows users to find what they need, understand what they have found, and then use this information to meet their needs [70] . On a global level, individual reading abilities and the readability levels of consumer information contribute to the overall health of all people and society [71] . Due to the importance of readability in the context of health literacy, health promotion, and patient self-care, we evaluated the readability and credibility of COVID-19 information available on websites intended for the general public that were retrieved via Google searches. Patient education materials should be easily understood by an average eleven-year-old or DOI: dx.doi.org/10.5195/jmla.2021.1020 Journal of the Medical Library Association 109 (1) January 2021 jmla.mlanet.org students in the sixth grade [34] . The results of this study show that the readability of COVID-19 information on websites is more advanced than the recommended level and is generally aimed at high school graduates or college students. This finding is consistent with studies examining online information on Ebola [72] and other diseases [73] [74] [75] . Moreover, our findings show that the readability level of website content published by international and national health organizations such as WHO and CDC was far above the recommended sixth-grade reading level. This is concerning because individuals consider these websites to be major sources of reliable health information, especially in health crises such as the current COVID-19 outbreak. We also examined the readability levels of websites based on their category. Government websites were expected to be more readable than other types of websites, because their purpose is usually to educate the general public [76] . However, we found that the readability scores of websites in all categories, including governmental websites, were far above the recommended level. Although previous studies reported that governmental websites were more readable than other types of websites [73] , we found that the readability of commercial websites was more suitable for a public audience than governmental websites. However, it should be noted that the information on commercial websites has been found to be of lower quality than other types of websites [77, 78] . Consequently, people looking for information on the symptoms, prevention, treatment, and management of COVID-19 may come across websites that contain readable but inaccurate information or, conversely, accurate but unreadable information, meaning that decisions based on this information could increase their anxiety and even threaten their health [17] [18] [19] . Most people tend to browse search results presented on the first page [74, 79] . Therefore, we expected that websites appearing on the first page of the search results would be more readable than those on the second and third pages. However, we found no significant difference in the mean readability scores of websites appearing on different pages of the search results. While organizations make efforts to increase the rankings of their websites in search engine results, we recommend that they also pay attention to the readability of their websites to ensure that their content can be understood. This would allow users to better understand the websites' content, satisfy their information needs, and help to prevent the spread of dangerous infectious diseases such as COVID-19. We found that most websites that we examined were not officially certified by the HON Foundation. Therefore, individuals searching for information on COVID-19 may encounter websites that contain misinformation, which could lead to incorrect decision-making and anxiety. Moreover, both HONcode-certified and non-certified websites had poor readability scores. This finding contrasts with a similar study on prostate health, in which more credible websites were found to have better readability [80] . Therefore, we recommend that authoritative organizations providing health information about various infectious diseases, including COVID-19, pay more attention to increasing the readability of their websites to help people understand the information that is provided. There were some limitations in this study. For example, searches were conducted through Google Search, and using other search engines might have generated different results. In addition, due to the dynamic characteristics of websites, alternative results might have been obtained if the searches were conducted at different times. World Health Organization (WHO). Coronavirus The Organization World Health Organization. Coronavirus disease (COVID-19) pandemic The Organization Coronavirus disease 2019 (COVID-19). situation report -100 The Organization COVID-19: what is next for public health? Lancet What is MERS-CoV, and what should I do? How do Qataris source health information? Use of the Internet as a health information resource among French young adults: results from a nationally representative survey Health information sources: trust and satisfaction Sources of health information and their impacts on medical knowledge perception among the Saudi Arabian population: cross-sectional study Using the Internet for healthrelated activities: findings from a national probability sample Websites as a tool for public health education: determining the trustworthiness of health websites on Ebola disease Persian language health websites on Ebola disease: less credible than you think? Glue ear: how good is the information on the World Wide Web? Evaluating the trustworthiness of consumer-oriented health websites on diabetes Middle East respiratory syndrome on health information websites: how much credible they are? Libr Philos Pract Evaluation of the quality and accessibility of available websites on kidney transplantation Diseases in a flash!: an interactive, flash-card approach Hypochondria in the age of COVID-19 Cyberchondria: challenges of problematic online searches for health-related information The future of truth and misinformation online. Pew Research Center When the quality of health information matters: Health on the Net is the quality standard for information you can trust [Internet]. Health On the Net Foundation Health websites visual structure: the necessity of developing a comprehensive design guideline Fake news in the age of COVID-19 False information fuels fear during disease outbreaks: there is an antidote. The Conversation [Internet Cyberchondria and its relationships with related constructs: a network analysis COVID-19 has killed more people than SARS and MERS combined, despite lower case fatality rate Why do we want health information? the goals associated with health information seeking (GAINS) questionnaire. Psychol Heal Internet use by patients with psychiatric disorders in search for general and medical informations Selective influence of test anxiety on reading processes Health information on the Internet: accessibility, quality, and readability in English and Spanish Communicating with patients who have limited literacy skills Readability of patient education materials: implications for clinical practice Health literacy: a manual for clinicians Medical Association Foundation Evaluation of literacy level of patient education pages in health-related journals Evaluation of quality and readability of health information websites identified through India's major search engines Reading level of online patient education materials from major obstetrics and gynecology societies Readability of patient education materials on the American Association for Surgery of Trauma website. Arch Trauma Res Quality and readability of English-language Internet information for tinnitus Readability of spine-related patient education materials from subspecialty organization and spine practitioner websites. Spine (Phila Pa 1976) Health literacy demands of written health information materials: an assessment of cervical cancer prevention materials. Cancer Control Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites Global stats: browser, OS, search engine including mobile usage share Top 15 best search engines [Internet]. eBiz MBA, eBusiness guide Blended search results study Critical analysis of the quality, readability, and technical aspects of online information provided for neck-lifts Information about oral cancer on the Internet: our patients cannot understand it Readability of patient education materials in hand surgery and health literacy best practices for improvement Evaluating the readability levels of medical infographic materials for public consumption Readability and quality assessment of websites related to microtia and aural atresia Clearly written, easily comprehended? the readability of websites providing information on epilepsy The quality and readability of colorectal cancer information on the Internet Quality and readability of websites for patient information on tonsillectomy and sleep apnea Choice rev online. National Library of Medicine Contact us Effectiveness of health education materials in general practice waiting rooms: a crosssectional study Preferences of ICT among patients with chronic kidney disease undergoing hemodialysis: an Ecuadorian cross-sectional study Analysis of readability, quality, and content of online information available for "stem cell" injections for knee osteoarthritis Quality and reliability of publicly accessible information on laser treatments for urinary incontinence: what is available to our patients? Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor-let the reader and viewer beware The DISCERN handbook: quality criteria for consumer health information on treatment choices Health on the Net Foundation. The commitment to reliable health and medical information on the Internet [Internet]. The Foundation Health on the Net Foundation. HONcode toolbar [Internet]. The Foundation Are personal health records safe? a review of free web-accessible personal health record privacy policies Female urinary incontinence health information quality on the Internet: a multilingual evaluation Quality of web-based information on social phobia: a cross-sectional study The quality of mental health information commonly searched for on the Internet Plain language for patient education Development of plain language supplemental materials for the Biobank informed consent process US Department of Health & Human Services. Writing for the web [Internet]. The Department Consumer literacy and the readability of health education materials Readability of Ebola information on websites of public health agencies, United States Readability of online health information: implications for health literacy. Inform Health Soc Care Using the Internet: skill related problems in users' online behavior Readability and implications for health literacy: a look at online patient-oriented disease information by Alzheimer's disease research centers Cancer information on the Internet Quality of web-based information on cocaine addiction The quality of nutritional information available on popular websites: a content analysis Cultural influences on seeking quality health information: an exploratory study of the Korean community How readable is BPH treatment information on the Internet? assessing barriers to literacy in prostate health