key: cord-0783905-29070mtv
authors: Joshi, Ashish; Kajal, Fnu; Bhuyan, Soumitra S.; Sharma, Priya; Bhatt, Ashruti; Kumar, Kanishk; Kaur, Mahima; Arora, Arushi
title: Quality of Novel Coronavirus Related Health Information over the Internet: An Evaluation Study
date: 2020-08-06
journal: ScientificWorldJournal
DOI: 10.1155/2020/1562028
sha: c093e9d77d574c8c8f62811086da66f5cd93f3d3
doc_id: 783905
cord_uid: 29070mtv

BACKGROUND: The novel coronavirus disease (COVID-19) has spread globally from its epicenter in Hubei, China, and was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. The most popular search engine worldwide is Google, and since March 2020, COVID-19 has been a global trending search term. Misinformation related to COVID-19 from these searches is a problem, and hence, it is of high importance to assess the quality of health information over the internet related to COVID-19. The objective of our study is to examine the quality of COVID-19 related health information over the internet using the DISCERN tool. METHODS: The keywords included in assessment of COVID-19 related information using Google's search engine were “Coronavirus,” “Coronavirus causes,” “Coronavirus diagnosis,” “Coronavirus prevention,” and “Coronavirus management”. The first 20 websites from each search term were gathered to generate a list of 100 URLs. Duplicate sites were excluded from this search, allowing analysis of unique sites only. Additional exclusion criteria included scientific journals, nonoperational links, nonfunctional websites (where the page was not loading, was not found, or was inactive), and websites in languages other than English. This resulted in a unique list of 48 websites. Four independent raters evaluated the websites using a 16-item DISCERN tool to assess the quality of novel coronavirus related information available on the internet. The interrater reliability agreement was calculated using the intracluster correlation coefficient. RESULTS: Results showed variation in how the raters assigned scores to different website categories. The .com websites received the lowest scores. Results showed that .edu and .org website category sites were excellent in communicating coronavirus related health information; however, they received lower scores for treatment effect and treatment choices. CONCLUSION: This study highlights the gaps in the quality of information that is available on the websites related to COVID-19 and study emphasizes the need for verified websites that provide evidence-based health information related to the novel coronavirus pandemic.

Novel coronavirus belongs to the large family of coronavirus like SARS-CoV [1] . On December 31, 2019, China reported a cluster of pneumonia cases in people associated with the seafood wholesale market in Wuhan, Hubei Province. On January 7, 2020, Chinese health authorities confirmed that this cluster was associated with a novel coronavirus, 2019-nCoV. e World Health Organization (WHO) initially named this coronavirus as the 2019-novel coronavirus (2019-nCoV) on January 12, 2020. e WHO officially named the disease as coronavirus disease 2019 (COVID- 19) , and the Coronavirus Study Group (CSG) of the International Committee proposed to name the new coronavirus as SARS-CoV-2 on February 11, 2020 [2] . Although cases were originally reported to be associated with exposure to the seafood market in Wuhan, current epidemiologic data indicate that person-to-person transmission of 2019-nCoV is occurring [3] . e WHO declared the COVID-19 virus outbreak a pandemic on March 11, 2020. Most people infected with SARS-CoV-2 experience mild to moderate respiratory illness and might recover without requiring special treatment. As of May 31, 2020, there have been nearly 5 million total coronavirus cases and 362,786 deaths [4] . ere is an urgent need to understand the epidemiology and evolution of this outbreak, as well as control strategies to stop the transmission [5] .

e COVID-19 outbreak has posed critical challenges for public health, research, and medical communities [6] . Every outbreak provides an opportunity to gain valuable information, some of which are associated with a limited window of opportunity. More and more individuals are relying on the internet for latest information on the pandemic, which makes it necessary to ensure credibility and accuracy of these sources [7] . e quality of internet-based health information is extremely variable as there are no mandatory standards for peer review of websites with the potential to provide wrong information to individuals seeking self-care [8] . Assessing credibility of health information over the internet is essential [9] .

A previous study suggested three basic requirements for quality information: (1) the information presented in a manner free from propaganda or disinformation (objectivity); (2) the information is a complete, not partial, picture of the subject (completeness); and (3) all aspects of the information are given, and the information is not restricted to present a particular viewpoint (pluralism) [10] .

Public health experts worry that the spread of COVID-19 worsened by misinformation. Experts worry dissemination of false or even unsafe information at an alarming rate.

e World Health Organization (WHO) described "massive infodemic," citing an overabundance of reported information, accurate and false, about the virus that makes it hard for people to find trustworthy sources and reliable guidance when they need it [11] . Experts suggest that false or even dangerous information about what can protect individuals during the pandemic is being disseminated at an alarming rate. Facebook, Twitter, and Google said they were working with the WHO to address "misinformation." A lack of credible, easy-to-access information for individuals has led to some pursuing unorthodox approaches to health care that can be harmful. Stores are selling hand sanitizers that do not meet the Centers for Disease Control and Prevention (CDC) guidelines and may not be effective, and bad actors are hawking homeopathic products that can harm unwitting individuals [12] .

Several tools exist to evaluate the quality of health information on the internet. ree tools that are widely used include the HON (Health on Net) Code, the JAMA benchmarks, and the DISCERN tool [13] . e DISCERN tool is the only tool currently available online for which substantive validation data are publicly available. e first section of this tool evaluates the reliability of the information, and the second section considers the quality of the information on treatment choices. Five-point Likert scales ranging from one (definite no) to five (definite yes) accompany these items. e final question assesses the overall rating of the publication on a five-point Likert scale ranging from one (low quality with serious or extensive shortcomings) to five (high quality with minimal shortcomings) [14] . Low scores indicate poor quality of information, and high scores indicate good quality. e DISCERN tool has an internal consistency of Cronbach's α (0.777) [15] . e DISCERN questionnaire is a reliable instrument and can be used as an assessment tool to evaluate health information not only by health professionals but also by patients and the general population [16] . e DISCERN tool is relatively easy to use as demonstrated in previous studies which also reported an excellent interrater reliability. e objective of our study was to evaluate the quality of COVID-19 related health information over the internet using the DISCERN tool.

In March of 2020, we used Google's search engine to assess the quality of coronavirus related information over the internet (Figure 1 ). e keywords used in this search included "Coronavirus," "Coronavirus causes," "Coronavirus diagnosis," "Coronavirus prevention," and "Coronavirus management." We recorded the first 20 websites that appeared in the results of various keyword searches on Google, generating 100 URLs. Prior studies have shown the importance of the first 20 websites as a source of the most reliable content [17] . Duplicate websites excluded from this search, allowing analysis of unique websites only. Additional exclusion criteria included scientific journals, nonoperational links, nonfunctional websites (where the page was not loading, was not found, or was inactive), and websites in languages other than English.

Previous studies [15, 18] have identified nine criteria for website assessment, including the following:

(1) Source (such as credentials, conflicts of interest, and biases) (2) Disclosure (statement of purpose and profiling) (3) Accuracy (4) e correctness of material (5) Statement of source (6) Levels of evidence (7) Disclaimers (8) Link content (evaluated according to selection, architecture, content, and back linkages) (9) Peer-review mechanisms (content reviewed by experts and colleagues in the related area)

Quality of health information on the internet must determine accuracy of the material, relevancy, topic clarity, and level of evidence in the form of citations of peerreviewed material [17, 19] . In our study, we used the 2 e Scientific World Journal DISCERN tool (http://www.discern.org.uk), which was designed for use by consumers of online health information and does not require previous knowledge of the subject [20] . It is a validated rating tool used by health consumers or professionals alike [21] . e tool is a 16-item questionnaire used to assess the quality of health information on a website and to help healthcare providers and consumers to evaluate any website containing health information [22] .

We divided the 16 items of the DISCERN tool in six categories: Relevance, Objectives, Information Credibility, Treatment Choices, Treatment Effect, and Prevention and Management (Table 1) . Each category scored on a Likert scale of 1 to 5. A higher score indicated that the website contains more useful and appropriate information while a lower score indicated lack of information in the identified categories. e content on these forty-eight websites identified in our search was considered accurate and complete only if the website provided the following information on novel coronavirus: risk factors for virus transmission, diagnosis, treatment effect, and treatment choices along with information on disease prevention and management.

In accordance with the URL suffixes, the websites were categorized as .edu, .com, .org, .gov, and others (for all remaining domains). Four raters belonging to diverse educational backgrounds also frequent internet users independently assigned a score between 1 and 5 to each of forty-eight websites in the six DISCERN categories (Relevance, Objectives, Information Credibility, Treatment Choices, Treatment Effect, and Prevention and Management). For final assessment, an average of the combined scores used. A high total score denotes that the website is providing high-quality consumer health information. Objectives, Information Credibility, Treatment Choices, Treatment Effect, and Prevention and Management) (Tables 2 and 3). We have also reported the p value for comparison between average DISCERN score and web domain groups using analysis of variance (ANOVA) (Table 2). We calculated the average DISCERN group score of the four raters and assessed whether the quality of information differed between domain groups and each DISCERN group score using ANOVA ( 

e initial search resulted in 100 websites, of which 52 excluded because they either contained insufficient information or were not in English, had nonfunctional links, or were duplicates. e remaining 48 unique websites were included in the final evaluation. Forty-six percent (n � 22) of the websites had .com website extension, 19% (n � 9) had .org website extension, and 23% (n � 11) had other website extension categories (Figure 1) .

Results showed that of all the raters, highest discern scores by one of the raters were given to website extension .edu (25) and .org (22.56) . Low DISCERN scores were seen for both .edu and .com website categories (Figure 2 ). Table 2 represents the average DISCERN score in each web domain by each of the four raters and the overall average rating of the four raters. Of the four raters, three raters consistently gave highest DISCERN scores to the website extension category .edu. An intracluster correlation coefficient value between 0-30 demonstrates poor interrater reliability. Overall, the scoring reliability between raters remained low for each web domain group except for .com extension (ICC � 0.43). e overall DIS-CERN score (based on average of all four raters) was highest for the website extension categories .edu (m � 20.13) and .gov (m � 16.19).

Lowest overall DISCERN score was assigned to the website extension category .com (m � 12.52) and can be attributed to limited information availability on COVID-19 treatment choices, treatment effect, objectives not defined clearly, poor information validity, and no or limited information on prevention strategies. e quality of information based on the overall DISCERN score was significantly different between the web domains ( Table 2 ). e DISCERN tool was broken down into six groups, and a score was assigned to each of the DISCERN category by the raters. An average DISCERN score for each DIS-CERN group for the websites is reported (Table 3 ). All raters consistently rated Objectives, Relevance, and Information Credibility of websites high. While quality of information on COVID-19 treatment choices and treatment effect consistently rated low by all the raters. e intracluster correlation coefficient value between 0 and 0.33 on all DISCERN tool categories demonstrates poor reliability across raters. We calculated the overall DISCERN Objective score based on Objective score values of all the four raters. e procedure repeated for all the other DISCERN groups including Relevance, Information Credibility, Treatment Choices, Treatment Effect, and Prevention and Management and compared them across the different website domains ( Table 4 ). As shown in the table, .com websites received the lowest average scores for website objectives, relevance, information credibility, treatment choices, treatment effect, and prevention and management. Websites in the .edu, category received the highest scores for objectives, relevance, information credibility, and prevention and management, indicating a better quality of novel coronavirus related health information. Among all .com website categories, objectives had the highest DISCERN score. Results were similar for objectives among .org and .gov website categories. Treatment choices scores were highest among the .edu website category. ere were no significant differences in how the raters assigned DISCERN scores to the objectives and relevance categories (Table 4 ). However, significant difference among the raters for various other DISCERN score categories including information credibility, treatment choices, treatment effect, and prevention and management was seen across different website extensions. Results showed that .edu, .org, and other website categories were excellent in conveying novel coronavirus related health information; however, they received lower scores for treatment effect and treatment choices.

e study results reveal that there is a lack of good quality websites with useful and quality novel coronavirus related health information. Results from sites like .edu, .gov, and .org and other website categories showed higher scores by the raters.

e findings of the current study hold high importance in the current times as it identifies critical gaps in the information on novel coronavirus and other aspects of it. is information paves a way to understand that putting information out on the internet is not the basis for accurate information. e reliability of the information needs consideration, but prioritizing information personalization and tailoring according to the needs of the users is prime focus of information generation and further dissemination. e study's findings can guide users to choose between various websites and help the user prioritize the information available. Several studies have suggested that the internet can be a useful source of health information and assist patients and providers with clinical decision making [23] [24] [25] [26] . e internet plays an important role in public health as it reaches a large part of the population and individuals increasingly turn to it for health information [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] . e ability to obtain online health information accurately, quickly, and conveniently offers consumers opportunity for better-informed decision making [8] . Searching useful and valid information on the internet can be difficult because of the speed and lack of control with which the information is accumulating. In this study, four independent raters who were frequent internet users were selected to search for information on the internet using the most common search engines. ere were differences in how the quality of the information rated by the four independent raters. However, there could be various reasons for these differences, including prior knowledge and familiarity with the health information content. Another reason for this lower interrater reliability could be the subjective nature of some of the questions. It is important to consider the user's perspective when presenting health information content over the internet [9] . e internet is recognized as a basis to educate and empower patients. e internet can provide information on an individual's health problems, prevention/management of diseases, and related health services. e internet is perceived to have the ability to reach those with limited access to information on a wide breadth of topics, and access information when needed. However, prior studies have documented that the information on the internet is of poor quality. Future studies should employ more than one rater [28] . A prior study has recommended having three to four raters to evaluate the quality of health information on the internet and resolve any differences by consensus [29] .

Voluntary organizations should regularly review information on their websites, specifically relating to the provision of up-to-date information on aspects of novel coronavirus and ensure quality of information on the website by providing author credentials and affiliations [30] . Variability in the quality of novel coronavirus related health information websites for core content points to the need for a grading system. is would allow individuals to reach trustworthy, up-to-date websites so that they can receive high-quality information to make informed decisions regarding prevention, treatment, and care. ere are no clear universal guidelines governing healthcare information. Several approaches, such as trust marks that sites can display and principles that websites can use, should be used to regulate their behavior. Also, there should be a mechanism established that will evaluate if wrong info is being spread, that site should be closed down/some regulatory provisions should also be framed in this regard. One recommendation could be that websites, once ranked with trust marks, be popularized for better information dissemination. Opportunities should be created for public health experts and officials to work more effectively with local journalists to increase the impact of public health messages available on websites.

is study has several limitations. It provides only a snapshot in time of information represented in a rapidly changing medium. We expect that changes to the websites that evaluated would already alter some of the findings from the date of the search. Earlier studies have recommended that methods of indexing web pages improved and that information on the web needs to made more readable for users of different socioeconomic backgrounds [31] . e internet has the potential to be a powerful resource for meeting some of the public's health information needs. A shared responsibility between health information consumers and website developers would enable the design and development of targeted websites to address the needs of the individuals. Consumers are generally not aware of characteristics that indicate quality information on the internet [9] . Results of our study help in recognizing the websites that might be useful in gathering health information regarding novel coronavirus on the internet. Education of the users regarding the approach to find valid and authentic health information may make them informed about decision making regarding the information they are accessing.

is study has addressed the gaps in the quality of information that is available on the websites for the subject of novel coronavirus. Further, this study acknowledges and emphasizes the need for websites to develop and provide a clear statement of purpose. It also highlights the need for adequate pondering before this information becomes accessible to users to serve as a means of consumer empowerment instead of putting them in a state of confusion or misguidance. To rapidly disseminate and share newly acquired scientific evidence and experiences in understanding the disease and its control measures, the health informatics are important sources. It is important to analyze the information available on the internet regarding this disease. In any crisis, the informer and the consumer share equal responsibility to solve the problem and share only the right information. e novel coronavirus (COVID-19) pandemic is a case that depicts how accurate passage of health information at the accurate time can save lives while also improving the way we respond to outbreaks in general.

ANOVA: Analysis of variance SD:

Standard deviation WHO:

World Health Organization CDC:

Centers for Disease Control and Prevention HON:

Health on Net.

Data Availability e data used to support the findings of this study are included within the article.

What is already known? (i) e novel coronavirus (COVID-19) pandemic continues to spread across the United States and the world. ere has been a lot of misinformation related to the novel coronavirus outbreak. e amount and nature of COVID-19 related information available to the public is changing and evolving constantly. Misinformation related to COVID-19 is recognized as a substantial public health challenge. (ii) Evidence-based and reliable health information of the disease from trustworthy sources is an urgent priority and a necessary step to disseminate accurate information to the public. (iii) Quality of health information related to COVID-19 requires ongoing evaluation. What are the new findings? (i) By using the quality of health information tools, we were able to demonstrate website extension categories rated as more reliable, with evidence-based information from trustworthy sources. (ii) It is also important to document information related to treatment choices as they become available for COVID-19 patients. What do the new findings imply? (i) e findings show that the quality of health information tools need to be applied to identify the most accurate and trusted sources of information related to COVID-19 outbreak. (ii) ere is a critical need to develop a repository of reliable and evidence-based resources for the public at large. (iii) Future research and practice should focus on innovative population health informatics tools and technologies to disseminate COVID-19 health information in a format that is easy to understand.

No patients were involved in developing the research question, nor were they involved in developing plans for design or implementation of the study. No patients advised on interpretation of the study findings and writing up of results.

e authors declare that they have no conflicts of interest.

First case of 2019 novel coronavirus in the United States

Transmission patterns and control of COVID-19 epidemic

Covid-19-navigating the uncharted

Social media and health care professionals: benefits, risks, and best practices

Accessibility, nature and quality of health information on the Internet: a survey on osteoarthritis

Quality Criteria for Consumer Health Information on Treatment Choices

Patients' views of a multimedia resource featuring experiences of rheumatoid arthritis: pilot evaluation of www.healthtalkonline.org

Public health experts worry about spread of COVID-19 misinformation

Quality of patient health information on the internet: reviewing a complex and evolving landscape

Validation of the translation of an instrument to measure reliability of written information on treatment choices: a study on attention deficit/hyperactivity disorder (ADHD)

Assessment of the quality and variability of health information on chronic pain websites using the DIS-CERN instrument

Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet

Quality of nutrition related information on the internet for osteoporosis patients: a critical review

A communitybased motivational personalised lifestyle intervention to reduce BMI in obese adolescents: results from the Healthy Eating and Lifestyle Programme (HELP) randomised controlled trial

Evaluating the quality of educational materials about schizophrenia

Assessing the quality of internet health information

A study of undue pain and surfing: using hierarchical criteria to assess website quality

Evaluating the reliability of DISCERN: a tool for assessing the quality of written e Scientific World Journal 7 patient information on treatment choices

An investigation of the quality of breast cancer information provided on the internet by voluntary organisations in Great Britain

Guidelines for medical and health information sites on the internet

e impact of health education on mother's knowledge, attitude and practice (KAP) of dengue haemorrhagic fever

How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and indepth interviews

e quality of health information on the internet

Reliability of health information on the internet: an examination of experts' ratings

Consumer health information seeking on the internet: the state of the art

Evaluation of medical web sites: interobserver and intraobserver reliability of an evaluation tool

Emerging challenges in using health information from the internet