key: cord-0960756-7mjrxyd2 authors: Kreutzer, Emma Pauline; Sauer, Sandra; Kriegsmann, Mark; Staemmler, Henrike; Egerer, Gerlinde; Kriegsmann, Katharina title: Accuracy and Reliability of Internet Resources for Information on Monoclonal Gammopathy of Undetermined Significance—What Information Is out There for Our Patients? date: 2021-09-07 journal: Cancers (Basel) DOI: 10.3390/cancers13184508 sha: e711ee1b89a0b34651ec44fb684d50db6b382664 doc_id: 960756 cord_uid: 7mjrxyd2 SIMPLE SUMMARY: In the current analysis, we provide a comprehensive evaluation of monoclonal gammopathy of undetermined significance (MGUS)-related health information available online. An analysis of general and patient- (user-) focused quality, readability, and content of websites and videos was performed. It revealed a generally medium to low quality of internet resources. Therefore, understandability, informative value, and support in a decision-making process can be attributed to single websites/videos only. Our study clearly highlights the risk of misinformation by insufficient, incorrect, misleading, and out-of-date information. Knowing what content is assessable to patients online may help clinicians to educate their patients and actively address misinformation. ABSTRACT: Background: Online information gathering can increase patients’ engagement in decision-making. The quality of online resources available for monoclonal gammopathy of undetermined significance (MGUS) was evaluated. Methods: 900 websites from Google, Bing, Yahoo, and 150 YouTube videos were assessed. Results: The websites did not differ regarding their search rank or between the search engines. The median time since last update was 24 months. The 86 unique websites showed a medium to poor general quality (JAMA score 3/4, only 8.1% websites with a valid HON certificate). The patient- (user-) focused quality was poor (sum DISCERN score 27/80 points). The reading level was difficult (11th US school grade). The content level was very low (13/50 points). 12.8% of websites contained misleading/wrong facts. Websites provided by scientific/governmental organizations had a higher content level. For the 61 unique videos, the median time since upload was 34 months. The videos showed a medium general quality (HON Foundation score). The patient- (user-) focused quality was poor (sum DISCERN score 24 points). The content level was very low (6 points). Conclusion: MGUS-relevant online sources showed a low quality that was provided on a high reading level. Incorporation of quality indices and regular review of online content is warranted. Monoclonal gammopathy of undetermined significance (MGUS) is a clonal, premalignant plasma cell or B-cell disorder, characterized by the presence of monoclonal protein and absence of multiple myeloma or lymphocytic/lymphoplasmacytic malignancies [1, 2] . Given the high prevalence, around 3% in persons ≥50 years and around 5% in persons Written online health information on MGUS in English was analyzed. North American access for the search engines Google, Bing, and Yahoo was used. Searches were performed on 18-26 March 2020 after the removal of the web browser cookies and history. The three search terms "monoclonal gammopathy", "monoclonal gammopathy of undetermined significance" and "MGUS" were applied. For each search term, the first 100 search hits in each search engine were selected for further evaluation, resulting in 300 websites per search engine and 900 overall initial websites. To identify videos with health information on MGUS, the online video-sharing platform YouTube was searched for the three terms. Searches were performed on 28-31 March 2020. For each search term, the first 50 search hits were selected for further evaluation, resulting in 150 overall initial videos. Websites and videos with no relevance to MGUS (e.g., only including the search term in a bullet list), requiring fees or registration, using other languages than English or scientific journal articles intended for professionals were defined as not eligible. For website evaluation, in a first step, websites not meeting the eligibility criteria and duplicates were excluded for each search engine separately and an evaluation by the search engine was performed. In a second step, the final overall list of websites was created by removing the duplicates between the search engines. These websites (called "unique") are meant to represent the primarily assessable body of information to the patients. For detailed evaluation, general website and video information, general quality of medical information online, patient-(user-) focused quality of medical information online, readability, and entity-related content were assessed (Table S1 ) [15] . General information on websites included the search rank of the respective search engine, paid advertising website, URL (Uniform Resource Locator), host continent, website category (scientific/governmental, e.g., the National Cancer Institute, Bethesda, United States; foundation/advocacy, e.g., the Leukemia Foundation, Brisbane, Australia; news/media, industry/for-profit; personal commentary/blog), update and access date. General information on videos included the search rank, URL/title, host continent, video category (identical to website category), upload and access date, video duration, number of views, likes, dislikes, and comments. The viewing rate was calculated as: views/days since upload. The engagement rate was calculated as: (likes + dislikes + comments)/views. The general quality of medical information online was evaluated by the Health on the Net (HON) Foundation certificate/score and the Journal of the American Medical Association (JAMA) score. HON is an international not-for-profit, non-governmental organization that promotes transparent and reliable health information online. Providers of health information websites can certify their website by the HON Foundation [16, 17] . As the HON certificate applies for websites only, a step-by-step evaluation of videos was performed according to the eight HON Foundation principle criteria (authority, complementarity, confidentiality, attribution, justifiability, transparency of authorship, transparency of sponsorship, honesty in advertising, and editorial policy; minimum points 0, maximum points 8). The achieved points translate into a low (0-2 points), medium (3-5 points) , and high (6-8 points) quality category according to the HON Foundation. The JAMA score evaluates a series of four criteria (authorship, attribution, disclosure, and currency; minimum points 0, maximum points 4) and aims to assess, control, and assure the quality of medical information on the internet [18] . The patient-(user-) focused quality of medical information online was evaluated by the DISCERN score, an instrument for judging the quality of written consumer health information on treatment choices. It addresses the following questions: is the publication reliable? (section 1, item 1-8); how good is the quality of information on treatment choices? (section 2, item 9-15); overall rating of the publication (item 16) [19] . A minimum of 1 point (not addressed) and a maximum of 5 (fully addressed) points can be achieved per DISCERN score item, resulting in a minimum of 16 (1 point × 16 items) and a maximum of 80 (5 points × 16 items) achievable points per website/video. As MGUS requires follow-up rather than treatment, section 2 was modified as follows: item 9-how treatment works-modified to the significance of MGUS diagnosis, item 10-benefits of treatmentmodified to benefits of follow-up, item 11-risks of treatment-modified to disadvantages of follow-up, item 12-no treatment options-modified to risks of no follow-up, item 13-quality of life-modified to uncertainty associated with follow-ups, item 14-other treatment options-modified to variation of follow-up intervals, item 15-shared decision making-not modified. To ensure the best possible objectivity, the evaluation of websites and videos according to the DISCERN was performed by two observers. The readability of websites was assessed according to the Flesch Reading Ease score and the Flesch Kincaid Grade level. The Flesch Reading Ease score was calculated as: 206.835 − (1.015 × average sentence length) − (84.6 × average number of syllables per word) and ranges between 0-30 (very difficult to understand text) and 90-100 (very easy to understand text) [20] . The Flesch Kincaid Grade level uses a modified Flesch Reading Ease formula ((0.39 × average sentence length) + (11.8 × average number of syllables per word) − 15.59) to produce a grade-level score according to the US school grade that is needed to understand a text [21] . The entity-related content was assessed according to 50 key facts on MGUS derived from current guidelines and addressing the categories definition, symptoms, risk factors, evaluation, management, outcome, and risk of progression (Table S2) [1, 2, 22] . 0 (not addressed), 0.5 (partially addressed), and 1 (fully addressed) points were achievable per key fact items, resulting in a minimum of 0 (0 points × 50 items) and a maximum of 50 (1 point × 50 items) achievable overall content points per website/video. If applicable, misleading, and wrong facts were recorded and classified according to the key fact category. Statistical analysis was performed in R studio (R version 4.0.2, 2020-06-22, The R Foundation for Statistical Computing). Data were presented as absolute numbers and percentages, medians and ranges as well as the means and standard deviations (SD) as appropriate. To analyze contingency tables Fisher's exact was used. The distribution of ordinal scaled variables was compared by the Mann-Whitney U test (two groups) or the Kruskal-Wallis H test (more than two groups). To identify differences between group means for continuous variables, comparisons were performed with unpaired two-tailed Student's t-tests (two groups) or analysis of variance (ANOVA, more than two groups). Linear regression was performed to investigate the correlation between the website search rank and the sum DISCERN score, sum key facts score, and time since update. Inverse Kaplan-Meier curves were chosen to demonstrate the proportion of website/video updates by time. An event represents a website update. The time to event was calculated as a difference in months between the update/upload and access date. Websites not indicating an update date were excluded from the Kaplan-Meier curve representation. Unsupervised hierarchical clustering (R package 'ComplexHeatmap') by the website and DISCERN or key fact score items was performed in order to identify an association between website category and addressed items. A p ≤ 0.05 was considered statistically significant. The initial search with three search terms resulted in 300 hits per search engine and, therefore, overall 900 websites. The removal of duplicates between the search terms and websites not meeting the eligibility criteria resulted in 53, 66, and 59 websites for Google, Bing, and Yahoo, respectively. The removal of duplicates between the three search engines resulted in 86 unique MGUS-relevant websites (Table 1) . MGUS, monoclonal gammopathy of undetermined significance. In order to identify differences between the three search engines a characterization and comparison of websites found on Google (n = 53), Bing (n = 66), and Yahoo (n = 59) was performed in the first step (Table S3) . We could not identify any statistically significant differences regarding the general website information. There was one promoted website on each search engine in terms of paid advertising. A comparison of websites with an indicated update date regarding the time since last update did not show any statistically significant differences (median time since last website update: Google 16 months, Bing 21 months, Yahoo 21 months, p = 0.720, Figure S1D ). The MGUS-relevant websites identified on Google, Bing and Yahoo also did not significantly differ in terms of the general quality of medical information online (availability of the HON Foundation certificate, JAMA score), patient-(user-) focused quality of medical information online (DISCERN score), readability (Flesch Reading Ease score, Flesch Kincaid Grade Level), and MGUS-related content (key fact score, misleading/wrong facts). Moreover, no correlation between the search rank and the sum of the DISCERN score, the sum of the key fact score, or the time since upload was identified for any search engine (Figure S1A-C). Eighty-six unique MGUS relevant websites, representing the primarily assessable body of information to the patients, were identified ( Figure 1A ) and subjected to an elaborate characterization ( Table 2) . The most websites were provided either by a scientific/governmental organization (n = 38, 44.2%) or a foundation/advocacy (n = 31, 36.1%). Around 20% of websites were provided by news/media, industry/for-profit organizations, or by single individuals in terms of a personal commentary/blog. The majority of websites originated from North America (n = 68, 79.1%), followed by Europe (n = 11, 12.8%). Sixty-two websites (72.1%) indicated an upload/update date ( Figure 1B ). The median time since upload/last update of those websites was 24 months ( Figure 1C) . The general quality of medical information online was acceptable in terms of the JAMA score: of 4 maximum achievable points on authorship, attribution, disclosure, and currency, in median 3 (range 0-4) points were addressed by the evaluated websites. However, only a few websites had a valid HON Foundation certificate (n = 7, 8.1%). The patient-(user-) focused quality of medical information online assessed by the DISCERN score was rather poor: of 16 minimum and 80 maximum possible points, in median 27 (range 16-43) points were achieved by the evaluated websites. While the items on the reliability of the provided information were at least addressed (section 1), information on treatment choices/follow-up of MGUS (section 2) was provided by very few websites ( Figure 1D , Figure S2B , Table S4 ). The readability according to the mean Flesch Reading Ease score of the evaluated websites was difficult (mean 48, SD 10). The mean Flesch Kincaid Grade level was 11 (SD 3), indicating that at least an 11th US school grade is necessary to understand the website content. The MGUS related content was evaluated by a set of 50 very detailed key facts ( Figure 1E , Figure S2C ). Of 50 maximum possible points, in median 13 (range 3-37) points were achieved. Approximately one-third of the evaluated websites at least partially addressed the aspects of MGUS diagnosis criteria. However, the absence of recently established SLiM criteria (clonal bone marrow plasma cells <60%, involved/uninvolved serum free light chain ratio <100, ≤1 focal lesions on magnetic resonance imaging) at MGUS first diagnosis were mentioned only by one website. A detailed enumeration of analyses to be performed for MGUS evaluation was not part of the content provided by most websites. The absence of symptoms, possible risk factors, management in terms of continuous follow-up, possible progression including progression rates into MM, B-cell non-Hodgkin lymphoma, and other associated conditions were frequently part of the content provided by the evaluated websites. Eleven websites (12.8%) containing, overall, 40 misleading/wrong facts were identified (Table 3) . Ten misleading/wrong facts (25.0%) related to the definition of MGUS diagnosis. Exemplarily, MGUS was falsely described to be associated with symptoms of end-organ damage-which in fact contradicts the diagnosis criteria of MGUS. Moreover, misleading information on risk factors of MGUS (n = 9, 22.5%), as a strong association with infections or environmental toxins, was provided. The most misleading/wrong facts related to the management of MGUS (n = 15, 37.5%). In this regard, misleading information on a treatment indication given an MGUS diagnosis was provided. Moreover, partially absurd suggestions on nutrition and diet were identified. Industry/for-profit 1 0 Personal commentary/blog 0 3 Overall, n 40 25 By key fact category, n (%) A data set containing the original data on selected quality, content scores, and citations of misleading/wrong facts obtained from the unique websites is provided along with this manuscript (Table S5 ). In order to identify differences between the website sources, a characterization and comparison of websites by providers was performed (scientific/governmental n = 38, foundation/advocacy n = 31, news/media n = 13, Table 4 ). Due to the low case number of websites, the industry/for-profit (n = 2) and personal commentary/blog (n = 2) categories were not included. Although the median time since last update was shorter in scientific/governmental website providers (13 months) compared to foundations/advocacies (25 months) or news/media (24 months), this difference was not statistically significant (p = 0.600, Figure S2A ). The MGUS-relevant websites of different providers also did not significantly differ in terms of the general quality of medical information online (availability of the HON Foundation certificate, JAMA score), patient-(user-) focused quality of medical information online (DISCERN score, Figure S2B ), and readability (Flesch Reading Ease score, Flesch Kincaid Grade Level). However, websites provided by scientific/governmental institutions have a significantly higher median sum key fact score (17, range 4-37) compared to foundations/advocacies (11, range 5-35) or news/media (11, range 5-33), (p = 0.022). This is also reflected by the hierarchically clustered heatmap, that shows clusters of websites provided by scien-tific/governmental institutions along with the most fully and partially addressed key fact items ( Figure S2C ). The initial YouTube search with 50 hits per search term resulted in overall 150 videos. The removal of duplicates between the search terms and videos not meeting the eligibility criteria resulted in 61 MGUS relevant videos (Table 1) . The 61 unique MGUS relevant videos were assumed as the primarily assessable body of information and subjected to an elaborate characterization ( Table 2) . Most videos were provided by news/media (n = 32, 52.5%), followed by foundations/advocacies (n = 21, 34.4%). Only a few videos were provided by a scientific/governmental organization (n = 2, 3.3%). Most videos originated from North America (n = 51, 83.6%). The median time since upload of the evaluated videos was 34 months (Figure 2A ). In median, the videos had a duration of 4 (range 1-57) minutes and 452 (range 24-55,869) views. A median viewing rate of 0.81 (range 0.02-28.16) indicated that in median the videos were watched less than once a day. The engagement rate was very low (median 0.01, range 0.00-0.05), resulting from low numbers of likes, dislikes, and comments. As the JAMA score and HON Foundation certificate refer to websites, the general quality of medical information online was assessed in a step-by-step evaluation of videos according to the eight HON Foundation principle criteria. Of 8 maximum points, in median 3 (range 1-6) points were achieved by the evaluated videos. While the items authority, confidentiality, and transparency of authorship were addressed by the majority of videos, aspects of complementarity, attribution, justifiability, transparency of sponsorship, honesty in advertising, and editorial policy were generally not met ( Figure 2B) . Overall, only one video (1.6%) reached a high quality according to the HON Foundation score. Thiry-five videos (57.4%) were ranked as medium and 25 videos (40.1%) as low quality according to the HON Foundation score. The patient-(user-) focused quality of medical information online assessed by the DISCERN score was rather poor: of 16 minimum and 80 maximum possible points, in median 24 (range 18-35) points were achieved by the evaluated videos. The items "explicit aims", "aims achieved", "relevance to patients" and "significance of MGUS diagnosis" were at least addressed in most videos. However, the remaining items, including the most items on the quality of information on treatment/follow-up choices (section 2) were provided by very few videos ( Figure 2C , Figure S3B , Table S4 ). The MGUS-related content was evaluated by a set of 50 very detailed key facts ( Figure 2D , Figure S3C ). Of 50 maximum possible points, in median 6 (range 0-22) points were achieved, demonstrating a low informational content. The content addressed in most videos related to a possible outcome of the precancerous condition MGUS, i.e., transition in SM and/or MM. Aspects of MGUS diagnosis criteria, absence of symptoms, risk factors, management, and risk of progression were content in not more than one-third of the evaluated videos. The details on the evaluation of MGUS were addressed only in a few videos. Eight videos (13.1%) containing an overall 25 misleading/wrong facts were identified (Table 3) . Twenty misleading/wrong facts (80.0%) related to the definition of MGUS diagnosis: MGUS was falsely described to be associated with symptoms of end-organ damage-which contradicts the diagnosis criteria of MGUS. The misleading/wrong facts on risk factors (n = 2, 8.0%) established a causal link between MGUS and environmental toxins or viral infection, which is not proven so far. Moreover, regarding the management (n = 2, 8.0%) steroid treatment and regular bone density assessment were suggested, which is not recommended in the current guidelines. A data set containing the original data on selected quality, content scores, and citations of misleading/wrong facts obtained from the unique videos is provided along with this manuscript (Table S6) . In order to identify differences, a comparison of videos by providers was performed (foundation/advocacy n = 21, news/media n = 32, Table 5 ). Due to the low number, videos provided by scientific/governmental institutions (n = 2), industry/for-profit organizations (n = 2), and personal commentary/blogs (n = 4) were not included. Table 5 . Characterization of videos by category. Video category, n 21 32 Host continent, n (%) Videos provided by foundations/advocacies had a significantly higher number of views (median 885, range 24-55,869) compared to news/media (median 207, range 26-6245), (p = 0.004). This resulted in a higher viewing rate (median 1.03 versus 053). However, the viewing rate difference was statistically not significant (p = 0.075). MGUS videos provided by foundations/advocacies provided a higher general quality of medical information online judged by the HON Foundation score. In the first-mentioned category the most videos (n = 18, 85.7%) achieved a medium HON quality rank, while in the latter category only 12 videos (37.5%) achieved this rank and other videos were ranked as low (n = 20, 62.5%, p < 0.001, Figure S3A ). No statistically significant differences were identified regarding the patient-(user-) focused quality of medical information online according to the DISCERN score (p = 0.857, Figure S3B ). There was also no difference in the MGUS key facts addressed in videos of both categories (p = 0.368, Figure S3C ). A comparison of unique websites (n = 86) and videos (n = 61) was possible, in some, but not all assessed variables (Table 2) . While scientific/governmental institutions (n = 38, 44.2%) and foundations/advocacies (n = 31, 36.1%) provided the most websites, the videos were mainly provided by news/media (n = 32, 52.5%) and foundations/advocacies (n = 21, 34.4%), (p < 0.001). The patient-(user-) focused quality of medical information online evaluated by the DISCERN score (minimum 16, maximum 80 points) was poor in both websites (median 27, range and videos (median 24, range [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] , with no statistically significant difference (p = 0.246). The amount of MGUS-related content evaluated by the sum key fact score (minimum 0, maximum 50 points) provided by websites (median 13, range 3-37) was higher compared to videos (median 6, range 0-22), (p < 0.001). The current analysis provides an extensive evaluation of MGUS-related online health information. Previously published analyses on the reliability and accuracy of online health information mainly focus on otorhinolaryngology-related topics [23] [24] [25] [26] [27] [28] [29] [30] . Studies on other medical conditions, such as idiopathic pulmonary fibrosis, SARS-CoV-2, neurological disorders, etc., are available as well [15, [31] [32] [33] [34] [35] [36] . However, except for oral precancerous conditions, other precancerous and cancer entities were not in the focus of such evaluations [37, 38] . As associated with a constant fear of facing cancer, precancerous conditions might be, in particular, the focus of a patients' online search and the quality of available online information is crucial. Besides its novelty, our study shows methodological strengths. A significant number of MGUS-related websites was analyzed upon a broad online search of three search engines. Going beyond written content, videos were considered. As Google and YouTube are the two most visited search platforms, the selection of analyzed websites and videos might be considered representative [39] . The evaluation was performed in a reproducible and objective score-based manner, applying a set of well-established scores covering the aspects of general, patient-(user-) focused quality, and MGUS-related content. The limitations of the study can mainly be attributed to constantly varying and potentially growing numbers, continuous updating, and changes in the rating of websites and videos. Therefore, the website and video selection in this analysis cannot be considered as ultimate. The website search on Google, Bing, and Yahoo revealed a lack of integrity/insufficient search results which was reflected by a high number of websites with no relevance to MGUS. The median time since last website update was 24 months, which reflects an outdated informational status. The actuality of health online information is of outstanding importance and should be critically considered [40] . Of importance, the website search rank in each search engine does not necessarily correlate with quality [41] . Indeed, in the current analysis, the most qualitative websites were not ranked at a top position. Moreover, the websites did not differ in terms of analyzed scores between the three search engines or regarding their search rank. Thus, not only the search engine market leader Google but also other search engines might be used, which is advantageous for users [39] . The 86 unique MGUS-relevant websites showed a medium general quality of medical information online and less than 10% of websites had a valid HON Foundation certificate. This is in line with the facts-only a small proportion of medical and health websites provide quality scores and certificates, and identification of websites with high-quality information is challenging for patients [42] . The patient-focused quality of MGUS-related information was poor, with, therefore, questionable potential to reduce anxiety and concerns and support the compliance for regular follow-up examinations. The overall reading level required at least an 11th US school grade. As "most adults read at an 8th grade level, and 20% of the population reads at or below a 5th grade level", the content would have been difficult to understand for the vast majority of patients [43] . The informational content level was very low. As we applied a large set of stringent criteria based on current diagnostic and clinical guidelines, a high sum content key fact score was not expected [1, 2, 22] . However, 12.8% of websites contained misleading/wrong facts, holding the risk of misinformation. Websites provided by scientific/governmental organizations were similar to those provided by foundations/advocacies or news/media, in terms of all analyzed variables except a slightly higher content level. Overall, our results are in line with these previously published studies. Those conclude on frequently outdated, mixed-, or low-quality, and incomplete online health information that requires high readability skills [15, [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] . Given a median time since video upload of 34 months, the MGUS-related video content can be considered as outdated. Judged by the HON Foundation score, the general quality of medical information online can be considered as medium. The patient-(user-) focused quality of MGUS-related information was poor and therefore similar to MGUSrelated websites. The content level was very low, even significantly lower compared to websites. MGUS videos provided by foundations/advocacies provided a higher general quality of medical information online judged by the HON Foundation score. To the best of our knowledge, this is the first systematical and score-based evaluation of disease-related videos that goes beyond the assessment of provided medical content. We could identify only one "snapshot analysis of information available on youtube.com" published by Tan et al., that evaluates the quality of content regarding breast reconstruction in breast cancer patients [44] . Similar to the results obtained in the current study, the authors conclude that the YouTube videos do not provide comprehensive information. The current analysis of online information on MGUS revealed an overall low general, patient-(user-) focused and content quality provided on a high reading level. Thus, understandability and informative value can be attributed to single sources only. Therefore, regular review, incorporation of indices indicating the quality of provided online health information, and consideration of current MGUS guidelines is warranted. Supplementary Materials: The following are available online at https://www.mdpi.com/article/ 10.3390/cancers13184508/s1, Figure S1 : Characterization of websites by search rank and time since last update, Figure S2 : Characterization of websites by category, Figure S3 : Characterization of videos by category, Table S1 : Scores, measures and certificates used for the evaluation of websites and videos, (Tables S5 and S6 ). How I manage monoclonal gammopathy of undetermined significance International Myeloma Working Group updated criteria for the diagnosis of multiple myeloma Prevalence of Monoclonal Gammopathy of Undetermined Significance Breast cancer on the world wide web: Cross sectional survey of quality of information and popularity of websites Evaluating and providing quality health information for adolescents and young adults with cancer. Pediatr 53% of EU Citizens Sought Health Information Online Internet Health Information Seeking and the Patient-Physician Relationship: A Systematic Review Health information on the Internet: Gold mine or minefield? Impact of internet use on health-related behaviors and the patient-physician relationship: A survey-based study and review Use of the Internet and E-mail for Health Care Information: Results from a National Survey How do Consumers Search for and Appraise Information on Medicines on the Internet? A Qualitative Study Using Focus Groups Shortcomings of health information on the Internet What do Web-Use Skill Differences Imply for Online Health Information Searches? Accuracy and Reliability of Internet Resources for Information on Idiopathic Pulmonary Fibrosis The Health on the Net Code of Conduct for medical and health web sites Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor-Let the reader and viewer beware DISCERN: An instrument for judging the quality of written consumer health information on treatment choices A new readability yardstick Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for NAVY Enlisted Personnel Monoklonale Gammopathie Unklarer Signif-Ikanz (MGUS) Quality and readability of internet information about stuttering Quality and readability of English-language Internet information for vestibular disorders Quality and readability of online information on dental treatment for snoring and obstructive sleep apnea Accuracy and Reliability of Internet Resources Providing Information on Obstructive Sleep Apnea Quality and readability of internet-based information on halitosis. Oral Surg. Oral Med. Oral Pathol Quality and readability of websites for patient information on tonsillectomy and sleep apnea Online Information on the Treatment of Burning Mouth Syndrome: Quality and Readability. J. Oral Facial Pain Headache The quality and accuracy of internet information on the subject of ear tubes Evaluating the Source and Content of Orthopaedic Information on the Internet. The case of carpal tunnel syndrome Quality and readability of online patient information for abdominal aortic aneurysms Quality of web based information on treatment of depression: Cross sectional survey Quality and readability of web-based Arabic health information on COVID-19: An infodemiological study Cauda Equina Syndrome: Assessing the readability and quality of patient information on the Internet Assessing the quality, reliability and readability of online health information regarding systemic lupus erythematosus Web-based information on oral dysplasia and precancer of the mouth-Quality and readability Web-based information on the treatment of oral leukoplakia-Quality and readability The Top 500 Sites on the Web Online Health Information: Is It Reliable? Available online Google's 200 Ranking Factors: The Complete List Health literacy: The gap between physicians and patients Patient information on breast reconstruction in the era of the world wide web. A snapshot analysis of information available on youtube The authors declare no conflict of interest.