key: cord-0931698-351rfpgn
authors: Man, Alice; van Ballegooie, Courtney
title: Assessment of the Readability of Web-Based Patient Education Material From Major Canadian Pediatric Associations: Cross-sectional Study
date: 2022-03-16
journal: JMIR Pediatr Parent
DOI: 10.2196/31820
sha: 580d0f4d4f086db97e50da1d310154deba300285
doc_id: 931698
cord_uid: 351rfpgn

BACKGROUND: Web-based patient education materials (PEMs) are frequently written above the recommended reading level in North America. Poor PEM readability limits the accessibility of medical information for individuals with average literacy levels or lower. Pediatric hospital and association websites have not only been shown to be a preferred source of information among caregivers but have also become a necessity during the COVID-19 pandemic. The readability of Canadian pediatric association websites has not yet been assessed. OBJECTIVE: The aim of this study is to determine if the content of PEMs from Canadian pediatric associations is written at a reading level that the majority of Canadians can understand. METHODS: A total of 258 PEMs were extracted from 10 Canadian pediatric associations and evaluated for their reading level using 10 validated readability scales. The PEMs underwent a difficult word analysis and comparisons between PEMs from different associations were conducted. RESULTS: Web-based PEMs were identified from 3 pediatric association websites, where the reading level (calculated as a grade level) was found to be an average of 8.8 (SD 1.8) for the Caring for Kids website, 9.5 (SD 2.2) for the Pediatric Endocrine Group website, and 13.1 (SD 2.1) for the Atlantic Pediatric Society website. The difficult word analysis identified that 19.9% (SD 6.6%) of words were unfamiliar, with 13.3% (SD 5.3%) and 31.9% (SD 6.1%) of words being considered complex (≥3 syllables) and long (≥6 letters), respectively. CONCLUSIONS: The web-based PEMs were found to be written above the recommended seventh-grade reading level for Canadians. Consideration should be made to create PEMs at an appropriate reading level for both patients and their caregivers to encourage health literacy and ultimately promote preventative health behaviors and improve child health outcomes.

doctor's visit but can also inform patients on best health practices and encourage them to seek medical treatment for symptoms. This is particularly relevant during the COVID-19 pandemic as patients are seeking information about unprecedented medical concerns with limited access to health care [4, 5] .

Pediatric hospital and association websites are preferred sources of health information by caregivers among internet resources [6] . It is especially important that these websites are accurate and accessible to prevent misinformation. If the content is too difficult to understand, caregivers and patients may resort to using less accurate internet sources. This may exacerbate disparities in health outcomes since individuals with higher health literacy will have greater functional access to health-related content than those with literary barriers. These disparities have been highlighted through multiple studies, where low caregiver health literacy was shown to be associated with poor preventative health behaviors, increased pediatric emergency department use, nonurgent visits, and poorer child health outcomes [7] [8] [9] .

In Canada, the average proficiency in literacy corresponds approximately to an eighth-to ninth-grade reading level, where over 45% of Canadian adults have been shown to have low literacy skills [10, 11] . The readability of educational material is recommended to be at least 2 or more grade levels below the average Canadian reading level to ensure comprehension [12] . Therefore, all patient-related material should be written at a maximum of a seventh-grade reading level. As health literacy commonly requires the use of a combination of prose literacy, document literacy, and/or numeracy skills, adults may have a harder time understanding health-related content than typical prose [13] . Even with adequate literacy skills, many caregivers still have difficulty understanding well-established health-related information in order to care for their infant [14] . Health-related reading materials should be further simplified to account for these additional challenges.

Pediatric health literacy has been explored globally, focusing on a variety of topics and subspecialties within pediatrics. Overall, studies in North America, France, Australia, the United Arab Emirates, Turkey, and Brazil have found that pediatric health information has been written above an acceptable reading level [15] [16] [17] [18] [19] [20] . Topics have ranged from mental health, otolaryngology, orthopedics, oral health, oncology, and consent and discharge forms [21] [22] [23] [24] [25] [26] [27] . In Canada, although a variety of topics and subspecialties have been studied as they relate to health literacy, such as oncology, microtia and aural atresia, and emergency medicine [28] [29] [30] , no study has evaluated the pediatric information developed by major pediatric associations and societies from multiple disciplines. This study aims to evaluate the reading level of the web-based Canadian pediatric patient education material (PEM) from pediatric associations and societies and to provide specific recommendations to improve readability.

During May and June 2020, all internet-based PEMs were downloaded from the pediatric associations' websites. A total of 10 national associations were identified and are listed in Table 1 along with the number of unique PEMs obtained from each association. The downloaded PEMs included materials describing any topic with intended use by parents, guardians, or children on the pediatric websites. Therefore, this excluded any material intended for health care providers. PDF files were manually converted to plain text for further analysis. Text sections containing nonmedical information such as page numbers, disclaimers, tables, diagrams, phone numbers, emails, and webpage navigation were removed from each of the PEMs before analysis. Gunning Fog index (GF); New Fog Count (NFC); New Dale-Chall readability formula (NDC); and Ford, Caylor, Sticht scale (FORCAST). The 2 graphical scales included the Raygor Readability Estimate Graph (RREG) and the Fry Readability Graph (FRG). These 10 scales are often used when assessing medical text and offer externally validated measures of readability [31] [32] [33] .

PEMs often contain text that must be modified before the analysis to appropriately apply the readability scales. This includes the removal of charts as well as the modification of bullet points to form complete sentences for analysis. To address the limitation of narrative-based readability scales, PEMs were individually edited to create high-and low-sentence documents, as performed by Perni et al [32] . For example, in high-sentence documents, each individual bullet point was treated as an independent sentence and resulted in a lower grade level estimate. On the other hand, low-sentence documents had each bullet point separated with a comma, with the final bullet point ending the sentence; this resulted in a higher grade level estimate [31, 32] . The high-and low-level estimates were then averaged for further analysis. The associations' readability level using the 8 numerical scales can be seen in Figure 1 , and the 2 graphical scales can be seen in Figure 2 . 

A difficult word analysis was performed to identify the number and percentage of complex words (composed of 3 or more syllables), long words (composed of 6 or more letters), and unfamiliar words in each PEM according to the NDC criteria [34, 35] . Once all the words were extracted from the PEMs, they were compared to the NDC word list as well as the New General Service List (NGSL). Words that appeared in either of the lists were removed and considered to be nonjargon words. All words that appeared in less than 3 PEMs were excluded from the analysis. Words with 3 or more syllables were then extracted, and the various tenses of the 10 most frequently identified words, where applicable, were combined. Alternative words were then proposed for any 3-syllable word that appeared in 3 or more PEMs, either using the Readability Studio Software, the Merriam-Webster thesaurus, or in consultation with a physician, to identify synonyms that can decrease the difficulty of the word.

Graphical data in Figure 1 were reported as the arithmetic means with the error bars representing the standard deviations. Data sets had their normality tested using a Shapiro-Wilk test when central limit theorem conditions were not met. Equal variance was tested using a Brown-Forsythe test to determine if the data would need to be transformed before analysis. Normally distributed data with equal variance then underwent a one-way analysis of variance (ANOVA). If the data were not normally distributed, then a nonparametric Kruskal-Wallis test was employed. Multiple comparisons tests, such as Tukey tests, were used to identify differences between sample means in the ANOVA analysis [36] . The data were analyzed using Graph Pad Prism (version 9; GraphPad Software Inc).

Following conversion to plain text, high-and low-sentence PEMs were subjected to 8 readability tests, including the DRP-GE, FK, SMOG, CLI, GF, NFC, NDC, and FORCAST. Figure 1 illustrates a summary of the results for the pediatric associations. The reading levels (reported as grade levels) measured by the 8 readability scales were averaged for each pediatric association, where the mean and standard deviations are reported as follows: Caring for Kids (8.8, SD 1.8), Pediatric Endocrine Group (9.5, SD 2.2), and the Atlantic Pediatric Society (13.1, SD 2.1). The overall mean was 9.1 (SD 2.4), with a grade-level range of 5 to 17. Note that Caring for Kids is a website developed by the Canadian Pediatric Society.

When the 8 readability scores of the individual PEMs were averaged, only 18 (7%) and 144 (55.8%) of the 258 PEMs, were below a seventh-grade and ninth-grade level, respectively. The RREG score of the high-sentence PEMs (Figure 2 ) ranges from a third-grade reading level to a grade level equivalent to that in university, with 26 (10.1%) and 127 (49.2%) of the 258 PEMs written at a grade level below 7 and 9, respectively. The FRG score of the high-sentence estimate, as seen in Figure 2 , ranges from a third-grade to a 17th-grade (university-educated) reading level, with 14 (5.4%) and 118 (45.7%) of the 258 PEMs written at a grade level below 7 and 9, respectively. The grade levels calculated by all 8 scales from the Atlantic Pediatric Society's PEMs were also compared to those from Caring for Kids and the Pediatric Endocrine Group. Comparisons with 7 out of 8 reading tests were found to be statistically significant for both pediatric associations, where the NFC test was the only test to show no statistical significance ( Table 2 ). 

From the difficult word analysis, it was determined that of all the words found in the PEMs, on average, 13.3% (SD 5.3%) were complex words which contained 3 or more syllables, 31.9% (SD 6.1%) contained 6 or more letters, and 19.9% (SD 6.6%) were unfamiliar words. All PEMs collected had a target audience of caregivers or pediatric patients (described as patients between the ages of 0 and 19). The most frequent terms included cannabis, marijuana, medication(-s), calcium, cortisol, and hepatitis. Table 3 describes the most frequent difficult words in compliance with the criteria described in the methods section. 

PEMs found on pediatric associations' websites serve as an important link between health care professionals and caregivers. Through these web-based resources, parents can access reputable information endorsed by health care professionals to inform childcare practices on a day-to-day basis [1] [2] [3] . Although these resources are readily available with internet access, they are not always functionally accessible to all caregivers and patients. Pediatric PEMs have consistently been shown globally to be written at higher reading levels than recommended for a public audience, which is consistent with this study's findings [9, [15] [16] [17] [18] [19] [20] .

Based on the analyses using the DRP-GE, FK, SMOG, CLI, GF, NFC, NDC, and FORCAST scales, PEMs available on Canadian pediatric association websites were found to be written at a ninth-grade reading level (mean 9.1, SD 2.4) on average, wherein only 7% (18/258) of PEMs were written below the recommended seventh-grade reading level. Similar results were shown by the RREG and FRG (Figure 2) , wherein only 5.4% (14/258) to 10.1% (26/258) of PEMs were found to be written below a seventh-grade reading level. This suggests that the PEMs cannot be easily understood by most Canadians, and even less so by pediatric patients. This is particularly true for the Atlantic Pediatric Society's PEMs, which are written at a university reading level (mean 13.1, SD 2.1). In Table 2 , it can be seen that the Atlantic Pediatric Society's PEMs were statistically significantly different from other pediatric associations' PEMs in a majority of the readability tests employed. This suggests that the Atlantic Pediatric Society should consider all parameters used in each readability test, such as word and sentence length, the number of syllables in each word, and the familiarity of the words used, to improve readability. In addition to reducing the reading level of the text directed to caregivers, pediatric associations should consider stratifying their websites to include simpler educational materials directed to children.

Although the study itself uses 8 numerical and 2 graphical readability indices to better represent the many parameters that are factored into readability, special emphasis should be given toward readability formulas designed for health care materials. This includes the SMOG and FRG, with the SMOG considered as the gold standard by many large institutes such as the National Cancer Institute [37] . When factoring this into account, the average reading level of the PEMs would be closer to an 11th-grade rather than a ninth-grade reading level. Although this study focused on PEMs derived from only Canadian pediatric hospitals and associations, patients retrieve information from a variety of other sources. Health content obtained from common sources, such as Wikipedia and other popular websites containing health information, has also been shown to be written at a reading level far above the seventh grade [38] [39] [40] . These findings suggest that even if pediatric patients use education materials outside of those analyzed in this study, they may still face challenges in identifying information that is at an appropriate reading level.

The difficult word analysis revealed long words were the most common type of difficult word in the PEMs. This is in line with another recent Canadian publication on cancer-related PEMs [34] . Difficult terminology should be replaced with more familiar alternatives whenever possible to improve the ease of understanding. For example, "beneficial" could be replaced with "helpful," and "clinician" could be replaced with "doctor" for simplicity. Although the substitution of terminology is ideal, it may not be applicable to cases in which information must be fully and accurately communicated. In these situations, a clear definition should be included when the word is introduced. Although the content of the PEMs was analyzed for reading level and word difficulty in this study, additional factors such as organization, layout, and design can impact PEMs' readability. Therefore, further studies should be undertaken once additional instruments, such as the Suitability Assessment of Materials and PMOSE/IKIRSCH document readability formula, are validated for medical literature [41, 42] .

The readability tests used in this study consider parameters such as word and sentence length, number of syllables per word, words per sentence, and the difficulty level of words [31] [32] [33] . Although the use of multiple tests allows for more dimensions of readability to be considered, there are still limitations to using readability tests overall. Tests that assess syllable count may overestimate the readability of the text. Monosyllabic medical terminology, such as the word "stent," contributes to a lower readability score, but that may not necessarily reflect a person's ability to understand the terminology [15, 31] . Conversely, tests that assess word familiarity may underestimate the readability of the text. Well-known medical terminology, such as the word "pediatrician," acts to increase readability scores but may not contribute to increased difficulty in understanding the text. Furthermore, the act of defining difficult words within the text, which would greatly improve comprehension, is also not considered to impact readability. Additionally, the readability tests in this study do not account for the formatting of the text or the inclusion of diagrams. Although bullet points were analyzed as both sentences and comma-separated phrases, this does not fully capture the improvement to readability that lists provide. The evaluation of communication tools such as tables and images should be considered for future studies, as they can also serve to improve the ease of understanding [12] . Lastly, interpretations of the results must be taken into context as only Canadian pediatric associations were assessed, with just 3 of the 10 associations having PEMs on their websites. The results are therefore not representative of the totality of the information that caregivers and pediatric patients would be exposed to.

Overall, web-based PEMs developed by Canadian pediatric associations exceed the recommended seventh-grade reading level. Difficult words should be replaced when possible or defined, and educational content directed specifically toward pediatric patients should be included. Additional consideration should be placed on the incorporation of multimedia PEMs [43] . Qualitative studies should be conducted in the future to better understand caregiver and provider information needs, as well as the barriers toward implementing more functionally accessible PEMs on pediatric association websites [44] [45] [46] . Additionally, the quality of the PEMs should be evaluated to determine if the information provided is accurate [47] [48] [49] . Once collected, this data can be used to inform changes that improve the usefulness, quality, and accessibility of pediatric PEMs. As the role of technology in health care increases, it is important that all individuals are able to understand and use reputable resources on the internet.

Quality, trustworthiness, readability, and accuracy of medical information regarding common pediatric emergency medicine-related complaints on the web

Dr Google in the ED: searching for online health information by adult emergency department patients

Patients' use of the internet for medical information

Social media for rapid knowledge dissemination: early experience from the COVID-19 pandemic

Rationing limited healthcare resources in the COVID-19 era and beyond: ethical considerations regarding older adults

Online pediatric information seeking among mothers of young children: results from a qualitative study using focus groups

The relationship between parent health literacy and pediatric emergency department utilization: a systematic review

Low caregiver health literacy is associated with higher pediatric emergency department use and nonurgent visits

Literacy and child health: a systematic review

First results from the survey of adult skills

The Canadian Encyclopedia. Toronto, Canada: Historica Canada

Simply put: a guide for creating easy-to-understand materials

Health literacy in Canada: a healthy understanding. eScholarship University of California

Parental understanding of infant health information: health literacy, numeracy, and the Parental Health Literacy Activities Test (PHLAT)

The readability of pediatric patient education materials on the World Wide Web

Readability of the written study information in pediatric research in France

How readable are Australian paediatric oral health education materials? BMC Oral Health

Mental health literacy among pediatric hospital staff in the United Arab Emirates

Accuracy of internet guidance on pediatric otolaryngology procedures

Comprehensiveness, readability, and reliability of Brazilian websites available for lay people's guidance on adenotonsillectomy

Improving the readability and processability of a pediatric informed consent document: effects on parents' understanding

Readability of pediatric otolaryngology information by children's hospitals and academic institutions

| iss. 1 | e31820 | p

Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites

Readability of pediatric health materials for preventive dental care

Assessment of readability, understandability, and completeness of pediatric hospital medicine discharge instructions

Readability of online information related to pediatric radiation safety From societal websites

Quality of health information on the Internet in pediatric neuro-oncology

A systematic review of readability and comprehension instruments used for print and web-based cancer information

Readability and quality assessment of websites related to microtia and aural atresia

Education attainment level of caregivers versus readability level of written instructions in a pediatric emergency department

Assessment of the readability of online patient education material from major geriatric associations

Assessment of use, specificity, and readability of written clinical informed consent forms for patients with cancer undergoing radiotherapy

Assessing readability of patient education materials: current role in orthopaedics

Health services: a mixed methods assessment of Canadian cancer patient education materials related to the 2019 novel coronavirus

Communication during radiation therapy education sessions: the role of medical jargon and emotional support in clarifying patient confusion

Assessment of Canadian patient education material for oncology pharmaceutics

Education Resources Information Center

Readability of Wikipedia pages on autoimmune Disorders: systematic quantitative assessment

Readability of online health information: implications for health literacy

Readability of online health information: a meta-narrative systematic review

Assessment of online patient education materials from major ophthalmologic associations

Readability assessment of patient education materials from the American Academy of Otolaryngology--Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg

Integrated digital patient education at the bedside for patients with chronic conditions: observational study

Internet use by parents of children with rare conditions: findings from a study on parents' web information needs

| iss. 1 | e31820 | p

Parents' perspectives, information needs and healthcare preferences when consulting for their children with bronchiolitis: a qualitative study

Patient-generated health data in pediatric asthma: exploratory study of providers' information needs. JMIR Pediatr Parent

Comparison of readability of official public health information about COVID-19 on websites of international agencies and the governments of 15 countries

Assessment of the quality, content, and readability of freely available online information for patients regarding diabetic retinopathy

Multilingual analysis of the quality and readability of online health information on the adverse effects of breast cancer treatments

The authors would like to thank and acknowledge Dr. Peter Hoang for his assistance with the difficult word analysis by providing his perspective as a physician. This research was supported by the Canadian Institutes of Health Research (Grant 21R04868) and the University of British Columbia Graduate Fellowships. The funding sources had no role in how the research was conducted or in the preparation of the paper.

None declared. 16.03.2022 . This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.