key: cord-0799797-vhgs397z authors: Stelzer, John W.; Wellington, Ian J.; Trudeau, Maxwell T.; Mancini, Michael R.; LeVasseur, Matthew R.; Messina, James C.; Mazzocca, Augustus D. title: Readability assessment of patient educational materials for shoulder arthroplasty from top academic orthopedic institutions date: 2021-09-28 journal: JSES Int DOI: 10.1016/j.jseint.2021.08.004 sha: 1aca8c414a53238ae35c39b196c2851d8b47593b doc_id: 799797 cord_uid: vhgs397z BACKGROUND: Previous studies have demonstrated that online patient educational materials are written at reading levels too advanced for the average patient. The average American reads at the eighth-grade reading level. To date, the readability of online educational material of academic centers for shoulder arthroplasty has not been analyzed. METHODS: Online patient educational materials from the top 25 orthopedic institutions, as ranked by U.S. News & World Report, were assessed utilizing the following readability assessments: Flesch-Kincaid (FK), Flesch Reading Ease, Gunning Fog Index, Coleman-Liau Index, Simple Measure of the Gobbledygook Index, Automated Readability Index, FORCAST, and the New Dale and Chall Readability. All of these scores, with the exception of the Flesch Reading Ease, provide an output indicating reading difficulty based on grade level. Correlations between academic institutional ranking and FK scores were evaluated using a Spearman regression. Lastly, additional factors including geographical location, private versus public institution, and use of concomitant multi-media modalities that may impact institutional readability scores (as determined by FK) were evaluated. RESULTS: Only 16.0% of the top 25 institutions included online material at or below the eighth-grade reading level. Moreover, half of the online resources evaluated (those with FK score ≥9.3) were not at a suitable reading level for more than two-thirds of the general United States population (∼70%). Overall, the composite mean scores were 9.5 ± 2.1 for FK, 52.8 ± 9. for 4 Flesch Reading Ease, 12.2 ± 2.4 for Gunning Fog, 11.6 ± 1.8 for Coleman-Liau, 12.3 ± 1.7 for Simple Measure of the Gobbledygook Index, 9.6 ± 2.6 for Automated Readability, 11.1 ± 0.6 for FORCAST, and 5.9 ± 0.6 for New Dale and Chall. There was no correlation between institutional ranking and FK scores (ρ = −0.15; P = .946). Geographical location, private versus public institution, and use of concomitant multi-media modalities were not significantly associated with readability. CONCLUSION: Shoulder arthroplasty online patient educational material at top-ranked orthopedic institutions have poor readability and are likely not suitable for the majority of patients in the United States. Technological advancements have vastly enhanced dissemination of educational resources to patients. This is especially true during the current COVID era, where patients are more dependent on online educational medical resources than ever before. 1, 6, 10, 21, 22 Although accessing this information may be easy, the true value of online resources is determined by the readers' health literacy and ability to comprehend the content of the resource. This is supported by previous research demonstrating strong associations between low health literacy and increased hospitalization rates, as well as diminished overall outcomes in a multitude of fields. 2, 3, 7, 18, 23 This relationship is of significant importance as the average reading ability in the United States approximates an eighth-grade reading level. 8, 14 Therefore, it is suggested that online educational material should be written at a sixth-grade reading level in an effort to increase health literacy, enhance informed consent, and promote better outcomes. 9 The number of studies investigating the readability of online health educational resources has increased, in an effort to improve the quality of patient care. 15 By assessing readability, clinicians can Institutional review board approval was not required for this survey study. critically evaluate the educational materials readily accessible to patients as well as provide an opportunity to advocate for education material that is more inclusive to all patients. Importantly, academic orthopedic centers are one of the primary contributors to accurate online information for patients. Expectedly, previous studies have analyzed the readability of educational resources from leading orthopedic organizations 4, 5, 11, 13, 16, 20, 24 and orthopedic academic centers for various pathologies. 15, 19 However, to the authors' knowledge, no previous study has investigated the readability of educational materials of academic centers for shoulder arthroplasty. The purpose of this study was to investigate the readability of patient online educational materials specific to shoulder arthroplasty at the top-ranked orthopedic institutions in the United States. It was hypothesized that online educational material for shoulder arthroplasty is above the reading level of the majority of patients in the United States. The U.S. News and World Report of orthopedic specialty rankings were used to identify the top 25 academic orthopedic institutions in the United States in 2021. 17 Online patient educational materials for shoulder arthroplasty, including both total shoulder arthroplasty and reverse total shoulder arthroplasty, were searched and collected for each institution. Institutions were excluded from analysis if no online patient educational material for shoulder arthroplasty was present. The patient education resources were converted into text-only format to exclude figures, disclaimers, acknowledgements, citations, references, and hyperlinks. Reformatted patient education files were then analyzed using Read-ablePro 2021 (Readable, Added Bytes Ltd.; Horsham, UK). Utilizing this software, the following readability test scores were calculated: Flesch-Kincaid (FK), Flesch Reading Ease, Gunning Fog Index, Coleman-Liau Index, Simple Measure of the Gobbledygook Index (SMOG), Automated Readability Index, FORCAST, and the New Dale and Chall Readability. Specific equations for these readability metrics have been detailed in previous studies and are displayed in Table I . 11, 15 All of these scores, with the exception of the Flesch Reading Ease, provide an output indicating reading difficulty based on grade level (eg, score of 8 equates to 8 th -grade reading level). A linear regression analysis was used to generate variance inflation factors, with values ! 10 indicating collinearity between various readability scores. 12 Continuous variables were presented as means and standard deviations. Correlation between academic institutional ranking and FK scores were evaluated using a Spearman regression. Additional factors including geographic location (urban versus rural), private versus public institution, and use of concomitant multi-media modalities (pictures or videos present on institutions website versus no media) that may impact institutional readability scores (as determined by FK) were analyzed with independent t-test and Mann-Whitney tests for parametric and nonparametric continuous variables, respectively. All tests were 2-sided. Analyses were performed with SPSS 27.0 (IBM Corp., Armonk, NY, USA). In total, 23 of top 25 institutions had educational patient resources that could be utilized for analysis. Readability scores were calculated for the top 23 institution's online resources for total and reverse total shoulder arthroplasty. A wide spectrum of FK scores were observed, ranging from 6.6 to 15.4. Notably, only 17.3% of the institutions included in this analysis had online material at or below the eighth-grade reading level ( Figure 1 ). Overall, the composite mean scores were 9.5 ± 2.1 for FK, 52.8 ± 9.4 for Flesch Reading Ease, 12.2 ± 2.4 for Gunning Fog, 11.6 ± 1.8 for Coleman-Liau, 12.3 ± 1.7 for SMOG, 9.6 ± 2.6 for Automated Readability, 11.1 ± 0.6 for FORCAST, and 5.9 ± 0.6 for New Dale and Chall. When assessing multicolinearity, it was determined that all scores (excluding Flesch-Kinaid Reading Ease score) demonstrated a high degree of collinearity relative to the FK score (variance inflation factor for each score: Individual readability scoring metrics for each institution can be appreciated in Table II . Considering the high degree of collinearity between scores, subsequent analyses focused on FK scores. There was no correlation between institutional ranking and FK scores (r ¼ À0.15; P ¼ .946). The following variables did not demonstrate a significant relationship with institutional online resource readability based on FK scores: geographic location (urban: 9.5 versus rural: 8.8, P ¼ .727), private versus public institution (9.4 versus 9.78, P ¼ .758), and use of concomitant multi-media modalities (presence of pictures or videos: 9.4 versus no media: 9.7, P ¼ .741). Beyond this, 17.4% of institutional websites had available educational photos and 36.4% had videos. Focusing on institutional websites with pictures, the average number of photos was 3.0 ± 2.4. Similarly, of the institutional websites with videos, the average number of videos was 1.25 ± 0.46. Upon analyzing our institutions online patient resources, the FK score was 11.3, placing our institution in the bottom quartile in terms of readability. The major finding of this study is that currently only 17.3% of the top orthopedic institutions provided online patient educational materials for shoulder arthroplasty at a suitable reading level (below 8 th -grade reading level) to provide value to patients. Academic orthopedic institutional ranking did not have any correlation with the readability of online patient educational materials (r ¼ À0.15; P ¼ .946). The scores utilized in this study demonstrated a high degree of collinearity, indicating that irrespective of the score being interpreted, educational resources across most institutions were too difficult to read based on estimated grade levels (eg, mean FK of 9.5). Geographic location, private versus public, and use of concomitant multi-media modalities did not appear to have a significant association with readability. Lastly, only a fraction of the top institutions provided pictures or videos as supplemental educational resources on their online websites (17.4 and 36.4%, respectively). These findings are in accordance with other domains of orthopedic surgery, which have also demonstrated poor readability for patient educational materials. 4, 5, 11, 13, 15, 16, 19, 20, 24 In the setting of hip preservationerelated educational materials, Parsa et al identified a statistically weak negative correlation between orthopedic institutional rank (r ¼ À0.21) and readability. 15 Conversely, the current study was not able to confirm these notions. Additionally, institution-related factors including geographic location and private versus public sector were not found to significantly impact readability scores. Instead, disseminated educational resources may be particular to an institutions' individual initiative to provide resources with appropriate levels of readability for their patient population. Surprisingly, only a minority of the institutions provided educational photos (17.4%) or videos (36.4%) for shoulder arthroplasty. Previous research utilizing a novel video assessment tool has demonstrated that when institutions do provide videos, they are usually of high quality in terms of content. 15 This appears to be a major missed opportunity as additional supplemental resources including pictures and videos may be a key secondary mediator to increase health literacy in the setting of poor readability. It appears at this current time, few institutions have fully incorporated all of these aspects to ensure the best health literacy for their patients. In fact, analysis of our own institution revealed an ample amount of pictures (9), but a FK score of 11.3. This would place our institution in the bottom quartile of readability relative to this current study cohort. Some key suggestions to improve readability are as follows: using shorter words with fewer syllables, using fewer words per sentence, and incorporation of more visual supplemental material. Ultimately, websites will derive the greatest benefit by substituting exhaustive explanations for brief descriptions that improve reader comprehension. This ideology seems to be the key to improving readability scores for online patient educational material for total and reverse total shoulder arthroplasty. Anecdotally, institutions with poorer readability scores commonly presented specific and complex information regarding anatomic references, procedural steps, anesthesia types, and implant designs and materials. Conversely, institutions with the best scores simplified these discussions, referencing the humerus as the "upper arm bone," the glenoid as the "socket of the shoulder," and implants as a "metal ball attached to a stem and socket." Additional common and difficult terms can be found in Table III with suggested alternatives. Overall, exchanging medical jargon for oversimplified descriptions and exclusion of technical aspects of the procedure will greatly enhance readability. This study has several noteworthy limitations. First, readability formulas do not fully capture all factors that may also affect the patient's ability to comprehend educational materials such as shorter unfamiliar words (medical jargon) and shorter sentences with increased complexity. Additional studies are required to determine the validity of the patient educational material, while also measuring the patient's comprehension of the material to accurately determine how well patients understand material presented in these formats. Furthermore, readability assessments do not account for educational benefit provided by visual supplemental materials such as pictures and videos. Lastly, this study utilized rankings provided by US News and World Report of orthopedic specialty rankings, which may not be the most accurate ranking system. Moreover, the educational material for shoulder arthroplasty from these institutions may not be representative of those across all orthopedic institutions. This analysis has demonstrated that online patient educational materials for shoulder arthroplasty provided by the nation's top orthopedic institutions, as well as our own, have poor readability. It is important to acknowledge this opportunity for improvement and strive to produce patient education materials with lower grade level readability standards, preferably at the eighth-grade level or below. This will ensure a higher degree of health literacy and ultimately lead to improved patient outcomes. Finally, this analysis revealed only a minority of institutions included supplemental educational pictures and videos. Future online patient educational materials should strive to include a greater number of supplemental materials as a secondary method to improve health literacy. Funding: The University of Connecticut Health Center/UConn Musculoskeletal Institute has received direct funding and material support from Arthrex Inc. (Naples, FL, USA). The company had no influence on study design, data collection, or interpretation of the results or the final manuscript. Conflict of Interest: Dr Mazzocca reports research grants and is a consultant for Arthrex, Inc. The other authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article. Public health communication in time of crisis: readability of on-Line COVID-19 information Low health literacy and health outcomes: an updated systematic review Health literacy interventions and outcomes: an updated systematic review Most American Academy of Orthopaedic Surgeons' online patient education material exceeds average patient reading level Quality of online pediatric orthopaedic education materials Communications in the time of a pandemic: the readability of documents for public consumption Association between functional health literacy and postoperative recovery, health care contacts, and health-related quality of life among patients undergoing day surgery: secondary analysis of a randomized clinical trial How to write easy to read health materials Health literacy: A Prescription to End Confusion Quality of online information for the general public on COVID-19 Poor readability of AOSSM patient education resources and opportunities for improvement Evaluation of variance inflation factors in regression models using latent variable modeling methods Readability of sports injury and prevention patient education materials from the American Academy of Orthopaedic Surgeons website Basic reading skills and the literacy of the America's least literate adults: results from the 2003 National Assessment of Adult Literacy (NAAL) supplemental studies Evaluation of hip preservation-related patient education materials from leading orthopaedic academic centers in the United States and description of a novel video assessment tool Readability of online patient education materials on adult reconstruction web sites Best hospitals for orthopedics Effects of health literacy on treatment outcome and satisfaction in patients with mallet finger injury Readability of Spine-related patient education materials from leading orthopedic academic centers. Spine (Phila Pa Readability assessment of American Shoulder and Elbow Surgeons patient brochures with suggestions for improvement Adapting to a new reality: COVID-19 coronavirus and online education in the health professions Readability of online patient education material for the novel coronavirus disease (COVID-19): a cross-sectional health literacy study Association of health literacy with postoperative outcomes in patients undergoing major abdominal surgery Readability of patient education materials from the AAHS