key: cord-0755566-qaqn7jo3
authors: Jo, Grace; Habib, Daniel; Varadaraj, Varshini; Smith, Jared; Epstein, Sabrina; Zhu, Jiafeng; Yenokyan, Gayane; Ayers, Kara; Swenor, Bonnielin K.
title: COVID-19 Vaccine Website Accessibility Dashboard
date: 2022-04-12
journal: Disabil Health J
DOI: 10.1016/j.dhjo.2022.101325
sha: 33de41a6bf54f0aeeed88cf66102808e296f2d40
doc_id: 755566
cord_uid: qaqn7jo3

BACKGROUND: The COVID-19 pandemic has exacerbated historical inequities for people with disabilities including barriers in accessing online information and healthcare appointment websites. These barriers were brought to the foreground during the vaccine rollout and registration process. OBJECTIVE: This cross-sectional study aimed to examine accessibility of U.S. state and territory COVID-19 information and registration centralized websites. METHODS: The Johns Hopkins Disability Health Research Center created a COVID-19 Vaccine Dashboard compiling COVID-19 information and vaccine registration web pages from 56 states and territories in the United States (U.S.) reviewed between March 30 through April 5, 2021 and analyzed accessibility using WAVE Web Accessibility Evaluation Tool (WAVE). WAVE identifies website accessibility barriers, including insufficient contrast, alternative text, unlabeled buttons, total number of errors, and error density. Web pages were ranked and grouped into three groups by number of errors, creating comparisons between states on accessibility barriers for people with disabilities. RESULTS: All 56 U.S states and territories had COVID-19 information web pages and 29 states had centralized state vaccine registration web pages. Total errors, error density, and alert data were utilized to generate accessibility scores for each web page, the median score was 259 (range=14 to 536 and IQR=237) for information pages, and 146 (range=10 to 281 and IQR=105) for registration pages. CONCLUSIONS: These results highlight barriers people with disabilities may encounter when accessing information and registering for the COVID-19 vaccine, which underscore inequities in the pandemic response for the disability community and elevate the need to prioritize accessibility of public health information.

utilized to generate accessibility scores for each web page, the median score was 259 (range=14 24 to 536 and IQR=237) for information pages, and 146 (range=10 to 281 and IQR=105) for 25 registration pages. 26 27

These results highlight barriers people with disabilities may encounter when accessing 29 information and registering for the COVID-19 vaccine, which underscore inequities in the 30 pandemic response for the disability community and elevate the need to prioritize accessibility of 31 public health information. 32

The COVID-19 pandemic has disproportionately impacted the disability community, 48 compounding already existing barriers that individuals with disabilities face in accessing 49 healthcare, as well as public health information. [1] [2] [3] [4] [5] These gaps in access create inequities in 50 accessing COVID-19 testing, treatment, and vaccination. 1, 2 People with disabilities are more likely 51

to have underlying conditions that impact the immune system and make them exponentially more 52 susceptible to severe and negative COVID-19 outcomes than the general population. 2,4 This may 53 result in some individuals with disabilities requiring personal care assistance, and the disruption 54 of caregiving and support services due to the need for social distancing may also increase the risk 55 for health complications unrelated to In particular, individuals with disabilities have 56 a multitude of unique obstacles within the health care system including, but not limited to, 57 difficulties with transportation and barriers in accessing information that have been exacerbated 58 by COVID-19 regulations such as the social distancing mandates. 6 People with disabilities are also 59 more likely than people without disabilities to live in nursing homes or congregate care settings 60 which have been associated with much higher risk of COVID-19 exposure 2-4 . In addition, for 61 certain disabilities (such as sensory disabilities), there can be differential societal implications of 62 social measures taken to combat the spread of COVID-19. For example, someone who is Blind 63 cannot tell if the people around them are wearing a mask or maintaining social distancing. For 64 these combinations of reasons, people with disabilities are more at risk and differentially impacted 65 by the pandemic, and therefore prioritized for COVID-19 vaccine allocation. 66

The disability community has historically faced barriers to accessing information. [1] [2] [3] [4] [5] The impact 68 of these gaps in access have been elevated during the pandemic. 1 the District of Columbia, and 5 territories collected by searching the internet for the most up-to-120 date web page Uniform Resource Locator (URL). As the data was publicly available and did not 121 involve human subjects, IRB approval was not required. Web pages were monitored weekly to 122 ensure that each URL was still active and to update the states and territories that created new web 123 pages. The entire website was not scored. Two types of web pages were analyzed: COVID-19 124 information and COVID-19 vaccine registration. COVID-19 information web pages were 125 determined based on containing state/territory-wide guidelines on business regulations, social 126 distancing policies, vaccine eligibility, etc. Our analysis focused on the main COVID-19 state 127 information webpage or landing page. As a result, no additional webpages were studied (only one 128 COVID-19 information webpage URL was analyzed for each U.S. state or territory). A registration 129 web page was determined based on containing a centralized portal for residents to register for 130 COVID-19 vaccine appointments. Our analysis was limited to single state/territory-level vaccine 131 pre-registration or registration government web pages and did not include data from states or 132 territories without a centralized state-level registration vaccine portal. For example, states or 133 territories that simply listed the contact information of each locale's health department or displayed 134 an interactive map of which providers and commercial pharmacies had vaccines available were 135 not captured. In addition, states or territories that listed multiple providers, each offering 136 independent pathways for registration, were not included. These web pages were excluded, despite 137 being centralized (i.e., hosted by state-and territory-wide departments of health), because these 138 web pages only served as intermediary information hubs between the resident and vaccine 139 provider. During data collection, state COVID-19 vaccine websites were split evenly between two 140 primary coders, G.J and D.H. Each person had 4 days to collect the data, after which checks were 141 completed, graphs created, and the data was published in the Johns Hopkins COVID-19 Vaccine 142

Dashboard each Wednesday. Data from reviews conducted between March 30 until April 5, 2021 143 are included in these analyses. The number of detected accessibility errors, the density of those errors on the page (errors divided 156 by number of page elements), and the number of likely/potential accessibility issues ("Alerts" 157 within WAVE) were considered when generating the rankings. Automatically detectable 158 accessibility issues do not represent all accessibility issues, but typically indicate barriers for users 159 with disabilities and non-compliance with accessibility guidelines and best practices. Accessibility 160 scores were generated for each web page by rank ordering the 56 COVID-19 information pages 161 J o u r n a l P r e -p r o o f (and 29 vaccine registration pages) on number of errors, error density, and likely/potential issues. 162

Weightings were then applied to each of the rank orders (ties were treated equally) to calculate an 163 overall access scorewith error ranks being assigned a score of 6, error density ranks being 164 assigned a score of 3, and likely/potential issues assigned a score of 1. This access score is not 165 necessarily a good measure of overall accessibility but provides a normalized metric for 166 comparisons across the web pages. The dashboard presented pages ordered or ranked by the overall 167 access score. 168 169 For information web pages, WAVE was run automatically, while for registration web pages, 170 accessibility scores were calculated by navigating to the first web page of the registration site 171

where user-specific information (e.g., name, age, etc.) was entered. Rather than capturing the entire 172 website, the first relevant information or registration page was considered reflective of all web 173 pages of each site. WAVE was run automatically if the URL was linked to the correct web page 174 of the vaccine portal. However, WAVE was run manually (using a browser extension) on the next 175 page of the registration portal if account creation or CAPTCHA (Completely Automated Public 176

Turing test to tell Computers and Humans Apart) was required on the first page before entering 177 user-specific data for 9 of the 20 COVID-19 vaccine registration web pages (leaving 20 of the 29 178 being scores using automated WAVE methods). In manual assessments, testers captured total 179 errors, error density, and alert datathe same data collected via the automated process for pages 180 that could be directly accessed by WAVE. However, automated WAVE data captured more in-181 depth analyses of the types of errors, such as contrast errors, empty links/buttons, images without 182 alternative text, and unlabeled form inputs. If a state or territory required the creation of a new 183 account to register for vaccination, the account creation page was used to assess accessibility: no 184 new accounts were created by the researchers during this process. Accessibility scores were calculated for 56 U.S. states and territories with COVID-19 information 231 web pages (Table 1) , and the 29 states that had a centralized (i.e., state/territory-wide) COVID-19 232 registration web page as of April 5, 2021 ( Table 2) . Total accessibility scores for each state and 233 territory information and registration web pages were examined (Figures 1a, b, c) . Mexico) exhibited barriers that made manual extraction necessary ( Table 5) The majority of state rankings did not have consistent accessibility rankings across COVID-19 314 vaccine information and registration websites. Only a few states had within-state consistency in 315 these rankings, such as California where the COVID-19 vaccine information website was ranked 316 4 th , and the COVID-19 vaccine registration ranked 2 nd . The reasons for the within-state 317 inconsistency in accessibility ranks are unknown. It is possible there are within-state variations in 318 the allocation of resources and personnel focusing on website accessibility that may contribute to 319 these differences. It is also possible that COVID-19 vaccine information websites may have been 320 created earlier than registration websites, allowing more time for states to improve information 321 website accessibility based on the public's feedback. States without centralized vaccine 322 registration websites also had discrepancies between information and registration rankings, as no 323 ranking was possible for registration websites in these cases. quantifying vaccine rates among people with disabilities to determine inequities. Additionally, the 351 accessibility score/ranking system is primarily designed around people with visual disabilities, and 352

there are barriers to online accessibility for people with other types of disabilities that were not 353 captured. These may include, for example, video captioning for people with hearing loss, or plain 354 language summaries for people with intellectual, executive and cognitive disabilities. Further, 355 some websites collect more information per page (i.e., long format) while other websites collect 356 less information per page but include multiple pages (i.e., wide format). A website with a wide 357 format may have fewer total errors per page but may have a higher percentage of errors with 358 respect to how much information is on a single page. Information web pages were more often in 359 long format, likely accounting for their notably higher accessibility scores compared to those for 360 registration web pages (Tables 3-4) . To mitigate this, the accessibility scores included error 361 densitythe number of errors by web page elements (i.e., how dense errors are within the page 362 content and functionality). Our method measured accessibility scores for the first page of the 363 registration website, which is indicative of patterns in subsequent pages; however, users may 364 abandon the process if significant accessibility barriers are encountered on subsequent pages, and 365 future research should focus on developing a metric for capturing accessibility data across many 366 pages of the same website rather than just the first page. Some COVID-19 vaccine registration 367 web pages required pre-registration, including entering driver's license credentialing, or creating 368 an account. This limitation may have created bias in our sample as we did not gain access to 369 websites requiring pre-registration to avoid entering false information and burdening the system. 370

While WAVE was used in this project, it is only one type of method to evaluate website 371 accessibility. This tool analyzes patterns in web site code and design that align with accessibility 372 compliance failures with a very high level of reliability. However, WAVE, like all automated tools, 373 cannot fully assess all aspects of end user accessibility, and the issues it does analyze are primarily 374 focused on users with visual disabilities. Analyzing plain language or assessing video captioning 375 presence/quality or testing for keyboard accessibility issues would necessitate manual testing that 376 was not conducted as part of this study. Additional aspects of accessibility, such as options for end 377 users to register for a vaccine or get additional supports via phone, were not considered, primarily 378 because web sites are a primary mechanism for accessing such information and functionality. 379

Despite these limitations, the issues detected by WAVE nearly always align with negative impact 380 on users with disabilities and thus provide a useful measure of accessibility. In addition, the 381 COVID-19 vaccine roll-out is rapidly evolving, and some states or territories may have adjusted 382 their information and vaccine registration portals since initial data collection. J o u r n a l P r e -p r o o f 

COVID-19 vaccine prioritisation for people with disabilities

People with Disabilities Are Experiencing Barriers to Covid-19 Vaccination

We Ran Tests on Every State's COVID-19 Vaccine Website -The

Markup

Accessibility and Usability of State

Health Department COVID-19 Vaccine Websites

A review of the accessibility of ACT COVID-19 information portals

COVID-19 Vaccination for People with

Civil Rights Division, Public Access Section

Americans with Disabilities Act : Title III Regulations : Part 36 Nondiscrimination on the 452 Basis of Disability in Public Accommodations and Commercial Facilities

Web content accessibility guidelines (WCAG) overview. Web 455 Accessibility Initiative (WAI)

Evaluating Accessibility of Malaysian Public Universities Websites 458 using AChecker and WAVE

Usability evaluation of academic websites using automated tools

3rd International Conference on User Science and Engineering

Egovernment and web accessibility in South America

First International Conference on eDemocracy & eGovernment (ICEDEG)

Vaccine Prioritization Dashboard . The Johns Hopkins Disability Health Research Center

The WebAIM MillionAn annual accessibility analysis of the top 1,000,000 home pages

Department of Health and Human Services

J o u r n a l P r e -p r o o f