key: cord-0686488-1bz6p70y
authors: Heinrich, Carolyn J.; Camacho, Sayil; Binsted, Kaitlin; Gale, Shadlan
title: An audit test evaluation of state practices for supporting access to and promoting Covid-19 vaccinations
date: 2022-03-12
journal: Soc Sci Med
DOI: 10.1016/j.socscimed.2022.114880
sha: 93433e25054ce53c8d1eb1be818573f7e0840508
doc_id: 686488
cord_uid: 1bz6p70y

A field experiment, using a paired audit testing design with testers of different racial and language profiles, was conducted to document and evaluate individual encounters in inquiring about COVID-19 vaccinations in the U.S. states. Testers communicated with state health department and major vaccination site staff about obtaining the COVID-19 vaccine and assessed the extent to which evidence-informed communication tactics for encouraging take-up were employed. The audit testers included individuals representing Latinx identities, given research showing they face greater hardships in navigating vaccine infrastructure and place less trust in public immunization efforts. Data were collected in phone and electronic communications between mid-June and mid-August of 2021. Empirical analyses confirmed that states vary considerably in how clearly officials communicate vaccination requirements and procedures, and in what they ask of individuals before providing the opportunity to receive the COVID-19 vaccine. The Spanish-speaking tester was more likely to encounter negative or racialized language—primarily implicit in nature—such as calls abruptly ended (vs. attempting to secure language support) and requests for additional identification or personal information before continuing with vaccination registration. Examples of overtly negative or racist encounters included condescending comments about Latinx testers' identification (or perceived undocumented status) and the Spanish-speaking tester's communication in Spanish. Analysis of an index of good practices constructed from the audit data revealed that very few strategies for promoting vaccinations were regularly implemented. In regression analyses, an additional point on the good practices index predicted a 0.133 percentage point increase in the percent of the population receiving the first vaccine dose, suggesting the lack of implementation of these good practices may represent a missed opportunity to increase COVID-19 vaccination rates. We identified exemplars in the communications that the federal government could disseminate to rapidly improve state practices and the accessibility of accurate information on COVID-19 vaccination opportunities.

"Nothing is more expensive than a missed opportunity." H. Jackson Brown, Jr.

The United States is struggling to reach the 70 percent (fully) vaccinated rate that a population needs to control COVID-19 through herd immunity, assuming no clustering of unvaccinated groups that can contribute to disease outbreaks (Shen et al., 2021) . As of the beginning of 2022, there were still states with rates of full vaccination below 50 percent. Among the factors consistently associated with lower intent to vaccinate are race, age (under 60 years), lower education, and political ideology, as well as fears driven by historical repression and harms associated with experimental medical research on communities of color, although disparities by race have been narrowing with time (Salmon et al., 2021; Shen et al., 2021) .

The medical literature provides clear guidance-including strategies for promoting COVID-19 vaccination and communication tactics based on vaccine hesitancy levels-that those in policy and health care roles can employ to encourage vaccination, particularly in communities of color that may be less likely to have information on COVID-19 safety procedures and vaccine access (CDC, 2020; Wood and Shulman, 2021) . Examples include: ensuring accessibility and ease of take-up; creating reminders and incentives through social media and networks; and developing targeted education and persuasive communications. In a randomized control trial (RCT) conducted in a California county, Chang et al. (2021) found positive impacts of public health video messages (e.g., affirming the safety of the vaccines) on vaccination intentions, although negatively framed messages (about the potential health J o u r n a l P r e -p r o o f consequences of not getting vaccinated) decreased 30-day vaccination rates for some subgroups.

While federal government policy explicitly conveys that everyone has a right to the coronavirus vaccine regardless of immigration status, the U.S. states vary considerably in how clearly officials communicate vaccination expectations and requirements, and in what they ask of individuals before administering vaccines (Shen and Orenstein, 2020) . This disparate messaging persists even though the Department of Homeland Security has called it "a moral and public health imperative to ensure that all individuals residing in the United States have access to the vaccine" (DHS, 2021) . As tracked by the Kaiser Family Foundation (2021), the registration process for COVID-19 vaccination differs across (and sometimes within) states, as vaccination sites can impose additional rules that create potential barriers to access, including policies or procedures that may exacerbate racial and ethnic gaps in vaccinations. For example, 26 states have restricted access to COVID-19 vaccinations to people who live and work in the state and require proof of residency with documents such as a driver's license or work identification (ID) . Moreover, only about one-fourth of state websites convey clearly that undocumented immigrants are eligible to be vaccinated or attempt to address fears about negative effects on immigration status, and relatively few states with residency requirements allow undocumented immigrants to obtain driver's licenses or state ID cards.

In this research, we aimed to document the experiences of individuals attempting to access COVID-19 vaccinations in all U.S. states through an audit study, that is, a field experiment that allows researchers to examine actions or behavior in real-world J o u r n a l P r e -p r o o f scenarios. In developing this study, we drew on the concept of "administrative burden," defined as overly burdensome or "onerous" individual encounters with government in policy implementation (Burden et al., 2012) . Examples of administrative burden include time and effort invested to understand how to access a public program or benefit and the costs associated with complying with rules or procedures for accessing the benefit, such as producing required documentation. Although administrative burdens can serve legitimate purposes in administering public programs-e.g., requiring the completion of forms for assessing the veracity of claims on public funds and implementing procedures that facilitate efficient rationing of limited resources-they may also impose excessive costs on those attempting to access public benefits (Herd and Moynihan, 2019) .

Moreover, historically underserved and disadvantaged groups often struggle more in overcoming the learning, compliance, and psychological costs (e.g., stigma) they encounter (Heinrich, 2016; Nisar, 2018) , which may exacerbate racialized inequities in access to public benefits (Ray, Herd, and Moynihan, 2020) . We therefore sought to understand how administrative burdens may contribute to government challenges in increasing COVID-19 vaccination rates, as well as how evidence-informed practices for promoting vaccinations might counter these burdens and boost COVID-19 vaccination rates. As vaccines were widely available at the time of our study, our focus is not on allocative exclusion that might prohibit vaccine access (Olsen, Kyhse-Anderson, and Moynihan, 2020) , but rather on the accessibility, quality and tone with which information about vaccinations was conveyed in interactions with agency staff.

In conducting the correspondence audit study, we created identities for hypothetical individuals-an English-speaking White, an English-speaking Latinx, and a J o u r n a l P r e -p r o o f Spanish-speaking Latinx-to request information from state health departments and a major vaccination site within each state on how to register for and receive the vaccine.

The decision to specify testers representing Latinx identities via names, accent, and English speaking abilities was based on evidence of added hardships they face when navigating the U.S. vaccine infrastructure, and lower trust in public immunization efforts among populations at disproportionate risk for COVID-19 (Salmon et al., 2021) , as well as research suggesting that administrative burdens are embedded in inherently racialized organizations that mediate citizen access to state-provided benefits (Ray, Herd, and Moynihan, 2020) . For example, in a recent audit study analysis of discrimination, Olsen et al. (2020) distinguished three types of discriminatory behavior: unconscious bias in interactions arising from implicit, negative beliefs about others; explicit, "taste-based" discrimination reflecting more hardened, negative attitudes toward others; and statistical discrimination based on assumptions about individuals as members of a group. Although we are not able to objectively differentiate among these three types of discrimination in tester interactions, we collect both quantitative and qualitative data to assess the nature of tester interactions and the accessibility and quality of information provided through them.

The testers called and sent emails to state health departments and major vaccination sites (MVS) in all U.S. states and the District of Columbia (DC). We included a MVS in each state given reports that these sites "often make up their own rules (KFF, 2021), i.e., exercising bureaucratic discretion that the administrative burden literature shows can discriminate with intent (Nisar, 2018; Ray, Herd, and Moynihan, 2020) . We also compiled state website documentation on COVID-19 vaccination registration J o u r n a l P r e -p r o o f processes, requirements, and communications, as well as other publicly available data on state vaccination rates, vaccine registration priorities and requirements, social distancing regulations, proportions of COVID-19 at-risk adults, political ideology, and other state population characteristics.

In the analyses we perform, we first quantify the extent to which we observed the application of recommended strategies and practices for promoting vaccination in interactions with state health departments and MVS. We examine differences in the application or use of these strategies and practices by: (i) state agencies vs. MVS, (ii) racial profile/language of the tester, and (iii) form of contact (phone, email, response form or chat). In addition, we construct an index of "good practices" for promoting COVID-19 vaccination based on existing research and generate index scores from the audit data collected, compiled across the audit testers.

We then use the good practices index to predict state COVID-19 vaccination ratesincluding first dose and series completion by age group-while controlling for state COVID-19 vaccination policies, social distancing regulations, and other state characteristics. We also present insights from qualitative data collected in the audit study that allow for deeper understanding of the individual encounters with state agencies and MVS and how language of communication and perceived immigration status affected access to COVID-19 vaccination information. We conclude with policy recommendations for reducing barriers to COVID-19 vaccinations and supporting our collective progress toward community immunity.

The correspondence audit study employed a paired testing design (Gaddie, 2018) in which the organizations being audited-state health departments (or other state agencies managing COVID-19 vaccination distribution) and a MVS within each state-received both emails and phone calls from testers with different (racial/ethnic) attributes. The three testers (English-speaking White, English-speaking Latinx, and a native language, Spanish-speaking Latinx) followed an Institutional Review Board (IRB)approved script (Appendix A) to request information on how to register for and receive the vaccine. We were limited by study resources to testing in a single gender; however, given that a previous U.S.-based study (Margolius et al., 2021) of COVID information hotline callers reported that 67 percent of callers were females, we went with the gender that is more likely to make these inquiries. Other identifying information that might be requested from the testers in the interactions (age, address, health insurance, employment status, and access to transportation) was established to be the same for each tester (see Appendix A). If asked what type of identification they had available, testers indicated they had a consular ID (a type of ID commonly used among some immigrants). Zip codes for the tester location were randomly selected within each state using a national zip code database and were offered when an agency representative sought to identify a vaccine location for them. The testers asked about ID requirements, if health insurance was required, and what to do when they got to the vaccination site (signaling some concern about the vaccine's potential effects). To minimize burden to the organizations, we asked for this information in a few simple questions that we expected would involve minimal effort to address.

J o u r n a l P r e -p r o o f All phone calls and electronic communications were initiated during typical business hours of 9 am to 5 pm (or within the identified business hours), and the testers made multiple attempts as necessary to establish contact, varying the days and times of the calls and recording the dates of attempts and notes on the efforts. For each attempt, the testers recorded the time they waited on hold, if they left a voice message and requested a call back, the content of automatic responses, and when a phone number was disconnected or any other reason for a failure to ultimately make contact. For electronic communications, the audit testers waited a minimum of two weeks for responses to email and chat attempts or response form requests before recording that no response was received, and copies of interactions (which might span multiple emails) were retained in email accounts or documents (for chat exchanges or response forms). With three testers collecting data from two entities (state agency and MVS) by phone and by email in the 50 states and DC, there were potentially 612 sets of interactions to result from the communication attempts and to evaluated.

Publicly available data on the state agencies managing COVID-19 vaccinations and the MVS in the 50 U.S. states and DC were compiled to facilitate the audit testing, including: the agency name (typically a state health department) and the agency's listed phone number for public inquiries (frequently a COVID information hotline), and email address for COVID-19 vaccine information; this same information was also extracted for the state's MVS. This information was identified in early June 2021, and the testers began contacting state agencies and MVS in mid-June 2021. In the first two weeks of June, the testers conducted inter-rater reliability tests of their execution of the protocol (script) by phone and email with a non-U.S. state site (Puerto Rico) to ensure consistent J o u r n a l P r e -p r o o f approaches in use of the prompts and follow-up questions in the script. Data collection was completed by mid-August 2021.

To capture information on tester interactions with state agencies and MVS, we created a REDCap data collection form that was IRB-approved and completed by the (Wood and Schulman, 2021) , which draws on consumer and survey research and behavioral economics to identify strategies to prioritize in interactions with the public based on vaccine-hesitancy levels.

The total number of tester-organization interactions captured in the database was 457 (of 612 possible), which reflects that some state agencies and vaccination sites could not be reached. In 40 of the 50 states plus DC, at least one tester failed to connect by phone and/or email with a state agency and/or MVS. Of the 155 failures to J o u r n a l P r e -p r o o f complete a connection, only 18 (11.6%) were attempts to communicate with a state health department; the rest were failed connections with MVS, and many of these (51) were breakdowns or non-response in electronic forms of communication. The large share of failed communications with MVS may have in part reflected their more limited infrastructure and some site closings as vaccine demand was waning. It was frequently noted by the testers that auto-responses or re-directions by an auto-attendant led to "dead ends" for information on COVID-19 vaccinations. For example, one tester recorded that she made three attempts at phone contact with an MVS and all went to a voicemail, which offered the auto-response: "COVID vaccine not available."

In addition to the original audit study data collection, we compiled publicly available data on the U.S. states and DC on vaccination rates (CDC, updated daily); vaccine registration priorities and requirements, social distancing regulations, shares of COVID-19 at-risk adults, and health insurance rates (Kaiser Family Foundation); political ideology and voting outcomes; immigration policy (National Conference of State Legislatures), and other state population characteristics. Table 1 presents a summary of these data and the specific measures that were included as control variables in the empirical models predicting state vaccination rates.

A first objective of the analysis was to assess the state-audit tester interactions, particularly the use of recommended strategies and practices for promoting vaccinations. The interactions were examined: by state agencies vs. MVS and by form of contact (phone, email, response form or chat) to evaluate the consistency of the practices and information provided, and by the racial profile/language of the tester to assess possible racial/ethnic discrimination in any differential responses. We performed J o u r n a l P r e -p r o o f cross-tabulations and computed chi-square tests to determine whether the relationships between the interactions and the agency/site, form of communication, and tester profile were statistically significant at α<0.05.

We next used the audit study data on state/MVS responses to tester inquiries, including the use of tactics or strategies for encouraging vaccine use, to construct an index of "good practices" for supporting and promoting vaccinations among state populations. The index consists of 15 audit study data elements, shown in Table 2 along with their means. We performed one-way analysis of variance tests to examine whether good practices for vaccine promotion varied by state agencies vs. MVS, form of communication, and tester language/racial profile.

We then estimated multivariate regression models to assess whether the application of good practices for vaccine access and encouragement (as measured with the index variable) predicted COVID-19 vaccination rates in U.S. states. The outcome variables included: the % of the state population that completed one dose of the COVID-19 vaccine; the % of the state population that completed the full (two dose) vaccine series; the % of those age 65 years and over that completed one dose (and the full vaccine series), and the % of those age 18 years and over that completed one dose (and the full vaccine series). The other explanatory variables, shown in Table 1 , are included to control for other state policies or attributes that might affect vaccination rates, such as vaccination requirements and priorities, regulations encouraging population health, region, and other state characteristics that might reduce (or increase) barriers to vaccination (e.g., paid sick leave, health insurance and immigration enforcement) or vaccination resistance (e.g., political ideology). Standard errors in the J o u r n a l P r e -p r o o f regressions were clustered by state and adjusted for heteroskedasticity (robust clustered standard errors). We interpret the model results as associations, not causal relationships, given there may be other factors that also affect state vaccination rates that we do not observe.

We analyzed the qualitative data captured through REDCap and identified experiences encountered primarily by the Latinx testers (English with a Spanish accent and Spanish speaking). We categorized these experiences of COVID-19 vaccine access as follows: translation services availability, state residency requirements for vaccine access, authorized immigration status for vaccine access, referral to vaccine sites for identification requirements, and negative connotations in relation to immigration status. We counted the frequency of each type of interaction and utilized the qualitative data to assess whether communication regarding language access or immigration status plausibly reflected explicit, taste-based discrimination vs. implicit or statistical discrimination.

In Table 3 , we present the findings of statistical tests for differences in state/MVS responses to the audit testers and their application of recommended strategies and practices for promoting COVID-19 vaccination by: state agencies vs. MVS, the racial profile/language used by the testers, and form of contact (phone vs. electronic communications). Looking across the columns in Table 3 , it is immediately apparent that there were very few differences between state agencies and MVS in their responses and communication practices as documented by the testers. For the three items showing statistically significant differences, health department representatives were less likely to provide information on how to receive the vaccine and were more J o u r n a l P r e -p r o o f likely to respond in ways that were perceived as being racially discriminatory or judgmental.

In contrast to the consistency in responses and practices between state agencies and MVS, the responses and practices as experienced by the audit testers varied substantially by racial profile/language of the testers. As the interactions of the two English-speaking testers (using English and Latinx names) with state agencies and MVS were experienced more similarly than the Latinx tester who communicated in Spanish only, we present the differences by tester language in Table 3 , and then we elaborate on the racialized experiences of the testers below. As expected, for the tester that communicated in Spanish, she was more likely to have translation made available by phone and to be supported in accessing the information in her native language, as communications could not proceed in the absence of translation. Yet in eight different interactions, the state agency or MVS staff abruptly ended the call rather than attempting to secure language support, and in another nine interactions of the Spanishspeaking tester, the call had to be terminated because they could not communicate.

While the English-speaking testers did not require translation to receive the information requested, they also documented cases in which translation was not available, such as when automated message conveyed information only in English and included no options for translation. A number of other statistically significant differences in the experiences of the English-vs. Spanish-speaking testers shown in Table 3 reflect that the Spanish-speaking tester (without access to translation) was subsequently less like to receive information on how and where to receive the COVID-19 vaccination or educational materials about the vaccine.

Even though all three audit testers responded that they possessed a consular ID if probed in the interactions, the Spanish-speaking tester was significantly less likely to have her questions about ID requirements addressed and her concerns dispelled (and in a nonjudgmental manner) than the English-speaking testers. In addition, only the two testers with Latinx identities experienced negative judgments about having a consulate ID. For example, in one phone encounter when the English-speaking Latinx tester indicated that she had a consular ID, the respondent asked (in a condescending tone) if that was "like an alien card." The two Latinx testers recorded five such instances of explicitly racialized interactions about identification (or perceived undocumented status), and the Spanish-speaking tester experienced three additional overtly negative retorts about her communication in Spanish.

This also suggests that the large majority of perceived negative or judgmental exchanges recorded by the testers were implicit in nature. For example, the Englishspeaking Latinx tester noted that a respondent would not register her for a vaccine appointment and provide any information on vaccine locations unless she gave a "full street [home] address," and this was experienced in 15 different interactions. Evidence of state residency was also requested from the testers in more than 20 different interactions in order to receive additional information or to be registered for the vaccine.

For instance, the Spanish-speaking tester was told that the representative could not answer any of her questions about identification requirements if her form of ID did not have a Georgia address, and she could therefore not be registered to receive the vaccine. In another 10 interactions, the Latinx testers were told that they would need to bring a social security number or permanent ID card to the vaccine location. It was also J o u r n a l P r e -p r o o f fairly common for all three of the testers to be referred to specific vaccine locations for the information on ID requirements. The English-speaking tester with the English name noted that one respondent expressed exasperation with the ID question and said: "Just take what you have and see what they say." Through the lens of administrative burden, the additional documentation/ID requirements imposed by state health department or MVS staff contributed to higher learning, compliance and sometimes psychological costs (e.g., stigma associated with not speaking English or not having a typical ID card like a driver's license) of getting vaccinated, with the added burdens falling disproportionately on the Spanish-speaking tester.

Regarding the form of communication, contacting state agencies and MVS by phone (vs. email or other electronic forms) had both advantages and disadvantages. In general, the audit testers encountered more practices oriented toward promoting COVID-19 vaccination in their phone inquiries, such as the use of nudges to make vaccination appointments and offers to send appointment reminders, although educational materials about the vaccine were more often provided electronically.

Translation was more likely to be facilitated by phone, although phone respondents were also more likely to ask about the tester's ID and to indicate that ID and residency were required to get vaccinated (a potentially stigmatizing practice). Communications by chat were less likely to provide information on how to get the vaccination and on where vaccinations could be received.

The summary statistics on good practices for promoting COVID-19 vaccinations indicate a wide range of index scores, from 0 to 13 on a scale with a maximum value of J o u r n a l P r e -p r o o f 15, and a mean and median of 6. The states with the highest average scores (above 7) on the good practices index (in rank order, beginning with the highest) included Indiana, Colorado, California, Tennessee, Wisconsin, Oregon and Kentucky. Looking at Table 2, which indicates the frequency with which particular practices were observed, it is clear that very few of the strategies recommended for promoting vaccinations 5 were implemented in practice. For example, persuasive language and nudges or narratives to encourage vaccination, which are no-or low-cost strategies, were employed by only 5-7 percent of state agency or MVS representatives in the interactions with audit testers. In addition, even though the script (tester characteristics) suggested challenges with transportation (no driver's license), transportation support was only offered in about 6 percent of encounters. This occurred despite the fact that publicly available, web-based information indicated that free rides to vaccination locations were available by Uber or Lyft and other publicly-funded supports.

The analysis of variance test by state agency vs. MVS found no statistically significant differences in the application of good practices by organization type (state agency mean=5.89, MVS mean=6.16, p=0.260). Alternatively, the analysis of variance test by racial profile/language of the testers showed statistically significant differences by the racial profile of the respondent, with the lowest average score on the good practices index experienced by the Latinx audit tester speaking Spanish (Spanishspeaking tester mean=5.08, English-speaking testers=6.43, p=0.000). Regarding the form of communication, inquiries made by phone scored the highest on the good practices index (phone mean=6.33, email mean=5.48, chat mean=5.55, response form mean=5.71, p=0.009). Table 4 presents the results of multivariate regression models predicting state COVID-19 vaccination rates (for completing the first dose and for completing the full series), including estimates for each covariate included in the models. The findings show that in both models, the good practices index was a statistically significant and influential predictor of vaccination rates, and the estimated association was stronger for the first dose. For each additional point on the good practices index, the percent of the population that received the first dose increased by 0.133 percentage points, while the percent of the population that completed the vaccine series increased by 0.085 percentage points (holding all else constant). More restrictive (preventative) state policies for addressing COVID-19 were positively associated with vaccination rates, along with political (2016 election) margins favoring the Democrats. The share of adults ages 18 and older at higher risk for COVID-19 was also positively associated with state vaccination rates, while the share of private sector self-insurance enrollees was negatively associated with state vaccination rates. In addition, states in the West and South had significantly lower predicted vaccine rates than the Northeast. Not only were the predicted relationships (covariate coefficients) in the direction expected based on existing knowledge, but the percentage of state variation in vaccination rates explained in both models was high, about 93% for the model estimating first dose completion and nearly 90% for the model predicting full vaccine series completion. Table 5 

As shown on the REDCap form in Appendix B, the audit testers recorded comments throughout their interactions with state agency and MVS staff, which we compiled and analyzed. We present additional insights from this qualitative analysis, focusing on those that are informative for improving state policies for increasing COVID-19 vaccinations. For example, as discussed above, some state agency staff, including those from Texas, New York, Georgia, Arizona, and Mississippi, would not register the testers for an appointment without a home address, while others indicated that proof of residency was required to obtain a vaccination in the state (e.g., Arkansas, Florida, Maryland, and New Jersey). These stated requirements were not always consistent with state policy documentation posted on websites or communicated through email. In conversation with a Mississippi MVS, a tester was asked to provide her social security number (SSN), but when the tester indicated she did not have one, she was told she J o u r n a l P r e -p r o o f could walk-in and would not need to bring anything. However, the same representative said that she would be asked to provide her name, date of birth, and SSN and to verify her address at the vaccination place.

The audit testers most frequently received inconsistent or imprecise information about state policies on ID requirements. Some were uncertain about how to answer the question, "Do I need to bring an ID with me?"; a common response was to recommend bringing a driver's license, state ID or some other photo ID. A respondent from Illinois indicated: "I believe probably any photo ID is probably fine, just to prove who you are."

Others (e.g., in Maine, Nebraska, Oregon, Virginia, West Virginia, and Wisconsin) made it clear that no ID was required and dispelled concerns about ID requirements. For example, an Oregon state health department staff member specifically stated that identification should not be required and encouraged the tester to report back if any providers said it was required. However, when the audit testers indicated that they did not have a driver's license, and if asked, offered that they had a consular ID, many state agency and MVS staff did not know what a consular ID was and would either direct them to the specific vaccination site (e.g., the nearest Walgreens) or convey uncertainty. Delaware and California were among the few states that explicitly conveyed that a consular ID was an acceptable form of identification. A few other states, including The state health department and MVS staff rarely acted on cues the testers provided by offering educational information on the types of vaccines and their side effects or reassuring the testers that side effects were known to be minimal. As indicated earlier, transportation options were rarely discussed (in only 6% of interactions), even with the cue that the inquirer did not have a driver's license. As noted above, the Spanish-speaking tester frequently experienced additional challenges in accessing information about how to get vaccinated for COVID-19, with some respondents ending the call when the tester began speaking in Spanish or stating "No Spanish" or "I don't speak Spanish" before terminating the call. The quality of translation, when provided, was widely varying, and key information was sometimes not translated by the third-party translator service. There was a lack of language-based resources via every form of communication (phone, email, chat, and response form). In some chat boxes (e.g., Georgia, Indiana, and Maryland), Spanish language was not recognized, which generated error messages.

The audit study we conducted to document the experiences of individuals inquiring about COVID-19 vaccinations in the U.S. drew on existing research (Wood and Schulman, 2021) to evaluate these encounters and the communication tactics used by health departments and MVS for promoting vaccinations. Our research confirmed assertions (Shen and Orenstein, 2020 ) that front-line staff in state health departments and MVS vary considerably (across states) in how clearly they communicate J o u r n a l P r e -p r o o f vaccination requirements and procedures, and in what they ask of individuals attempting to find out how to access the COVID-19 vaccine. We identified some exemplars, such as California, which via both verbal and electronic forms of communication, was more likely to provide educational materials about the vaccine and thorough information about where to go and what to expect at vaccination sites, as well as to dispel concerns about ID and other requirements for vaccine access. While the use of communications practices was fairly consistent between state health departments and MVS, respondents from state health departments were more likely to use (implicitly or explicitly) negative and sometimes racialized language. In addition, although translation was more likely to be facilitated by phone, respondents were also more likely to indicate that ID and state residency were required to get the COVID-19 vaccination.

Analyses of the good practices index constructed from the audit datadocumenting communication tactics employed for encouraging vaccination take-up and the provision of information on how and where to receive the COVID-19 vaccine and on ID, state residency and health insurance requirements-revealed that very few of the strategies and communications tactics recommended for promoting vaccinations were regularly implemented in practice. In addition, we found that the audit tester speaking Spanish was less likely to receive information on how and where to get the vaccine, and she was more likely to encounter negative language in interactions with state health department and MVS staff. More generally, the lack of implementation of these good practices may represent a missed opportunity for states to increase their COVID-19 vaccination rates, given that we found statistically significant associations between the J o u r n a l P r e -p r o o f good practices index scores and state vaccination rates. Although we do not claim to have identified a causal relationship (in the absence of an RCT), we believe that these findings, combined with qualitative analysis, point to important ways in which states may potentially improve their vaccination promotion efforts.

For example, electronic forms of communication were highly varied by state and often very limited in the information they provided in response to basic questions about vaccination access. The federal government might consider disseminating a model for these electronic communications, such as California's chat box option, which could quickly expand the accessibility of rich and accurate information regarding the COVID-19 vaccine and how to get it. Auto-generated emails and chat boxes could also be configured to provide high-quality translation, which would eliminate disparities for nonnative English speakers. Another straightforward recommendation that follows from this study is to provide better training to state agency and MVS staff who interact with the public to support vaccination take-up. This training could ensure that staff have correct and complete information on any requirements for accessing the vaccine in the state, including ID requirements, as well as transportation, education and other supports that might encourage follow-through to obtain the vaccination. In addition, state agencies might go a long way toward reducing burdens for non-native English speakers by establishing phrases in relevant languages to be played or conveyed when an inquiry is made in another language, such as "please hold while I get a translator" (as we observed in DC, Kansas and South Carolina). Lastly, federal agencies such as the Department of Health and Human Services might play a stronger role in disseminating good practices to support vaccination access and take-up and in coordinating outreach J o u r n a l P r e -p r o o f across state agencies, focusing in particular on states with lower vaccination rates and weaker existing practices. This would be consistent with the current administration's "National Strategy for the COVID-19 Response and Pandemic Preparedness" and recent calls for a nationwide "comprehensive, digital, real-time, integrated data infrastructure for public health" that would help to restore public in government and our public health institutions (Emanuel, Osterholm and Gounder, 2021) .

J o u r n a l P r e -p r o o f Source/Notes: SOURCE Authors' original audit study data collection. NOTES One-way analysis of variance tests used to assess statistical significance (at α<0.05) of differences in means. Source/Notes: SOURCE Authors' original audit study data analysis. NOTES Statistically significant coefficients at α<0.05 indicated in boldface. 

The effect of administrative burden on bureaucratic perception of policies: evidence from election administration

Center for Disease Control and Prevention. Covid-19 vaccine equity

Financial incentives and other nudges do not increase COVID-19 vaccinations among the vaccine hesitant

Department of Homeland Security. DHS Statement on equal access to covid-19 vaccines and vaccine distribution sites

A national strategy for the "new normal" of life with covid

Audit studies: behind the scenes with theory, method, and nuance

The bite of administrative burden: a theoretical and empirical investigation

Administrative burden: policymaking by other means

Kaiser Family Foundation. State covid-19 data and policy solutions. KFF

On the front (phone) lines: results of a covid-19hotline

Latest Data on covid-19 vaccinations by race/ethnicity

Children of a lesser god: administrative burden and social equity in citizenstate interactions

The unequal distribution of opportunity: a national audit study of bureaucratic discrimination in primary school access

Racialized burdens: applying racialized organization theory to the administrative state

Reflections on governance, communication, and equity: challenges and opportunities in covid-19 vaccination

Ensuring equitable access to covid-19 vaccines in the us: current system challenges and opportunities

Continued challenges with medicaid coverage of adult vaccines and vaccination services

Beyond politics-promoting covid-19 vaccination in the united states

Three different communication strategies for emails and phone calls:1. Tester with white profile communicates in English. 2. Tester with Latinx profile communicates in English but with a Latin accent. 3. Tester with Latinx profile communicates in Spanish.Audit tester names (typical female, common names for the ethnic profile) and language of communication: J o u r n a l P r e -p r o o f Highlights • Audit study assesses individual encounters inquiring about COVID-19 vaccination • State staff vary in how clearly they convey vaccination requirements, processes • Few evidence-based strategies to promote vaccination regularly used by states • Spanish-speaking audit tester was less likely to receive vaccination information • Implementation of good practices predicts higher state vaccination rates J o u r n a l P r e -p r o o f