key: cord-0735057-rt24q1d4
authors: Han, Hyemin
title: Testing the validity of the modified vaccine attitude question battery across 22 languages with a large-scale international survey dataset: within the context of COVID-19 vaccination
date: 2022-02-01
journal: Human vaccines & immunotherapeutics
DOI: 10.1080/21645515.2021.2024066
sha: b139e45dffecf7d7ffac3bcc306fdf9e8dd0e23f
doc_id: 735057
cord_uid: rt24q1d4

In this study, we tested the validity of the modified version of the Vaccine Attitude Question Battery (VAQB) across 22 different languages. Validity test was conducted with a large-scale international survey dataset, COVIDiSTRESSII Global Survey, collected from 20,601 participants from 62 countries. We employed exploratory and confirmatory factor analysis, measurement invariance test, and measurement alignment for internal validity test. Moreover, we examined correlation between the VAQB score, vaccination intent, compliance with preventive measures, and trust in public health-related agents. The results reported that the modified VAQB, which included five items, showed good validity across 22 languages with measurement alignment. Furthermore, the VAQB score showed negative association with vaccination intent, compliance, and trust as expected. The findings from this study provide additional evidence supporting the validity of the modified VAQB in 22 languages for future large-scale international research on COVID-19 and vaccination.

COVID-19 vaccines have been reported as an effective and reliable mean to prevent COVID-19 caused hospitalization and death across the globe. 1 Although scientific research has consistently supported their safety and effectiveness, negative attitudes toward vaccines have become a serious issue. 2 Because how to deal with such negative attitudes is one of the most fundamental problems that shall be addressed to promote wide distribution of the COVID-19 vaccines, and eventually, to end the COVID-19 pandemic, it is necessary to have a reliable and valid way to examine vaccine attitude. For example, 3 developed and tested the Vaccination Hesitancy Scale (VHS) consisting of ten items; in the study, the VHS was validated in two languages, English and French, among Canadian populations. Since its development, the VHS has been widely employed in studies focusing on people's attitude on vaccination. 4 Although the VHS has been tested in validated in prior research, whether it can be applicable across different languages in a consistent way has not been fully examined. Given the COVID-19 pandemic is a global issue, international and cross-cultural research is strongly required. The majority of the previous studies employing the VHS tested the scale within a single-language study context, so they would not sufficiently support cross-language validity of the scale, which requires for international and cross-cultural comparisons or investigations. 5 Similarly, other scales that have been less employed than the VHS have not been well tested for their validity in multi-lingual settings (e.g., 2 ).

In this study, we aimed at testing the validity of the Vaccine Attitude Question Battery (VAQB), which was modified from the original VAQB employed in a large-scale survey regarding vaccine attitude conducted in 2020, 2 across different languages by examining a large-scale dataset, COVIDiSTRESSII Global Survey (COVIDiSTRESSII), collected 20,601 participants from 62 countries. 6 First, we performed exploratory and confirmatory factor analysis (EFA and CFA) with the English version of the modified VAQB. Second, we conducted measurement invariance (MI) test across 22 languages to examine whether the measurement structure of the modified VAQB was valid across different languages. Third, we performed measurement alignment to address the existing measurement noninvariance. Finally, we examined correlation between the calculated VAQB score and other variables that are expected to be positively associated with positive vaccine attitude, i.e., vaccination intent, compliance with preventive measures, trust in public health-related agents. 7, 8 

COVIDiSTRESSII dataset was collected from 20,601 participants using 48 different languages across the globe (see Supplementary Methods for further details about translation, data collection, and data preprocessing procedures). 6 For MI test and measurement alignment, which involve CFA, only the language versions with N ≥ 200 were analyzed. As a result, responses from 14,271 participants using 22 different languages were used in the present study (see Table S1 for the list of languages and demographics 

The modified VAQB with six items was employed to assess participants' attitude to get COVID-19 vaccines. The six items were extracted and modified from the original VAQB 2 for a feasibility within the context of the large-scale international survey project based on discussions between COVIDiSTRESSII Consortium members. Responses were anchored to a 7-point Likert scale (1 -strongly disagree to 7 -strongly agree). Four items, Items 4 and 5 were reverse coded.

Intent to get COVID-19 vaccines was assessed one item. Responses were anchored to a 5-point Likert scale (1 -not willing at all to 5 -very willing).

We surveyed participants' compliance with non-pharmaceutical measures to prevent spread of COVID-19. These items were adapted from the previous round of COVIDiSTRESS Global Survey. 9,10 We examined compliance in three different behavioral domains, i.e., indoor and outdoor mask use, social distancing. Each type of compliance was measured with one item. Responses were anchored to a 7-point Likert scale (1 -strongly disagree to 7 -strongly agree).

Participants' trust in four different agents, i.e., health system, the World Health Organization (WHO), governmental effort to handle the pandemic, science research, related to vaccination was also assessed. Similar to the items for compliance, the trust items were also adapted from COVIDiSTRESS Global Survey. 9,10 Trust in each agent was surveyed with one item. Responses were anchored to a 11-point Likert scale (0 -no trust to 10 -complete trust).

In this section, we described how the reliability and validity of the modified VAQB were tested. All R source code and data files are available to the public via the Open Science Framework, https://doi.org/10.17605/OSF.IO/QCPZX.

The whole English version dataset was randomly separated into two subsets, one for EFA (50%) and one for CFA (50%), to prevent overfitting. Then, we performed EFA of the modified VAQB with the first subset with R packages, EFA.dimensions and EFAtools. First, we tested whether EFA can be adequately performed with Kaiser-Meyer-Olkin (KMO) and Bartlett's test. Second, we employed diverse measures, i.e., parallel analysis (PA), minimum average partial (MAP) test, hull method, Kaiser-Guttman criterion (KGC), to determine the number of factors to be explored. 11 Third, we performed EFA with the determined factor number. In general, we assumed that factor loadings smaller than .50 as inappropriate. 12 Then, we conducted CFA with the second subset. CFA was performed with the measurement model identified by EFA with lavaan package. Because responses were anchored to a Likert scale, we employed WLSMV estimator, which is suitable for CFA with ordinal responses. CFA was conducted again with the whole data with the model. I examined whether RMSEA and SRMR < .08 and CFI and TLI ≥ .90 at the least. 13 Furthermore, we also investigated whether each factor loading was significant at p < .05. 12 If the aforementioned requirements for EFA and CFA indicators were not fulfilled, we revised the modified VAQB by adjusting items and examined the fit indicators once again with the modified version.

Additionally, we examined the internal consistency of the modified VAQB across different languages in term of Cronbach α. The brief descriptive statistics and internal consistency of the modified VAQB in α for each language version are presented in Table S1 .

MI test was performed to examine whether the measurement model of the modified VAQB validated in the English version can also be validated across different language versions. It was performed by setting the group variable and equal parameter conditions in lavaan.

There are different levels of MI depending on the different equal parameter assumption. First, configural invariance only assumes the equal measurement structure across different groups. Second, metric invariance additionally assumes that factor loadings are equal. Third, scalar invariance additionally requires equal intercepts. Finally, the strictest invariance, scalar invariance, is achieved when the equal residual assumption is additionally satisfied. 14 Whether a specific level of invariance was achieved was examined by comparing fit indicators, i.e., RMSEA, SRMR, CFI, TLI, between two different levels of invariance. In the case of metric invariance, indicator changes should be smaller than −.01 CFI, +.015 RMSEA, and +.30 SRMR. For scalar invariance, changes should be less than −.01 CFI, +.015 RMSEA, and +.15 SRMR. 15 For between-group comparison, scalar invariance must be satisfied at the least. 5

If scalar invariance was not supported in the MI test, we performed measurement alignment to address the existing non-invariance. We employed sirt package to implement measurement alignment in R. 5 Measurement alignment is a procedure to adjust factor loadings, intercepts, and group means to address non-invariance. 5, 16 To examine whether non-invariance was successfully addressed, we checked whether the resultant R 2 loadings and R 2 intercepts , which indicate to what extent the non-invariance was absorbed via measurement alignment, approached 100%. 16, 17 If both R 2 values exceeded 95% as shown in the prior research using measurement alignment, 18 we assumed achievement of scalar invariance through measurement alignment. 16, 17 Then, we calculated an adjusted VAQB factor score for each language group with adjusted factor loadings and intercepts. The factor score was calculated with the pseudo inverse matrix of factor loadings with MASS package. 19

We conducted correlation analysis to acquire additional evidence supporting the convergent validity of the modified VAQB. Correlation between the VAQB score and other surveyed variables, vaccination intent, compliance, and trust, which were supposed to be positively associated with vaccine attitude, 7,8 was examined. We assumed that the convergent validity of the VAQB was supported once significant positive correlation between the vaccine attitude, vaccination intent, compliance, and trust was found. For additional information, correlation between the VAQB items was also investigated.

Both KMO, .82, and Bartlett's test, χ 2 (15) = 1,545.28, p < .001, indicated that EFA was able to be adequately performed with the current scale and first English-version subset. All factor number determination methods, i.e., PA, MAP test, hull method, KGC, unequivocally suggested that only one factor was sufficient in the measurement model. The result of EFA with the one-factor model is presented in Table 1 . Because the factor loading of Item 4 did not exceed the cutoff, .50, we conducted CFA with caution.

When CFA was performed with the second English-version subset, the original measurement model including all six items did not report good model fit indicators, RMSEA = .15, PCLOSE = .00, SRMR = .06, CFI = .75, TLI = .58. Thus, we excluded the fourth item that showed the lowest standardized factor loading (.42) and did not show the satisfactory factor loading from EFA.

Hence, we performed CFA once again with the updated version with five items. The modified five-item scale well fitted the data, RMSEA = .04, PCLOSE = .69, SRMR = .02, CFI = .99, TLI = .98. All factor loading reported p < .001. When CFA was performed with the whole dataset with the five-item model, acceptable model fit indicators were reported as well, RMSEA = .07, PCLOSE = .00, SRMR = .02, CFI = .98, TLI = .96. Similar to the case of the English version, all factor loadings demonstrated p < .001. All resulting standardized factors loadings are presented in Table 1 .

When configural invariance was tested, the resultant model fit indicators suggested mediocre fit, RMSEA = .09, PCLOSE = .00, SRMR = .03, CFI = .93, TLI = .87. With the equal loading assumption, metric invariance was tested. However, the model did not fit data well, RMSEA = .11, PCLOSE = .00, SRMR = .09, CFI = .81, TLI = .78. Additionally, the large changes in RMSEA (+.02), SRMR (+.06), CFI (−.12), and TLI (−.08), suggested metric invariance could not be achieved. Given scalar invariance, which is minimally required for between-group comparison, could not be supported by data, we performed measurement alignment to address the non-invariance across different languages.

The results from measurement alignment demonstrated that measurement non-invariance across different languages was successfully resolved by adjusting factor loadings and intercepts. The resultant R 2 loadings = .97 and R 2 intercepts = .98. They indicate that approximately 100% of existing non-invariance was absorbed by latent factor means and variances varying across languages, and thus, scalar invariance can be achieved via measurement alignment.

Correlation between surveyed variables, the VAQB factor score, general vaccination intent, compliance, and trust, is presented in Table 2 . As expected, vaccine attitude was positively associated with all other variables, so the convergent validity of the modified VAQB was supported. Moreover, all VAQB items showed significant correlation with each other (see Table S2 ).

In this study, we tested the validity of the modified VAQB across 22 different languages. We found that the original measurement model should be modified to achieve good model fit, so one item All correlation reported p < .05 after false discovery rate correction.

was excluded from the original six-item scale. The five-item scale reported acceptable model fit. However, measurement noninvariance was reported from the MI test. When measurement alignment was performed, the existing non-invariance was successfully absorbed; the large resultant R 2 values suggest achievement of scalar invariance through measurement alignment. 18 The correlation analysis demonstrated that the calculated factor score of vaccine attitude after alignment was positively associated with general vaccination intent, compliance with non-pharmaceutical preventive measures, and trust in health organizations, governmental efforts, and science research, in line with previous research. 7, 8 Although the findings suggested the potential utility of the modified VAQB, several limitations warrant further investigations. First, although our dataset is a large-scale international survey dataset, it has been collected via convenient samples (i.e., internet users). Thus, the generalizability of the findings could be limited due to the sample quality and bias issues. Second, we employed self-report measures for vaccine intent and compliance, so predictive validity in terms of whether the measured vaccine attitude can predict actual behavioral outcomes could not be fully supported.

The modified VAQB, which contains five items, can be reliably and validly administrated in 22 different languages with assistance of measurement alignment addressing measurement noninvariance. Because the version with six items did not show good psychometrical quality, we excluded one item and re-tested the version with five items. The correlational analysis also provides additional evidence supporting the convergent validity of the scale. In conclusion, the modified VAQB will be able to be widely utilized in future large-scale international studies regarding COVID-19 vaccination, particularly those involving crosscultural or cross-national comparisons, with measurement alignment.

Progress of the COVID-19 vaccine effort: viruses, vaccines and variants versus efficacy, effectiveness and escape

The sources and correlates of exposure to vaccine-related (mis)information online

The vaccine hesitancy scale: psychometric properties and validation. Vaccine

Current tools available for investigating vaccine hesitancy: a scoping review protocol

A primer to (cross-cultural) multi-group invariance testing possibilities in R

COVIDiSTRESS diverse dataset on psychological and behavioural outcomes one year into the COVID-19 pandemic

Unwillingness to engage in behaviors that protect against COVID-19: the role of conspiracy beliefs, trust, and endorsement of complementary and alternative medicine

Mask usage, social distancing, racial, and gender correlates of COVID-19 vaccine intentions among adults in the US

COVIDiSTRESS Global Survey dataset on psychological and behavioural consequences of the COVID-19 outbreak

Stress and worry in the 2020 coronavirus pandemic: relationships to trust and compliance with preventive measures across 48 countries

Representative sampling of the via assessment suite for adults

Interfirm business-to-business networks: theory, strategy, and behavior

Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives

Measurement invariance conventions and reporting: the state of the art and future directions for psychological research

Replicating the disease framing problem during the 2020 COVID-19 pandemic: a study of stress, worry, trust, and choice under risk

Package "sirt

Multiple-group factor analysis alignment

Exploring the association between compliance with measures to prevent the spread of COVID-19 and big five traits with Bayesian generalized linear model

Understanding and using factor scores: considerations for the applied researcher

The author thanks Sara Vestergren and COVIDiSTRESSII Global Survey Consortium members for their help regarding data collection and preprocessing.

No potential conflict of interest was reported by the authors.

The authors reported there is no funding associated with the work featured in this article.

http://orcid.org/0000-0001-7181-2565