key: cord-0717813-jn6zofl7
authors: Taylor, Steven; Landry, Caeleigh; Paluszek, Michelle; Fergus, Thomas A.; McKay, Dean; Asmundson, Gordon J.G.
title: Development and Initial Validation of the COVID Stress Scales
date: 2020-05-04
journal: J Anxiety Disord
DOI: 10.1016/j.janxdis.2020.102232
sha: 577ff5b5b67f3367ba8ee8f469983f6270c0c8ee
doc_id: 717813
cord_uid: jn6zofl7

Abstract Research and clinical observations suggest that during times of pandemic many people exhibit stress- or anxiety-related psychopathology that include fear of becoming infected, fear of coming into contact with possibly contaminated objects or surfaces, fear of foreigners who might be carrying infection (i.e., disease-related xenophobia), fear of the socio-economic consequences of the pandemic, compulsive checking and reassurance-seeking regarding possible pandemic-related threats, and traumatic stress symptoms about the pandemic (e.g., nightmares, intrusive thoughts). We developed the 36-item COVID Stress Scales (CSS) to measure these features, as they pertain to COVID-19. The CSS were developed to better understand and assess COVID-19-related psychopathology. The scales were intentionally designed so they could be readily adapted for future pandemics. The CSS were developed and initially validated in population-representative samples from Canada (N = 3,479) and the United States (N = 3,375). A stable 5-factor solution was identified, corresponding to scales assessing COVID-related stress and anxiety symptoms: (1) Danger and contamination fears, (2) fears about economic consequences, (3) xenophobia, (4) compulsive checking, and (5) traumatic stress symptoms about COVID-19. The scales performed well on various indices of reliability and validity. The scales were intercorrelated, providing evidence of a COVID Stress Syndrome. The scales offer promise as tools for better understanding the psychopathology associated with COVID-19 and for identifying people in need of mental health services.

Emerging findings from China suggest that more than 25% of the general population experienced moderate to severe levels of stress-or anxiety-related symptoms in response to COVID-19 (Qiu et al., 2020; Wang et al., 2020) . These findings are similar to those reported during the SARS outbreak (Cheng et al., 2004) and in the 2009 H1N1 pandemic (Rubin et al., 2009 ). Studies of previous epidemics and pandemics show that anxiety, or the lack thereof, is an important driver of behavior (Taylor, 2019) . People with too little anxiety about a viral outbreak are less likely to engage in hygiene behaviors (e.g., handwashing), less likely to adhere to physical distancing mandates, and are less likely to get vaccinated if a vaccine is available (Taylor, 2019) . On the other hand, people with excessive anxiety are more likely to engage in socially disruptive behaviors, such as panic buying and surging unnecessarily into hospitals and clinics when they misinterpret their minor ailments as signs of serious infection (Asmundson & Taylor, 2020a,b) .

Given the role that anxiety plays in shaping behavioral responses to viral outbreaksboth behaviors that can mitigate as well as those that can facilitate the spread of infection-it is critical that public health decision-makers, health officials, and health care providers understand the nature and degree of adverse psychological responses to the current COVID-19 crisis. To date, there has been little empirical attention devoted to this issue; indeed, while several measures of COVID-19-related fears have recently emerged (e.g., Ahorsu et al., 2020) , they tend to be unidimensional (i.e., focused on general fear aspects of and are based on limited psychometric evaluation. There is a pressing need to develop an empiricallysound measure of COVID-19-related stress and anxiety-related symptoms. Research and clinical observations (e.g., Taylor, 2019) suggest that during times of pandemic many people exhibit stress-or anxiety-related psychopathology that include the following: Fear of becoming infected, fear of coming into contact with possibly contaminated objects or surfaces, fear of foreigners who might be carrying infection (i.e., disease-related xenophobia), fear of the socio-economic consequences of the pandemic (e.g., job loss), compulsive checking and reassurance-seeking regarding possible pandemic-related threats, and traumatic stress symptoms about the pandemic (e.g., nightmares, intrusive thoughts). The COVID-19 Stress Scales (CSS) were developed to measure the aforementioned features as well as to better understand and assess COVID-19-related psychopathology. The scales were intentionally designed so they could be readily adapted for future pandemics. In the present study we examined the factor structure, reliability as internal consistency, and convergent and discriminant validity of the CSS. To determine the robustness (replicability) of the findings, results from a Canadian sample were replicated in an American sample.

Data were collected from Canada and the United States using an internet-based selfreport survey delivered in English by Qualtrics, a commercial survey sampling and administration company, between March 21 and April 1, 2020. The data collection protocol was approved by the University of Regina Institutional Research Ethics Board and all respondents consented prior to beginning the survey. Participation was solicited by Qualtrics using sampling of web-panels to meet quotas based on age, sex, ethnicity, socioeconomic status, and geographic region within each country in order to obtain a population representative sample.

Filters were used to eliminate data from careless or incomplete responders. The final sample comprised 6,854 adults (United States: 3, 375; Canada: 3, 479) . Respondents were aged 18-94 years (M=49.8 years, SD=16.2). Almost half (47%) were female and most (52.3%) were employed full-or part-time. A total of 10.1% were on leave or unemployed, 25.0% were retired, 4.3% were homemakers, and 4.4% were students. Most (78.8%) had completed full or partial college, 17.6% had only completed high school or equivalent, and 2.9% did not graduate from high school. Most (68.1%) were Caucasian, with the remainder being Asian (11.5%), African American/Black (9.4%), Latino/Hispanic (6.4%), Native American/Indigenous (1.4%), or other (3.2%).

In addition to questions regarding COVID-19-related psychopathology, the survey comprised measures regarding demographics, current anxiety and depression, and various trait characteristics. Data collection began after the first cases of COVID-19 had been confirmed in both the United States and Canada. In order to assess pre-COVID-19 trait characteristics, we instructed participants to respond to the trait measures (see 2.2.2 Validation Scales) as they would have prior to the COVID-19 outbreak. chain, fears of looting or rioting; 10 items), (5) COVID-19-related checking (e.g., checking news media or social media, seeking reassurance from friends or medical professionals; 7 items), and (6) traumatic stress symptoms related to COVID-19 (e.g., unwanted intrusive thoughts or nightmares relating to COVID-19; 7 items). 1 Items assessing these domains were generated (58 items in all, listed in the appendix of supplementary materials). Five items were culled on rational bases (see supplement), leaving 53 items for analysis. To simplify the instructions, we referred to COVID-19 as "the virus." COVID-19 actually refers to the disease and SARS-CoV-2 is the virus; however, we expected that many respondents would not be aware of this distinction. Based on feedback from pilot testing, respondents readily understood what we were referring to. We assessed a 7-day window because fears about COVID-19 may change over time as the pandemic unfolds. A 7-day window provides the ability to assess these changes while also keeping the window broadly consistent with the other symptom measures included in our assessment battery.

Instructions for the fear-related items (domains 1-4) were as follows: "The following questions ask about various kinds of worries that you might have experienced over the past seven days… about the virus." Items were rated on a 5-point scale ranging from 0=not at all to 4=extremely. We used the term "worries" to assess feared (anticipated) outcomes. The checking and traumatic stress items were rated on a 5-point scale ranging from 0 (never) to 4 (almost always).

Several measures of trait characteristic were used to assess convergent and discriminant validity. These measures were not tied to COVID-19. As mentioned, we asked respondents to respond to these measures as they would have before the J o u r n a l P r e -p r o o f 9 COVID Stress Scales outbreak of COVID-19. We also included a measure to assess a bias toward socially desirable responding.

Questionnaire-4 (PHQ-4; Kroenke, Spitzer, Williams, & Löwe, 2009 ). The PHQ-4 provides a brief 4-item measure of current anxiety and depression using a 4point Likert scale ranging from 0 (not at all) to 3 (nearly every day). Participants were asked to rate their past week anxiety and depression. Factor analysis supported a two-factor structure corresponding to depression and anxiety. The PHQ-4 has demonstrated good reliability and validity in both clinical and non-clinical samples (Kroenke et al., 2009; Löwe et al., 2010) .

Internal consistency for the full scale in the present study was excellent, Cronbach alpha = .90. The SHAI has good reliability and validity in both clinical and non-clinical samples Salkovskis et al., 2002; Wheaton et al., 2010) . Internal consistency for the 14-item subscale in the present study was excellent, alpha = .90. Foa et al., 2002) . The OCI-R measures symptoms characteristic of obsessive-compulsive disorder (OCD). The two subscales of checking (e.g., "I check more things than necessary") and washing (e.g., "I sometimes have to wash or clean myself simply because I feel contaminated") were used in this study, each comprising three items rated on a 5-point Likert scale ranging from 0 (not at all) to 4 (extremely). The OCI-R demonstrates good reliability, good convergent and discriminant validity J o u r n a l P r e -p r o o f 10 COVID Stress Scales (Abramowitz & Deacon, 2006; Foa et al., 2002; Hajcak et al., 2004) . Internal consistency in the present study was good for the subscales, alpha = .86 for both. (XS; Van Zalk, Kerr, van Zalk, & Stattin, 2013) . The XS measures negative attitudes towards immigrants (e.g., "Immigrants increase criminality") using 12 items rated on a 4-point Likert scale ranging from 1 (don't agree at all) to 4 (agree completely). The original measure made reference to Sweden. For the current measure, all references to "Sweden" were changed to "our country" so that items were applicable to all participants. The XS has demonstrated good reliability, high temporal stability, and convergent validity (Van Zalk & Kerr, 2014; Van Zalk et al., 2013) . One item from the Xenophobia Scale (Wilson-Daily, Kemmelmeier, & Prats, 2018), "Immigrants abuse the health system and fill up our emergency rooms," was added to our modified XS, given its potential relevance to the COVID-19 pandemic. The addition of this item improved internal consistency of the xenophobia subscale from alpha = .86 to .90 and, therefore, was retained. (MCSD-SF; Reynolds, 1982) . The MCSD-SF Scale measures the tendency to respond in socially desirable ways using 13 items (e.g., "I sometimes try to get even rather than forgive and forget") with "true" (1) or "false" (0) response options. The 13-item version has demonstrated to have better reliability and fit compared to the original Marlowe-Crowne Scale (Crowne & Marlowe, 1960) ; but, the shortened version is still highly correlated with the original scale (Fischer & Fick, 1993; Reynolds, 1982) . Internal consistency for the full scale in the present study was acceptable, alpha = .72.

(OCI-R;

J o u r n a l P r e -p r o o f 11 COVID Stress Scales

The goal was to construct short, internally consistent scales, which likely would be intercorrelated to form a coherent syndrome. Results were conducted separately for each country to determine whether the findings were robust (i.e., replicable). For each country and for each of the six scales of the CSS, an exploratory factor analysis was conducted using robust Maximum Likelihood (RML) using MPlus (Muthen & Muthen, 2017) . That is, Maximum Likelihood using robust standard errors. RML was used because it is robust to departures from normality in the data distribution. For each scale, parallel analysis (Patil et al., 2017 ) indicated a single factor. That is, each scale had a single underlying factor. For the factor for each scale, the items with the six highest loadings were retained. This resulted in the creation of six 6-item scales. Six items were selected because we had previously found, in studies of other anxietyrelated phenomena, that 6-item subscales provided a good balance between brevity and reliability (Taylor et al., 2007) . Item selection was replicated across countries; that is, for each scale, the items with the top six loadings in the Canadian sample were the same as the items with the top six loadings in the United States sample (see supplement).

Exploratory factor analysis of the resulting pool of 36 items was conducted using the Canadian sample. This was done using RML with oblique (Oblimin) rotation using MPlus.

Parallel analysis was used to determine the number of factors to retain. The robustness (stability) of the multi-factor structure was then investigated by conducting RML confirmatory factor analysis using MPlus on data from the 36 items from the United States sample.

The selection of goodness-of-fit indices was based on conventional guidelines (Hu & Bentler, 1998) . We used the standardized-root-mean-square residual (SRMR), the root-mean-J o u r n a l P r e -p r o o f 12 COVID Stress Scales square error of approximation (RMSEA), and the comparative fit index (CFI). The SRMR was used because it is among the most sensitive to misspecified factor correlations, and the RMSEA is sensitive to misspecified factor loadings (Hu & Bentler, 1998) . To interpret whether a given factor model provided a good fit to the data, we used Hu and Bentler's (1999) empirically derived cut-off values. These values minimize errors in deciding whether a model provides a good fit to the data. Excellent fit is indicated by SRMR ≤ .08, RMSEA ≤ .06, and CFI ≥ .95. Good fit is indicated by CFI ≥ .90.

Reliability as internal consistency was assessed by Cronbach's coefficient alpha.

Correlational analyses were conducted to assess various indices of validity. Given the number of statistical analyses reported in this study, the alpha level for statistical significance was set at 0.01. To interpret the substantive significance of correlations, we used Cohen's (1988) criteria: r = 0.10 (small), 0.30 (moderate), 0.50 (large).

The study was conducted during the early stages of the pandemic in the United States and Canada, in which many people were experiencing emotional distress. Based on the cut-offs 

Parallel analysis indicated a 5-factor solution, rather than a 6-factor solution in which each factor corresponded to each of the six scales of the CSS. That is, two of the subscales loaded on a single factor (see below). The first six eigenvalues were as follows: 15.84, 2.86, Table 1 . The 5-factor solution had an excellent simple structure; that is, each item had a salient loading on only one factor. Each factor corresponded to one of the scales, with the exception that the COVID-related danger and COVID-related contamination scales loaded on a single factor. Rather than reducing this factor to a 6-item scale, it was retained as a 12-item scale so that, if needed in future studies, it would be possible to assess danger separately from contamination.

The 5-factor model, obtained in the exploratory factor analysis from the Canadian sample, was tested in RML confirmatory factor analysis in the United States sample. In the latter sample, the model performed well in terms of the goodness-of-fit indicates: RMSEA = J o u r n a l P r e -p r o o f 14 COVID Stress Scales 0.050 (90% confidence interval: 0.049-0.051), SRMR = 0.042, and CFI = .93. To further test the robustness (replicability) of the factor structure, a 2-group confirmatory factor analysis was conducted in which the 5-factor model was simultaneously fitted to the United States and Canadian samples, in which factor loadings and factor correlations were constrained to be the same for each sample. The models performed well in goodness-of-fit: RMSEA = 0.050 (90% confidence interval: 0.049-0.051), SRMR = 0.053, and CFI = .92. That is, constraining loadings and correlations to be equal across samples yielded a factor model that had a good fit to the data, indicating that the factor structure was replicable across countries. For the two samples, the correlations among factors ranged from .48 to .77 (see supplement for the complex matrix of correlations). The factors loaded on a single higher-order factor. The present study focuses on the psychometric properties of the lower-order factors.

Based on the results of the factor analysis, five scales were constructed to form the CSS.

The complete CSS appears in the supplement. The scales were scored by adding the unitweighted items together. Higher scores indicate greater levels of COVID-19-related stress. Items were unit-weighted instead of being weighted according to factor score coefficients, because unit-weightings are more likely to be reliable (i.e., replicable) in future studies (Cohen, 1990) . Table 2 presents Cronbach alpha coefficients for each scale in each sample. Here, it can be seen that all of the coefficients were > .80, indicating good-to-excellent reliability as internal consistency (Tavakol & Dennick, 2011) . Table 3 presents the correlations among the scales.

Here, it can be seen that all the scales of the CSS were intercorrelated. This suggests, for people with high scores, that symptoms assessed in the CSS form a coherent COVID Stress Syndrome. Table 4 shows the correlations of the scales of CSS with the pre-COVID trait measures of health anxiety and obsessive-compulsive (OC) contamination and checking symptoms. Here, it can be seen that all correlations were significant (p<.001) and almost all were medium-to-large in magnitude. These findings support the convergent validity of the CSS.

Due to the large sample sizes, the correlations between the five scales of the CSS and social desirability were statistically significant for each country (p<.001); but, they were substantively trivial in their absolute values, smaller than Cohen's classification of "small" correlation (i.e., the correlations ranged from -.14 to -.05). This finding indicates that a socially desirable response set was essentially unrelated to scores on the CSS. Table 5 shows the tests of the differences between correlations with general anxiety versus depression for each of the subscales of the CSS. This is a highly stringent and, in some ways, contentious test of discriminant validity, in part because if a given variable leads to anxiety, depression is a common consequence. Accordingly, for many measures of anxietyrelated symptoms (e.g., OC symptoms), it has historically been very difficult to show that they are more strongly correlated with anxiety than depression (e.g., Taylor, 1995) . Nevertheless, Table 5 shows that for most scales of the CSS, the correlations with current anxiety were significantly greater than the correlations with current depression. This finding supports the discriminant validity of the CSS. Table 6 provides further support for discriminant validity. Here, the mean correlations for general distress (i.e., correlations of the CSS with the measures of general anxiety, J o u r n a l P r e -p r o o f 16 COVID Stress Scales depression, and the pre-COVID trait measures of health anxiety and OC symptoms) were compared to the correlations with general, pre-COVID xenophobia as measured using the XS.

These correlations were conducted for each of the scales of the CSS. Table 6 shows that the pattern of correlations supports the discriminant validity of the CSS. That is, the COVID xenophobia scale was more strongly correlated with the general xenophobia scale versus the mean of the distress scales. Table 6 shows that the converse pattern was observed for the other CSS scales. Those scales were more strongly correlated with distress than general xenophobia. In addition, the COVID xenophobia scale, as compared to the other CSS scales, was more strongly correlated with general xenophobia: Canadian sample, Z=26.75, p<.001; United

States sample, Z=23.46, p<.001.

The CSS were developed and initially validated in large, population-representative samples from Canada and the United States. A stable 5-factor solution was identified, corresponding to five scales assessing COVID-19 stress and anxiety symptoms: (1) COVID danger and contamination fears, (2) COVID fears about economic consequences, (3) COVID xenophobia, (4) COVID compulsive checking, and (5) COVID traumatic stress symptoms. The scales performed well on various indices of reliability and validity. The scales were intercorrelated, loading on a single higher-order factor, thereby providing evidence of a COVID-19 Stress Syndrome. The scales offer promise as tools for better understanding the psychopathology associated with COVID-19 and for identifying people in need of mental health services. The CSS can also be used in studies to predict which people are most likely to engage in safety behaviors. For example, the CSS could be used to investigate which people are most J o u r n a l P r e -p r o o f 17 COVID Stress Scales likely to engage in hygiene behaviors, social distancing, and the uptake of a vaccine, when one becomes available.

It is anticipated that when this pandemic passes, significant mental health needs will emerge in the public. These predictions are based on prior pandemics, where anxiety, depression, and traumatic reactions were observed (such as following quarantine due to SARS; Hawryluck, et al., 2004; Taylor, 2019) . Accordingly, the development of a pandemic-specific measure such as the CSS can serve to aid in identifying individuals at risk for adverse emotional reactions both during and post-pandemic. This can then aid public health officials in allocating resources for mental health interventions. The measure can also be further investigated for its predictive utility for returning to functioning post-pandemic. This is also, to our knowledge, the first assessment of pandemic-related emotional responses that includes specific evaluation of xenophobia as a contributory factor in fear and avoidance. This is an important consideration in any preparation for public health officials in addressing emotional and behavioral responses to potential pandemics. The role of xenophobic reactions, in conjunction with other emotional indicators such as those identified in the CSS (i.e., COVID danger and contamination, COVID socioeconomic consequences, COVID traumatic stress, and COVID checking and reassurance seeking) deserves additional investigation, such as with individuals at risk for post-pandemic adverse reactions.

In terms of limitations, the present study did not include structured diagnostic assessments (i.e., DSM-5 or ICD-11 diagnoses), which would have been useful in evaluating criterion-related (known-groups) validity of the CSS. It would be predicted that scores on the CSS would be higher in people with current anxiety-related disorders (e.g., generalized anxiety J o u r n a l P r e -p r o o f 18 COVID Stress Scales disorder, obsessive-compulsive disorder), as compared to nonclinical controls. Future research is needed to investigate this issue. A further limitation was that we were unable to compare the CSS to other COVID-related anxiety measures (e.g., Mertens et al., in press) , because the latter scales had not been published when we were conducting our study. Despite these limitations, the present findings provide encouraging support for the CSS as a brief, multidimensional measure of COVID-related stress and anxiety.

This study is limited by the reliance on an online survey method of evaluation, and selfreport measurement. Additional research involving expanded breadth of content may provide incremental increases in the validity of the CSS. Likewise, additional methods of assessment, such as interviews to evaluate the scope of avoidance and the inclusion of new safety behaviors, may reveal additional indicators of emotional responses to pandemic response.

Notwithstanding these limitations, the introduction of a robust instrument to assess COVOD-19-related stress reactions, developed and evaluated with a large bi-national community sample during the peak period of COVID-19, is of considerable importance. It is expected that the CSS will lead to important new empirical findings on the nature of reactions to COVID-19 in particular and future pandemics in general.

J o u r n a l P r e -p r o o f COVID Stress Scales 

Psychometric properties and construct validity of the Obsessive-Compulsive Inventory-Revised: Replication and extension with a clinical sample

The Short Health Anxiety Inventory: Psychometric properties and construct validity in a non-clinical sample

The Fear of COVID-19 Scale: Development and Initial Validation

Coronaphobia: Fear and the 2019-nCoV outbreak

How health anxiety influences responses to viral outbreaks like COVID-19: What all decision-makers, health authorities, and health care professionals need to know

Psychological distress and negative appraisals in survivors of severe acute respiratory syndrome (SARS)

Statistical power analyses for the behavioral sciences

Things I have learned (so far)

A new scale of social desirability independent of psychopathology

Measuring social desirability: Short forms of the Marlowe-Crowne social desirability scale

The Obsessive-Compulsive Inventory: Development and validation of a short version

Trajectories of resilience and dysfunction following potential trauma: A review and statistical evaluation

Psychometric properties of the OCI-R in a college sample

SARS control and psychological effects of quarantine

Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification

Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives

An ultra-brief screening scale for anxiety and depression: The PHQ-4

A 4-item measure of depression and anxiety: Validation and standardization of the Patient Health Questionnaire-4 (PHQ-4) in the general population

Predictors in an online study

Mplus users guide

Parallel analysis engine to aid in determining number of factors to retain using R

A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: Implications and policy recommendations

Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale

Public perceptions, anxiety, and behaviour change in relation to the swine flu outbreak: Cross sectional telephone survey

The Health Anxiety Inventory: Development and validation of scales for the measurement of health anxiety and hypochondriasis

Making sense of Cronbach's alpha

Assessment of obsessions and compulsions: Reliability, validity, and sensitivity to treatment effects

The psychology of pandemics: Preparing for the next global outbreak of infectious disease

Robust dimensions of anxiety sensitivity: Development and initial validation of the Anxiety Sensitivity Index-3

Developmental trajectories of prejudice and tolerance toward immigrants from early to late adolescence

Xenophobia and tolerance toward immigrants in adolescence: Cross-influence processes within friendships

Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China