key: cord-0924259-fe3gkqoh authors: Korhonen, Joonas; Axelin, Anna; Katajisto, Jouko; Lahti, Mari title: Construct validity and internal consistency of the revised Mental Health Literacy Scale in South African and Zambian contexts date: 2021-11-25 journal: Nurs Open DOI: 10.1002/nop2.1132 sha: 1406d0e591e5490fda9b8af56c47cbfc5ed1d86d doc_id: 924259 cord_uid: fe3gkqoh AIM: The aim of this study was to evaluate the construct validity and internal consistency of the revised Mental Health Literacy Scale (MHLS) in South Africa (SA) and Zambia. DESIGN: This cross‐sectional study was conducted between October 2018 and December 2019. METHODS: The study population comprised PHC workers (n = 454) in five districts in SA and Zambia. Principal component analysis (PCA) was used to explore the construct validity, and Cronbach's alpha was applied to measure the internal consistency of the MHLS. RESULTS: Cronbach's alpha values for three attributes were below the appropriate level, but the value was strong (0.804) for the whole scale. The study found nine components explaining ~59% of the total variance of variables. All MHLS items loaded to main attributes based on the theory of MHL. The results stated that the revised version of the MHLS is a construct valid instrument with strong internal consistency. of health literacy relates to worse health status and greater needs for health services (Sørensen et al., 2015) . Likewise, poor MHL, as public knowledge and understanding of factors of mental disorders, has been one of the major challenges in developing mental health services (Atilola, 2016; Jorm, 2000) . Nevertheless, previous studies have stated that more relevant, context-related and empirically tested research measuring MHL should be conducted (Brooks et al., 2011; Jack et al., 2014; . In sub-Saharan Africa, primary healthcare (PHC) workers play a significant role in delivering and promoting mental health care in their respective societies (Atilola, 2016; Munakampe, 2020; Mwape et al., 2012) , but they lack knowledge on mental health-related issues and hold negative stereotypes towards mentally sick persons (Kapungwe et al., 2011; Mwape et al., 2010) . These negative attitudes, knowledge, beliefs and recognition of mental health issues (Jorm et al., 1997; O'Connor et al., 2014) may also reflect PHC workers' poor MHL (Atilola, 2016) . This highlights an obvious need for training of PHC workers of low-and middle-income countries (LMICs) on MHL (Alburquerque-Sendín et al. 2018; Ganasen et al., 2008; Kapungwe et al., 2011; Kutcher et al., 2017 Kutcher et al., , 2019 . Training potentially strengthens health professionals' skills related to MHL and improves the quality of care in LMICs (Atilola, 2016; Kutcher et al., 2017) . PHC workers' outcomes of training on the concept of MHL cannot be verified without psychometrically tested, valid research tools (Kutcher, Wei, & Coniglio, 2016) . Traditionally MHL researchers have faced difficulties in assessing MHL as a whole concept in terms of recognition, knowledge and attitudes towards mental health; rather, they measured parts of it and its different core components of knowledge and beliefs about mental health with separate vignettes (Jorm et al., 1997; O'Connor & Casey, 2015; O'Connor et al., 2014) . Particularly psychometrically robust instruments measuring MHL constructively are lacking (O'Connor & Casey, 2015; O'Connor et al., 2014) . Limited use of scale-based, valid and reliable instruments in MHL-related studies may also lead to greater limitations in study results (Kutcher, Wei, & Coniglio, 2016) . The aim of this study was to clarify the psychometrics of the Mental Health Literacy Scale (MHLS) (O'Connor & Casey, 2015) in South Africa (SA) and Zambia. Differing from earlier MHL measures, the MHLS has been acknowledged as a psychometrically and conceptually reliable instrument with excellent psychometric properties regarding internal consistency, content and structural validity (O'Connor & Casey, 2015; Wei et al., 2016) , and internal and test-retest reliability (O'Connor & Casey, 2015) . The MHLS, which includes all attributes of MHL, was recently revised and content-validated in SA and Zambia using a heterogeneous expert panel with PHC workers and workers and professional research experts . The study revealed the MHLS to have sufficient content validity also in LMICs' context. Only minor changes were made to the original MHLS by O'Connor and Casey (2015) . The MHLS instrument requires further validation to be used in this new context to explore MHL of PHC workers from a cultural perspective and understand their training needs (Atilola, 2016; Rathod et al., 2017) in SA and Zambia. In this study, we evaluate the psychometric properties (DeVon et al., 2007) of the content-validated version of the MHLS in SA and Zambia. The research questions are as follows: (1) What is the construct validity of the MHLS in SA and Zambia? (2) What is the internal consistency of the MHLS in SA and Zambia? This cross-sectional study (Polit & Beck, 2018) was conducted between October 2018 and December 2019. The study is part of a larger European Union-funded project, "MEGA-Building capacity by implementing mhGAP mobile intervention in SADC countries" (funding number 585827-EPP-1-2017-1-FI-EPPKA2-CBHE-JP). In the MEGA project, PHC workers are trained for screening youths' and adolescents' mental health problems using a new mobile application . The Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) Study Design checklist 2019 version Prinsen et al., 2018; Terwee et al., 2018) was used to guide the methodological quality of the study. The COSMIN follows "the worst score counts" principle so that only the lowest rating on each domain is reported. In this study, four out of ten domains of the checklist were applicable in terms of construct validity and internal consistency. Three out of ten domains (content validity, structural validity and internal consistency) were determined to be "very good." The fourth domain, "general recommendation for the design of a study on measurement properties," was rated as adequate. The evaluation of content validation was based on our previous findings PHC workers for this study were selected and invited to participate from the MEGA project research field in the five following regions: Free State, Gauteng and Western Cape Provinces of SA, and Lusaka and Central Provinces in Lusaka, Zambia. The aim was to obtain a culturally diverse sample to identify construct validity and reliable of the revised MHLS in sub-Saharan Africa. Following the MEGA project protocol , participating PHC workers who meet and screen youths and adolescents for mental health issues in their daily clinical practice were targeted. Inclusion criteria were as follows: (a) registered or enrolled nurses or clinical officers working in PHC in the three provinces of SA and two provinces in Lusaka, Zambia, who (b) were able to speak and read English. PHC practitioners or clinical officers who were retiring during the course of the project (2017-2020) were excluded from the study. At the beginning of data collection, adequate and comprehensive oral and written information with full disclosure was given to PHC workers in the local clinics to obtain informed consent (Polit & Beck, 2018; World Medical Association, 2013) . Participants had the power to voluntarily consent to or refuse participation. Participants' rights were stated and discussed before the data collection. After the informed consent procedure was carried out, the participants were asked to fill in the revised version of the MHLS with a background questionnaire. Collected data were personally handed to the PIs as original copies and in SPSS data form. In one case, the original copies from the participating university were not able to be handed out to the PIs due to COVID-19 travel restrictions. Table 1 ). Theoretically, the concept is divided into six attributes of recognition, knowledge and attitudes relating to mental health. The MHLS consists of 35 items. The total score of the MHLS is produced by summing all items together (minimum score 35, maximum score 160). Questions 1-15 are answered on a 4-point scale ranging from 1 (very unlikely/unhelpful) to 4 (very likely/helpful), and Questions 16-35 are answered on a 5-point scale ranging from 1 (strongly disagree/ definitely unwilling) to 5 (strongly agree/definitely willing). The instrument also includes the following reverse-scored items: Q10, Q12, Q15 and Q20-Q28. No cut-off points have yet been introduced for the appropriate level of scoring. The data were analysed using statistical analyses with SPSS26.0. The characteristics of the sample were reported using descriptive statistics. Principal component analysis (PCA; Grove et al., 2013; Maćkiewicz & Ratajczak, 1993; Mishra et al., 2017) was used to explore the construct validity, and Cronbach's alpha (Polit & Beck, 2018; Waltz, Strickland, & Lenz, 2016) This study was conducted according to the basic principles and A convenience sample of PHC practitioners (n = 505) was recruited by the MEGA project researchers, of which n = 454 were included in the final study. After cleaning the data, n = 343 complete answers were obtained with the MHL survey. The majority of these participants were Zambian (53%, n = 181) and female (74%, n = 251). Diploma level was presented regarding the background of professional education in 66% (n = 219) of cases. Figure 1 . Loadings of PCs aligned with the theoretical background of the MHLS (Table 1) Table 3 . Internal consistency reliability was explored by determining Cronbach's alpha for the whole scale and six attributes ( and "Attitudes that promote recognition or appropriate help seeking behaviour (stigma)" (Q20-Q35)-met the appropriate level for the alpha coefficient (≥0.70). These questions represented seven out of nine previously formulated PCs. Three of six attributes, representing the majority of knowledge-related questions (Q9-Q15), fell below the appropriate Cronbach's alpha level. However, Cronbach's alpha for the whole scale was 0.804, which can be seen as a strong indicator of internal consistency reliability. All coefficient correlations for the different attributes and the whole scale are presented in Table 4 . can range between 0.6 and 0.69. Moreover, the MHLS consists of 35 items, and scales with ≥20 items usually score better in terms of internal consistency (Gray et al., 2017) . Alpha's adequacy as an indicator of internal consistency reliability has been critically discussed by several authors (Sijtsma, 2009; Trizano-Hermosilla & Alvarado, 2016; Waltz et al., 2016) , and the value itself cannot been seen as a symbol of hetero-or homogeneity of test items (Sijitsma, 2009; Tavakol & Dennick, 2011) . Our findings indicate that all items of the MHLS fit together in terms of internal consistency (DeVon et al., 2007) and together with proper factor loadings support the instrument's construct validity (Tavakol & Dennick, 2011 (Gray et al., 2017; Waltz et al., 2016) , as Cronbach's alpha is traditionally applied to examine items indicating the same latent trait on the scale (tau equivalency), and diverse items may violate that assumption (Tavakol & Dennick, 2011) . Nevertheless, our findings indicate that for better reliability, avoiding use of knowledge "traps" such as reverse scoring and multiple scales in development of instruments measuring multidimensional concepts may prevent inflation of alpha values. Moreover, our findings strengthen previous findings and suggestions to determine and report alpha values for different subconcepts separately (Gray et al., 2017; Tavakol & Dennick, 2011) . This study has methodological strengths and limitations, which should be considered when interpreting the results. First, the psy- Even including multiple attributes, the revised version of the MHLS has been shown to be a construct valid and internally consistent scale when measuring MHL as a larger concept. Considering the mentioned methodological limitations, our findings reveal that the revised version of the MHLS is a convenient instrument for studying African PHC workers' recognition, knowledge and attitudes related to mental health issues. By understanding MHL among PHC workers, context-and cultural-specific, quality education and training can be provided for planning and implementation of proper care for people suffering from mental health disorders in sub-Saharan Africa. This study was supported by MEGA project consortium. This study was supported by [Erasmus+ Capacity Building] under grant [58582 7-EPP-1-2017-1-FI-EPPKA2-CBHE-JP]. This publication reflects the views of the authors. The commission cannot be held responsible for any use of the information contained herein. Authors have no conflicts of interest. J Korhonen (JK) conceived the study and the manuscript. ML and JK were responsible for the study design. MEGA consortium, JK and ML were responsible for collecting the data. ML and AA contributed to the manuscript by commenting and modifying. J Katajisto led the statistical analysis of data and drafted the manuscript. ML and AA supervised the data analysis and interpretation. The MEGA research team provided overall guidance. All authors read and approved the final manuscript. Research data are not shared. https://orcid.org/0000-0002-5324-1727 The experience of being a psychiatric nurse in South Africa: A qualitative systematic review Ethical principles of psychologists and code of conduct Mental health service utilization in sub-Saharan Africa: Is public mental health literacy the problem? Setting the perspectives right. Global Health Promotion Global Health Promotion Innovation in mental health services: What are the key components of success? Implementation Science, 6, 120 A psychometric toolbox for testing validity and reliability Mental health literacy: Focus on developing countries Burns and Grove's the practice of nursing research: Appraisal, synthesis, and generation of evidence The practice of nursing research: Appraisal, synthesis, and generation of evidence Challenges and opportunities for implementing integrated mental health care: A district level situation analysis from five low-and middle-income countries The ICN code of ethics for nurses. International Council of Nursing Closing the mental health treatment gap in South Africa: A review of costs and costeffectiveness Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: A systematic analysis for the global burden of disease study 2017 Mental health literacy. Public knowledge and beliefs about mental disorders Mental health literacy": A survey of the public's ability to recognise mental disorders and their beliefs about the effectiveness of treatment Attitudes of primary health care providers toward people with mental illness: Evidence from two districts in Zambia Content validation of Mental Health Literacy Scale (MHLS) for primary health care workers in South Africa and Zambia ─ A heterogeneous expert panel method Creating evidence-based youth mental health policy in sub-Saharan Africa: A description of the integrated approach to addressing the issue of youth depression in Malawi and Tanzania Mental health literacy: Past, present, and future Addressing adolescent depression in Tanzania: Positive primary care workforce outcomes using a training cascade model A school mental health literacy curriculum resource training approach: Effects on Tanzanian teachers' mental health knowledge, stigma and help-seeking efficacy Design and development process of a youth depression screening m-health application for primary health care workers in South Africa and Zambia: An overview of the MEGA project Principal components analysis (PCA) Multivariate statistical data analysis-Principal component analysis (PCA) COSMIN risk of bias checklist for systematic reviews of patient-reported outcome measures Strengthening mental health systems in Zambia Strengthening the health system to enhance mental health in Zambia: A policy brief Integrating mental health into primary health care in Zambia: A care provider's perspective The mental health literacy scale (MHLS): A new scale-based measure of mental health literacy Measuring mental health literacy-A review of scale-based measures Mental health in low-and middle-income countries Essentials of nursing research: Appraising evidence for nursing practice COSMIN guideline for systematic reviews of patient-reported outcome measures Mental health service provision in low-and middle-income countries On the use, the misuse, and the very limited usefulness of Cronbach's alpha Health literacy in Europe: Comparative results of the European health literacy survey (HLS-EU) Lifetime prevalence of psychiatric disorders in South Africa Making sense of Cronbach's alpha COSMIN methodology for evaluating the content validity of patient-reported outcome measures: A delphi study Best alternatives to Cronbach's alpha reliability in realistic conditions: Congeneric and asymmetrical measurements Sustainable development. Ensure healthy lives and promote wellbeing for all at all ages. Retrieved from February 11 Measurement in nursing and health research Measurement properties of tools measuring mental health knowledge: A systematic review Depression and other common mental disorders: Global health estimates World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects Construct validity and internal consistency of the revised Mental Health Literacy Scale in South African and Zambian contexts