key: cord-0020883-6jkqa9qb
authors: Lee, Eun Sol; Ryu, Vin; Lee, Ji Hyun; Hong, Hyeon; Han, Hyeree; Park, Subin
title: Psychometric Properties of the Korean Version of the Mental Health Professionals Stress Scale
date: 2021-08-30
journal: Front Psychiatry
DOI: 10.3389/fpsyt.2021.685423
sha: 9b61eca719226341099f9a1539bed2dce9078c75
doc_id: 20883
cord_uid: 6jkqa9qb

Background: Job stress of mental health professionals can have a negative impact on them, particularly their psychological health and mortality, and may also affect organizations' and institutions' ability to provide quality mental health services to patients. Aim: This study aimed to: (1) investigate the validity and reliability of the Korean Mental Health Professionals Stress Scale (K-MHPSS), (2) develop K-MHPSS cut-off points to measure clinical depression and anxiety, and (3) examine whether specific stressors vary by area of expertise. Methodology: Data were collected via an online survey over 3 months, from August to October 2020. An online survey using a survey website was administered to volunteers who accessed the link and consented to participate. Data from 558 participants (200 clinical psychologists, 157 nurses, and 201 social workers) were included in the final analysis. Confirmatory and exploratory factor analyses were conducted to examine the factor structure of the K-MHPSS; concurrent validity of the scale was determined by analyzing correlation; internal consistency was determined by Cronbach's alpha coefficient. In addition, ROC curve analysis and Youden's index were used to estimate optimal cut-off points for K-MHPSS; one-way ANOVA was performed to investigate the difference among the three groups. Results: The seven-factor model of the original scale did not be replicated by Korean mental health professionals. The K-MHPSS had the best fit with the six-factor model, which consists of 34 items. Concurrent validity was confirmed, and overall reliability was found to be good. The K-MHPSS cut-off points for depression and anxiety appeared to slightly different by professional groups. Furthermore, nurses and social workers showed significantly higher total scores compared to clinical psychologists, and there are significant differences in subscale scores among professionals. Conclusion: The Korean version of the MHPSS has appropriate psychometric properties and can be used to assess the occupational stress of mental health professionals. It can also serve as a reference point for screening clinical level of depression and anxiety in mental health professionals.

Mental health professionals' job stress and mental suffering are occasionally underestimated because they have more psychiatric information and knowledge than the general public. In addition, sometimes, mental health professionals themselves demonstrate a negative attitude toward mental disorders (1) or hesitate to seek help for their problems (2) . However, given the nature of their jobs, mental health professionals often attend to patients and clients with psychiatric problems for prolonged periods, which consequently exposes them to emotional exhaustion and mental stress (3) .

Previous studies have shown that job stress has detrimental effects on mental health professionals' mental health (4) (5) (6) . For instance, job stress reported by nursing professionals working in mental health services was significantly related to depression (5) . Further, association between job stress and burnout was observed among mental health professionals (7) (8) (9) . Burnout is characterized as a syndrome of emotional exhaustion and cynical attitude toward clients, which is often experienced by humanservice workers (10) . A meta-analysis study estimated that 40% of mental health professionals experience emotional exhaustion, a key dimension of burnout (11) . Moreover, a recent study showed that the effect of job stress on psychological problems was mediated by burnout (6) . Specifically, nurses' job stress can cause mental health problems, such as depression and anxiety, through the mediation of emotional exhaustion. Compared to nurses in other specialties, psychiatric nurses not only experience emotional exhaustion more frequently but also show a markedly higher risk of suicide (12) .

Stressors experienced by mental health professionals are also known to impact their job satisfaction, absenteeism, job performance, and turnover (13) (14) (15) (16) . A study of community mental health professionals in Korea also revealed that job stress influences turnover intention (17, 18) . Further, poor work attitudes such as turnover intention, turnover, and absenteeism resulting from stress deteriorate the quality of patient and client service, while adding to organization and institution losses by increasing the cost and effort to train and foster new personnel (19) . Given previous findings, job stress of mental health professionals can affect individuals, particularly their psychological health and mortality, and may also affect the organizations' and institutions' ability to provide quality mental health services to patients. Consequently, identifying mental health professionals' job stress and providing effective interventions is crucial.

Among professionals who provide mental health services, the key stressors differ across areas of expertise. Past studies have revealed the stressors specific to each mental health discipline. For clinical psychologists, the major stressors are workload, lack of resources, poor management quality, conflicting roles and relationships with other professionals, lack of work-life balance, and treating clients with chronic and complex mental health problems (20) (21) (22) . One study reported that professional selfdoubt is the greatest stressor for clinical psychologists (23) ; that is, doubts about the effectiveness of therapy they provide and perceived limitations in their competence can cause stress.

Moreover, since therapists must immerse themselves in client sessions and engage in an emotional relationship with clients to provide effective therapy, it is a fundamentally stressful and emotionally burdening activity (23) . Since providing therapy is the main work description of clinical psychologists, factors related to therapy are anticipated to be greater stressors for clinical psychologists than for professionals in other fields.

Stressors experienced by mental health nurses include lack of resources, workload, client-related difficulties, and organizational structures and processes (24) (25) (26) . A qualitative study on mental health nurses presented detailed descriptions of these stressors (27) . Nurses are put in charge of clinical work, management, and administrative work because of understaffing; are exposed to patients' violent and aggressive behaviors when organizations fail to establish appropriate measures; and lack support and recognition from management for their positive practice. They experience all these while being placed in a critical work environment with no leniency for errors, thus elevating their stress. Such stressors seem to have a larger impact on nurses than other mental health professionals because they work in an environment that places them in close contact with patients for prolonged periods (28) .

Mental health social workers experience stress from lack of resources and staffing, relationships and conflicts with other professionals, workload, and organizational structures and processes (29, 30) . In addition, being undervalued at work, having limited latitude in decision-making, and ambiguous roles as social workers in mental health service are also stressors for this group (31) . Social workers play a wide-ranging and comprehensive role in providing mental health services, from direct client care to management; thus, role conflicts and ambiguity undermine their job satisfaction (32) .

In addition to mental health professionals' stressors differing by job characteristics, their work is distinguished from other jobs, including general physical health workers' jobs; therefore, general job stress scales or job stress scales developed for healthcare providers cannot accurately detect mental health professionals' stressors (33) . In Korea, few studies have measured mental health professionals' stress, and some studies did utilize the Occupational Stress Scale for Korean Employees (34) , which is a valid and reliable scale developed to evaluate employees in various fields, including health and social work, but the scale lacks items specific to mental health and thus is not appropriate for mental health professionals in Korea. Another commonly used scale is the Occupational Stress Measurement for Psychiatric Nurses (35) , but it contains 88 items and is thus time-consuming, which limits its utility for large-scale studies. Moreover, since it is specifically tailored to mental health nurses, its use is limited with other types of mental health professionals.

The Mental Health Professionals Stress Scale (MHPSS) was developed by Cushway et al. (33) to measure stressors experienced by mental health professionals. The items were based on stressors affecting mental health professionals identified in previous studies. The scale consists of seven subscales, each containing six items for a total of 42 items. The MHPSS was validated with 154 clinical psychologists and 111 psychiatric nurses and has been used to identify stressors in studies of clinical psychologists, psychiatric nurses, and mental health social workers (30, 36, 37) . In Korea, the MHPSS was validated for use with counselors in a study by Choi and Lee (38) , and a factor analysis showed that a four-factor, 36-item structure was appropriate for the purpose. This study limited the sample to counselors; therefore, whether the finalized factor structure and items apply to other mental health professionals is unknown. Thus, the present study aimed to validate the MHPSS for use with mental health professionals in Korea, including clinical psychologists, mental health nurses, and mental health social workers so that the tool can be used to identify stressors in these professions.

The study objectives were to: (1) investigate the validity and reliability of the Korean MHPSS by conducting factor and correlation analyses and examining internal consistency to determine whether the tool has adequate reliability; (2) develop MHPSS cut-off points to screen for clinical depression and anxiety that can be used by mental health professionals to determine their stress level and screen for depression and anxiety, adding to its clinical utility; and (3) examine whether specific stressors vary by area of expertise, and test the relative contributions of stressors to psychological problems. This is particularly helpful to establish a framework for prevention and intervention strategies for mental health professionals' mental health issues.

To validate the MHPSS for mental health professionals in Korea, we collected data via an online survey over 3 months, from August to October 2020. Notifications outlining the study purpose and participant eligibility were posted to relevant mental health expert associations and academic societies. An online survey using a survey website was administered to volunteers who accessed the link and consented to participate.

The inclusion criterion was professionals working in a mental health-related organization, namely clinical psychologists, nurses, and social workers, whose work directly involved patients and clients. A total of 571 volunteers participated in the online survey. After excluding 13 duplicate responses, non-mental health experts, or professionals working in other areas of mental health, data from 558 participants (200 clinical psychologists, 157 nurses, 201 social workers) were included in the final analysis ( Table 1) . This study was approved by the Institutional Review Board of the National Center for Mental Health (116271-2020-26). Table 1 shows participant characteristics for the full sample (n = 558) and the sub-groups (clinical psychologists, nurses, and social workers). A chi-squared test of independence was conducted, and overall sociodemographic characteristics showed a significant difference (see Table 1 for clinical psychologists, 90.4% for nurses, 73.1% for social workers). While social workers and nurses predominantly had a bachelor's degree, clinical psychologists predominantly had a master's degree, which is a major qualification needed for clinical psychologist certification. Most participants in all three groups had a career length between one and 10 years (81.5% for clinical psychologists, 50.3% for nurses, 75.6% for social workers). The main work was therapy/counseling (31%) and psychological evaluation (34.9%) for clinical psychologists, patient care (27.2%) and patient/caregiver education (20.1%) for nurses, and case management (29.1%) and patient/caregiver education (19.2%) for social workers.

The original English version of MHPSS was translated into Korean by two individuals with a master's degree in psychology and one with a master's degree in occupational therapy master with English proficiency; a bilingual psychology professor reviewed and revised the translation. One bilingual professional back-translated the original translation, and then the back translation was modified to accurately convey the source text meanings by a psychiatrist and researchers with master's degrees in psychology and occupational therapy. Conflicts were resolved through discussion. Then, three certified clinical psychologists again reviewed the translation to check for any inappropriate Korean culture context before finalizing the translated version.

Sociodemographic characteristics included age, sex, education level, length of career, weekly work hours, income, and type of work.

The MHPSS measures work stressors on seven subscales: "workload (WORKL), " "client-related difficulties (CRD), " "organizational structures and process (ORG), " "relationships and conflicts with other professionals (REC), " "lack of resources (RES), " "professional self-doubt (DOUBT), " and "home-work conflict (HWC)." Each subscale consists of six items, for a total of 42 items; the items are rated on a four-point Likert scale (0 = does not apply to me, 3 = does apply to me). The average total score and subscale scores were calculated, where higher scores indicated greater stress. The Cronbach's alpha of the total score in the original MHPSS was 0.87 for clinical psychologists and 0.94 for mental health nurses, showing good reliability (33) .

Concurrent validity of the K-MHPSS was tested against the CBI-K (39), which was developed to measure fatigue and burnout. The items are rated on a five-point Likert scale (0 = never/almost never or to a very low degree, 25 = seldom or to a low degree, 50 = sometimes or somewhat, 75 = often or to a high degree, 100 = always or to a very high degree), and subscale scores were used for analysis, where a higher score indicated a higher level of burnout. The scale consists of three subscales (personal burnout, workrelated burnout, and client-related burnout), for a total of 19 items. The Korean version has been confirmed to have adequate reliability and validity (40) . The Cronbach's alpha for the CBI-K subscales in this study were 0.91 for personal burnout, 0.82 for work-related burnout, and 0.90 for patient-related burnout.

Concurrent validity of the K-MHPSS was tested against the Job Satisfaction Scale developed by Park (41) based on the Job Satisfaction Scale items developed by Scarpello and Campbell (42) . The scale consists of 25 items on satisfaction with the boss, work, compensation, colleagues, and work conditions, with each item rated on a seven-point Likert scale (1 = extremely dissatisfied, 7 = extremely satisfied) where higher scores indicate higher job satisfaction. The Cronbach's alpha in this study was 0.93, which was similar to that of the original scale (41) .

The SCL-90-R assesses overall psychological problems and psychopathological symptoms (43) ; the Korean version has been standardized (44). The items are rated on a five-point Likert scale (0 = not at all, 4 = extremely). A T score of 63 or higher for the global severity index (GSI) or 63 or higher for two or more major symptom dimensions is deemed clinically relevant (45) . In this study, only the depression and anxiety scores were used to identify cut-off points for the K-MHPSS and examine relationships with stressors. The Cronbach's alpha in this study was 0.92 for depression and 0.91 for anxiety.

First, confirmatory factor analysis (CFA) was conducted to estimate the stability of the factor structure of the MHPSS in Korean mental health professionals. The original MHPSS had a seven-factor structure, but an Indian study of clinical psychologists (46) and a Korean study of counselors (38) identified a four-factor structure. Seven-factor and four-factor structures obtained from previous research (33, 38) were investigated via CFA. In addition, we conducted exploratory factor analysis (EFA) because this was the first attempt to use the MHPSS with clinical psychologists, nurses, and social workers in Korea, the potential for a cultural gap between Europe and Asia, and the inconsistency in the MHPSS factor structure in previous studies. Factors were extracted using the maximum likelihood method, and Geomin oblique rotation was used because of the possibility of correlations among the extracted factors. The number of factors was determined based on the Kaiser criterion (eigenvalues >1), RMSEA, and information criterion. Starting with a one-factor model, the number of factors was increased to select the model with a RMSEA of 0.05 or lower with the least number of factors (47) and lowest BIC (information criterion) (48) . Only factors with at least three items and a factor loading >0.30 were retained (49, 50) . Items loaded on two or more factors with a loading of 0.32 or higher (cross-loading) were deleted (51) . Model fit indices where CFI and TLI were 0.90 or higher (52) and RMSEA and SRMR were 0.05 or lower (53) were considered to indicate a good model fit.

Concurrent validity was tested by analyzing the correlation between the K-MHPSS and the CBI-K and Job Satisfaction Scale. Cronbach's alpha for the entire scale and each subscale was calculated, and a value of 0.60 or higher was considered to indicate good reliability (54) . Sensitivity and specificity were computed using receiver operating characteristics (ROC) curve analysis, and Youden's index was used to estimate the K-MHPSS cut-off points for depression and anxiety (55) . Further, the area under the curve (AUC) was examined; AUC indicates the probability that the tool accurately classifies a randomly selected pair including a case with symptoms and a case without symptoms, that is, the tool's discriminatory ability. An AUC value of 0.70 or higher indicates fair discrimination (56, 57) . Finally, between-group differences in the K-MHPSS total score and subscale scores were analyzed with oneway ANOVA, and the subfactor effects on depression and anxiety were analyzed using multiple regression. Considered the presence of gender differences in depression and anxiety (58, 59) , gender was set as a control variable. Statistical analyses were performed using IBM SPSS version 22.0 and Mplus 8.5.

The result from the CFA models for each professional group could not demonstrate reasonable fit indices (see Table 2 ). It showed that current data could not replicate the factor structure of MHPSS. Therefore, we undertook an EFA by combining the three professional groups. Because it is recommended to conduct EFA with a large enough sample size (60), unlike CFA, which can perform with a relatively small sample size (61) . To determine the number of factors, the fit indices were analyzed for a one-factor to a seven-factor model with an eigenvalue greater than one, and the seven-factor model (χ 2 = 1,225.348, df = 588, p < 0.001, CFI = 0.935, TLI = 0.904, RMSEA = 0.044) was found to have the best fit for the data. However, there were items with loadings below 0.32 and cross-loaded items, so a second analysis was performed after deleting these eight items (1, 8, 14, 26, (33) (34) (35) 41) . In the second analysis, both the six-factor model ( (Table 3) ; however, the sevenfactor model had one factor with fewer than three items loaded, and considering the parsimony of the model, the model with fewer factors was deemed more appropriate. Hence, the six-factor model was selected. Factor 1 consisted of three items (4, 11, 25) , with a loading range of 0.373-0.737, and was labeled "conflict with other professionals (COP)." Factor 2 consisted of ten items (3, 10, 17, 18, 24, 28, 31, 32, 38, 39) , with a loading range of 0.307-0.788, and was labeled "organizational structure and processes (ORG)." Factor 3 consisted of four items (6, 13, 20, 27) , with a loading range of 0.360-821, and was labeled "professional self-doubt (DOUBT)." Factor 4 consisted of seven items (7, 15, 21, 22, 29, 36, 42) , with a loading range of 0.346-0.878, and was labeled "workload (WORKL)." Factor 5 consisted of six items (2, 9, 16, 23, 30, 37) with a loading range of 0.405-0.593 and was labeled "client-related difficulties (CRD)." Finally, Factor 6 consisted of four items (5, 12, 19, 40) , with a loading range of 0.313-0.575, and was labeled "lack of resources (RES)" ( Table 4 ). All items had factor loadings of 0.30 or higher for the six factors. Table 5 shows the K-MHPSS scores and relevant evaluation scale scores for the entire sample. The mean K-MHPSS score was 1.25 (SD = 0.50), and the subscale scores ranged from 0.94 to 1.46, with the lowest mean score for COP and highest mean score for DOUBT. The absolute values of skewness and kurtosis did not exceed one, and thus the assumption of normal distribution does not seem to be violated (62) .

The correlation between the K-MHPSS and other scales was analyzed using Pearson product-moment correlation to determine the K-MHPSS's concurrent validity. The K-MHPSS score was significantly positively correlated with the personal (r = 0.62, p < 0.01), work-related (r = 0.62, p < 0.01), and client-related (r = 56, p < 0.01) CBI-K factors, and significantly negatively correlated with the total job satisfaction score (r = −0.46, p < 0.01). Table 6 shows the correlations between the K-MHPSS total score and subscale scores and other relevant scales.

The internal consistency of the K-MHPSS was examined based on Cronbach's alpha for the total score and subscale scores for the entire sample ( 

The optimal cut-off point for the K-MHPSS was estimated in each mental health professional group using the depression and anxiety subscales of the SCL-90-R. The cut-off point for diagnosing clinical depression and anxiety in Koreans is 63T, which is equivalent to that used in the United States. With 63T as the reference, cases with a score of 63 or higher were considered to have symptoms, and cases with a score below 63 were considered to have no symptoms for the ROC analysis. In clinical psychologist, the K-MHPSS cut-off point for depression was 1.78, with a sensitivity of 38% and specificity of 95% (Youden's Index = 0.34); the cut-off point for anxiety was 1.45, with a sensitivity of 67% and specificity of 78% (Youden's Index = 0.45). In nurses, the K-MHPSS cut-off point for depression was 1.66, with a sensitivity of 75% and specificity of 76% (Youden's index = 0.51); the cut-off point for anxiety was 1.78, with a sensitivity of 88% and specificity of 85% (Youden's index = 0.73).

In social workers, the K-MHPSS cut-off point for depression was 1.46, with a sensitivity of 85% and specificity of 69% (Youden's index = 0.54); the cut-off point for anxiety was 1.46, with a sensitivity of 81% and specificity if 68% (Youden's index = 0.49). Table 7 show the ROC curve, AUC, and other applicable data for the MHPSS. The values of AUC were ranged from 0.696 to 0.810 for depression and ranged from 0.725 to 0.922 for anxiety in each profession, showing that the K-MHPSS had good discrimination for depression and anxiety (57) . 

As shown in Table 8 , there were between-group differences for the K-MHPSS total score and subscale scores. The data for WORKL did not satisfy the homogeneity of variance assumption, so Welch's F test and Dunnett' T3 post hoc comparison were performed. The remaining factors all satisfied the homogeneity of variance assumption, so Scheffe's post hoc comparison was performed. There was a significant betweengroup difference in the K-MHPSS total score [F (2, 

The effects of the K-MHPSS sub-factors of depression and anxiety in clinical psychologists, nurses, and social workers were analyzed with multiple regression (Tables 9, 10) . 

This study standardized the K-MHPSS for use with mental health professionals in Korea, including clinical psychologists, nurses, and social workers. The study's first objective was to test the validity and reliability of the K-MHPSS using factor analysis, correlation analysis with relevant scales, and internal consistency testing for the K-MHPSS total score and subscale scores. We tested the factor structure obtained from prior research via CFA to Korean mental health professionals but did not show an acceptable level of fit indices. A subsequent EFA revealed that a six-factor structure could be appropriate in Korean mental health professionals. However, construct validity could not be confirmed because CFA for the six-factor structure did not conduct. The K-MHPSS's total score and subscale scores were significantly positively correlated with the personal, work-related, and client-related burnout scales in the CBI-K and significantly negatively correlated with the Job Satisfaction Scale, thereby confirming good concurrent validity. Moreover, the Cronbach's alpha for the K-MHPSS total score was 0.92, and that for the subscale scores ranged from 0.68 to 0.87, showing good reliability overall. The K-MHPSS six-factor structure was found to have the best fit, which contrasts with the original scale's seven-factor structure and the four-factor structure found to be optimal in a study that validated the MHPSS with Korean counselors (38), and a study that examined the factor structure for MHPSS use with Indian clinical psychologists (46) . The difference in the factor structures may be attributable to the sample size and characteristics. Costello and Osborne (49) suggested that the likelihood of misclassifying an item with the wrong factor increases with small samples, which then undermines generalizability and replicability, and that the sample size should be close to 20 times the number of items for highest accuracy. In Cushway et al.'s (33) study of 154 clinical psychologists and 111 nurses, the original seven-factor structure was found to be the optimal structure for clinical psychologists, but only six of the seven factors were effective, with the remaining items included in a new factor ("shortage of time") for nurses. An MHPSS study of clinical psychologists in India had only 116 participants, which may explain differences in our study results, where factor analysis was conducted on 558 mental health professionals. A study of counselors in Korea had a larger sample size compared to previous studies, but the organization characteristics and type of work performed by counselors differed from that of the mental health professionals who participated in this study, which might have led to the difference in the factor structure.

In this study, the six factors obtained through EFA were labeled similar to the original scale: conflict with other professionals, organizational structure and process, professional self-doubt, workload, client-related difficulties, and lack of resources. The first factor, COP, was equivalent to REC in the original scale, but items 18, 32, and 39 from the six items included in the original study's REC loaded on ORG in this study. In a study of Korean counselors, the ORG and REC items were merged to form new factors "organization-related difficulties" and "relationship conflicts" (38), and similarly, the items related to conflict with other professionals were classified as COP, while items related to relationships with colleagues were classified as ORG in this study.

Among the six HWC items in the original scale, items 7, 21, and 42 loaded on WORKL, while item 28 loaded on ORG. Items 14 and 35 were not loaded on a single factor and thus were removed. Choi and Lee (38) also had items 7, 21, and 42 loaded on WORKL, presumably because of cultural differences. Kim and Kim (63) investigated factors affecting work-family conflict experienced by married hospital workers, including health care professionals, and found that factors related to workload and work burden, such as overtime and unpredictability of work, contributed to work-family conflict. While all participants were married, only those with a child age six or younger had adverse impacts on work-family conflict. Other factors related to family roles were not associated with work-family conflict. Based on these results, we can predict that work-family conflict and workload are similar and closely linked concepts, which may have led to items 7 ("not enough time with family"), 21 ("taking work home"), and 42 ("inadequate time for friendships/social relationships") loading on workload, while items 14 ("inability to separate personal from professional role") and 35 ("work emphasizes feelings of emptiness and/or isolation") were not classified into a specific factor because they are not largely relevant to work-family conflict as perceived by Koreans.

The second objective of this study was to present the optimal K-MHPSS cut-off points for initial clinical depression and anxiety screening. The K-MHPSS cut-off points for depression and anxiety were 1.78 and 1.45 in clinical psychologists, 1.66 and 1.78 in nurses, and 1.46 and 1.46 in social workers. However, they could not be compared with that of the original scale and other studies, because cut-off points were not reported. The cutoff points were higher than the mean scores for each group, with 7.0 and 22.0% of clinical psychologists, 26.2 and 17.9% of nurses, and 36.8% of social workers showing scores above the cut-off points for depression and anxiety, respectively. The K-MHPSS had good discriminatory power for depression and anxiety; however, since this study was conducted on general mental health professionals and not a clinical patient group diagnosed with depression and anxiety-related disorders, using the K-MHPSS as the primary screening tool for clinical patients may have limitations. Although stress is not a disease in itself, it can lead to psychological problems, including depression and anxiety. Therefore, the K-MHPSS may be used as the initial screening tool for mental health professionals, and prophylactic interventions could be provided to professionals exhibiting stress that exceeds the cut-off points

The study's third objective was to examine differences in the stressors experienced by the type of profession. Nurses and social workers showed significantly higher total, COP, ORG, and CRD scores as compared to clinical psychologists, and nurses showed significantly higher RES scores than clinical psychologists. Cushway et al. (33) also reported that nurses showed significantly higher scores on the same subscales compared to clinical psychologists. Moreover, one study that administered the MHPSS to 108 social workers and compared the results with Cushway et al.'s clinical psychologist and nurse scores, reported higher total, REC, ORG, and CRD scores among social workers as compared to clinical psychologists, which was consistent with our results, although it is not known whether the differences were significant (30) .

The lower COP and ORG scores in clinical psychologists may result from clinical psychologists working alone because of the nature of their work. Clinical psychologists who work in a medical institution, practice or work as a freelancer in a private treatment facility, rarely engage in activities within the organization or interact with other professionals because their major work is providing therapy, which may have resulted in lower scores for the corresponding stressor as compared to other professions. Further, nurses and social workers had significantly higher CRD scores than clinical psychologists. Jobs involving patients/clients may pose greater stressors for nurses and social workers despite that all three professions deal directly with patients, which could be because nurses spend most of their time with patients and social workers manage numerous cases and provide wide-ranging services. Among the three professions, nurses reported the highest RES score. The number of mental health nurses per 100,000 population in Korea is 14.7, which is more than twice as few as in high-income countries (64) , and such short staffing may have contributed to the high RES score among mental health nurses.

Regression analysis was performed for each group to examine the effects of stressors on depression and anxiety with controlling gender. In all three groups, WORKL was a significant predictor of depression or anxiety. Past studies have also reported that excessive workload predicts depression and anxiety in healthcare providers, including nurses, as well as in general office workers (65) (66) (67) , showing consistent findings among mental health professionals as well. Another factor affecting depression and anxiety in mental health professionals was DOUBT. A previous study argued that professional self-doubt is the greatest stressor for clinical psychologists because they sense limitations in therapy effectiveness and their competence as therapists (23) . However, although clinical psychologists in our study had DOUBT scores that were higher than their other MHPSS subscale scores, DOUBT did not predict their depression or anxiety. In contrast, DOUBT was a significant predictor of depression or anxiety among nurses and social workers. Professional selfdoubt specifically addresses characteristic mental health-related job descriptions, so studies examining its effect on psychological problems in mental health professionals are rare. However, given the finding that excessive workload increases emotional distress and reduces professional efficacy in nurses (65) , incompetency in professional tasks may lead to professional self-doubt, which in turn is predicted to adversely impact professionals' mental health. Further studies are needed to examine the relationship between stressors and psychological problems in mental health professionals.

This study has a few limitations. First, although the K-MHPSS with a six-factor structure was appropriate to Korean mental health professionals, the construct validity could not be confirmed since the CFA for this structure has not been performed. Several previous research randomly divided samples into two equal parts and conducted EFA and CFA, respectively, but this method requires a large enough sample size (68) (69) (70) . The sample size in this study was not suitable for the cross-validation method. Therefore, subsequent studies should verify the six-factor structure derived from this study with mental health professionals, including clinical psychologists, nurses, and social workers. Second, self-report questionnaires were used, so there is a potential for underreporting or biased responses relating to social desirability. In addition, the study parameters were measured cross-sectionally, so causality between stressors and psychological symptoms could not be examined. Thus, subsequent studies should address the limitations of the questionnaire and study design. Third, study participants were mental health professionals as recognized by the Korea Ministry of Health and Welfare, including clinical psychologists, nurses, and social workers. However, various other professionals also provide mental health services to patients and clients, including school psychologists, and occupational therapists. Subsequent studies should examine the reliability and validity of the K-MHPSS in other types of mental health professionals that were not included in this study.

Despite these limitations, this study provides evidence for the reliability and validity of the K-MHPSS for use with clinical psychologists, nurses, and social workers. Moreover, the results which identify the difference in stressors experienced among the professionals and the relationships between the specific stressors and depression and anxiety give knowledge for establishing an effective intervention approach. As there are few or no studies to verify the effectiveness of stress management intervention for mental health professionals (71) , exploring interventions primarily based on the stressors specific to each mental health profession with reference to our findings would contribute to promoting mental health professionals' health and thereby facilitating quality mental health service to patients.

The datasets presented in this article are not readily available because the participants did not consent to share their raw data. Requests to access the datasets should be directed to Subin Park, subin21@korea.kr.

The studies involving human participants were reviewed and approved by National Center for Mental Health. The patients/participants provided their written informed consent to participate in this study.

Do mental health professionals stigmatize their patients?

Seeking help for psychological distress: barriers for mental health professionals

Stress in mental health professionals: a theoretical overview

Occupational stress, coping strategies, and psychological-related outcomes of nurses working in psychiatric hospitals

The relationships among work stress, resourcefulness, and depression level in psychiatric nurses

Mediating effects of selfefficacy, coping, burnout, and social support between job stress and mental health among young Chinese nurses

The relationship of work environment and client contact to burnout in mental health professionals

Assessing burn-out and occupational stressors in a medium secure service

The relationship between perfectionism, stress and burnout in clinical psychologists

The measurement of experienced burnout

Burnout in mental health professionals: a systematic review and meta-analysis of prevalence and determinants

Hopelessness and suicide risk emerge in psychiatric nurses suffering from burnout and using specific defense mechanisms

Stress management for mental health professionals: a review of effective techniques

Workplace stressors, ways of coping and demographic characteristics as predictors of physical and mental health of Japanese hospital nurses

Fernández-Ríos L. Occupational stress and state of health among clinical psychologists and psychiatrists

Burnout, self-and supervisor-rated job performance, and absenteeism among nurses

Factors Associated With Turnover Intention in Community Mental Health Professionals

Report on training and current situation of mental health professionals

Job satisfaction of social workers in mental health settings: focus on the young-nam area

Contemporary profiles of clinical and health psychologists in Australia

Stress and coping in clinical psychologists

Burnout amongst clinical and counselling psychologist: the role of early maladaptive schemas and coping modes as vulnerability factors

Stress in clinical psychologists

Occupational stress, social support, and quality of life among Jordanian mental health nurses

burnout and social support: nurses in acute mental health settings

A measurement tool for assessing stress among mental health nurses

The lived experience and meaning of stress in acute mental health nurses

A systematic review of stress and stress management interventions for mental health nurses

Stress and pressures in mental health social work: the worker speaks

Sources of stress experienced by occupational therapists and social workers in mental health settings

Mental health, burnout and job satisfaction among mental health social workers in England and Wales

A systematic review of stress among mental health social workers

Development of a stress scale for mental health professionals

Developing an occupational stress scale for Korean employees

The development of occupational stress measurement tool for psychiatric nurses

Cognitive therapy and behavioural coping in the management of work-related stress: an intervention study

Exploring the relationship between work stress, years of experience and emotional competency using a sample of Australian mental health nurses

The Copenhagen Burnout Inventory: a new tool for the assessment of burnout

Psychometric properties of the Korean version of the Copenhagen Burnout Inventory in Korean homecare workers for older adults

Employee's preference of the allocation principles and job satisfaction

Job satisfaction: are all the parts there? Pers Psychol

Confirmation of the dimensional structure of the SCL-90: a study in construct validation

Manual for the Symptom Checklist-90-R. Seoul: HUNO

Reliability and factor invariance of Symptom Chekclist-90-Revised

Factor structure of the mental health professionals stress scale (MHPSS) among clinical psychologists in India

Choosing the optimal number of factors in exploratory factor analysis: a model selection perspective

Bayesian model assessment in factor analysis

Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis

The quality of factor solutions in exploratory factor analysis: the influence of sample size, communality, and overdetermination

Using Multivariate Statistics

Principal-Components Analysis and Exploratory and Confirmatory Factor Analysis. Reading and Understanding Multivariate Statistics

Primer of LISREL: Basic Applications and Programming for Confirmatory Factor Analytic Models

Psychometric Theory

Estimation of the Youden Index and its associated cutoff point

The meaning and use of the area under a receiver operating characteristic (ROC) curve

Can routine laboratory tests discriminate between severe acute respiratory syndrome and other causes of community-acquired pneumonia?

Epidemiology of women and depression

Are gender differences in depression explained by gender differences in co-morbid anxiety?

Principles and Practice of Structural Equation Modeling

How to use a Monte Carlo study to decide on sample size and determine power

The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis

Hospital workers' work-family interface: demand and resource approach. Korean Acad Org Manag

Reactions to increased workload: effects on professional efficacy of nurses

Effect of the occupational stress and self esteem on mental health among nurses

Work stress precipitates depression and anxiety in young, working women and men

Factors influencing cross-validation of confirmatory factor analysis models

A confirmatory factor analysis and validation of the forms of self-criticism/reassurance scale

Age and gender effect on alexithymia in large

Stress, burnout, coping and stress management in psychiatrists: findings from a systematic review

EL, VR, and JL conceived and designed the study. EL, JL, and HHo collected data. EL, HHo, and HHa contributed to performing statistical analysis and interpreting the data. EL and HHo wrote the manuscript. VR and SP reviewed critically and revised the manuscript. All authors contributed to the article and agreed to be accountable for the content of the final version of the manuscript. 

The authors would like to thank Delia Cushway and her colleagues for willingly approving the validation of the Korean version of MHPSS. In addition, we express deep gratitude toward the mental health professionals who gave time and effort to contribute to this study. August 2021 | Volume 12 | Article 685423

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.