key: cord-0077416-nx264rat
authors: Tang, Ying; Liao, Zhenjiang; Huang, Shucai; Hao, Jingyue; Huang, Qiuping; Chen, Xinxin; Lin, Shuhong; Li, YiFan; Qi, Jing; Shen, Hongxian
title: Development and Validation of a Risk Assessment Tool for Gaming Disorder in China: The Gaming Hazard Assessment Scale
date: 2022-04-11
journal: Front Public Health
DOI: 10.3389/fpubh.2022.870358
sha: 68fe3aca179b986081dbe617ebd98163ceb0d106
doc_id: 77416
cord_uid: nx264rat

Despite the growing research interest in gaming disorder, risk screening tools developed specifically for the Chinese population are still lacking. This study aimed to construct a screening tool to evaluate the risk of gaming disorder (GD) development, by assessing the severity of GD symptoms among Chinese gamers, based on clinical expert interviews, structured interviews with GD patients, a background literature review, and IGD/GD criteria proposed by the DSM-5 and ICD-11. It introduced the Gaming Hazard Assessment Scale—a multidimensional GD risk screening tool—and evaluated the dimension structure, reliability, and validity of the scale among 959 Chinese gamers. A three-level structure, consisting of 18 items scored from 0 to 54, ultimately indicated satisfactory reliability, good validity, and acceptable model fit. The scale will help large-scale initial screening and early identification of patients with a high risk of GD. Further evaluation of the Gaming Hazard Assessment Scale in clinical settings is highly recommended.

In 2013, the American Psychiatric Association officially included Internet gaming disorder (IGD) in the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (1). In 2019, the World Health Organization (WHO) proposed the inclusion of Gaming disorder (GD), as a psychiatric disorder in the category of behavioral addiction and included it in the 11th edition of the International Classification of Diseases (ICD-11) (2) . IGD is characterized by gaming behavior that lasts for at least 12 months, which is accompanied by preoccupation, loss of control and tolerance, and impairment of social functioning. Data from the Cyberspace Administration of China notes, the number of game users in China has reached 665 million as of December 2020 (3), the majority of which consisted of adolescents. Considering the vulnerability of adolescents and the accompanying impairment caused by GD, problems related to high levels of gaming and internet usage are increasingly recognized as a serious public health burden across China and the whole world (4) . Depending on the differences in study design, measurement methods, and study populations used in various studies, the prevalence of IGD ranged from 0.21 to 57.5% (5) . Studies have shown an alarming prevalence of IGD among adolescents, especially in males (6, 7) . Further, IGD is associated with multiple health issues (e.g., physical symptoms, depressive symptoms, and sleep disturbances), especially in adolescents (8, 9) . Presently, the diagnosis and treatment of GD is relatively difficult to accomplish (10) , so this study focused on early identification to facilitate early preventive intervention.

Although many studies have explored new identification methods [e.g., electroencephalograph, functional magnetic resonance imaging, and biomarkers (11) (12) (13) (14) ], a convenient scale with high reliability and validity, as well as wide adaptability, is more conducive to large-scale initial screening, and it can achieve an effective balance between economic and test efficiency.

Multiple diagnostic screening tools for GD exist (15) . For example, the Personal Internet Gaming Disorder Evaluation-9 (PIE-9), covered both the DSM-5 and ICD-11 diagnostic criteria, and was used to screen for IGD; however, it did not include structured interviews in its development, and it excluded participants younger than 18 years old (16) . Another example is the Problematic Online Gaming Questionnaire (POGQ), which did not assess functional impairment, and it was not test-retested in the Chinese population (17) . Most of these previous tools or studies are based on IGD diagnostic criteria in the DSM-5; however, relatively few tools are based on the ICD-11 (18) . The Internet Gaming Disorder Test (IGD-20 Test) and the Internet Gaming Disorder Scale-Short Form (IGDS9-SF), showed good psychometric properties, though they did not account for the latest conceptualization of IGD proposed by the WHO in the ICD-11 (19, 20) . Therefore, these two diagnostic criteria should be integrated to further improve the psychometric evaluation of IGD. Many studies have found that the above two criteria share components of IGD features, such as preoccupation, loss of control, and prioritizing gaming over other activities (21) . However, some differences are notable. First, ICD-11 no longer emphasizes withdrawal and tolerance, which is highlighted in the DSM-5 (15) . Second, functional impairment is stressed in the ICD-11, which is key for differentiating between people with gaming disorder and the large proportion of individuals engaged in intense or persistent patterns of gaming (e.g., 20-30 h per week), without experiencing associated negative consequences (22) . Third, for the presentation of common symptoms, the two diagnostic criteria are not exactly the same, such as the "diminished interests" in DSM-5 and "increasing priority given to gaming" in ICD-11. Considering these limitations, this study attempted to establish a GD risk assessment scale based on both the ICD-11 and DSM-5 standards, so that it can be used for the large-scale risk screening of GD. It should be emphasized that the purpose of this scale is to identify high-risk game users at an early stage, who can be intervened in time to reduce the likelihood that they will eventually develop GD, rather than a GD diagnosis. Owing to the complexity and social characteristics of behavioral addiction, clinical diagnosis may be the best method to evaluate and diagnose GD.

The purpose of this study was to use representative samples, to construct a risk assessment scale for GD with strong applicability, high reliability and validity, and easy operation based on DSM-5 and ICD-11 criteria. This scale will especially focus on the adverse consequences (e.g., significant sleep deprivation or changes in sleep patterns, physical impairment, etc.), resulting from GD, to identify gamers who are at high risk of developing GD.

First, we invited four Chinese experts majoring in addiction, to conduct in-depth individual interviews and gather the initial item pool, based on the DSM-5 and ICD-11 criteria and their clinical experience. Concurrently, item collection was conducted by two psychiatrists, based on DSM-5 criteria in outpatients who were diagnosed with IGD, with semi-structured interviews being used to compile the items. Twenty patients who met the diagnostic criteria were selected, and they were interviewed regarding issues such as preoccupation, withdrawal, and tolerance. After the interviews, the content was sorted and classified, and keywords were extracted. Next, we reviewed relevant screening tools, such as the Internet Gaming Disorder-20 Test (23) . Under the guidance of experts, we used patient interviews and references to other relevant scales to finally form an initial item pool of 47 items related to GD symptoms, which covered both the DSM-5 and ICD-11 diagnostic criteria. Specifically, we constructed a multidimensional GD model. Of the 47 item questions that were compiled, 15 candidate questions that related to the DSM-5's definition of IGD were analyzed. Each candidate question fell under one of the following categories found in the DSM-5's definition: preoccupation, tolerance, withdrawal symptoms, deception, escapism, unsuccessful control, and adventure. Fourteen candidate questions, that related to the ICD-11's definition of GD, were also analyzed. Each candidate question fell under one of two categories found in ICD-11's definition: out-of-control over gaming, and the prioritization of gaming over other hobbies and daily activities. "Continued or enhanced gaming behavior despite adverse consequences/excessive gaming use despite psychosocial problems, " is a common entry in both diagnostic criteria, and it comprised 16 questions. A close relationship between peer influence and IGD has been reported in the literature (24) (25) (26) , and combined with communication with clinical patients. We also added the dimension of peer influence, including "playing games alone without friends" and "can't wait to play games with friends when they are invited to play games."

Another four experts were then invited to conduct content validity tests, to provide suggestions on the initial scale's structure and content, the suitability of each dimension, and the semantic accuracy of each item. Some items were modified or deleted according to expert opinions. Experts repeatedly reviewed all the items and confirmed that the scale had satisfactory content validity. A pretest survey was then conducted among 113 Chinese gamers, to determine the relevance of items with GD. Based on the positive feedback provided by all participants, and after obtaining the consensus of all experts, items with weak applicability, low relevance, and poor understanding were removed, and 34 items were selected for the final test in the broader target population.

Sample 1 included 572 adolescents and young adults, who were recruited from three middle schools (n = 372) in Changsha city, Hunan Province, by cluster sampling and online surveys (n = 200). To increase sample size, in-person surveys and online surveys were combined. We randomly selected different quality ranking classes and then two classes were sampled from each grade level. Three hundred and seventy-two students used a pencil and paper to complete the original 34-item scale, and participants recruited online used a QR code of Questionnaire star platform to complete the same scale anonymously, in September 2021. Each participant was questioned whether they had used the game in the past 12 months before completing the scale, and those who answered "no" were excluded. They were asked to select scale items using a four-point Likert scale, that ranged from "never" to "always" (selecting 0, 1, 2, and 3 points, respectively), with a total score ranging from 0 to 54. We performed a scale structure item analysis and an exploratory factor analysis. We then recruited another 274 participants (sample 2), from a general high school and vocational high school in Kunming City, Yunnan Province, to complete a confirmatory factor analysis (CFA). Of these 274 participants, 100 completed the Internet Addiction Test (IAT), to test criterion validity and underwent a test-retest of the Gaming Hazard Assessment Scale after 2 weeks, to assess test-retest reliability. Informed consent was obtained from all the participants before the scale was completed.

The structural analysis was performed using exploratory factor analysis, and item analysis was used to explore the differences between subjects with high and low scores for each item, or to conduct homogeneity research between items. The homogeneity test included a reliability test to obtain the reliability index values of scales and dimensions, and a validity test to verify the level of questionnaire validity through confirmatory factor analysis. After determining the final items and dividing the dimensions, calibration and test-retests were completed on September 7, 2021, and September 22, 2021, respectively. Spearman correlation analysis was performed to evaluate the test-retest reliability of the participants who test-retested for the Gaming Hazard Assessment Scale. Criterion validity was tested using Spearman's correlation between the Gaming Hazard Assessment Scale and IAT (27) . SPSS 24.0 and AMOS version 25.0 were used in this study. 

The mean age of the participants in sample 1 was 19.7 years (SD = 3.929 years), of which 53.3% were male (male, n = 305; female, n = 267). The age range was 15-30 years (28, 29) . The mean age of the participants in sample 2 was 16.5 years (SD = 1.765 years), of which 50.7% were male (male, n = 139; female, n = 135). The age range was 14-22 years.

Since the dimensions of the questions were classified according to diagnostic criteria, expert opinions, and practical experience during the preparation of the questionnaire, exploratory factor analysis was used for the 34 initial questions. First, Kaiser-Meyer-Olkin (KMO) and Bartlett sphericity tests were conducted, to verify whether the sample was suitable for exploratory factor analysis. The KMO value of the 34 items was 0.963 (p < 0.05), which indicated that they were suitable for factor analysis. Principal component analysis and the maximum variance method were used to determine the final factors (see Table 1 ). Exploratory factor analysis was conducted on the scale of 34 items, and the final rotation component matrix revealed that the sample data converged after five rotations. Some factor load coefficients of items were less than 0.4 in the factor, which deviated from the corresponding relationship with the factor. Therefore, deletion was considered. A total of 16 questions were deleted, for example, "play games regardless of time or place, " "I think playing games is more important than participating in other activities, " "1 day seems like a year if I don't play games, " "I feel angry or impatient when I can't play" and so on. The final 18 items were divided into three dimensions after the deviation items were deleted, and the items with high load coefficients were selected (see Table 2 ). 9, 22 , and 25 had a high load on Factor 1 ("out-of-control" dimension); items 13, 30-32, 34, and 37-38 had a high load on Factor 2 ("impairment of social functioning" dimension); and items 12, 14-16, and 18 had a high load on Factor 3 ("cognitive impairment" dimension).

In general, the results obtained by the factor load rotation component matrix of the scale data, were consistent with the scale and dimensions that were divided in the research design. Additionally, the load coefficients of the corresponding items in each dimension were all >0.45, indicating that the Gaming Hazard Assessment Scale can be used for further research and analysis. 

The 18 items obtained after eliminating the deviation items, were tested by an independent sample t-test for the high (The top 27%), and low (The bottom 27%) score groups (see Table 3 ). All 18 items presented 

The Gaming Hazard Assessment Scale showed high internal consistency (α = 0.938) in the study, with Cronbach's α ranging from 0.856 to 0.896 for the three factors (see Table 4 ). Pearson correlation coefficients showed that the three factors were significantly and positively correlated with each other (r = 0.924-0.944, p < 0.001) on the 18-item scale (see Table 5 ).

This study further examined the relationship between the whole Gaming Hazard Assessment Scale and each dimension. Through dimensionality reduction of the dimensions' data, the Pearson correlation coefficient was used to test the correlation Frontiers in Public Health | www.frontiersin.org between each dimension and the whole scale. Pearson correlation coefficients showed that the three factors were significantly and positively correlated with each other and the whole 18-item scale (r = 0.924-0.944, p < 0.001). Among these dimensions, the correlation coefficient between the "out-ofcontrol" dimension and the whole scale was the highest, indicating that this dimension more greatly impacts the game hazard rating level.

The fit of the original three-factor model was also tested, with the results indicating that the model fit was not very good [χ 2 (274) = 3.846, p < 0.001, CFI = 0.857, and RMSEA = 0.102]. To optimize the model selection, model modification was considered. As all path coefficients were statistically significant, no paths were removed. We selected the Modification Index (MI), with the largest value and established a correlation between e9 and e10, e11 and e12 (see Figure 1) . The results revealed that the revised model fits better than the original model [χ 2 (274) = 3.062, p < 0.001, CFI = 0.898, and RMSEA = 0.087]. Although the adjusted goodness-of-fit index was slightly lower, its overall model fit was better. No further corrections were made because the revised model did not result in a significant improvement.

The evaluation criteria of convergence validity in this study were the reliability combination (CR) value and average extraction variance (aVE) value. The basic aVE value and CR value of each dimension indicate that the convergence validity of this dimension is high (see Table 5 ), but the square root of aVE is lower than the correlation value with other factors, so the discriminant validity of internal factors of each variable is not very good.

Twenty IAT items were used as the validity criteria of the Gaming Hazard Assessment Scale for correlation analysis. The scale was found to highly correlate with IAT scores (r = 0.779, p < 0.01), and the criterion validity was satisfied.

Two weeks later, after CFA, a total of 100 participants were test-retested, with an overall test-retest reliability of 0.535 (p < 0.01) (30) . 

In this study, we developed the Gaming Hazard Assessment Scale, which was developed specifically for the Chinese population, and was the first risk screening tool based on the dual diagnostic criteria of DSM-5 and ICD-11. Among the existing GD risk assessment tools, the Gaming Hazard Assessment Scale is a relatively convenient and comprehensive tool that can help clinicians effectively evaluate the risk of GD and provide targeted intervention measures. This study examined the Gaming Hazard Assessment Scale's development, as well as its psychological characteristics. The results suggest that although the Gaming Hazard Assessment Scale model fit did not reach the recommended level, it nevertheless showed satisfactory results in other aspects in the Chinese gamer sample-which supports its use as a rapid screening tool for risk assessment. Specifically, 18 items in three dimensions had factor loadings higher than the recommended level, and the overall scale and each of its factors displayed high internal consistency and acceptable testretest reliability, and demonstrated good criterion-related and concurrent validity. This result suggests that this scale has good reliability and validity, which confirms that the Gaming Hazard Assessment Scale can be used as a relatively reliable tool for target group initial screening of patients at high risk of GD. It also provides an effective quantitative tool and evaluation standard for risk assessment of GD, to achieve early identification and intervention for GD. The relatively low test-retest reliability could be owing to memory bias caused by the time interval between tests. In addition to what has been mentioned above, the small number of participants may also be accounted for the result, which we will continue to improve in the follow-up work.

Considering that the original model's degree of fit was not satisfactory in this study, minor modifications were made to achieve sufficient model fitting (i.e., two correlations were added). One possible reason for the correlation between items 27 and 28, is that internet addiction and social phobia are considered relatively common; people with social phobia seldom go out or engage in social activities, and internet games provide them a way to reduce face-to-face contact with others while participating in game interaction (31, 32) . One reason for the correlation between 30 and 33, may be that patients with severe GD can experience sleep disturbance and spend large amounts of money on games (33) (34) (35) . Another possible explanation, is that players who spend money on loot boxes are more likely to play games longer on weekends than non-paying players, which indirectly leads to sleep problems (36, 37) .

Although the Gaming Hazard Assessment Scale aims to identify the risk of GD development in the Chinese population, a symptomatic assessment of GD reveals only a partial picture of the clinical status of gamers with GD (38) . Compared to the IGD diagnostic criteria proposed by DSM-5, the criteria proposed by ICD-11 emphasize functional impairment and consider it a core diagnostic indicator (39) . The assessment of functional impairment raises the diagnostic threshold of ICD-11 for GD, avoiding the fear of pathological normal gaming behavior (40, 41) , and it may thus be more advanced and applicable. The Gaming Hazard Assessment Scale displayed easyto-operate screening applicability in assessing ICD-11's proposed GD functional impairment. Further, the three-dimensional, 18item Gaming Hazard Assessment Scale not only included the diagnostic criteria of DSM-5, but also accounted for the clinical characteristics and game behavior patterns of ICD-11's GD diagnostic criteria, which improves the efficacy of large-scale risk screening tests for GD. Subsequent studies should consider combining the Gaming Hazard Assessment Scale with other approaches (e.g., long-term dynamic assessments of social and cognitive function, somatization disorder, and quality of life assessments), to better inform prognosis and treatment plans (42) (43) (44) . Isolated studies that create multiple tools, produce an incoherent and unconvincing evidence base, so there is a need to provide a more robust theoretical and methodological basis for GD or other behavioral addictions in the population (e.g., a more integrated screening, diagnosis, and assessment system) (18) . The authors intend for the development of the Gaming Hazard Assessment Scale to facilitate further empirical research on GD.

This study had some limitations and unresolved problems. First, the scale was developed specifically for Chinese participants. In this context, and considering the small sample size of the case group limited by the COVID-19 pandemic, this study can be considered a pilot study; the validity and reliability of the tool should thus be tested in future studies that involve more GD patients and other ethnic groups with different gaming cultures. Second, the test-retest reliability is not satisfactory and more than half of the study participants commenced the survey, but did not complete the test-retest. It cannot be feasibly known why the dropout rate was so high, but reasons could include the long time between tests or participants wanting to know what the survey was about and not having any intention to complete it (i.e., commencing the test out of curiosity). Whether participants who did not complete the survey differed from those who did is unclear, but this result should be accounted for when considering the study's results. Third, the discriminant validity of internal factors of each dimension is not very good, thus further studies may adjust the factor or dimension to enhance its discriminant validity. In addition, although mixed study population will benefit the availability of the scale, validity of which will also be decreased. Finally, there are some inherent disadvantages of self-reported online questionnaires, such as memory bias and social expectation bias. Future studies could attempt to add structured interviews to validate this test in a clinical setting and then estimate cutoff scores for screening purposes.

The Gaming Hazard Assessment Scale is a multidimensional GD risk screening tool, which was developed specifically for the Chinese population, and was the first risk screening tool based on the dual diagnostic criteria of DSM-5 and ICD-11. This scale's development accounted for the two existing diagnostic index systems, which were characterized by ease of use, high reliability and validity, and acceptability for certain groups. Referring to IAT, the authors recommend a three categories cut-off when using this scale, which refers to score 0-18, indicating no risk, 19-36 indicating moderate risk, and 37-54 indicating high risk for GD. Considering the limitation of this study, the scale needs further validation, especially in clinical populations, to be helpful for the early identification of gamers with high-risk of GD in large scale. Reliability and validity can be further verified to expand the scope of application of the scale. Although more research is needed to confirm the Gaming Hazard Assessment Scale's ability to test in specific settings (e.g., the presence of comorbidities with other psychiatric disorders or cut off), it has great potential in terms of helping health researchers and practitioners identify potentially high-risk GD cases early. The authors also recommend that future studies should promptly apply the Gaming Hazard Assessment Scale to the clinical evaluation of GD-related risk in Chinese populations.

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Written informed consent was obtained from the individual(s), and minor(s)' legal guardian/next of kin, for the publication of any potentially identifiable images or data included in this article.

Diagnostic and Statistical Manual of Mental Disorders (DSM-5)

Available online at

Available online at

Policy and prevention approaches for disordered and hazardous gaming and internet use: an international perspective

Exploring the prevalence of gaming disorder and Internet gaming disorder: a rapid scoping review

Internet gaming disorder: trends in prevalence 1998-2016

Internet gaming disorder: an emergent health issue for men

Prevalence and risk factors for Internet gaming disorder

Associations of Internet addiction severity with psychopathology, serious mental illness, and suicidality: large-sample cross-sectional study

A Cohort study of patients seeking Internet gaming disorder treatment

Identification of Internet gaming disorder individuals based on ventral tegmental area resting-state functional connectivity

Exploratory metabolomics of biomarker identification for the Internet gaming disorder in young Korean males

Associations between prospective symptom changes and slow-wave activity in patients with Internet gaming disorder: a resting-state EEG study

Dorsal striatal functional connectivity changes in internet gaming disorder: a longitudinal magnetic resonance imaging study

Measuring Internet gaming disorder and gaming disorder: a qualitative content validity analysis of validated scales

Psychometric testing of the personal Internet gaming disorder evaluation-9: a new measure designed to assess Internet gaming disorder

The development of the Problematic Online Gaming Questionnaire (POGQ)

Screening and assessment tools for gaming disorder: a comprehensive systematic review

The conceptualisation and measurement of DSM-5 internet gaming disorder: the development of the IGD-20 test

Portuguese validation of the Internet gaming disorder scale-short-form

Rationale for and usefulness of the inclusion of gaming disorder in the ICD-11

The need for publicly funded research on therapeutic use of psychedelic drugs

Psychometric validation of the Internet gaming disorder-20 test among chinese middle school and University students

Prevalence of Internet gaming disorder and its association with personality traits and gaming characteristics among Chinese adolescent gamers. Front Psychiatry

Gender swapping and socializing in cyberspace: an exploratory study

Associations of social support, friends only known through the internet, and health-related quality of life with internet gaming disorder in adolescence

Caught in the net: how to recognize the signs of internet addiction and a winning strategy for recovery

A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial

Incentive-elicited brain activation in adolescents: similarities and differences from young adults

Development and validation of a nine-item short screening test for ICD-11 gaming disorder (games test) and estimation of the prevalence in the general young population

Individual and interpersonal factors associated with the incidence, persistence, and remission of Internet gaming disorders symptoms in an adolescents sample

Psychiatric symptoms in adolescents with internet addiction: comparison with substance use

The relationship of loot box purchases to problem video gaming and problem gambling

Beyond loot boxes: a variety of gambling-like practices in video games are linked to both problem gambling and disordered gaming

Relationships between severity of Internet gaming disorder, severity of problematic social media use, sleep quality and psychological distress

Characteristics of gamers who purchase loot box: a systematic literature review

Fortnite microtransaction spending was associated with peers' purchasing behaviors but not gaming disorder symptoms

Are we overpathologizing everyday life? A tenable blueprint for behavioral addiction research

Clinical characteristics of diagnosis for internet gaming disorder: comparison of DSM-5 IGD and ICD-11 GD diagnosis

Functional impairment matters in the screening and diagnosis of gaming disorder

Validity, functional impairment and complications related to Internet gaming disorder in the DSM-5 and gaming disorder in the ICD-11

Changes of quality of life and cognitive function in individuals with internet gaming disorder: a 6-month follow-up

Measures of behavioral function predict duration of video game play: utilization of the video game functional assessment -revised

Excessive smartphone use is associated with health problems in adolescents and young adults

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.