key: cord-0752728-h4fesn4j authors: An, Hae Jeong; Choi, Jung Suk; Roh, Min; Cho, Hyun Mi; Choi, Eun Ju title: The Korean version of the Virtual Patient Learning System Evaluation Tool: Assessment of reliability and validity date: 2021-08-10 journal: Nurse Educ Today DOI: 10.1016/j.nedt.2021.105093 sha: 917382c66a239fe260c2f39cf367419bd895eec9 doc_id: 752728 cord_uid: h4fesn4j BACKGROUND: Due to the recent spread of coronavirus disease 2019, Korean nursing colleges are increasingly using virtual patient simulation to make up for a lack of available clinical practice in medical institutions. Therefore, an instrument is required to evaluate the effects of the virtual patient learning system in South Korea. OBJECTIVE: To assess the validity and reliability of the Korean version of the Virtual Patient Learning System Evaluation Tool (K-VPLSET). DESIGN: This is a methodological study. SETTINGS: This study was conducted via an online survey for Korean nursing students. PARTICIPANTS: The present study included 373 participants who were 3rd and 4th year Korean nursing students. METHODS: After translating the English version of VPLSET into Korean, a pilot test with a cognitive interview was undertaken to ensure that the meaning of original instrument and appropriateness for Korean nursing students had been retained. The content validity of the K-VPLSET was examined by a panel of six experts. Convenience sampling was used to recruit 3rd and 4th year Korean nursing students, among whom 170 were recruited for exploratory factor analysis (EFA) and 203 for confirmatory factor analysis (CFA). SPSS version 26.0 was used for EFA, whereas AMOS version 22.0 was used for CFA. RESULTS: From the 32 initial items, the final version of the K-VPLSET ultimately included 20 items, with a Cronbach's α of 0.89. EFA identified four factors (“Nursing Knowledge Improvement,” “Clinical Competency Development,” “Confidence in Nursing Performance,” and “Nursing Care Plan Application”) that explained 56.9% of the total variance. CFA confirmed the validity of the instrument. CONCLUSIONS: Our findings confirmed that the K-VPLSET is a valid and reliable instrument for assessing the effects of the virtual patient learning system, through which the quality of e-learning for Korea nursing students can be determined. Virtual patient simulation in nursing education provides access to virtual patients in various and immersive clinical environments from hospitals to communities (Padilha et al., 2019) . Moreover, virtual patient simulation has been suggested as a key educational tool for online learning environments given that it is safe for nursing students to practice their nursing knowledge and skills before being exposed to complex clinical environments (Guise et al., 2012) . Nursing education using virtual patient simulations has been found to improve competency, including prioritization, clinical skills, critical thinking, learning readiness, and decision-making (Berman et al., 2016; Liaw et al., 2015) . The assessment of simulation primarily focuses on the learner's performance in various areas, such as nursing knowledge, academic satisfaction, and clinical competence (Berman et al., 2016; Padilha et al., 2019) . Moreover, emphasis should be placed on not only the learner's performance but also other aspects, including program evaluation, transition of learning into clinical practice, and the impact of learning on patient outcomes (Oermann, 2018) . Therefore, selecting and developing an evaluation instrument for virtual patient simulation education is fundamental and crucial. Despite the continuous emphasis regarding the need for valid and dependable instruments that assess education effectiveness in nursing training (Abuatiq, 2019) , hardly any of the available evaluation instruments assess virtual patient simulation education programs for Korean nursing students. This study therefore aimed to translate the Virtual Patient Learning System Evaluation Tool (VPLSET) into the Korean language and verified the reliability and validity of the Korean version of the VPLSET (K-VPLSET). Virtual patient simulation education is conducted by multiple users (Multi-User Virtual Environment, MUVE) in an online environment that is safer than clinical practice or in-school practice. Therefore, it is possible to individually apply and perform nursing procedures for virtual patients, thereby reducing clinical errors and improving the quality and safety of nursing (Padilha et al., 2019) . In addition, since education is possible anytime, anywhere, as long as Internet use is available, there is an advantage in that it is less constrained by resources because there is less cost for training space, equipment installation compared to traditional high-fidelity simulation practice (Consorti et al., 2012; Dutile et al., 2011; ) . Previous studies about the effectiveness of virtual patient simulation education are as follows. Virtual patient simulation in nursing education was found to increase learner's immersion, improve nursing knowledge, clinical reasoning ability and satisfaction by providing various virtual environments from not only hospitals to local communities (Berman et al., 2016; Padilha et al., 2019) . In addition, as well as positive effects such as reinforcing active learning, determining nursing priority, learning readiness, and improving critical thinking, it is predicted that the virtual patient simulation will be used more and more in the future as an important educational tool in clinical education (Berman et al., 2016) . However, virtual patient simulation studies in Korea mainly focus on standardized patients or scenarios, Studies on the effectiveness of virtual patient simulation education are still in the early stages . Virtual patient simulation education helps improve nursing knowledge, nursing performance ability, learning confidence, communication, critical thinking, and problem solving ability (Kim and Choi, 2018; Kim et al., 2019; Jeon, 2019) . However, due to the absence of a virtual patient simulation evaluation tool, learning outcomes have been evaluated using various indicators (Shin et al., 2019) . Previous studies either used a tool that the researchers had developed according to the study's education content, which they themselves had used (Kim and Choi, 2018) , or the education effect was measured using general evaluation tools after the researcher selected specific variables to evaluate the education effects . Thus, it was determined that in Korea, there is a need for a standardized tool that reflects the characteristics of virtual patient simulation. Abuatiq's (2019) VPLSET for nursing students in the United States (U.S.) was developed to evaluate the effectiveness of using virtual patient simulation. Considering that this tool was developed for nursing students in the U.S., it is necessary to confirm the applicability of this tool in Korean nursing students who have experienced different nursing educational environments. Because, Unlike the U.S., where students use their intuition by recognizing and observing patient information and clues on their own (White, 2011) , the clinical practice environment in Korea has low autonomy in clinical decision-making among nursing students (Yu et al., 2019) . Since they have been educated in different nursing educational environments, there may be differences in how students accept and learn from virtual patient simulation education, so it is necessary to confirm its applicability to Korean nursing students. For these reasons, this study aimed to verify the validity and reliability of the K-VPLSET. This is a methodological study that translated the VPLSET instrument developed by Abuatiq into Korean and adapted it to be compatible with the domestic nursing education program for providing an evaluation instrument based on the virtual patient simulation education program; then, its validity and reliability were tested. Data was collected between November and December 2020. Since nursing education in Korea had established the legal foundation for a 4year Bachelor in nursing system in 2011, all nursing colleges in Korea have used a 4-year bachelor system (Cha, 2020) , wherein 3rd and 4th year students experience clinical practice. Therefore, the study sample consisted of 3rd and 4th year Korean baccalaureate nursing students who completed a practical course according to the domestic nursing curriculum and experienced virtual patient simulation education. Given that the number of participants should be 5-10 times the number of items for exploratory factor analysis (EFA) of the composition validity test (Tabachnick and Fidell, 2007) , for the first sample, we planned to recruit at least 160 participants (i.e., 5 times the number of items based on 32 items). The second sample was recruited considering that at least 200 participants were needed for confirmatory factor analysis (CFA) (Hinkin, 1998) . Therefore, a minimum of 360 participants were required for both EFA and CFA. Considering a dropout rate of 10%, 180 participants were recruited for the first sample and 220 for the second sample. After excluding 10 copies from the first sample and 17 from the second sample using a different educational method other than virtual patient simulation education or duplicate participation, data from 170 participants were obtained for EFA and from 203 participants for CFA. The VPLSET was developed by Abuatiq (2019) based on Quality and Safety in Education competency development for nursing students in U. S. who experienced a virtual patient learning program. The VPLSET consists of 32 items in total divided into four domains, including communication, nursing care plan application, competency development using virtual patient simulation, and performance evaluation. A 5point Likert scale from 0 (not applicable, N/A) to 5 (strongly agree) was used for each item. Cronbach's α of the total 32-item scale was reported to be 0.973. The original version was translated into Korean according to standardized guidelines proposed by the World Health Organization (2015). Forward translation was performed by two bilingual translators, both of whom held a PhD in nursing. Thereafter, back translation into English was performed at Graduate School of Interpretation and Translation. After receiving advice from a Korean literature scholar to confirm the naturalness of Korean phrases and context, the final version was completed. To verify the content validity of the completed K-VPLSET, a group consisting of three experts with >10 years of nursing experience and three nursing professors was formed. The group was asked to rate each scale item using a 5-point Likert scale: 1 (Not relevant), 2 (Needs modification), 3 (Some modification required), and 4 (Relevant and concise). After verifying the validity, the Item Content Validity Index (I-CVI) and the Average of Content Validity Index for Scale, averaging calculation method (S-CVI/Ave) was determined. After confirming the content validity, a preliminary survey was conducted on 10 3rd and 4th year Korean nursing students who satisfied the target selection criteria. The time required to complete the questionnaire, appropriateness of the number of questions, and degree of comprehending the questions were evaluated. A cognitive interview was conducted to evaluate the wording and appropriateness of the translated items and to determine how the respondents understood the intentions of the developer toward a translated tool language (Nápoles-Springer et al., 2006) . Afterward, the researchers discussed all the findings until no substantial disagreements were present, and the Korean version of VPLSET was finalized. The E university ethics committee approved the study (institutional review board number: 202011-0018-01). Participants were recruited from an online community and website for Korean nursing students. The author (HJA) contacted the web community administrator to provide information regarding the study and obtained permission to post the recruitment advertisement that included the online survey website link. Data were collected via an online survey created using Google Forms. Participation was voluntary, and withdrawal was possible at any time. The data obtained in this study will be used only for research, and the personal information of participants will not be disclosed to others. To construct a validity test, EFA and CFA were conducted. When translating tools whose validity has been confirmed in other languages, it is necessary to conduct a CFA (Harrington, 2009) . However, in this study, the contents of learner evaluation and system evaluation were mixed among the items of the original instrument, so an EFA was conducted according to the researchers' judgment. Criterion-related validity evaluates the relationship among the measurement value and the newly developed instrument under the study and the criterion instrument. To verify the criterion-related validity of the K-VPLSET, the "Critical Thinking Disposition Scale" and "Clinical Reasoning Competency Scale" of the preceding literature were used as a criterion instrument. The critical thinking disposition makes it possible to find respondents' problems and solve these problems using the best practices of nursing intervention (Guise et al., 2012) . This can relate to the problems of virtual patients during virtual patient simulation and solving nursing problems. Clinical reasoning is a required skill for the capability of collection and analyzing patient information and making effective decisions regarding how to approach the situation and patient (Berman et al., 2016; Padilha et al., 2019) . Criterion-related validity was verified through correlation analysis; a high correlation represented high criterion-related validity. Reliability was verified by calculating Cronbach's α, which is a coefficient of internal consistency reliability. Cronbach's ⍺ for the items in each factor and for all items as a whole was analyzed for the 20 questions contained in the K-VPLSET. SPSS version 22.0 (SPSS Inc., Chicago, IL, USA) and AMOS 23.0 (Amos Development Corporation, Meadville, PA, USA) were used for statistical analysis. Descriptive statistics were used to summarize the demographic characteristics and item analysis. To evaluate the itemtotal correlation, bivariate correlation analysis was performed. Internal consistency was analyzed using Cronbach's α, whereas the intraclass correlation coefficient for each factor was confirmed. Construct validity was determined through EFA and CFA. In this study, EFA was also conducted when the researchers determined that the validity of the item needed reconfirmation due to the mixing of the content regarding learner and system evaluation. EFA used principal component analysis and varimax rotations. To confirm the structural suitability of the factors derived through this, CFA was conducted to determine the chi-squared statistic divided by the degrees of freedom (CMIN/df), comparative fit index (CFI), goodness of fit index (GFI), standardized root mean square residual (SRMR), and root mean square error approximation (RMSEA). A model fit is acceptable if CMIN/df is <3 (Bentler and Bonett, 1989) , CFI is >0.80 (Hair et al., 2006) , RMSEA is <0.08 (Hair et al., 2006) , GFI is >0.90 (Seyal et al., 2002) , and SRMR is near or <0.08 (Hu and Bentler, 1999) . To verify the criterion-related validity of the K-VPLSET, "critical thinking disposition" and "clinical reasoning competence" were used as the criteria, whereas correlation analysis was performed via Pearson's correlation coefficient. Content validity was determined using I-CVI and S-CVI/Ave (Table 2) . Accordingly, I-CVI was ≥0.8, whereas S-CVI/Ave was 1.0. No questions were removed based on the results of content validity verification, whereas additional opinions from the six experts to enhance the understanding of the participants and the sentence flow were reflected in the modifications to some questions. The results from the preliminary survey that was conducted on 10 nursing students, including 5 3rd year and 5 4th year students, were used. The average time it took to take the survey was 5.05 ± 2.01 min. The question comprehension level was 3.80 ± 0.75 points out of 5 points, and the survey layout was rated as 4.10 ± 0.54 points. As a result EFA, exploratory factor analysis; CFA, confirmatory factor analysis; VSIM, virtual simulation. of the cognitive interview, an N/A alternative, in which the participants answered that the presence or absence of clinical practice experience did not affect their responses, was deleted from the tool as it could make the test ambiguous. In this study, both EFA and CFA were performed for all 32 questions to confirm the possibility for reclassifying unclassified items in the original tool. The verification process for construct validity is presented in Fig. 1 . Accordingly, the Kaiser-Meyer-Olkin was 0.90, whereas the Bartlett sphericity was x 2 = 2554.05 (p < 0.001), which indicated suitability for factor analysis (Kaiser, 1974) . Principal component analysis and varimax rotation revealed that the commonality between the 32 questions was 0.53-0.79, with no question requiring deletion (Costello and Osborne, 2005) . A total of six factors were extracted explaining 61.6% of total variance, with all factors having a loading of 0.37-0.79. Thereafter, CFA was done to confirm the six-factor structure's model fit based on the results of EFA. The results included a chi-square of 912.14 (p < 0.001), CMIN/df of 2.03, GFI of 0.78, SRMR of 0.78, RMSEA of 0.71, CFI of 0.75, TLI of 0.79, and RMSEA of 0.07. Considering that the standardized regression weight should exceed 0.50 (Han and Lee, 2012) , 12 items (number 4, 5, 6, 7, 9, 11, 15, 18, 19, 24, 29, and 30) were removed for not meeting the criteria. EFA was performed again with the final 20 questions, resulting in a Kaiser-Meyer-Olkin of 0.89 and Batlett sphericity of x 2 = 1313.55 (p < 0.001), which indicated suitability for EFA. Through principal component analysis and varimax rotation for 20 questions, four factors were extracted, explaining 56.9% of the total variance. The percentage of variance explained by each factor is presented in Table 3 . Factor loading ranged from 0.44 to 0.85, with all factors containing >3 items. According to each factors, factor 1 was labeled "Nursing Knowledge Improvement" and contained items regarding theoretical knowledge and knowing something. Factor 2 was labeled "Clinical Competency Development" contained items related to actual clinical field competence. Factor 3 was labeled "Confidence in Nursing Performance" and contained items regarding self-confidence in performing tasks. Lastly, factor 4 was labeled "Nursing Care Plan Application" and contained items related to the nursing process. Finally, CFA was conducted to confirm the model fit of the fourfactor structure of the 20 items based on the results of EFA. The result showed an x 2 of 313.71 (p < 0.001), x 2 /df of 1.99, CFI of 0.86, GFI of 0.87, RMSEA of 0.70, and SRMR of 0.81. All criteria except for SRMR reached the recommended levels, with the SRMR almost reaching the recommended level of 0.80, which indicated that the model obtained was suitable (Hu and Bentler, 1999) . Convergent validity assessment to verify the validity of the items constituting the model was performed using construct reliability, with the standard value of 0.70 falling within the satisfactory range of 0.70-0.75 (Yu, 2012) (Table 3) . As shown in Table 4 , the mean score of each item was 3.23-3.86, with a standard deviation of 0.81-1.01, thus indicating no extreme value. The skewness of questions ranged from − 0.89 to − 0.31 and the kurtosis ranged from − 0.61 to 1.02, with no skewed values. The itemtotal correlation coefficient ranged from 0.35 to 0.64, suggesting that no item had a low contribution to the tool (Tabachnick and Fidell, 2007) . Moreover, the alpha if item deleted was 0.88-0.89 (Table 4) . Internal consistency verification of questions with identified construct validity showed a Cronbach's α value of 0.89 for the overall 20 questions in the K-VPLSET, with the internal consistency for each factor shown in Table 4 . Our findings showed that the VPLSET had a significantly positive correlation with critical thinking disposition (r = 0.76, p < 0.001) and clinical reasoning competency (r = 0.74, p < 0.001). These results supported the criterion-related validity of the K-VPLSET. Virtual patient simulation is considered to be a key education tool in nursing education (Abuatiq, 2019) , and the need for a valid, reliable measurement system has always been emphasized in studies (Guise et al., 2012) . This study evaluated the content validity, construct validity, criterion-related validity, and reliability of the K-VPLSET, as well as its compatibility with nursing education circumstances in Korea. This study is significant in that it provided a basis for using the K-VPLSET as a suitable instrument for measuring the effectiveness of virtual patient simulation education. The e-learning system helped me improve my written communication? The e-learning system helped me in understanding (Anatomy, physiology, microbiology, and labs). The e-learning system helped me implement the nursing process? 1.0 6 The e-learning system helped me implement basic intervention skills in preventive and rehabilitative nursing across the life span. 1.0 7 The e-learning system helped me understand physical, behavioral, and social aspects of human development from birth through all age levels. 1.0 8 The e-learning system helped me improve my writing principles. 1.0 9 The e-learning system helped me understand different societal and cultural patterns, and human behavior relevant to health-illness. The virtual patient learning system helped me understand pharmacodynamics. 1.0 The virtual patient learning system helped me understand drug categories and their effects. 1.0 The virtual patient learning system helped me understand and memorize side effects of drugs. 1.0 The virtual patient learning system improved my critical thinking skills related to medication administration. The K-VPLSET initially underwent EFA, CFA, and criterion-related validity analysis to examine for construct validity. Since the original tool was developed for the U.S. nursing students, reactions to measurement questions may differ depending on the characteristics of Korean nursing students and the nursing education environment (Pett et al., 2003) . Therefore, EFA and CFA were performed to confirm the implicit characteristics of the tool without restrictions on the number or structure of factors. EFA for the four subfactors obtained results that were equivalent to those for the original tool. However, given the difference in the number of factors per questions and content between both tools, the second factor (Clinical Competency Development) and fourth factor (Nursing Care Plan Application) were named based on the subfields from the original version, whereas the last two factors (Nursing Knowledge Improvement and Confidence in Nursing Performance) were newly added after considering the significance of questions and naming in the original version. Although the composition of the questions for each of the four subfactors was relatively different from the original version, the questions were grouped in such a way that they were conceptually similar, with the four factors having high explanatory power (56.9%). Therefore, the theoretical substance of the original tool was maintained. Based on the results of CFA, 6 of the 12 deleted questions (number 4, 6, 7, 9, 29, and 30) from the Korean version of the tool were also deleted in the original version during the tool development procedure, with 3 questions (number of 5, 11, 15) belonging to Nursing Care Plan Application of the original version. The last three questions were "I was able to check medications information from the drug book icon. (#18)," "I developed more competency in pharmacology (#19)," and "I was able to apply infection control standards (#24)." According to the scenario of the characteristics for virtual patient simulation learning system, the six deleted questions from the Korean version of the tool can elicit different answers from the respondents depending on the virtual patient learning system to which they were exposed to, regardless of the tool was used. The U.S., where the original version had been developed, has several diverse web-based virtual patient simulation programs, such as Virtual Clinical Excursions®, Virtual World®, vSIM® for nursing, and Shadow Health®. Although the original version reflects the various effects of a diverse virtual patient learning system for online nursing training in the U.S., South Korea has barely started with web-based simulation education since the beginning of 2020 due to the recent coronavirus disease pandemic. Moreover, for domestic nursing colleges, virtual patient simulation learning has progressed using mainly vSIM® for nursing. Therefore, construction differences between tools, the original and Korean version, should certainly exist. This study verified the criterion-related validity of the virtual patient simulation learning system by examining the relationship between critical thinking and clinical reasoning ability, which are essential elements for the responsibilities and nursing performance of nurses as professional health care providers (Berman et al., 2016; Padilha et al., 2019) . The K-VPLSET comprises 20 questions that have a statistically significant positive correlation with critical thinking disposition and clinical reasoning. Therefore, it was determined to be a valid tool for evaluating the effectiveness of the virtual patient simulation learning system for Korean nursing students. The original VPLSET questions serve as evidence-based tools that were developed based on the Quality and Safety Education for Nurses in the U.S. In light of Korea's nursing education, the subfactors of the K-VPLSET were consistent with the program outcome, including Clinical Competency, Nursing Care Plan Application, and Nursing Knowledge Improvement, which is an evaluation standard for Korean nursing education (Korean Accreditation Board of Nursing Education, KABONE). Therefore, the K-VPLSET is considered to be a tool based on Korea's nursing education evaluation system. So far, most evaluations for virtual patient simulation education effectiveness have focused on learner performance, such as learners' nursing knowledge, academic satisfaction, and clinical performance (Berman et al., 2016; Padilha et al., 2019) . However, an evaluation of the virtual patient simulation education effectiveness has to be made in various ways, considering factors such as the learner's outcome and evaluation of the program, such as design characteristics and teaching method (Oermann, 2018) . Therefore, this tool is meaningful in that it verified the validity and reliability of the VPLSET, which is a tool that was developed to evaluate the education effectiveness that is specialized in virtual patient simulation education for Korean nursing students. The present study has some limitations worth noting. We acknowledge that including various virtual patient simulation learning methods would enhance the generalizability of the study. However, this study was limited to VSIM and shadow health given that South Korea had just recently started virtual patient simulation learning. Moreover, given that this study did not establish a cutoff point for the learning effect of virtual patient simulation, follow-up studies are needed to effectively utilize this tool. The currents study confirmed the reliability and validity of the K-VPLSET, which consists of four subordinate factors and 20 questions, for the evaluation of virtual patient simulation learning. The instrument uses a 5-point Likert scale and has scores ranging from 20 to 100, with higher scores implying higher virtual patient simulation learning effectiveness. This instrument can be easily used by college nursing students due to the few number of questions and has the advantage of being an instrument that assesses various aspects of learning performance. Moreover, it is also meaningful that an instrument designed to evaluate the effectiveness of the virtual patient simulation learning system abroad had been evaluated for its applicability in South Korea's nursing education due to its reliability and validity being verified, which indicates its applicability. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. This study was approved by the Ethics committee of Ewha Womans University; approval code: ewha-202011-0018-01. E-learning in nursing: tool development for evaluating virtual patient learning systems Significance tests and goodness of fit in the analysis of covariance structures The role for virtual patients in the future of medical education The substantial content and current significance of revised ?Higher Education Act? for the unified four-year bachelor in nursing Efficacy of virtual patients in medical education: a meta-analysis of randomized studies Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis Virtual clinical education: going the full distance in nursing education What can virtual patient simulation offer mental health nursing education? Multivariate Data Analysis Nursing and Health Statistical Analysis Confirmatory Factor Analysis A brief tutorial on the development of measures for use in survey questionnaires Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives Exploring study on virtual reality utilization strategies in scenariobased nursing simulation: an integrative review An index of factorial simplicity The effect of virtual simulation in nursing education: an application of care for acute heart disease patients Effects of a virtual reality simulation and a blended simulation of care for pediatric patient with asthma Designing and evaluating an interactive multimedia web-based simulation for developing nurses' competencies in acute nursing care: randomized controlled trial Teaching in Nursing and Role of the Educator: The Complete Guide to Best Practice in Teaching, Evaluation, and Curriculum Development Clinical virtual simulation in nursing education: randomized controlled trial Making Sense of Factor Analysis: the Use of Factor Analysis for Instrument Development in Health Care Research Determinants of academic use of the internet: a structural equation model Educational characteristics of virtual simulation in nursing: an integrative review Using Multivariate Statistics The Development and Validation of a Tool to Measure Selfconfidence and Anxiety in Nursing Students While Making Clinical Decisions Process of translation and adaptation of instruments The Concept and Understanding of Structural Equation Modeling Reliability and validity of korean version of nursing students' anxiety and self-confidence with a clinical decision making scale Supplementary data to this article can be found online at https://doi. org/10.1016/j.nedt.2021.105093.