key: cord-0970092-x10chiz6 authors: Lowry, Bryce; Johnson, Garrett G. R. J.; Vergis, Ashley title: Merged virtual reality teaching of the fundamentals of laparoscopic surgery: a randomized controlled trial date: 2022-01-03 journal: Surg Endosc DOI: 10.1007/s00464-021-08939-4 sha: 7123be33f720873e61c0fccc36c2434c33ca7170 doc_id: 970092 cord_uid: x10chiz6 BACKGROUND: The COVID-19 pandemic challenges our ability to provide surgical education, as our ability to gather and train together has been restricted due to safety concerns. However, the importance of quality surgical education has remained. High-fidelity simulation platforms have been developed that merge virtual reality video streams to allow for remote instruction and collaboration. This study sought to validate the use of a merged virtual reality (MVR) platform for the instruction and assessment of the fundamentals of laparoscopic surgery (FLS) skills. METHODS: This was a prospective randomized controlled non-inferiority study. Thirty participants were randomized between three groups: The standard group received in-person instruction and expert feedback, the experimental group received identical training via the MVR platform, and the control group practiced on their own, but received no feedback. All participants were pre-tested for baseline performance at the beginning of the study. Change in performance was evaluated immediately after training and one month later for retention. Ordinary one-way analysis of variance was used to evaluate the effects of time, group, and time-on-group. RESULTS: The pre-test confirmed baseline homogeneity between the groups. MVR was non-inferior to standard in-person training for total FLS times on either the post-test (p = 0.632) or the retention test (p = 0.829). Performance was also identical between MVR and standard training groups for each of the individual FLS tasks. Each group improved significantly in nearly all tasks after practice; however, the standard and MVR training groups both improved significantly more than controls for the ligating loop, extracorporeal suturing, intracorporeal suturing, and total FLS task training but did not reach statistical significance for peg transfer and pattern cut tasks. CONCLUSION: This randomized, controlled trial has demonstrated the use of an MVR platform as non-inferior to in-person instruction for the FLS program, forming the foundation for future work on remote instruction and collaboration. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00464-021-08939-4. and evaluating laparoscopic skills and has been endorsed by the American Board of Surgery [5] [6] [7] . Recently, new high-fidelity mobile simulation platforms have been developed that merge real-time views from local and remote video streams, allowing for live, interactive instruction, and collaboration by distance [8] . These so called "merged virtual-reality" (MVR) platforms have seen wide acceptance in a variety of commercial and educational environments. For example, MVR has been used to subjectively enhance patient engagement in postoperative care following orthopedic surgery [9] and to facilitate neurology virtual consultation [10] . It has also been used as an educational tool in a pilot study to train infant car seat installation [11, 12] . To our knowledge, these platforms have not been used as an instructional tool for medical education, but are a promising candidate for adaptation to laparoscopic surgery training and practice. Use of MVR in surgical education could allow for dissemination of robust, high-fidelity technical skills training even when learners and educators are not in close physical proximity. This is of particular importance during a pandemic, when physical distancing and isolation are required. However, even as the current safety restrictions begin to relax, MVR is potentially an asset for enhancing connectivity between geographically diverse populations. As FLS moves toward becoming an international standard, MVR could help ensure dissemination of laparoscopic skills worldwide [8] . Finally, MVR could be used in patient care environments. For example, a geographically remote surgeon could initiate a virtual intraoperative consultation with a tertiary-care center specialist for difficult cases. The subspecialist could view the entire operative field, annotate relevant anatomy on the referring surgeon's screen, or even virtually put his or her hands into the field [8] . However, no comprehensive investigations have yet sought to support MVR adoption for laparoscopic teaching. Herein we describe a non-inferiority randomized comparative study to obtain validity evidence for the use of an MVR platform (HelpLightning™, 2015-2021 Help Lightning Inc., Alabama, USA) in laparoscopic skills education. The study purpose was to compare novice laparoscopic skills acquisition between this MVR platform and conventional in-person teaching. We hypothesized that training using MVR is non-inferior to standard in-person methods, thus laying the foundations for future research into MVR applications in laparoscopic surgery. This was a prospective randomized controlled parallel group non-inferiority study with three arms. Participants were assigned equally into one of three groups, each with two hours of supervised practice. Group 1 (control) received no formal mentorship or feedback; group 2 (standard) received in-person instruction and expert feedback; and group 3 (experimental) received remote instruction and feedback via the MVR platform (Fig. 1) . Approval was obtained from the Health Research Ethics Board and all participants signed an informed consent form (protocol #HS20908 (H2017:218) . This study is reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) [13] . Messick's validity framework was used to guide this research. This framework has become the gold standard for evaluation of validity evidence in performance assessment and is supported by multiple education and research organizations [14] . In brief, this framework uses five sources of validity evidence: content, response process, internal structure, relations with other variables, and consequences of the Inc., Alabama, USA) system. A The learners' set-up, with the learner grasping both needle drivers inside of the simulation box. Virtually overlaid Maryland forceps and drawn red arrow belongs to the remote proctor. B The remote proctor's set-up assessment or test. A more detailed review of validity in surgical education is beyond the scope of this paper, therefore we refer the reader to a recent systematic review on this topic [15] . According to this framework, we attempted to establish initial validity evidence for MVR by measuring participant performance in "relation to other variables." We compared novices trained using three different modalities and compared their performance using the previously established FLS tasks. Medical students and pre-medical (undergraduate students with an interest in medicine) students from the University of Manitoba were invited to participate. Eligibility criteria included no previous exposure to FLS training and no laparoscopic surgery experience. All participants viewed a demonstration of the five FLS tasks with instructions for each read from a preformed script. The tasks were as follows: (1) peg transfer: six plastic objects are transferred from one side of a pegboard to the other. (2) Precision cutting: a circular pattern is cut along a pre-marked line. (3) Ligating loop: a loop is placed at a pre-marked line on a foam appendage. (4) Suture with extracorporeal knot: a long suture is passed through two targets on a Penrose drain and three knots tied in an extracorporeal fashion are placed using a knot pusher. (5) Suture with intracorporeal knot: a short suture is used and three knots are tied in an intracorporeal fashion. Following initial instruction and demonstration, the participants completed a pre-test to ensure baseline homogeneity. After the pre-test, Group 1 (control) underwent a 2-h self-practice session with no further instruction or feedback. Group 2 (standard) underwent a 2-h session consisting of in-person instruction and expert summary feedback. Group 3 (experimental) also underwent a 2-h session, with instruction and feedback provided via our MVR platform. To guard against potential confounders between the standard and experimental groups, the nature of instruction provided was purposefully semi-scripted and nearly identical between groups. For example, the instructor did not physically intervene in either group to demonstrate additional tips and tricks, aside from pre-determined scripted verbal feedback. Furthermore, the same instructor mentored both groups. Immediately following these sessions, all groups completed a post-test. One month later, they completed a retention test. The study protocol was pilot tested on a small number of non-participant students to ensure flow and homogeneity of instructions prior to study commencement. Standard FLS assessment metrics were measured for each of the five tasks during the testing phases. The FLS scoring system is previously described and supported by past validity evidence [5-7, 16, 17] . In brief, the FLS scoring system uses a cut-off time assigned for each of the tasks. A raw score is calculated by subtracting the total time taken to complete a task in seconds, plus any predefined penalty points. A higher score indicates greater performance. Upon initial pilot testing, we noted that even the trained participants would often obtain a score of zero on both the pre-and the post-tests. We hypothesized that equivalency between scores of zero is more easily demonstrated and could falsely conclude equivalency of each training regimen. Therefore, to increase the sensitivity of our non-inferiority study, total time to task completion was used rather than the FLS scoring system. Primary outcomes were the change in total FLS times from pre-test to post-test for each group, and differences between groups. Secondary outcomes were the change from pre-test to post-test times for each of the individual FLS tasks and skills retention. Skill retention was calculated by subtracting the post-test time from the retention test time. Normality of data was tested using Shapiro-Wilk test and visually using Q-Q plot and histogram. Data were normally distributed, so ordinary one-way analysis of variance (ANOVA) was used to evaluate the effects of group (control, standard, MVR) on test time. Pairwise comparisons were done using post hoc Tukey HSD. Paired t tests were used to assess the amount of skill acquisition and retention that occurred within each group. All data were analyzed using SPSS for Windows (Version 27, SPSS Inc., Chicago, IL). Statistical significance was set a priori at a p < 0.05. Sample size calculation was performed using a non-inferiority margin set at 10% of the mean improvement in expected total FLS scores (post-test minus pre-test) after training using the standard regimen. The sample size was calculated by applying a single-tailed alpha of 0.05 and a power of 0.80. Calculation was based upon previous FLS tasks of novice trainees showing a mean improvement in mean total scores of 281.6 points with a standard deviation (SD) of 25 points [18] . Ten participants were required in each training arm. For this sample size calculation, mean and SD were converted from median and interquartile range data using methods reported by Wan et al. [19] . Three participants dropped out near the beginning of study recruitment, therefore they were replaced with three additional volunteers to maintain the target sample size. Randomization was done using blocked computer-generated random sequence. Due to the nature of the study design, blinding was not possible. Randomization, assessment, and participant coaching were all administered by a single investigator. Thirty-three individuals agreed to participate and were assigned equally to one of the three study interventions (Fig. 2 ). Thirty participants completed study follow-up. Seventeen participants were female and 16 were male. Participants were all students with no laparoscopic surgical experience (simulated or otherwise). Eleven participants were medical students and 22 were pre-medical students. Three participants (one in each group) did not return for retention testing within the study timeline due to scheduling conflicts and were therefore excluded. The pre-test undertaken by all participants confirmed baseline homogeneity between groups (Table 1) . There were statistically significant improvements in performance for all three groups from comparisons between pre-test and post-test scores for each of the FLS tasks and total task scores, except for in the ligating loop in the control arm (p = 0.051) (Supplement 1). Pairwise comparisons between groups following the post-test are demonstrated in Fig. 3 . There were no differences between the standard and MVR groups for any of the individual FLS tasks. Trained groups (Standard or MVR) were each significantly better than controls for the ligating loop, extracorporeal suturing, intracorporeal suturing, and total FLS task completion time but not for peg transfer and pattern cut tasks. There were no significant differences seen between the standard and MVR groups for skill retention for any of the FLS tasks or the total FLS task completion time (Fig. 4) . Trained groups (Standard or MVR) were each significantly better than controls for the extracorporeal suturing task and total FLS task completion time, but not for peg transfer, pattern cutting, or ligating loop. The MVR group compared to control were significantly different for intracorporeal suture retention, but not the standard group compared to control. Advances in virtual simulation, such as the MVR Help-Lightning™ platform, hold much promise for surgical education as it confronts new and ongoing challenges. However, proper evaluation of any new approach is imperative before incorporating into a surgical training program. Gallagher et al. found that for virtual reality to be successful in improving surgical skills, it must be backed up by validity evidence [20] . Therefore, according to the Messick's validity framework [14, 15] , we provide data which fall under the "relations to other variables" category of validity evidence. According to this framework, there are four other possible sources of validity evidence, which we did not explicitly assess: content, response process, internal structure, or consequences. However, by building upon the substantial work of those who have previously rigorously studied the FLS evaluation system [5, 6] , we are able to assess the equivalence of two training modalities. In this randomized controlled trial, we have demonstrated that training and assessment of FLS tasks by MVR are non-inferior to in-person training on a standard laparoscopic box trainer, both in the immediate posttraining period and on a skills retention test 4 weeks later. Significant improvement of skills occurred for both training regimens when compared to the control group. Our study provides important data in support of the validity of a mobile MVR platform for laparoscopic skills training, demonstrating it to be a non-inferior alternative to in-person participation. This study used the tasks and assessment strategies from the FLS program. FLS is one of the most extensively studied laparoscopic surgical skills training programs and is supported by a breadth of validity evidence [5-7, 16, 17] . FLS has known standards and widely published expected training improvements, which allowed us to power our study with relative accuracy. We are also able to compare our results to a wide breadth of FLS literature. For example, the observed improvements in laparoscopic skills in the present study closely correspond with those reported previously. In all practice arms, the greatest improvements were seen in the suturing and pattern cutting tasks, compared to less improvements seen in the ligating loop or the peg transfer. These are widely observed phenomenon in the FLS literature, further supporting the validity of MVR as a training method compared to standard approaches [17, 18] . The effects of expert instruction in both the MVR and standard arms were most pronounced for enhancing speed on the suturing tasks. Suturing is widely hypothesized to be the most difficult task and expert instruction and feedback have previously been shown to speed acquisition of this skill in novices [21] . We did, however, modify the FLS assessment strategies employed compared to traditional FLS assessment metrics. For feasibility purposes, two hours of dedicated practice were chosen for the instruction component. This was based on past literature, which demonstrated that two hours of training led to significant and measurable improvements in laparoscopic suturing performance [22, 23] . A longer training session we felt would lead to more attrition or difficulty recruiting participants [24] . On pilot testing, we noted, however, that the majority of untrained participants and many participants after 2 h of training would often attain a score of 0 when assessed using traditional FLS metrics, despite demonstrating marked objective improvements in overall task performance. We felt that equivalency between scores of zero is more easily demonstrated and could falsely conclude equivalency of each training regimen. For example, the maximum time allocated to total FLS tasks to score above zero is 1800 s. Even without tabulation of error scores, many participants in all groups would have obtained a score of zero on many of the tasks (Figs. 3, 4) . Therefore, the decision was made to use overall task completion time rather than a calculated FLS performance score. Our findings support previous work on tele-simulation in laparoscopic surgery and provide important evidence for future implementation of such a teaching program into a surgical training. Prior to the present study, the use of remote tele-simulation had already seen preliminary successes in administering the FLS skills test. The FLS certifying exam Fig. 4 Task completion time for post-test (light colors) and retention test (dark colors) for the control, standard, and merged virtual reality (MVR) groups for all Fundamentals of Laparoscopic Surgery (FLS) tasks. p values indicate comparisons for relative difference between groups using † 1-way ANOVA and pairwise Tukey HSD tests (for significant ANOVA). EC extracorporeal, IC intracorporeal has now been taken by physicians from more than 20 countries worldwide, yet there are very few international test centers, necessitating costly and time-consuming travel. To this end, O'Krainec et al. undertook a study in 2016 in which 20 participants completed the FLS test while being examined by both an on-site and a remote proctor [25] . They found excellent inter-rater reliability between the two proctors and high participant satisfaction with the process. This study was an important first step in addressing the dilemma of geographical diversity, proving the efficacy of remote proctoring and supporting our study in taking the next step of both instructing and assessing the FLS skills remotely. Such efforts have brought resource-rich and resource-poor learners and physicians together from around the world and effectively put them in the same room, marking a breakthrough in the accessibility of the FLS program and the benefits it brings to physicians and their patients. MVR has the potential to further enhance existing distance skills training, such as the FLS tele-simulation project developed by the University of Toronto, in which Canadian surgeons proctored surgeons in resource-limited countries allowing them to develop technical proficiency in basic laparoscopic skills [26] . Tele-mentoring has also seen successful adoption in collaboration between expert physicians. As early as 1998, Rosser et al. at the Yale University School of Medicine helped guide laparoscopic cholecystectomies in Ecuador [27] . Schlachta et al. from the Schulich School of Medicine at the University of Western Ontario in London, Ontario, successfully incorporated tele-mentoring in laparoscopic colon resections between their tertiary center and surrounding community hospitals [28] . More recently, a multiinstitutional initiative set forward by SAGES demonstrated tele-mentoring to be a practical, feasible, and successful method of instruction for laparoscopic sleeve gastrectomy [29] . Despite these important steps, until now no prospective randomized controlled trials had demonstrated the efficacy of tele-simulation or tele-mentoring in both the instruction and the assessment of the FLS skills, which form the foundation of any laparoscopic surgery. Considering the direct impact that remote virtual intraoperative consultations can have on patient care, this is an exciting area of future research for MVR technology. Some important adjuncts already in development include devices for automated surgical skills assessment. These tools can capture data in real-time, such as surgeon's hand-motion efficiency and measures of manual dexterity. These data could potentially be fed back to surgeons or trainees and used to enhance performance [30] . When used in conjunction with MVR tele-mentoring, these novel technologies could prove revolutionary in remote surgical proctoring, education, and skills development. Furthermore, there appears to be a demand for such innovations. A large descriptive study conducted in association with the American College of Surgeons in 2017 surveyed more than 150 rural surgeons in which 79% reported that use of tele-simulation would be helpful in their practice and 68% expressed interest in virtual intraoperative consultation for unexpected findings [31] . There are two important limitations to this study. First, our study was powered to detect a clinically significant difference in total FLS test scores of > 10% between arms. However, the sample size was not powered a priori to adequately detect differences for each of the individual FLS tasks. Therefore, we can only conclude in the non-inferiority between training arms on the total FLS scores and not scores on each of the individual FLS tasks. Despite this limitation, improvements were observed for the majority of FLS tasks when pairwise comparisons were made between controls and each of the training arms. Improvement in the peg transfer task and pattern cutting task were numerically higher than controls, but were not statistically significant. It is possible that with more power, a difference in performance for these tasks would have been measured as well. Second, this study was completed at an academic center equipped with reliable, high-speed internet connectivity that facilitated a strong connection between the mentor and mentee while using the MVR platform. Without such a powerful internet connection, this remote collaboration would have been potentially more difficult. This factor limits applicability of MVR to rural and remote settings where internet quality can be poor. Applicability to developing countries may also be an issue. However, ongoing globalization and advances in fiber and wireless internet such as LTE and 4G are expanding high-speed internet access more broadly [32] . Therefore, we expect in the not-too-distant future that high-speed internet will be available near universally, potentially facilitating the dissemination of distant virtual tele-mentoring. In summary, our prospective, randomized, controlled study has demonstrated that the use of a mobile MVR platform is a non-inferior method of teaching and assessing the FLS skills compared to in-person instruction. In doing so, it builds on the existing literature on remote proctoring and collaboration, provides evidence for deeper integration of remote instruction of laparoscopic surgical skills, and paves the way for inclusion of an MVR platform in surgical education. With the aid of future work, including the development of automated surgical skills assessment and virtual collaboration, MVR may be a method of safely enhancing global connectivity, while remaining physically distant. The online version contains supplementary material available at https:// doi. org/ 10. 1007/ s00464-021-08939-4. What is wrong with the training of general surgery Simulation in paediatric urology and surgery Covid-19 leaves surgical training in crisis Virtual reality training compared with apprenticeship training in laparoscopic surgery: a meta-analysis Trends and results of the first 5 years of Fundamentals of Laparoscopic Surgery (FLS) certification testing Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery Feasibility of remote administration of the Fundamentals of Laparoscopic Surgery (FLS) skills test Commercial videoconferencing for use in telementoring laparoscopic surgery Telemedicine with mobile devices and augmented reality for early postoperative care Letter: emergency response plan during the COVID-19 pandemic: The University of Alabama at Birmingham Experience Teaching infant car seat installation via interactive visual presence: an experimental trial Virtual car seat checks during a pandemic CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials Standards for educational & psychological testing, 2014th edn Gathering validity evidence for surgical simulation: a systematic review Does laparoscopic simulation predict intraoperative performance? A comparison between the Fundamentals of Laparoscopic Surgery and LapVR evaluation metrics The effect of blocked versus random task practice schedules on the acquisition and retention of surgical skills Basic laparoscopic skills training is equally effective using 2D compared to 3D visualization: a randomized controlled trial Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range Prospective, randomized assessment of the acquisition, maintenance, and loss of laparoscopic skills The use of simulation in the acquisition of laparoscopic suturing skills Teaching basic video skills as an aid in laparoscopic suturing Initial laparoscopic basic skills training shortens the learning curve of laparoscopic suturing and is cost-effective Integrating simulation into a surgical residency program Remote FLS testing in the real world: ready for "prime time Surgical simulation in Africa: the feasibility and impact of a 3-day fundamentals of laparoscopic surgery course Use of mobile low-bandwith telemedical techniques for extreme telemedicine applications A model for longitudinal mentoring and telementoring of laparoscopic colon surgery Project 6 Summit: SAGES telementoring initiative Automated methods of technical skill assessment in surgery: a systematic review Rural surgeons would embrace surgical telementoring for help with difficult cases and acquisition of new skills Optical wireless communications for broadband global internet connectivity: fundamentals and potential applications Disclosures Dr. Bryce Lowry, Dr. Garrett Johnson, and Dr. Ashley Vergis have no conflicts of interest or financial ties to disclose.