key: cord-0905880-hzfzj4p7
authors: Martini, Michael L.; Yaeger, Kurt A.; Kellner, Christopher P.; Hadjipanayis, Constantinos; Shrivastava, Raj; Mocco, J; Morgenstern, Peter F.
title: Student Survey Results of a Virtual Medical Student Course Developed as a Platform for Neurosurgical Education During the Coronavirus Disease 2019 Pandemic
date: 2021-05-28
journal: World Neurosurg
DOI: 10.1016/j.wneu.2021.05.076
sha: 5dafef27727d2f413fd17aa28d77ce0c8f58fd38
doc_id: 905880
cord_uid: hzfzj4p7

BACKGROUND: Before the coronavirus disease 2019 (COVID-19) pandemic, medical students training in neurosurgery relied on external subinternships at institutions nationwide for immersive educational experiences and to increase their odds of matching. However, external rotations for the 2020–2021 cycle were suspended given concerns of spreading COVID-19. Our objective was to provide foundational neurosurgical knowledge expected of interns, bootcamp-style instruction in basic procedures, and preinterview networking opportunities for students in an accessible, virtual format. METHODS: The virtual neurosurgery course consisted of 16 biweekly 1-hour seminars over a 2-month period. Participants completed comprehensive precourse and postcourse surveys assessing their backgrounds, confidence in diverse neurosurgical concepts, and opinions of the qualities of the seminars. Responses from students completing both precourse and postcourse surveys were included. RESULTS: An average of 82 students participated live in each weekly lecture (range, 41–150). Thirty-two participants completed both surveys. On a 1–10 scale self-assessing baseline confidence in neurosurgical concepts, participants were most confident in neuroendocrinology (6.79 ± 0.31) and least confident in spine oncology (4.24 ± 0.44), with an average of 5.05 ± 0.32 across all topics. Quality ratings for all seminars were favorable. The mean postcourse confidence was 7.79 ± 0.19, representing an improvement of 3.13 ± 0.38 (P < 0.0001). CONCLUSIONS: Feedback on seminar quality and improvements in confidence in neurosurgical topics suggest that an interactive virtual course may be an effective means of improving students' foundational neurosurgical knowledge and providing networking opportunities before application cycles. Comparison with in-person rotations when these are reestablished may help define roles for these tools.

-BACKGROUND: Before the coronavirus disease 2019 (COVID-19) pandemic, medical students training in neurosurgery relied on external subinternships at institutions nationwide for immersive educational experiences and to increase their odds of matching. However, external rotations for the 2020e2021 cycle were suspended given concerns of spreading COVID-19. Our objective was to provide foundational neurosurgical knowledge expected of interns, bootcamp-style instruction in basic procedures, and preinterview networking opportunities for students in an accessible, virtual format.

-METHODS: The virtual neurosurgery course consisted of 16 biweekly 1-hour seminars over a 2-month period. Participants completed comprehensive precourse and postcourse surveys assessing their backgrounds, confidence in diverse neurosurgical concepts, and opinions of the qualities of the seminars. Responses from students completing both precourse and postcourse surveys were included.

-RESULTS: An average of 82 students participated live in each weekly lecture (range, 41e150). Thirty-two participants completed both surveys. On a 1e10 scale selfassessing baseline confidence in neurosurgical concepts, participants were most confident in neuroendocrinology (6.79 AE 0.31) and least confident in spine oncology (4.24 AE 0.44), with an average of 5.05 AE 0.32 across all topics. Quality ratings for all seminars were favorable. The mean postcourse confidence was 7.79 AE 0.19, representing an improvement of 3.13 AE 0.38 (P < 0.0001).

-CONCLUSIONS: Feedback on seminar quality and improvements in confidence in neurosurgical topics suggest that an interactive virtual course may be an effective means of improving students' foundational neurosurgical knowledge and providing networking opportunities before application cycles. Comparison with in-person rotations when these are reestablished may help define roles for these tools. INTRODUCTION T he coronavirus disease 2019 (COVID-19) pandemic has caused large-scale disruptions in the daily operations of hospitals and medical schools worldwide. Medical schools have adapted to using remote virtual platforms to achieve student learning objectives and follow social distancing guidelines. Typically, medical students applying for residency positions in competitive surgical subspecialties in the United States have relied on visiting student externships at institutions of interest, as well as in-person interviews, to successfully match. 1 These external rotations also play a fundamental role in medical student education. They provide students with an immersive experience in the field that exposes them to a diverse sampling of cases, program idiosyncrasies, and training cultures that can influence how good a fit the applicant is for a program.

In April 2020, the Society of Neurological Surgeons recommended deferral of all medical student external rotations for the 2020e2021 academic year. 2 We recognized the need for a virtual neurosurgical educational resource for current and future residency applicants. [3] [4] [5] We designed a course that would provide 1) foundational knowledge that subinterns would be expected to learn, 2) bootcamp-style instruction in basic neurosurgical procedures with which subinterns frequently assist, 3) opportunities for interactions among students, faculty, and residents before interviews, and 4) tips for virtual interviews and navigating the interview season. 6 Precourse and postcourse surveys were administered to all course participants.

The primary purpose of this study was to assess the effectiveness of a virtual neurosurgery course in preparing medical students for residency applications and neurosurgery residency through self-assessed measures in student confidence across a wide range of neurosurgical topics.

The virtual neurosurgery course for medical students consisted of 16 one-hour seminars that were conducted biweekly over the course of a 2-month period. The course was sponsored entirely by the host institution's department of neurosurgery without any financial support from external organizations. It was advertised on social media platforms and interest was spread largely by word of mouth. The course was open to any interested student regardless of medical school status or country of origin. An overview of the curriculum is provided in Table 1 . The virtual neurosurgery course for medical students was hosted remotely on the Zoom video conferencing platform ( Figure 1A ) in June and July 2020, divided into evening sessions lasting 1 hour. Students were encouraged to type questions into the online chat room for a quick response from the course director or they could hold their questions until the end to engage the speaker directly ( Figure 1B) . At the beginning of each session, participants were asked to raise their hands if they were final-year students applying into a neurosurgery course this cycle. Of these students, 15e20 were chosen for each session to serve as panelists, to whom the speaker could pose questions to assess core knowledge and reasoning. After each live webinar was completed, the recording of the session was posted online for on-demand viewing.

Three days before the start of the course, registered students were asked to complete a comprehensive survey collecting basic demographic and background information, as well as selfassessments of confidence and knowledge of specific topics within neurosurgery. On completion of the course, participants were asked to complete a follow-up survey (see Supplementary Content for the full precourse and postcourse surveys). All survey responses were kept confidential and anonymized for analysis, as communicated to students through e-mail and on the surveys. Student survey responses were compared before and after the virtual course using paired analyses. In addition, subanalyses comparing responses between U.S. medical students and international medical graduates (IMGs) were performed. Prism 7 (GraphPad, La Jolla, California, USA) was used for statistical analysis. Continuous variables were compared using 2-sided paired t tests. c 2 tests were used to analyze categorical variables.

Statistical significance was determined using P < 0.05. Students were also given an opportunity to provide anonymous freeresponse feedback on the course survey, including suggestions for future improvements.

A total of 595 students originally registered for this course when the registration Web site went live. The online seminars were well attended, with an average of 82 students participating live in each weekly lecture (range, 41e150). Across all seminars, there was a group of approximately 40 students who attended almost every lecture. A total of 32 participants completed both the precourse and postcourse surveys (32/82 average participants each week; 39.0%), with most comprising the students who consistently watched each lecture live or on recording (average, 27/32 viewers per lecture; 84.4%). Among the respondents, the average age was 25.5 AE 0.56 years, 21 (65.6%) were male, and 15 (46.9%) identified as an underrepresented minority in medicine ( Table 2) . Nineteen respondents (59.4%) were IMGs, whereas the other 13 (40.6%) were U.S. medical students. Among the U.S. medical students, 12 (92.3%) had a home neurosurgery program. At the time of the lecture series, most respondents had completed either the third year of medical school (or equivalent) or a research year (34.4% and 28.1%, respectively) during 2019e2020. Most students were planning to complete their neurosurgical subinternships during 2020e2021 (11 students, 34.4%) or 2021e 2022 (11 students, 34.4%) and were completely confident that they would apply into a neurosurgery course (25 students; 78.1%). Twelve students (37.5%) indicated that they would pursue an additional degree before applying. In comparing the demographic characteristics of the U.S. medical student and IMG cohorts, we found that both groups were remarkably similar, with the exception of a greater proportion of IMG students reporting that they would earn another degree during medical school (P ¼ 0.02) and more U.S. medical students stating that they would be undertaking their neurosurgical subinternships during the 2020e2021 academic year (P ¼ 0.01; Table 3 ). Medical school year completed in 2019e2020 0.35 

When asked to identify perceived challenges or obstacles for medical students in acquiring neurosurgical knowledge before subinternships ( Table 2) , lack of formal instruction on neurosurgical decision making in courses/electives was the most commonly identified factor (29 respondents; 90.6%). Regarding resources used to learn about neurosurgical concepts, home institution lectures and the Neurosurgical Atlas (https://www.neurosurgicalatlas.com/) were the most commonly cited resources (71.9% and 68.8%, respectively). When asked what they hoped to gain from the virtual course, respondents identified a desire to learn more about the host institution's neurosurgical residency program (29 students; 90.6%) as well as learning skills and concepts that would be useful during subinternships (87.5%), and gaining networking opportunities (87.5%).

The baseline experiences and confidence levels of participants were assessed in the prelecture survey. Most respondents had approximately 101e250 hours of clinical exposure to neurosurgery before the start of the course (13 respondents, 40.6%). Spine (22 respondents, 68.8%) and cerebrovascular neurosurgery (15 respondents, 46.9%) were the most cited neurosurgical subdisciplines to which participants believed they had the most exposure, whereas pediatric (6 respondents, 18.8%) and functional neurosurgery (7 respondents, 21.9%) were the least cited.

Participants were also asked to rate their confidence level on a scale of 1e10 (1, not confident at all; 10, very confident) with material pertaining to core concepts across various neurosurgical subdisciplines. Participants were most confident in neuroendocrinology (6.79 AE 0.31) and least confident in spine oncology (4.24 AE 0.44), with an average of 5.05 AE 0.32 across all topics. All of the other baseline confidences are summarized in Figure 2A and Table 4 .

Quality ratings on a scale of 1e5 were generally very favorable across all seminars in the series, with most scoring 4 or 5 in each domain. Students generally found the lectures to be interesting and engaging, at an appropriate level for medical students, and with minimal commercial bias. Furthermore, most participants indicated that they would strongly recommend the lectures to their friends and colleagues ( Table 5) .

Postcourse survey responses indicated improved confidence levels across all surveyed topics across neurosurgical subdisciplines. The mean postcourse confidence level across all surveyed topics was 7.79 AE 0.19 out of 10, representing a mean increase of 3.13 AE 0.38 (P < 0.0001). Changes in self-assessed participant confidence levels for each surveyed subdiscipline may be found in Figure 2B . When these trends were re-examined separately in U.S. medical student and IMG cohorts, we found that there were no significant differences in the changes between precourse and postcourse confidence levels reported by participants across nearly all neurosurgical topics surveyed ( Table 6 ). For example, the change in overall confidence across all surveyed topics was 2.99 AE 0.23 for U.S. medical students and 3.23 AE 0.38 for IMGs (P ¼ 0.63).

Other follow-up questions in the postcourse survey asked participants to evaluate the personal impact of the course and what they had gained from it ( Table 7) . On a scale of 1e10 (1, no influence; 10, very influential), the mean course influence on participants' decisions to pursue neurosurgery was 7.41 AE 0.48. Among the respondents, 16 (50.0%) indicated that they were more likely or much more likely to pursue neurosurgery since completing the virtual course, whereas the other 50% of respondents indicated that their decision had not changed. The postcourse survey also showed that the course was well received by participants, because most stated that the course met or exceeded precourse expectations and learning goals (mean score, 4.69 AE 0.10; scale of 1e5) and that the course was valuable to the participants' learning and career development (mean, 4.63 AE 0.13; scale of 1e5). All participants stated that they would recommend the course to other students interested in neurosurgery (32 respondents; 100.0%) and most believed that the course would still be useful after the pandemic (31 respondents; 96.9%).

To our knowledge, this virtual seminar series was among the first offered to medical students during the height of the COVID-19 pandemic. Because of the significant impact that the pandemic has had on neurosurgical practice, 7 education, 6,8-18 residency interviewing, 19 and subinternships, 2,20 we recognized the need for a virtual alternative for medical students to gain exposure to neurosurgery. 6 Accordingly, this course included faculty-led didactic sessions in each neurosurgical subdiscipline, virtual bootcamp-style sessions introducing students to basic neurosurgical procedures, and special sessions focused on practical advice for virtual interviews and navigating a neurosurgical career. It is likely for these reasons that this course received significant interest from students (595 registrants) worldwide within a matter of weeks of being announced, also highlighting a key advantage of the online format of the course. We asked registrants to fill out comprehensive precourse and postcourse surveys that would allow us to evaluate the efficacy of this medium for student learning and identify areas for improvement. The virtual neurosurgery course was well received by participants. The discrepancy between total registrants and survey respondents is partially explained by recorded lecture availability. Some students chose to watch the lectures in recorded format after they were released online. Only students participating in the live webinars were asked to complete both precourse and postcourse surveys.

Most students stated that the seminar series met or exceeded their precourse expectations and believed that the course was valuable to their learning and career development ( Table 7) . Students most commonly believed that they gained knowledge of the host institution's neurosurgery residency program (93.8%), important skills and concepts useful for subinternships (90.6%), and exposure to fundamental topics in neurosurgery (87.5%) through the course. Although most students still believed that they gained networking opportunities (59.4%) and learned more about virtual interviews from the course (65.6%), these rates were below the precourse numbers for what students were hoping to gain from the course (87.5% and 71.9%, respectively), suggesting that a greater emphasis on these areas may lead to improved student satisfaction and benefit in future iterations of the course. This situation may also show challenges associated with a virtual environment, which could be addressed through software adaptations in the right setting.

Before the virtual course, participants most frequently indicated that they had previous clinical or research exposure to spine (68.8%) and cerebrovascular (46.9%) neurosurgery ( Table 2) . These exposures generally mirrored the baseline confidence levels of participants by neurosurgical subdiscipline (Figure 2A) . Although confidence in all neurosurgical concepts and subdisciplines increased significantly by the end of the course, comparatively smaller increases were noted for cerebrovascular and degenerative spine surgery, likely because precourse 

exposures to these areas were already greater among participants than for other subspecialties ( Figure 2B) . It is also possible that these areas also represent topics that are complicated enough such that a single hour-long seminar may not be enough to significantly increase medical student confidence in the topic. Conversely, the most improved domains pertained to virtual interviews, spine tumors, spine trauma, and functional neurosurgery, in which participants all identified as having minimal exposure and confidence at baseline (Figure 2A ). In addition, despite resulting in significant improvements in student confidence in all surveyed neurosurgical topics, the course did not significantly alter participant confidence in applying into neurosurgical residency (P ¼ 0.882), likely because baseline confidence in this area was already so high (precourse, 9.34 AE 0.30; postcourse, 9.41 AE 0.29). We found that changes in confidence levels in neurosurgical topics ( Table 6 ) and the overall interest in neurosurgery and the value of the course to students' learning and career development ( Table 8) were similar between U.S. and IMG participants, suggesting that the course experience for students interested in neurosurgery is likely a shared one. Given the novelty of circumstances necessitating a virtual course format, participant feedback regarding the quality of the audio/video, the presenters, and the presentations was paramount. In previous years, efforts have been made to implement neurosurgery training camps to prepare students for subinternships. [21] [22] [23] [24] [25] [26] These efforts similarly resulted in near-universal improvements in student preparedness and confidence in applicable neurosurgical skills. However, these training camps benefited from being held in person with hands-on components to facilitate student learning and skills acquisition. This virtual medical student course, although it adopted similar learning goals for students, required a different line of thinking to adapt the course to a Web-based format and retain the learning efficacy and skills acquisition that have been reported in previous neurosurgical training camps. Results from our analyses suggest that the functionality of the video conferencing platform was sufficient for facilitating live, interactive seminars and discussions among faculty and students. Participants reported widespread satisfaction with the qualities of the presentations, presenters, and audio/video components ( Table 5) . Furthermore, most students reported that each seminar enhanced their understanding of the topics discussed and that they would recommend the seminars to other interested students.

On the postcourse survey, 96.9% of participants indicated they believed that the virtual course would still be useful for future students after the pandemic. These findings mirror those of previous survey studies that found that resident, fellow, and faculty respondents from numerous neurosurgical programs carried a certain preference for continued use of virtual neurosurgical educational conferences beyond the pandemic. 27 These findings seem to suggest that the current learners with varying levels of experience may even prefer a virtual format for certain aspects of neurosurgical education. These findings also suggest an ongoing need for accessible and high-quality neurosurgical resources geared toward medical students. This need was similarly echoed by participants on the precourse survey ( Table 2) , as well as in previous studies in medical student education in neurosurgery. 1, 3, 4 Accordingly, it is likely that future iterations of this course or others like it will be produced for students in subsequent application cycles. Identifying areas to target for improvement is critical to optimize the usefulness of the course. Several areas for improvement can be identified. First, relatively fewer participants believed that they gained sufficient networking opportunities with students and faculty outside their home institutions ( Table 7) . Incorporation of a dedicated networking session or use of the smaller breakout group sessions may better accomplish this element during the virtual course. Another area of improvement is the use of more objective precourse and postcourse measures of participant evaluation to better gauge learning. Several participants suggested incorporating live procedures so that participants can better appreciate neurosurgical decision making in real time. We anticipate that incorporating some of these suggestions into future iterations of 

this course will further increase its value and appeal to students, even after in-person subinternships are reinstated.

Several study limitations must be acknowledged. First, because this study relied on participant survey data, the results are subject to a response bias, particularly toward providing positive feedback given that the course was offered free of charge. Furthermore, applicants may be concerned that the host institution may use their responses in the process of applicant selection, thereby withholding negative comments. In addition, this study had a relatively low postcourse survey response rate, meaning that some of the opinions of course participants might not have reflected the data collected. Another limitation is that this study did not use an objective measure of participant improvement. Instead, a selfassessed confidence level was surveyed, which is an imperfect measure of improvement and may not directly correlate with knowledge and skill acquisition. In addition, because a 1e10 scale was used to assess confidence, whereas a 1e5 scale was used to assess lecture quality, it is possible that this could have introduced a potential bias into the results. Future use of objective knowledge assessments would address this issue. First-year and second-year medical students were underrepresented in this course. A possible reason for this situation is that the course was advertised within neurosurgical circles on social media that may have less visibility for younger medical students. Increasing outreach to other medical schools, possibly through partnerships or neurosurgical professional organizations, may better promote this course to a wider student audience in the future.

Participant feedback on seminar quality and changes in selfassessed confidence in neurosurgical topics suggest that a virtual Web-based medical student seminar series may be an effective means of providing a high-quality interactive educational environment to improve students' foundational knowledge across neurosurgical subdisciplines for the level of subinternship. In addition, this format can be used to provide networking opportunities before residency interviews. Future studies incorporating cohort match data and more objective measures of concept mastery may further support the findings from this study.

Neurosurgical rotations or clerkships in US medical schools

Neurosurgery elective for preclinical medical students: early exposure and changing attitudes

Improving medical student recruitment into neurological surgery: a single institution's experience

Fostering student interest in neurologic surgery: the University of Pittsburgh experience

Medical student concerns relating to neurosurgery education during COVID-19

Neurosurgical practice during coronavirus disease 2019 (COVID-19) pandemic. World Neurosurg

COVID-19 and neurosurgical training and education: an Italian perspective

Democratizing access to neurosurgical medical education: national efforts in a medical student training camp during coronavirus disease 2019

Virtual Spine: A novel, international teleconferencing program developed to increase the accessibility of spine education during the COVID-19 pandemic

Medicine and surgery residents' perspectives on the impact of COVID-19 on graduate medical education

Live-streaming surgery for medical student education -educational solutions in neurosurgery during the COVID-19 pandemic

A review of anatomy education during and after the COVID-19 pandemic: revisiting traditional and modern methods to achieve future innovation

Virtual learning during the COVID-19 pandemic: a turning point in neurosurgical education

A comprehensive multicomponent neurosurgical course with use of virtual reality: modernizing the medical classroom

Neurosurgical education during COVID-19: challenges and lessons learned in Egypt

Asynchronous, digital teaching in times of COVID-19: a teaching example from general practice

Utilizing virtual interviews in residency selection beyond COVID-19

A virtual COVID-19 ophthalmology rotation

Neurosurgery training camp for sub-internship preparation: lessons from the inaugural course

Medical student neurosurgery training camp: updates following 2019 course expansion

Medical school surgical boot camps: a systematic review

The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies

Fundamental skills for entering neurosurgery residents: report of a Pacific region

A national fundamentals curriculum for neurosurgery PGY1 residents: the 2010 Society of Neurological Surgeons boot camp courses

Virtual education in neurosurgery during the COVID-19 pandemic

Lazarus Effect, Medina, and Pulsar Vascular, is an investor in Blockade, Medina, Lazarus Effect, and TSP Inc., and receives research support from Penumbra. The authors do not have any personal or institutional financial