key: cord-0784493-eqswzzjh
authors: Byrnes, Yasmeen M.; Luu s, Neil N.; Frost, Ariel S.; Chao, Tiffany N.; Brody, Robert M.; Cannady, Steven B.; Rajasekaran, Karthik; Shanti, Rabie M.; Newman, Jason G.
title: Evaluation of an interactive virtual surgical rotation during the COVID-19 pandemic
date: 2021-04-24
journal: World J Otorhinolaryngol Head Neck Surg
DOI: 10.1016/j.wjorl.2021.04.001
sha: c4fec58961e7472519d3ae11df7c6c66a69cc929
doc_id: 784493
cord_uid: eqswzzjh

OBJECTIVE: To evaluate medical student and attending surgeon experiences with a novel interactive virtual Otolaryngology - Head and Neck Surgery (OHNS) medical student elective during the COVID-19 pandemic. STUDY DESIGN: A virtual OHNS elective was created, with three components: (1) interactive virtual operating room (OR) experience using live-stream video-conferencing, (2) telehealth clinic, (3) virtual didactics. SETTING: OHNS Department at the University of Pennsylvania (May 2020 to June 2020). METHODS: Six medical students from the University of Pennsylvania; five attending otolaryngologists. Two surveys were designed and distributed to participating medical students and attending surgeons. Surveys included 5-point Likert scale items, with 1 indicating “not at all” and 5 indicating “very much so”. RESULTS: Response rate was 100% for both surveys. Students on average rated the educational value of the telehealth experience as 4.2±1.2, and the virtual OR experience as 4.0±0.6. Most students (n=5, 83%) indicated that they had enough exposure to faculty they met on this rotation to ask for a letter of recommendation (LOR) for residency if needed, while attending surgeons had an average response of 3.0±1.0 when asked how comfortable they would feel writing a LOR for a student they met through the rotation. A majority of students (n=4, 67%) felt they connected enough with faculty during the rotation to ask for mentorship. Half the students (n=5, 50%) indicated that the rotation allowed them to evaluate the department’s culture either “extremely well” or “somewhat well”. CONCLUSIONS: Overall, participating students described this innovative virtual surgical rotation as an educationally and professionally valuable experience. With the continued suspension of visiting student rotations due to the COVID-19 pandemic, this virtual model may have continued relevance to medical education.

The novel coronavirus 2019 (COVID-19) pandemic has disrupted every facet of the United States (US) healthcare system, including undergraduate medical education. On March 17, 2020 , the American Association of Medical Colleges released recommendations that in-person clinical learning for medical students across the country should be halted. 1 This suspension continued from March 2020 until June 2020, when hospitals slowly began to re-integrate students into clinical care teams. 2 During this prolonged suspension of clinical rotations, many institutions developed virtual curricula for medical students. [3] [4] [5] Despite the return of medical students to clinical rotations at their home institutions, visiting student learning opportunities (VSLOs), rotations which for many specialties usually play a J o u r n a l P r e -p r o o f critical role in the residency application process, 6,7 remain indefinitely suspended. 8 Hence, research examining the implementation and effect of virtual medical education remains of high importance. For surgical specialties, these virtual curricula have included learning tools such as targeted readings, case-based discussions, simulated or real patient interactions, and instructional surgical videos. [9] [10] [11] [12] [13] However, literature evaluating the effectiveness of these interventions is limited, and few studies have addressed the significant problem of lack of student access to the operating room (OR) for interactive learning. One study described a virtual surgical subinternship that included simulated surgical skills workshops, 13 and another presented a case report of livestreaming from the OR for medical student education, 14 but neither have yet evaluated the educational value of these virtual experiences from student or faculty perspectives.

The authors present a single institution's critical evaluation of an innovative virtual Otolaryngology -Head and Neck Surgery (OHNS) medical student elective. This previously described virtual curriculum emphasizes active participation and includes an innovative and interactive live-stream OR experience, along with virtual clinic and didactic components. 15 This survey-based study aims to analyze the value of our virtual curriculum to medical students, both educationally and professionally, as they choose a specialty and prepare to apply to residency.

Virtual elective structure A 2-week virtual elective in OHNS was offered to medical students at our institution from May 2020 to June 2020. Four sessions of the course were completed, each with 1-2 students enrolled, for a total of 6 students over two months. Five attending surgeons participated in the course. The course comprised three major components, which we have previously described J o u r n a l P r e -p r o o f in greater detail: 15 (1) Virtual OR: Attending surgeons wore a head-mounted GoPro® camera which live-streamed video of the surgical field via a Health Insurance Portability and Accountability Act (HIPAA) compliant, video-conferencing platform, BlueJeans (BlueJeans, Verizon Enterprise Solutions LLC, Mountainview, California). Students on the video conference were then able to, in real time, view the operation and converse with attending surgeons, who were wearing wireless headphones connected to the call.

(2) Telehealth: Students joined attending surgeons' video-conferencing visits with patients, conducted via BlueJeans or Doximity (Video Dialer Beta, Doximity Inc., San Francisco, California).

For some visits, students performed an independent history and virtual "physical", presented the patient to the attending, and then returned to the video call with the attending for the full visit. 

The value of the course was determined by six (6) outcome measures, assessed both through student perceptions (items 1-3 below) and attending surgeon perceptions (items 4-6 below). See Appendix A for an outline of which survey items correspond to which of these six outcome measures.

• Student-related outcome measures: ① Feasibility, ② Educational benefit, ③ Professional benefit.

• Attending-related outcome measures: ④ Feasibility, ⑤ Ability to provide professional J o u r n a l P r e -p r o o f support to students, ⑥ Overall impressions Of note, content-based learning objectives were provided to students upon enrollment.

These served as a guidepost for student learning throughout the course. However, these learning objectives should be distinguished from the course objective, which was to introduce students to the subspecialty of Head and Neck oncologic surgery and to offer the opportunity to interact directly with attending surgeons through live-streamed surgeries, telemedicine, and case-based discussions. Students 'perception of the educational value of the course components, as assessed by our study, was likely based on the content-based learning objectives, which were:

(1) Understand the risk factors and epidemiology of head and neck cancer (2) Create a management algorithm for patients presenting with a neck mass (3) Identify key portions and pitfalls of head and neck surgical operations (i.e. neck dissection, parotidectomy)

Two surveys were created to evaluate student and attending experiences, respectively [see Appendices B and C]. The surveys were designed to assess the outcome measures of the study (see section 2.2), and underwent at least 3 rounds of edits with different members of the research team to ensure they achieved this goal. For some questions, a 5-point Likert scale was used with 1 indicating "not at all" and 5 indicating "very much so". Optional comment boxes were included in both surveys for gathering of qualitative data. Surveys were distributed to students and attendings by email. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Pennsylvania. 16, 17 Survey data were analyzed using mean and standard deviation (SD). All statistics were calculated using Numbers [version 

Response rate was 100% for both the attending (n=5) and student (n=6) post-course surveys. Survey results regarding student and attending perceptions of course components (telehealth, operating room, didactics, tumor board) are presented in Table 1 .

During the telehealth portion of the course, all but one student (83%, n=5) had the opportunity to perform an independent history and virtual "physical exam" and formally present the patient to the attending surgeon prior to observing the attending's visit with the patient. All students (n=6, 100%) reported an increase in their level of comfort with telehealth from before beginning the course to after completing it. Reported comfort levels in telehealth increased from a mean of 2.3±0.8 to a mean of 4.0±0.6. Students on average rated the educational value of the telehealth experience as 4.2±1.2 and attending surgeons on average ranked the experience of having medical students in telehealth clinic as 3.8 ±1.3.

When asked for additional comments, students voiced particular appreciation for the telehealth component of the course. One commented that "'seeing' the patient on my own first, and then presenting to the attending…… really helped with my comprehension", while another similarly noted that "the best learning experiences were seeing patients on my own and presenting to the attending…… this prepared me better……to see how the attending proceeded with the patient". A third student remarked that this rotation was a great "primer" for their subsequent in-J o u r n a l P r e -p r o o f person OHNS elective. Students described the experience as "enjoyable" and "educational".

All students (100%, n=6) reported that, for the virtual operating room (OR) experience, their level of interaction with the attending surgeon was "about the right amount". On average, students rated the educational value of the virtual OR experience as 4.0 ±0.6. All but one student (n=5, 83%) were either "somewhat likely" or "very likely" to recommend the virtual OR experience to a classmate. Attending surgeons on average ranked the experience of having medical students in the virtual OR as 4.0±1.3.

The majority of students (n=5, 83%) reported that the amount of time spent on virtual didactics was "just right"; the remaining student thought it was "too little" (n=1, 17%). Out of the three didactic components (student-only didactics, resident didactics, and tumor board), a majority (n=5, 83%) enjoyed student-only didactics the most. Within the student-only didactics, 100% of students (n=6) preferred the interactive case discussion format over the lecture format.

When asked whether this virtual elective helped them make a decision on whether to apply into OHNS, four students (67%) said "no", and two (33%) said "yes". Of those that said "no", the majority (n=3, 75%) had already decided to apply prior to the start of the course, and the rest (n=1, 25%) decided based on other factors.

When asked about letters of recommendation (LORs) for residency applications, most students (n=5, 83%) indicated that they had enough exposure to faculty they met for the first J o u r n a l P r e -p r o o f time on this rotation to ask for a LOR "only if 'they'really needed it". Attending surgeons were also asked how comfortable they would feel writing a LOR for a student they met for the first time through the virtual elective; the average response was 3.0±1.0.

The majority of students (n=4, 67%) felt they connected enough with faculty during the rotation to feel comfortable asking for advice or mentorship. When asked the same question about residents, half said they would feel comfortable (n=3, 50%). When asked how well they were able to evaluate the department's culture and fit, half of students selected either "extremely well" (n=1, 17%) or "somewhat well" (n=2, 33%).

According to student responses, difficulties with technology were present in both the telehealth and virtual OR sections of the course but were more prevalent in the latter (Table 3 ).

One attending commented that this elective, and the virtual experience in general, would likely be "much easier" a second time if a resurgence of the pandemic necessitates it. Another thought the virtual elective was an "excellent option" given the situation but that it "cannot and should not replace a real rotation".

In this study, the authors describe a single institution's experience with and evaluation of an innovative and interactive 2-week virtual OHNS surgical elective utilizing head-mounted Go-Pro® video technology and live HIPAA compliant two-way audiovisual video conferencing.

Overall, participating students described the rotation as an educationally and professionally valuable experience.

In this study, student perception of the educational value of the interactive telehealth experience was high. This is supported by existing literature, in which learning about and experiencing telemedicine has been well-received by medical students both during and prior to the J o u r n a l P r e -p r o o f COVID-19 pandemic. [18] [19] [20] Satisfaction with the virtual OR experience was also high among our respondents. Data against which to compare these results is limited, as few virtual OR experiences have been described. One study of an ophthalmology course for pre-clerkship students suggested that virtual surgical simulation was well-received. 21 Recent publications discuss use of intraoperative videos for subsequent surgical teaching, virtual suturing and laparoscopic skills workshops, or even a case report of an OR live-stream being used to teach medical students, 13, 14, 22 but their effect on learners has not been investigated. Finally, our didactic component was also well received. The body of literature on virtual didactics during the COVID-19 pandemic is more robust than for other methods. Studies of didactic-based virtual electives show high student satisfaction as well as significant improvement in medical student performance in post-versus pre-tests on specialty-specific material. 10, 11 Our results are particularly relevant given the aforementioned continued suspension of visiting student learning opportunities (VSLOs), or colloquially, "away rotations". 8 Participation in VSLOs is especially prevalent in highly competitive surgical subspecialties such as otolaryngology, plastic surgery, orthopedic surgery, and urology. In many of these fields, more than half of residency applicants ultimately match at an institution in which they completed an away rotation. 6 While many institutions have created "virtual sub-internships" geared toward students from other institutions, these experiences have focused on providing information about a program's features, creating a platform for students to interact with faculty and residents, and offering virtual access to departmental conferences. 23 Efforts at improving virtual surgical elective options may remain relevant even after the COVID-19 pandemic fades, especially for students without home programs in their chosen specialties, who are less likely to match at their top choice program despite being equally competitive applicants. 7

Limitations of this study included small sample sizes, subjective-only measurement of educational value (ie, lack of content-based pre-and post-test), as well as technology challenges.

For the latter, in the authors' experience with the course, these technology issues demonstrated a learning curve and were most prevalent in the first iterations of the course, with substantial improvement as problems were identified and corrected. Frequent feedback was sought from students for this purpose. Another somewhat inevitable limitation was the inability for faculty to assess students' surgical skills due to the virtual nature of the OR experience. Further development of this course could include creative solutions to this problem, such as a virtual attendingor resident-led skills workshop using university-issued suture kits.

Though a virtual elective should not replace an in-person OR experience, our model for a surgical elective provides a feasible alternative that allows students to gain some insight into the culture and fit of the program. Furthermore, given the void left by the suspension of VSLOs, our surgical curriculum could serve as a basis for developing a virtual VSLO with higher fidelity to an in-person rotation than the options currently offered by surgical residency programs.

Traditional paradigms in medical student surgical education have shifted due to the COVID-19 pandemic. Though students have largely returned to home clinical rotations, the ability to rotate at outside institutions through VSLOs remains suspended. Our two-week interactive virtual surgical elective, including an innovative live-stream OR experience, has been wellreceived by our home institution students and faculty during its preliminary iterations. Students reported that the virtual OHNS elective was not only educationally valuable but also gave them the opportunity to connect with department faculty. This study provides a promising model for a virtual surgical elective. As VSLOs remain suspended indefinitely, the implementation of virtual J o u r n a l P r e -p r o o f surgical curricula enhances medical students' understanding of the specialty and builds relationships within the department. 

Important guidance for medical students on clinical rotations during the coronavirus

Excited, scared, ready: Medical students head back to clerkships

The impact of CO-VID-19 on medical student surgical education: implementing extreme pandemic response measures in a widely distributed surgical clerkship experience

Medical student education roadblock due to COVID-19: virtual radiology core clerkship to the rescue

Continuing medical student education during the coronavirus disease 2019 (COVID-19) Pandemic: development of a virtual radiation oncology clerkship

Do audition electives impact match success

Competitiveness of otolaryngology residency applicants without a home program

The coalition for physician accountability's work group on medical students in the class of 2021 moving across institutions for post graduate training. Final report and recommendations for medical education institutions of LCME-accredited

A multimodal multi-institutional solution to remote medical student education for otolaryngology during COVID-19

Imple-J o u r n a l P r e -p r o o f mentation and evaluation of a virtual elective in otolaryngology in the time of COVID-19

Adapting to the educational challenges of a pandemic: development of a novel virtual urology subinternship during the time of COVID-19

Improving diversity and inclusion in the post-coronavirus disease 2019 era through a radiation oncology intensive shadowing experience (RISE)

Establishing a virtual curriculum for surgical subinternships

Teaching operative surgery to medical students using live streaming during COVID-19 pandemic

Creation of an interactive virtual surgical rotation for undergraduate medical education during the COVID-19 pandemic

The REDCap consortium: building an international community of software platform partners

Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support

Integrating telemedicine instruction into the curriculum: expanding student perspectives of the scope of clinical practice

An interdisciplinary, multi-institution te-J o u r n a l P r e -p r o o f lehealth course for third-year medical students

Use of telemedicine in undergraduate teaching of surgery

A self-directed preclinical course in ophthalmic surgery

Innovating surgical education using video in the otolaryngology operating room

Addressing the impact of COVID-19 on the residency application process through a virtual subinternship