key: cord-0710821-3jwh4zi3
authors: Sandhu, Navjot; Frank, Jessica; von Eyben, Rie; Miller, Jacob; Obeid, Jean-Pierre; Kastelowitz, Noah; Panjwani, Neil; Soltys, Scott; Bagshaw, Hilary P.; Donaldson, Sarah S.; Horst, Kathleen; Beadle, Beth M.; Chang, Daniel T.; Gibbs, Iris C.; Pollom, Erqi
title: Virtual Radiation Oncology Clerkship During the COVID-19 Pandemic and Beyond
date: 2020-10-01
journal: Int J Radiat Oncol Biol Phys
DOI: 10.1016/j.ijrobp.2020.06.050
sha: 530581975bdb904b052c34b1176fdd5721374f76
doc_id: 710821
cord_uid: 3jwh4zi3

PURPOSE: We evaluated the impact of a virtual radiation oncology clerkship. METHODS AND MATERIALS: We developed a 2-week virtual radiation oncology clerkship that launched on April 27, 2020. Clerkship components included a virtual clinic with radiation oncology faculty and residents, didactic lectures, student talks, and supplemental sessions such as tumor boards and chart rounds. Medical students completed pre- and post-clerkship self-assessments. Faculty and resident participants also completed surveys on their experience with virtual lectures and clinics. Pre- and post-clerkship results were compared using a 2-sided paired t test. An analysis of variance model was used to analyze the clerkship components. RESULTS: Twenty-six medical students, including 4 visiting students, enrolled over 2 clerkship periods (4 weeks). All students completed the pre- and post-clerkship self-assessments and agreed that the clerkship improved their understanding of radiation oncology. Compared with 3 (11.5%) students who agreed that they understood the daily responsibilities of a radiation oncologist before the clerkship, 22 (84.6%) students agreed and 3 (11.5%) strongly agreed that they understood the daily responsibilities of a radiation oncologist after the clerkship (P < .0001). Although 15 students (57.7%) reported an increased interest in radiation oncology because of the clerkship, the mean level of interest in radiation oncology as a career remained the same, with pre- and post-clerkship scores of 3.0 (±0.9) and 3.0 (±1.1) on a 5-point scale, respectively (P = .7). Students found virtual clinic and didactic lectures to be the most valuable components of the clerkship. Most respondents agreed (30.8%) or strongly agreed (65.4%) to recommend the clerkship to their classmates. CONCLUSIONS: Our virtual clerkship was effective in increasing medical student interest in and knowledge about radiation oncology. These data will help optimize a new paradigm of virtual radiation oncology education for medical students during COVID-19 and beyond.

Medical students generally have limited exposure to radiation oncology during their preclinical and clinical years and are required to independently seek educational opportunities in this specialty. For many students, the clerkship is their first formal experience with radiation oncology and is therefore a critical tool for educating and recruiting physicians-in-training. [1] [2] [3] At our institution, we have offered 2and 4-week clerkship electives to internal and visiting medical students.

However, with the onset of the COVID-19 pandemic in our state in early 2020 and subsequent recommendations from the Association of American Medical Colleges, 4 on-site clerkships were suspended. To maintain an educational opportunity for students, we rapidly developed and implemented a virtual radiation oncology clerkship using telemedicine tools. 5 The virtual clerkship was met with enthusiasm at our medical school and allowed us to not only maintain but also to broaden one of the few avenues that connect medical students with radiation oncology. We report here student and educator feedback and impact of the clerkship on student interest in and knowledge about radiation oncology.

The virtual clerkship curriculum was designed by an advisory committee of key stakeholders, as described previously. 5 Canvas (Instructure Inc, Salt Lake City, UT), a cloudbased learning management system, was used to host our clerkship. We used Canvas's (Instructure Inc) integrated calendaring system, communication stream, built-in web conferencing functionality, and assignment modules to organize clerkship activities. Synchronous ("live") didactic lectures with faculty and residents were held on Zoom (Zoom Video Communications, San Jose, CA), a cloudbased video conferencing tool. All Zoom (Zoom Video Communications) sessions were recorded and password protected and required registration of attendees for attendance tracking. Zoom (Zoom Video Communications) features such as poll questions and live chat moderated by resident cohosts were used to help with student engagement. The virtual clinic was conducted on our institution's Epic Systems (Epic Systems Corporation, Verona, WI) telemedicine portal. Students could log into video visits via a web browser using the VidyoConnect (Vidyo Inc, Hackensack, NJ) application. Multidisciplinary tumor boards and chart rounds were held on WebEx (Cisco Webex, Milpitas, CA), another video conferencing tool. In addition, a journal club, during which students chose a relevant and recent oncology paper in the literature to present, was hosted via Zoom (Zoom Video Communications).

Although our institution did not allow visiting students to formally rotate during the COVID-19 school closure, we allowed interested students from outside institutions to audit the course and participate in the didactic sessions and student journal club presentations.

This study was exempted by our institutional review board.

Student pre-and post-clerkship self-assessments and faculty and resident surveys (Figs. E1-4; available online at https://doi.org/10.1016/j.ijrobp.2020.06.050) were collected using Research Electronic Data Capture (REDCap), a web-based application for capturing clinical research data hosted by our institution. Residents and faculty who gave lectures and/or worked with students in the clinic received surveys at the end of the clerkship on their experience with teaching in the virtual format. To pass the clerkship, students were required to complete the preclerkship assessment on the first day of the rotation, before participating in any clerkship activity, and the post-clerkship assessment on the last day of the clerkship, after the final clerkship activity. For the questions that were identical on the pre-and post-clerkship assessments, the order of questions and answer choices were altered to decrease recall bias. Although pre-and post-assessments were linked by respondent, responses were anonymized and aggregated, and we encouraged students to be honest in their feedback.

Paired t tests were used to compare the pre-and postclerkship assessment results. Clerkship components were assessed using an analysis of variance model. Statistical analyses were performed using SAS software (version 9.4; SAS Institute, Cary, NC). All tests were 2-sided with an alpha level of 0.05.

At the time of this analysis, 26 students had enrolled in our virtual clerkship. Twelve home-institution students were enrolled in the first offering of the clerkship (period 1) from April 27 through May 10, 2020. Ten home-institution students and 4 students from outside institutions were enrolled in the second offering (period 2) from May 27 through June 7, 2020.

All students reported the clerkship as their first radiation oncology rotation. Half of the students were female. Although the 4 visiting students ranked radiation oncology as their top choice for specialty, the home-institution students ranked oncology (n Z 4), internal medicine (n Z 3), surgery (n Z 3), neurology (n Z 2), dermatology (n Z 2), otolaryngology (n Z 2), and other (n Z 6; urology, pediatrics, physical medicine and rehabilitation, anesthesia, cardiology, and ophthalmology) as their top choice for specialties. Additional student characteristics are provided in Table 1 .

After clerkship period 1, a 2-week break was used to improve the clerkship based on feedback from students, residents, and faculty.

Students from clerkship period 1 requested that didactics cover a broader range of disease sites ("having additional didactics on treating specific types of cancer would be informative"), more virtual clinic experience, and that they be required to enable their camera for all of the lectures ["Force students to show faces on Zoom (Zoom Video Communications) for lectures; this forces us to pay more attention"].

Faculty and resident responses reflected similar themes: "The availability of more interactive options (poll question variety, live Q&A) would be helpful"; "Being able to see all participants (encouraging them to be seen rather than disabling video camera) was also helpful"; and "It would be better for students to see more patients in virtual clinic."

Students also requested to have more time allotted for their end-of-clerkship talks ("It would be nice to have a little more time to present the papers. Some of them have a lot to discuss, and for students new to radiation oncology, it's helpful to discuss even the more basic things.") and for attendance in chart rounds to not be mandatory ("I did not find chart rounds useful as it was fast paced and often above my level of understanding making it difficult to follow.").

Based on the feedback, we revised the clerkship to include didactics for every disease site using a combination of prerecorded and synchronous lectures. We requested that students have their cameras on for synchronous lectures and encouraged all speakers to use interactive polls in their lectures. We also incorporated more virtual clinic into the student schedules. Given the number of students and time constraints, we did not increase the allotted time for student talks; however, we assigned students faculty mentors to provide guidance for their journal club papers and talks. We also continued to require attendance at our weekly chart rounds, which we believed encapsulated important aspects of our specialty. The core components of the revised curriculum are summarized in Table 2 .

Clerkship impact on knowledge about radiation oncology and learner satisfaction Student response rate for the postclerkship assessment was 100%. Students ranked the curriculum components on a scale from 1 (most valuable) to 5 (least valuable). Students found the clinic to be most valuable (mean, 1.3; standard deviation [SD], AE0.7), followed by didactic lectures (2.5; AE1.1), journal club presentations (3.4; AE1.0), treatment planning sessions (3.5; AE1.1), and chart rounds (4.3; AE1.0) (P < .0001; Fig. 1 ). Students from period 2 reaffirmed the value of the virtual clinic: "Virtual clinic was the most valuable in showing the day-to-day decision making involved in radiation oncology"; "Particularly appreciated attending teaching during clinic." Although many students commented that the virtual curriculum was conducive to learning ("Great mix of didactics, clinic and journal club presentations"; "The clerkship was very well-organized, and I appreciated the diverse opportunities despite us interacting with the course virtually"), 1 student found the virtual lecture format challenging ["I personally find the lectures hard to follow, especially over Zoom (Zoom Video Communications)."] All students agreed that the clerkship improved their understanding of radiation oncology. Compared with 3 (11.5%) students who agreed that they understood the daily responsibilities of a radiation oncologist before the clerkship, 22 (84.6%) students agreed and 3 (11.5%) strongly agreed that they understood the daily responsibilities of a radiation oncologist after the clerkship (P < .0001; Fig. 2A ).

Most respondents agreed (30.8%) or strongly agreed (65.4%) to recommend the clerkship to their classmates.

Sixteen (61.5%) students agreed or strongly agreed that the clerkship helped inform their decision to choose a specialty; 6 (23.1%) remained neutral, and 4 (15.3%) disagreed or strongly disagreed. One student commented, "I wish I had had more exposure earlier in my medical school career when I might've considered radiation oncology."

Fifteen students (57.7%) reported an increase in their interest in radiation oncology after this clerkship; 34.6% reported their interest stayed the same, and 7.7% reported a decrease in their interest in radiation oncology.

Reasons for increased interest in radiation oncology included learning about the number of diverse therapies in our field ("There are more therapies in radiation oncology than I thought"; "didn't previously know that radiation oncology does procedures, like brachytherapy"; "nice balance of clinic and procedural work"), the technical nature of our specialty ("There is a real tangible relationship to physics and technology in ways that other fields may not have"; "I have really enjoyed learning about the technology used in radiation oncology"; "This clerkship invalidated preconceived notions that radiation is a nonspecific treatment."), the important role our specialty plays in multidisciplinary cancer care ("Much more applicable to specialties I am interested in than I previously thought"; "gained an appreciation of how central radiation oncology is in the treatment of cancer today and how integral it will remain despite much of the headline attention to immunotherapy"; "I learned about the coordination among teams for oncology treatment"), and the people in our department ("Great and supportive residents and faculty"; "I liked the people I met in this clerkship"). Reasons given for decreased interest included "The dayto-day work was less interesting than I thought" and "My interest in pursuing the field personally is lower than prior, although I do have a new appreciation for the field and will be a better clinician for understanding/knowing more about radiation oncology."

Mean level of interest in radiation oncology as a career remained the same with pre-and post-scores of 3.0 (AE0.9) and 3.0 (AE1.1) on a 5-point scale, respectively (P Z .7). The 5-point Likert scale was defined as (1) not interested at all; (2) would consider oncology but not necessarily radiation oncology; (3) consider learning more about radiation oncology; (4) considering applying to radiation oncology residency; and (5) likely to apply to radiation oncology residency (Fig. 2B) . However, after the clerkship, 5 (19%) home-institution students had radiation oncology as 1 of their top 3 specialty choices (compared with only 1 homeinstitution student before the clerkship).

Thirteen faculty and 9 residents participated in the virtual clerkship through giving lectures or working with students in the virtual clinic over the 2 clerkship periods. Their survey response rate was 95%. Overall, they thought virtual clinics and lectures yielded an educational benefit to the students and found ease of educating students via the virtual format acceptable (Table 3 ).

The COVID-19 pandemic and resulting shelter-in-place orders both required and allowed us to redesign medical student education. By successfully implementing a virtual radiation oncology clerkship, we broadened our specialty's reach to medical students. Over 2 virtual clerkship periods, we had 22 home-institution students participate, representing a substantial increase compared with the 4 total home-institution students who enrolled from July 2018 to March 2020.

This increased exposure to radiation oncology for medical students is important given recent trends of declining radiation oncology residency applications. [6] [7] [8] In the 2019 match, 30 residency positions across 22 US radiation oncology residency programs were unfilled, with the unmatched rate (14.5%) nearly doubled from the previous year. 7 Furthermore, racial and ethnic groups underrepresented in medicine and women continue to be disproportionately low in our field. 9-11 Deville et al found that radiation oncology had less women, Black, and Hispanic/Latinx representation compared with the 20 largest training specialties. 12 Chapman showed an ominous trend of declining Black representation in radiation oncology trainees in recent years. 13 However, half of our virtual clerkship students were female, and more than one-fifth were Black and/or Latinx. Thus, the virtual clerkship can be a feasible instrument to help recruit the best students and a more diverse workforce to radiation oncology.

However, most medical students will not pursue radiation oncology as a career. Thus, the clerkship is also important for educating students about radiation therapy and the roles of the radiation oncology care team. Because cancer remains the second leading cause of death with estimates of 17 million new cases and 9.5 million deaths globally, 14,15 these future physicians, regardless of specialty, will undoubtedly care for patients with cancer. More than 50% of patients with cancer will undergo radiation at some point during their disease course, 16 and medical students need to be aware of our treatment modality and how it affects their patients. Despite this, students are rarely exposed to formal radiation oncology curricula during their medical education. 17, 18 Indeed, the majority of our home-institution students taking our virtual clerkship (86%) had no prior exposure to our field. Before the clerkship, 90% did not understand the daily responsibilities of a radiation oncologist. We were able to significantly increase students' overall knowledge of radiation oncology over the 2-week virtual clerkship. Students commented that through the clerkship, they became aware of the broad range of treatments available in radiation oncology, were able to correct preconceived notions they had about our field, and wished that they had earlier exposure to radiation oncology during their medical training.

We designed the virtual curriculum to reproduce key components of our in-person clerkship, including facultyled didactic sessions, clinic, student talks, and supplemental sessions, such as quality assurance rounds and multidisciplinary tumor boards. Didactics and student talks were hosted over Zoom (Zoom Video Communications). To recreate the interpersonal aspects of these typically in-person activities, most lectures were synchronous, and all participants were required to turn on their videos. We further encouraged the use of poll questions and chat to help engage students. Residents were crucial to faculty-led *Four students from external institutions did not participate in virtual clinics; n = 22 for this category. didactics; each session was cohosted by a resident moderator who helped address chat questions while faculty spoke. Although 1 student found the virtual lecture format challenging for learning, there were several advantages with the virtual format. There was increased participation by faculty in giving lectures to students and increased departmental attendance to student talks, potentially facilitated by the increased flexibility with the virtual format. Of all the clerkship components, students rated virtual clinics as the most valuable. The clinic allows students to develop their skills in assessing and presenting a cancer patient's history and pertinent findings and allows faculty to evaluate students' clinical performance. Although integrating medical students into our evolving virtual clinic workflow posed additional logistical challenges, we considered the clinic as an indispensable component of the clerkship. Several factors allowed us to accommodate these students into the virtual clinic workflow. First, our institution quickly implemented a robust telehealth platform, with more than 70% of clinic visits in our department being over video as early as March 2020. Second, many faculty in our department had developed mostly virtual workflows that could accommodate students, with implementation of personal Zoom (Zoom Video Communications) "workrooms" with their residents, nurse coordinators, and nurse practitioners to hold patient presentations and team discussions. Finally, the residents again played a key role in facilitating the virtual clinic experience for medical students; they connected with their assigned students before the clinic on Zoom (Zoom Video Communications) to introduce students to the video visit platform and their team's specific virtual workflow. This helped streamline clinical workflow with respect to student integration and reduce burden on faculty and their teams. Table 4 lists key pointers for implementing effective virtual clerkship didactics and clinic.

It is important to note that most home-institution students taking this virtual clerkship were not interested in applying to radiation oncology residency. Thus, their feedback about the clerkship would be expected to differ from that of students who are interested in matching into a radiation oncology residency and take our clerkship as an "audition" elective. Prior research found that medical students applying into radiation oncology considered the following educational activities as moderately to extremely important to include in a clerkship curriculum: an opportunity to perform an unsupervised history and physical examination; an opportunity to give a formal lecture; a didactic hands-on session on radiation contouring and planning; a formal case presentation to faculty; and lectures on treatment planning, radiobiology, physics, and evidencebased medicine. 19 Our students found didactic sessions with faculty and residents to be second most valuable after the virtual clinic. Interestingly, our students reported treatment planning sessions and quality assurance rounds (chart rounds) to be least valuable. This likely reflects the background of the students, who largely were without prior experience or interest in radiation oncology as a career. Thus, they might not derive educational value in the more technical and specialized nature of these sessions. These data from our students are valuable to enhance the incorporation of radiation oncology into the general medical school curriculum.

A key question is how medical education will evolve moving forward. For our institution, we hosted 1 more 2week virtual rotation from June 8 through June 21, 2020 (period 3), before resuming in-person clerkship activities for the 2020 to 2021 academic year. However, our institution will resume in-person clerkships only for internal students and will not be hosting visiting clerkships. This policy will be reviewed later this year but is currently in line with Association of American Medical Colleges recommendations 4 and the policies adopted by many medical schools and residency training programs. The cessation of the visiting clerkship will have important implications for the residency application process; many programs and students use the away elective as an opportunity to evaluate one another. Students without radiation oncology programs at their home institutions will likely be most affected by these policy changes.

As telehealth reduces the burden on patients while increasing access to care, virtual clerkships can similarly improve access to our specialty for students. In the COVID-19 era, virtual clerkships allow reduced department crowding and reduced exposure risk for students. After the COVID-19 pandemic, virtual clerkships may continue to have a role in overcoming geographic and financial barriers associated with away rotations. 20 We plan to continue our virtual clerkship for the near future given that social distancing measures will remain in place in our institution. Furthermore, cancer center providers at our institution anticipate that 50% of visits will be virtual moving forward, with 75% agreeing or strongly agreeing to use video visits to a greater extent after the COVID-19 restrictions are lifted (internal data). Thus, virtual clerkship components, such as virtual clinics, should naturally continue to train future physicians in the practice of outpatient medicine. Although we have welcomed visiting students to audit our 2-week clerkship thus far, we plan to formally host visiting students for credit in the summer and fall so that visiting students can participate in patient-facing activities, such as the virtual clinic. These clinical activities were rated by students as the most valuable component of our virtual clerkship and are necessary for the full educational experience.

We acknowledge that our study is based on the small and unique experience of a single tertiary academic medical center and may be challenging to generalize to other settings. We benefited from a quickly implemented and comprehensive telehealth infrastructure and a patient population largely capable of participation in video visits. Also critical to the success of our virtual clerkship was the support of faculty and residents who were engaged in the education of our medical students and receptive to hosting them in virtual clinic and leading lectures.

Finally, although our virtual clerkship was initially created to primarily address the educational needs of medical students during this pandemic, a critical need still exists to address the loss of mentorship, which is the other key purpose of visiting clerkships. Medical students often select institutions for their away clerkships for a number of reasons, including but not limited to (1) learning more about the field of radiation oncology, (2) learning more about a program they wish to consider for training, (3) participating in research, (4) establishing a mentorship or personal connection with faculty in specific areas of interest, and (5) obtaining a referee letter. To continue encouraging and developing the next generation of entrants into radiation oncology, it will be critical to overcome these gaps, given the already decreasing applicant pool. Although this broader goal will require creativity and participation from all training institutions, some suggested proposals are included in Table 4 . We hope that our efforts are simply the beginning of the development and optimization of a new paradigm for virtual radiation oncology education for medical students.

The implementation of a 2-week virtual clerkship in radiation oncology at our institution was received positively by medical students, faculty, and resident educators in our department and holds promise as a tool to potentially Assign students to faculty/resident teams to facilitate more one-on-one interaction Encourage resident and faculty check-in with assigned students by email or phone call after clerkship is complete for establishing longitudinal relationship Zoom (Zoom Video Communications) happy hour with faculty and residents Host additional sessions for students on workelife balance, leadership, and research opportunities mitigate inequities in access by expanding exposure to our field. Based on the impact of this program, we will continue to offer the virtual rotation to home-institution students and visiting students in the 2020 to 2021 academic year.

A national radiation oncology medical student clerkship survey: Didactic curricular components increase confidence in clinical competency

Assessing the value of an optional radiation oncology clinical rotation during the core clerkships in medical school

Radiation oncology residency selection: A postgraduate evaluation of factor importance and survey of variables associated with job securement

Association of American Medical Colleges. Final report and recommendations for medical education institutions of LCMEaccredited, U.S. osteopathic, and non-U.S. medical school applicants

Continuing medical student education during the COVID-19 pandemic: Development of virtual radiation oncology clerkship

Trends in radiation oncology residency applicant interview experiences and post-interview communication

The high number of unfilled positions in the 2019 radiation oncology residency match: Temporary variation or indicator of important change?

Attracting future radiation oncologists: An analysis of the national resident matching program data trends from 2004 to 2015

Long-term outcomes of radiotherapy for pituitary adenomas

Female representation in the academic oncology physician workforce: Radiation oncology losing ground to hematology oncology

Representation trends of underrepresented minority physicians in the US radiation oncology (RO) workforce

Diversity in graduate medical education in the United States by race, ethnicity, and sex

Diversity based on race, ethnicity, and sex, of the US radiation oncology physician workforce

World Health Organization. International Agency of Research on Cancer

Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries

Cancer and radiation therapy: Current advances and future directions

Quantitatively and qualitatively augmenting medical student knowledge of oncology and radiation oncology: An update on the impact of the oncology education initiative

The need for a standard, systematic oncology curriculum for U.S. medical schools

Radiation oncology medical student clerkship: Implementation and evaluation of a bi-institutional pilot curriculum

Patterns and perceptions of "away" rotations among radiation oncology residency applicants