key: cord-0962229-ep79v1af
authors: Gomez, Erin; Azadi, Javad; Magid, Donna
title: Innovation Born in Isolation: Rapid Transformation of an In-Person Medical Student Radiology Elective to a Remote Learning Experience During the COVID-19 Pandemic
date: 2020-06-18
journal: Acad Radiol
DOI: 10.1016/j.acra.2020.06.001
sha: 824bd28dcf36f271aa4865d28622da587e1055b7
doc_id: 962229
cord_uid: ep79v1af

RATIONALE AND OBJECTIVES: With social distancing measures in place both nationally and globally, the current COVID-19 pandemic has forced the cancellation of in-person classes at universities and medical schools across the country. This has presented unique challenges for educators in all fields as they have embarked, many for the first time, on the journey into remote education in order to provide distance learning opportunities for students. MATERIALS AND METHODS: In this article we will review our approach to rapidly converting an in-person diagnostic radiology elective to an entirely remote learning experience for medical students at our institution, including modification of course structure, software tools and materials utilized, and strategies for learner engagement and collaboration. RESULTS: Development of an online elective led to a 10-fold increase in student enrollment compared to the traditional course offering, providing a unique opportunity to reach a broad number of students, many of whom were in the early clinical stages of their medical school career. Lastly, we will review faculty feedback after participating in the remote course as well as lessons learned from the transition to distance learning and its implications for future work CONCLUSION: The current state of technology makes radiology particularly well-suited for distance learning, and with the proper tools and approaches, effective remote radiology instruction can be achieved.

T he rapidly evolving clinical practice guidelines surrounding the COVID-19 pandemic have had a profound effect on academic medicine, forcing students and educators alike around the world to venture into uncharted territory. When the virus was first diagnosed in the United States in mid-January, many governmental and educational institutions adopted a watchful waiting approach toward the novel corona virus during January and February 2020. However, the rapid spread of confirmed cases throughout the United States during March 2020 led to the implementation of COVID-19 mitigation strategies based on strict social distancing which altered all aspects of pre-COVID daily living, including in academic medicine.

On March 17, 2020, the Association of American Medical Colleges recommended the temporary suspension of medical student clinical rotations (1), both for the safety of students and patients and for the conservation of personal protective equipment. Medical educators had little or no warning nor time to develop strategies for meaningful alternatives to inperson clinical experiences.

At our institution, in-person classes were cancelled for all university students and preclinical medical students on March 11. On March 17, in conjunction with the Association of American Medical Colleges recommendations, in-person clinical experiences were also suspended for all medical students. On March 23, we hosted the first sessions of a 3-week remote diagnostic radiology elective.

In this paper, we will describe the strategies and tools used to rapidly transform an in-person medical student radiology elective into a comprehensive, interactive remote learning experience, effects on course enrollment and lessons learned from our students and teachers.

The timeline listed in Table 1 details the chain of events leading up to and through the first iteration of the remote course, which launched less than 2 weeks after all in-person classes were canceled for medical students and just 8 days after the suspension of on-campus clinical rotations and electives. In addition to the remote diagnostic radiology tutorial, online electives in dermatology, pathology, and library science were initially available

Our school of medicine is a major academic institution with approximately 120 medical students per class. At our institution, diagnostic radiology is an elective rotation, but is also considered one of the "Horizontal Strands" woven into the curriculum throughout all 4 years of medical school. Typically, the standard version of the diagnostic radiology elective is taken by students in their third-or fourth year of medical training and is 4.5 weeks in length. The course is offered 5 times per academic year and averages 5À15 students enrolled per half quarter.

In July of 2019 the current course directors, both of whom are junior faculty, assumed leadership of the elective and implemented major revisions to the structure and content of the course, including the addition of interactive workshops covering skill related to ultrasound scanning, body computed tomography (CT) interpretation and radiography.

Digitizing the Classroom: Transitions From In-Person to Remote Learning

Due to the urgency of the transition to remote teaching, the course directors worked together to abbreviate the traditional course and adapt to an online version. All in-person components of the course that could not be adapted for distance learning were eliminated. This included 1 week of shadowing experiences, a radiology-themed escape room and hands-on workshops in ultrasound scanning and radiography.

Our university officially endorses Zoom for video conferencing. Most lecturers were able to present their talks while working offsite due to alterations made in our department's clinical schedule. Departmental IT and the School of Medicine worked rapidly to upgrade the VPN in order to ensure secure, reliable network access. Staggering of the radiology residents' rotations also allowed for some remote case sessions to be led by several residents. Faculty who were unavailable to give lectures via Zoom pre-recorded their lecture content.

All faculty were asked to acquire free basic Zoom accounts and were able to purchase premium Zoom accounts with their discretionary funds at an affordable rate. A website was launched by the Office of Faculty Development which included video tutorials for installing, setting up and using Zoom. All medical students and trainees were granted free basic Zoom accounts and given access to the tutorials available to faculty. In order to assist the faculty with the transition to Zoom, trial Zoom calls were conducted between the course directors and individual faculty members prior to the start of the course. One of the two course directors also signed on to the School of Medicine's Zoom platform with the assigned lecturers ten minutes prior to the start of each session to confirm function and introduce the lecturer, and remained signed in for as long as necessary to troubleshoot any issues that arose.

Several interactive workshops were held throughout the course, including day-long body CT and head CT workshops focusing on identifying normal anatomic structures and reviewing commonly encountered pathology. During these workshops, students accessed anonymized cases which had been uploaded by the session facilitators to a free online HIPAA compliant PACS called Pacsbin (http://get.pacsbin.com); this eliminated any potential issues caused by a large number of students using the hospital's PACS remotely. Cases were viewed by the learners on their personal devices followed by "readouts" with the workshop facilitators via Zoom's screen sharing feature. A similar format was used for "hot seat" case sessions which were largely delivered by residents.

In addition to Zoom, several resident-led sessions utilized the Nearpod platform, which allows the presenter to embed assessments and interactive polls into lecture slides. Nearpod is used frequently in our resident morning conference curriculum and is a resource familiar to our trainees. Others chose to embed quiz questions into their PowerPoint presentations Available online courses announced to students March 23

First day of remote radiology elective April 13

Final day of remote radiology elective via Microsoft Forms, sharing lecture slides and quiz links with students prior to the start of the session. Multiple asynchronous learning activities were included in the course including a series of online quizzes which were created to assess student retention of e-lecture content. These were distributed to the students via Google Forms. Several lecturers also hosted Zoom quiz-bowls or "Jeopardy" sessions via screen sharing of PowerPoint slides.

All students have Microsoft Office 365 accounts provided by the university. Course-related messages were largely communicated by the course directors via Microsoft Teams. In our opinion, Microsoft Teams was one of the most valuable resources utilized in the remote course, as it significantly improved our ability to communicate with the students and engage in real-time conversations not typically permitted by e-mail. Announcements were distributed to all enrollees via the group chat feature, which also served as a parking lot for questions about course content and logistics. All Zoom lecture links and attendance forms were posted to the class homepage in Microsoft Teams. Microsoft Teams was also trialed as an alternative to Zoom by a lecturer presenting one of the PowerPoint presentations to the class through Microsoft Teams. The students' feedback was that both Zoom and Microsoft Teams were comparable with regard to ease of use and video/audio quality. Blackboard is the established course management software at our institution, thus course materials including lecture slides, assignments, and external links were hosted on an existing Blackboard site maintained by the course directors.

When transitioning the course to a distance learning experience, we remained cognizant of the fact that the circumstances of the pandemic had likely placed an enormous amount of stress on the students. Many informed us that they were sheltering in place off-campus or with family or friends, some out of state and across the country. Others who had been abroad found themselves unable to return home immediately due to travel restrictions and quarantine measures. With this in mind, we opted to convert the course to a pass-fail experience in an attempt to alleviate the amount of stress placed on the learners. In addition to modifying the final exam to reflect the remote course content, the test was also made open book and was offered online via the course Blackboard site.

Students were assigned work related to multiple publicly available radiology education websites as in the following list: -TeamRads.com -LearningRadiology.com -CTisUs.com -Cleveland Clinic pediatric radiology modules (https:// www.cchs.net) -"One Night in the ED" emergency radiology case file (http://radiology.cornfeld.org) -High Value Practice Academic Alliance Ordering Wisely E-Lectures (https://hvpaa.org/order-wisely/)

During live lectures, in addition to microphone and video capabilities, students used the chat window to ask and answer questions. Private chats between students and with the session facilitator were also available which allowed for anonymous questions that could be shared with the group if applicable.

The course directors hosted virtual "office hours" via Zoom for a 30À60-minute period 3 times per week. These sessions were blocked off as protected time in the course schedule for students to ask questions about lectures and assignments. Students often arrived individually, but also occasionally logged in with classmates to clarify points that they had been discussing amongst themselves offline. After office hours sessions, if appropriate for the larger group, a summary of the topics discussed was posted to Microsoft Teams along with additional references or examples. Virtual office hours provided students with the opportunity to connect one-on-one in an environment that was less formal and potentially less intimidating than the larger Zoom classroom, and likely reduced the overall number of e-mails from students throughout the course because discussions took place in a dedicated space rather than back and forth in our inboxes.

In order to receive credit for attending a session, students submitted lecture feedback forms which were generated in Microsoft Teams. These were due by midnight on same the day of the lecture. Each lecturer provided a unique six-digit attendance code which students typed into the form along with comments and a Likert-scale rating of the session. Feedback forms were sent to faculty and resident lecturers in order to improve future sessions. Lecture attendance was also randomly monitored via Zoom reports, which generate data regarding students' online presence as well as "attention" based on whether the Zoom application was in the foreground of their computer screens or minimized.

As one of the final assignments in the course, students submitted video recordings of narrated PowerPoint presentations about the imaging-related topic of their choice. Topics covered included specific imaging modalities, disease processes, the history of radiology, radiology-based art and the role of imaging in various fields and global events. These presentations were used to build a library of peer-sourced mini e-lectures for future course participants.

In lieu of an in-person journal club, students participated in an electronic version of this activity by using an online discussion board. After reading several assigned articles, multiple prompts were posted to a forum generated on the course Blackboard. Students were then asked to engage in a discussion about the papers by contributing to relevant threads, responding to each other's posts and debating in real-time.

The topic of discussion was artificial intelligence in Radiology and its role in the COVID-19 pandemic.

Enrollment for the course has remained consistent for the past 3À5 years, with 5À15 students enrolled per 4.5 week (halfquarter) offering. The majority of students enrolled in the traditional course are fourth-year medical students, although during certain quarters up to 50% of the course may be comprised of second-and third-year students. Following the announcement of the online elective, we enrolled 116 students (Fig 1) . Of the enrollees, 50% were third-year medical students, 40% were second years, and 10% were in their fourth year of training.

In order to receive credit, strict deadlines for submission of the aforementioned quizzes, online modules and assignments were enforced. Students' grades were posted to the course Blackboard site and updated at regular intervals throughout the course. All students completed the assignments in a timely manner. Hundred percent of students achieved a passing grade on the final exam and in the overall course.

While the Office of the Registrar is still processing the students' formal course evaluations, feedback obtained from an anonymous end of course survey distributed by the course directors has been largely positive. Representative student comments are listed in Table 2 . When asked about their favorite part of the course, most listed the Pacsbin workshops and trainee-facilitated case sessions due to their interactive nature. None indicated that the remote learning environment was suboptimal when compared to traditional in-person rotations. Students expressed gratitude for the opportunity to continue learning despite the world around them grinding to a halt. Many stated that convening with classmates on a daily basis, although remote, helped them maintain a sense of community and mental well-being. Suggested improvements included small group breakout discussions for certain assignments, an increased number of workshops and interactive case sessions, and more trainee-led teaching sessions. Students also suggested limiting in-person lectures to 4À5 hours per day and creating more accompanying pre-and postquizzes to assess retention of lecture content.

A post-course evaluation survey was distributed to the 15 faculty who taught in the course. A response rate of 73% (11/ 15) was achieved. The majority of respondents (72.7%) indicated that they had never used Zoom prior to teaching in the course. All faculty reported an improved sense of comfort with teaching students via distance learning after participating in the remote elective (Fig 2) . Most (10/11) agreed or strongly agreed that they had the tools and resources necessary to convert their traditional in-person lecture to a remote format.

When asked about their concerns at the start of their experience as a remote instructor, most faculty indicated that were worried about student participation and engagement. A few also expressed concern about potential IT issues, but clarified that these were not encountered. When asked what they enjoyed about teaching remotely, many listed conveniences such as the flexibility of the schedule and ability to work from home. Several also felt that participation was increased in the remote setting. One faculty member commented, "In some ways, the students were more interactive. They could annotate directly on the images and ask questions in a less pressured environment. Also, given the large numbers that could actually attend, I felt for the first time that I was reaching a large audience and introducing them to the joys of radiology." Suggestions for course improvement included implementation of an audience response system beyond the Zoom chat, smaller class size and access to computers with cameras for future teaching sessions.

Overall, excellent pilot course. I am happy to have taken the course and many of the resources given to us were very helpful. I think many parts of this course should be seriously adapted into the medical curriculum earlier, potentially built into the anatomy course or even have the course first...it is very useful and important for our future medical education. Thank you for this wonderful course! I gained a much better understanding of different imaging modalities and the appearance of normal structures, common pathologies, and foreign bodies, which I think will help me during all of my clinical rotations. The course also piqued my interest in radiology, a field which I did not know much about before, and which I now am considering as a possible specialty to pursue. Thank you so much for putting this together. It makes a big difference to be able to learn something so worthwhile during this time and to get to know you all as well. I am very grateful this course happened and feel FAR more competent reading a CT. One day, I will find the appendix. Overall, I really loved this course, and I think it transitioned to the virtual platform really well. All of the resident lecturers were phenomenal and engaging. Thanks for a wonderful 3 weeks! Thank you so much for putting this course together and allowing so many of us to participate. I know many people wanted an opportunity to learn more about radiology regardless of their career paths, and this was a great way to gain exposure to radiology while also stuck at home. Thank you so much for making such a great course for us so quickly! It helped me stay sane for the first part of my quarantine. After implementing this remote course, it is apparent that with the proper tools, interactive and engaging teaching is achievable in the absence of in-person instruction. While the existing structure and framework of our traditional in-person course lent itself well to a digital platform, radiology is an ideal field for distance learning, especially given the current state of radiology IT. Many attending physicians in our department were well-equipped to deliver remote lectures due to the availability of home workstations and a robust VPN, and the availability of Zoom licenses to faculty, residents and fellows provided opportunities for trainees to teach. Students were able to view images from home with the use of online and Cloud PACS systems, although we recognize that access to internet and computers is a privilege not afforded to all. It was also apparent that the variability of the students' personal computer equipment contributed to the diagnostic quality of their received images.

Following the conclusion of the initial pilot, we are ready to offer additional iterations of the course during the upcoming academic year. Having reached such a large number of students at once, we plan to evaluate students' early exposure to radiology with the number of residency applicants in the coming years. For those expressing interest in diagnostic or interventional radiology after taking the course, a focused inperson experience comprised of shadowing, mentoring and research opportunities is planned once in-person clinical activities resume. Additionally, we have met with the directors of other clinical electives and shared our methodologies as a framework for developing remote course content. Since the initial launch of the course, a total of twenty-eight additional online elective experiences have been developed for medical students at our institution. Moving forward, these rapidly developed on-line educational opportunities can be refined and further developed in the anticipation of increased emphasis on on-line learning in the wake of the COVID-19 pandemic.

Several students also voiced opinions about making radiology a required core rotation at our institution. We plan to evaluate the possibility of broader radiology instruction throughout all 4 years of medical training. Additionally, the creation of a remote radiology elective for students at our institution presents an opportunity to further develop a remote course that could be offered to students at other institutions on a national or international scale.

Both instructors and learners in this course expressed feeling a sense of normalcy in the virtual classroom despite the unique circumstances that brought about its need. Through distance learning we are reminded that education is capable of bringing people together, even when social distancing necessitates that we remain physically apart.

Important guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. Press release. Association of American Medical Colleges

The authors would like to acknowledge our Department Chair, Dr. Karen Horton, Dr. Pamela Johnson, Vice Chair of Quality and Safety, Dr. Nancy Hueppchen, Associate Dean for Curriculum and Dr. Roy Ziegelstein, Vice Dean for Education. Without their support this endeavor would not have been possible.