key: cord-0973770-ud3gde1u
authors: Jones, Henderson M; Ankem, Akhila; Seroogy, Erik A; Kalantar, Alborz; Goldsmith, Dylan C; Rizenbergs, Kyle C; Van Meter, Tracy L
title: Impact of COVID-19 on radiology residency selection process: a survey of radiology residency programs in the US
date: 2022-01-07
journal: Acad Radiol
DOI: 10.1016/j.acra.2021.12.033
sha: de09429814ea214aef0f2c2c3378beba9f2da532
doc_id: 973770
cord_uid: ud3gde1u

OBJECTIVE: Virtual residency interviews have become crucial in maintaining CDC-recommended precautionary measures throughout the ongoing COVID-era. However, scant literature exists regarding the direct impact the pandemic has had on the radiology program selection process, including preferred modalities of residency interviews and social media utilization. This survey aimed to understand how radiology programs adapted to changes in the 2020-21 virtual application cycle. METHODS: A 32-question survey was distributed to 194 residency programs between July and August of 2021. Data were analyzed primarily by using descriptive statistics and Paired Student's T-testing. RESULTS: A total of 31 programs completed the survey, yielding a response rate of 16%. When queried about the perceived success of virtual interviews during the 2020-21 application cycle, 21 programs (68%) said the new interview format was very successful. 27 of the programs (87%) believed they were able to adequately gauge applicants through virtual interviews. However, when surveyed about personal preferences for interviews, the responses were more evenly distributed between virtual (11/31, 35%) and in-person (14/31, 45%). Pre-COVID, the top three criteria programs used to rank candidates were USMLE Step 1 score (20/31, 65%), letters of recommendation (17/31, 55%), and MSPE (12/31, 39%). Within the virtual, COVID-19 era, these criteria remained statistically unchanged (p = 0.22): USMLE Step 1 score (21/31, 68%), letters of recommendation (17/31, 55%), and MSPE (14/31, 45%). About half of programs who had not previously utilized social media (12 of 23, 52%) created accounts, mostly via Twitter or Instagram. CONCLUSION: The primary findings indicate that programs perceived success with virtually interviewing and ranking applicants, the criteria to rank applicants remain unchanged despite the virtual environment, and programs’ social media utilization increased.

The COVID-19 pandemic has prompted responses from several administrative bodies, including the Coalition for Physician Accountability, Association of American Medical Colleges, and Electronic Residency Application Service, to issue major changes to the residency application process.

[1] Residency programs were mandated to conduct fully virtual interviews and called on to explore novel virtual approaches during pandemic-era constraints. Social media and additional online platforms were highly utilized by residency programs of all specialties, including virtual sub-internships, webinars, and updated social media accounts. [2, 3] Radiology programs had to consider the following adaptations to the traditional application process: implementation of virtual interviews, virtual away rotations, and the limited number of specialty-specific virtual electives. [4] Additionally, programs have needed to devise virtual substitutes for department tours and pre-interview dinners with residents, often seen as integral portions of the interview process. On the applicant side, the virtual environment also brought changes to the medical school curriculum, resulting in a transition to remote radiology electives and elimination of away rotations for some time, potentially creating further obstacles for students to obtain meaningful recommendation letters from radiology attendings.

We surveyed PDs from all the Accreditation Council for Graduate Medical Education (ACGME)accredited residency programs across the United States to determine the impact of COVID-19 on the radiology application cycle. The investigators aim to elucidate the PDs' attitudes on the virtual delivery of interviews and whether limited in-person experiences affected application criteria, ranking decisions, and social media usage.

Under an IRB-approved protocol, diagnostic radiology residency program directors (PDs) of all ACGME-accredited residencies (194 programs) were identified in the United States. PD contact information and email addresses were found on the ACGME public website. An e-mail invitation was sent to the PDs to complete a 32-item electronic survey throughout July and August of 2021. One reminder was sent after a two-week period to promote participation and completion. Program coordinators (PCs) listed by ACGME were also sent an invitation if the PD was unable to complete the survey.

Survey participation was completely voluntary, and PDs could choose not to respond at any point throughout the study. Only fully completed surveys were included in data analysis, and no incentives were offered to respondents. E-mail addresses of participating PDs and PCs were collected for the purpose of ensuring only a single survey response from each program and verifying the legitimacy of the source of each submission.

The 32-question survey included four questions on a five-point Likert scale, seven dichotomous yes/no questions, nine demographic information questions, and two free-text questions. It was designed to take less than ten minutes to complete and was distributed using Microsoft Forms.

The survey was created to query PDs' and PCs' perceptions on the impact of COVID-19 during the 2020-21 radiology residency application cycle, ranking criteria for potential candidates, preferred interview modalities, and social media presence. The survey questions can be referenced in the appendix section.

Data were analyzed primarily by using descriptive statistics, reporting percentages, mean ± SD, and 95% confidence interval (CI) as appropriate. Paired Student's T-testing was utilized to measure the differences in the programs' top three application criteria for ranking applicants.

A total of 31 programs completed the online survey, yielding a response rate of 16%. Of the 31 programs who responded, 29 are affiliated with an academic institution. There was a fairly equal geographic distribution of responses, with 8 programs from the Northeast, 9 from the Midwest, and 7 each from Southern and Western programs. These programs offer an average 6.52 + 2.63 residency positions annually. Of the 29 programs affiliated with a medical school, the average percentage of residency positions filled by seniors from their home institution was 23.44% (95% CI, 15.71%-31.17%). 61.29% of responding programs reported offering more interview invitations in the 2020-21 application cycle than in the previous application cycle. 38 .71% of programs said that they offered roughly the same number of interview invitations in this application cycle, and none of the programs surveyed reported a decrease in this cycle's invitations.

When asked about the perceived success of virtual interviews during the 2020-21 application cycle, 21 programs (68%) said the new interview format was very successful, and 9 (29%) said they were somewhat successful ( Figure 1) . Moreover, 27 of the programs (87%) believed that they were able to adequately gauge applicants through virtual interviews. However, when asked about their personal preferences for virtual versus in-person interviews, the responses were more evenly distributed between virtual (11/31, 35%) and in-person (14/31, 45%) ( Table 1) .

Most programs (22/31, 71%) remain undecided whether they will resume in-person interviews once the pandemic ends, citing program cost, applicants' travel time, and applicants' socioeconomic barriers as considerations in their decision-making process. Additional common changes made during the COVID-19 era were website updates (28/31, 90%), virtual departmental tours (20/31, 65%), and virtual Q&A sessions with residents (23/31, 74%) and/or with the PD (15/31, 48%). Pre-COVID, the top three applicant criteria that programs reported using to rank candidates were their United States Medical Licensing Examination (USMLE)

Step 1 score (20/31, 65%), letters of recommendation (17/31, 55%), and medical student performance evaluation (12/31, 39%) ( Figure 2 ). Within the COVID era, these criteria -

Step 1 score (21/31, 68%), letters of recommendation (17/31, 55%), and MSPE (14/31, 45%) -remained essentially unchanged (p = 0.2254). Some additional, less-emphasized top pre-COVID criteria were USMLE Step 2 CK score (10/31, 32%), research activity (7/31, 22%), and leadership roles (6/31, 19%). These additional three proportions similarly stayed the same in the COVID era.

Lastly, eight of the programs (26%) claimed a robust social media presence before the onset of COVID-19, with over half of those using Twitter. After the pandemic began, 52% of those programs who had not previously utilized social media (12/23) created accounts, mostly via Twitter or Instagram ( Figure 3 ). However, when asked if social media interactions with applicants influenced interview invitations and/or ranking, most programs (27/31, 87%) answered either "neutral", "disagree", or "strongly disagree".

The purpose of this study was to provide information for radiology applicants and residency programs about the 2020-21 match and to guide future application cycles. In May of 2020, the Association of Program Directors in Radiology and Interventional Radiology supported the Coalition for Physician Accountability recommendations for virtual interviews and visits for all students participating in the 2020-21 cycle. [5] The virtual interview environment offers socially distanced interactions to assist in evaluating prospective applicants. Radiology residency programs employed various video-conferencing software applications during the interview season, which presented both opportunities and challenges.

Advantages to virtual interviewing includes reduced cost and travel time for applicants, increased flexibility of interview scheduling, and an increased number of interview days offered by residency programs. [6] However, several obstacles accounted for a varying quality of participants' experiences during the virtual interview process. These include applicants' distractions. [7] While this survey did not collect information on technological capabilities, the virtual delivery model could theoretically be biased towards students who have access to higher quality resources. However, survey respondents indicated that they were able to adequately gauge applicants via virtual interviews throughout the 2020-21 application cycle. The overwhelming majority of respondents answered that this new interview format was successful,

indicating an overall positive perception of the unprecedented experience.

When PDs were surveyed about personal preferences for virtual versus in-person interviews, the responses were more evenly distributed. Previous studies comparing virtual and in-person residency interviews across many medical specialties have produced nebulous results. [6] Although one cycle of complete virtual interviews has been administered, this is insufficient to fully support the exclusive use of virtual platforms. The lack of in-person experiences, stemming from the inability to visit the program site, is one commonly referenced issue that may contribute to PDs' hesitancy to permanently adopt a virtual model. [6] The three main selection criteria reported by PDs were USMLE Step 1 score, letters of recommendation, and the Medical Student Performance Evaluation. This is congruent with previous research identifying USMLE scores and letters of reference as top factors in ranking radiology applicants. [8] These applicant-screening criteria are correlated with higher clinical performance during radiology residency. [9] While letters of recommendation are still widely regarded as a primary ranking criterion, it is possible that remote electives have decreased the value of letters on average, presumably because it is harder to form personal connections and/or display one's knowledge with attendings in a virtual setting. Our survey results show that ranking criteria for candidates remain unchanged in light of the pandemic and the resulting transition to virtual interviews. It should be noted that these ranking criteria are subject to a significant change given the impending pass/fail scoring of USMLE Step 1. While this change is technically independent of virtual interviewing, how it will impact ranking remains to be seen, given the mixed reception from program directors. [11] It has been further suggested that additional criteria must be considered, including interpersonal and communication skills, professionalism, and the overall fit of an applicant at the unique program. [7] These attributes clearly influence the ranking process, yet it is already difficult to evaluate these skills by reading an application and then interviewing in-person, not to mention virtually. While the survey did not directly ask about PDs' perspectives on interpersonal skills and professionalism, they were queried about gauging the overall fit of candidates for their program. Most PDs felt able to properly evaluate applicants through virtual interviews in the 2020-2021 application cycle.

With no significant changes in interview quality or selection criteria, these data indicate that a virtual model may be a viable alternative to mirror traditional in-person interviews, supporting the Coalition for Physician Accountability's recommendation for mandatory virtual interviews.

Many programs remain undecided on whether they will continue to offer virtual interviews once it is again safe to conduct in-person interviews, whereas some programs have already decided to continue offering virtual interviews after the pandemic ends. As such, for upcoming radiology application cycles, it will be crucial for programs to facilitate an easily accessible and organized virtual interview day experience.

The surveyed PDs, evenly distributed by geographical region, commonly cited socioeconomic barriers as affecting their decision-making process regarding the possibility of maintaining virtual interviewing post-pandemic. In-person interviews incur financial burdens on applicants -a diagnostic radiology applicant, holding an average of $200,000 in medical school debt, would spend a median of $4000 interviewing in the pre-pandemic era on travel, hotels, and food. [12] Inherently, this process more heavily impacts applicants who have significant financial limitations, disproportionately affecting underrepresented students. Additionally, the virtual model offers flexibility by eliminating travel time, reducing schedule conflicts, and minimizing disruptions to an applicants' fourth year clinical rotation schedules. Virtual interviewing attenuates these barriers, allowing applicants to apply more widely and complete as many interviews as desired. Moreover, the financial savings for each program should be considered, as they need not provide multiple meals for each invited applicant. It follows that smaller programs potentially stand to benefit disproportionately from the virtual environment's cost-savings; applicants might also be more likely to apply to such programs when they cast a wider net. However, given the recent nature of the COVID-19 pandemic, information on specific cost-savings is limited, precluding any definitive cost-related conclusions at this time.

Nevertheless, the Coalition for Physician Accountability has again recommended that interviews be conducted virtually for 2021-22 application cycle.

Overall, these COVID-era changes will likely result in far more interviews conducted per radiology program going forward, as applicants apply more widely and as programs can afford to conduct more interviews. [13] This is supported by our data showing that over 60% of PDs offered more interviews in the 2020-21 cycle compared to previous years. Notably, away rotations were potentially disproportionally reduced and/or conducted remotely compared to home institution radiology electives during the 2020-21 cycle. While these electives may have been modified for an online format, home programs likely continued to offer them regularly. The reduced availability of in-person away rotations may have led to more home program engagement across the nation, yielding a home field advantage. Yet, this research was restricted to data from one application cycle and did not survey PDs on whether limited inperson away rotations created such an advantage. Further research analyzing the coming years' geographic matching trends could uncover whether such changes occurred throughout the virtual era.

However, possible negatives of virtual interviewing must be considered. Foremost is the potential for applicants to fail to adequately evaluate programs, partly owing to lack of physical exposure to the facilities, research laboratories, reading rooms, and city itself.

Moreover, perceived resident-faculty relationships and interactions with current residents are among the top five criteria used by applicants to evaluate programs. [14] These aspects may be more difficult to assess in a virtual environment. Furthermore, Rozenstein et al. (2021) postulate that, because students will apply to an increasing number of programs each year, the overapplication effect will favor the most competitive applicants to the detriment of still-qualified students who would have otherwise been granted an interview in years past. This process may lead to preferential interviewing of this same cohort of hypercompetitive applicants across many programs, yielding a smaller overall rank pool, thus increasing both the number of unmatched applicants and unfilled positions. There are no obvious solutions to this issue, though some ideas have been proposed, such as having the Electronic Residency Application Service implement an early, "special interest" period for students to apply to a limited number of their chosen programs. [13] In 2020, the Undergraduate Medical Education to Graduate Medical Education Review Committeea subsection of the Coalition for Physician Accountability with representatives from the NBME, ACGME, and AMA, among others -met to form recommendations for the future of residency transition. Regarding the optimization of residency applications and interviews, they recommended: 1) Creating a common interview invitation/scheduling platform for all programs, 2) Pilot studies testing innovations in reducing overapplication and concentrating applications to mutually interested programs, and 3)

Introducing a cap on how many interviews may be attended, based on evidence for each specialty's number of interviews required until an applicant no longer significantly increases their match rate. [10] As such, the specific, long-term outcomes of rapidly rising numbers of both applications and interviews should be monitored as the situation progresses.

Demonstrating increased utilization, social media has served as an educational and advertising platform for the radiology community at large. Major pages, including "FOAMrad" on Twitter and @radiopedia on Instagram have played a major role in advancing patient advocacy and outreach. [3] In the COVID-19 era, social media applications increase program visibility and engagement with prospective applicants. Nguyen et al. (2020) highlights the extent of communication lost by the virtual transition, including pre-interview dinners with residents, touring the program facilities and city, and attending noon conference.

[15] While social media outreach may not fully bridge the gap between virtual and in-person interview formats, it may afford applicants more detailed information with which to evaluate programs. Our survey found that radiology programs bolstered their social media presence by mainly creating Twitter and Instagram accounts. While engagement between programs and applicants increased, most PDs reported that these interactions did not influence the ranking of applicants. Even if programs increase their online presence, applicants must then actively utilize these platforms for their effects to be truly measured.

Yet, programs are not limited to Facebook, Instagram, or Twitter to reach applicants online.

Departmental residency program websites are particularly useful for candidates to collect important information and better prepare for their interview day. Program overview, listing of faculty members and residents, the curriculum, salary/benefits, and a description of the application process have been described as the most important contents of a radiology residency program website. [16] Without the familiarity afforded by in-person visits, PDs have found innovative ways to showcase their programs, including virtual webinars, live-narrated video tours, and pre-recorded residency recruitment videos. The content should generally be concise and targeted while avoiding overly produced videos. [17] Information from program websites, webinar and virtual tours influence where prospective applicants apply and guides interview preparation. As such, comprehensive improvements and maintenance of program websites are recommended to disseminate up-to-date program information and attract highly qualified applicants. [18] Limitations:

The study is subject to several limiting factors, and foremost a low response rate. Of the surveyed residency programs, 16% (31/194) responded. Although this survey evenly represents programs by geographic distribution, this may still bias the results and limit external validity. The mean number of residents per class for the surveyed programs was 6.5, indicating that we received more survey responses from smaller programs than larger ones. Differently sized programs may encounter altogether different sets of virtual interviewing challenges, so the survey results cannot be generalized to programs of all sizes. Both PDs and PCs were invited to complete this survey; given the subjective nature of some questions, specific opinions may differ between the director and coordinator within each program. Moreover, respondents were asked to gauge the success of the virtual interview process at their program. However, the definition of "success" was not specified in the survey, so what exactly comprises perceived success may vary between programs. Some survey questions asked for purely comparative information, rather than -for example -the exact number of interview slots, interview days required, and matched home-institution applicants pre-and post-COVID, thereby limiting the potential statistical analysis of these questions' data. Lastly, this survey pertains to an unprecedented, ongoing instance of history in the residency application process. Therefore, it is challenging to identify causative factors, especially given that only a small number of relevant studies are appropriate to reference and guide study design.

In response to the COVID-19 pandemic, radiology residency programs had to promptly adapt to the systematic transition to the virtual environment. Herein, we surveyed radiology PDs on the perceived success of virtual interviewing, preference of virtual versus in-person interviews, primary applicant-ranking criteria, and social media usage. Programs generally perceived success with the virtual model in selecting applicants, and programs' social media utilization increased overall. Additionally, the main criteria used to rank applicants remained unchanged. Based on these data and the socioeconomic factors involved, we posit that virtual interviews represent an adequate substitution for the in-person model and will persist even after the pandemic ends. A follow-up survey is planned for after the 2021-22 match to evaluate the impact of COVID-19 on two consecutive virtual radiology application cycles. Future research should be conducted to similarly measure matched radiology applicants' perceptions of the virtual interview process, showcasing views from both sides of the application cycle.

Lastly, the feasibility of hybrid models for radiology residency interviews should be investigated.

A complete version of the survey has been submitted separately in the appendix section.

All listed authors of this study have no conflicts of interest or disclosures. 

Evaluating Applicant Perceptions of the Impact of Social Media on the 2020-2021 Residency Application Cycle Occurring During the COVID-19 Pandemic: Survey Study

Use of Social Media in Radiology Education

Potential Implications of COVID-19 for the 2020-2021 Residency Application Cycle

APDR/APDIR position statement regarding residency recruitment in the 2020-2021 cycle

From Beginning to End: A Single Radiology Residency Program's Experience with Web-Based Resident Recruitment during COVID-19 and a Review of the Literature

Brave New World: Challenges and Opportunities in the COVID-19 Virtual Interview Season

Interview Techniques Utilized in Radiology Resident Selection-A Survey of the APDR

Do Residency Selection Factors Predict Radiology Resident Performance?

The Coalition for Physician Accountability's Undergraduate Medical Education-Graduate Medical Education Review Committee. Recommendations for comprehensive improvement of the UME-GME transition

& Motuzas C. Pass/Fail USMLE Step 1 Scoring -A Radiology Program Director Survey

Video Interviewing: A Review and Recommendations for Implementation in the Era of COVID-19 and Beyond

Residency Match During the COVID-19 Pandemic: The Clear and Present Danger of the Remote Interview

Evaluating Factors and Resources Affecting Ranking of Diagnostic Radiology Residency Programs by Medical Students in 2016-2017

COVID-19 and the Radiology Match: A Residency Program's Survival Guide to the Virtual Interview Season

Radiology resident recruitment: A study of the impact of web-based information and interview day activities

Best Practices for Video-Based Branding During Virtual Residency Recruitment

An Assessment of Radiology Residency Program Websites