key: cord-1025632-qho91yv9
authors: Sequeira, Nicola; Park, Keon Min; O’Sullivan, Patricia; Fee, Christopher; Foster-Barber, Audrey; McNamara, Margaret; Shinkai, Kanade; Vener, Margo; Kim, Hubert; Cornett, Patricia; Alseidi, Adnan
title: Designing successful virtual residency interviews: a survey-based needs assessment of applicants across medical specialties
date: 2022-03-09
journal: Global Surg Educ
DOI: 10.1007/s44186-022-00004-5
sha: 5952f792d31d8301a188871bce90bf1d19d8c427
doc_id: 1025632
cord_uid: qho91yv9

BACKGROUND: Residency selection in the United States relied on in-person interviews for many decades. The COVID-19 pandemic and recommendations from the Coalition for Physician Accountability (COPA) required programs to implement virtual interviews for the 2020–2021 residency selection cycle. Although virtual interviews may become the norm in the future, there is scant data at the institutional level to inform how to best approach this process. OBJECTIVE: To describe the perceptions of applicants to several residency programs at one institution on the importance of virtual recruitment features and assess the impact on their overall ranking decisions. METHODS: Applicants who interviewed for 12 medical and surgical residency programs during the 2020–2021 cycle at the University of California San Francisco were invited to participate in an anonymous survey in March 2021, after all interviews were completed. A survey consisting of 26 questions was administered to applicants on features that are important during interviews and the impact on their ranking decisions scored on a 5-point Likert scale. RESULTS: Of the 1422 participating applicants, 303 (21%) completed the survey. The most important feature for applicants during the interview day was getting a feel of the program (92%). Conversations with residents (91%) and faculty (79%) were also highly rated. Respondents reported morale and happiness of residents (71%) as an extremely important factor in their overall ranking decision . CONCLUSION: Programs should consider prioritizing features that aid in alignment with getting to know residents and faculty and provide a sense of morale over emphasis on the institutional and location features. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s44186-022-00004-5.

Residency interviews are an essential component of the residency selection process. Since the establishment of the National Resident Matching Program (NRMP) in 1952, residency programs across the United States have offered in-person interviews to thousands of applicants to learn more about the applicant and assess compatibility with their program. In-person interviews allow applicants to gain an in-depth understanding of the program including the culture, strengths, and weaknesses. However, applicants and programs alike find in-person interviews burdensome. Studies report that interview days lead to significant financial cost with applicants spending an average of $4000 on interview-related expenses and loss in productivity [1, 2] . Residency training programs also incur significant expenses with in-person interviews that could be reduced by virtual interviews. One study reported the average cost of $5031.68 per in-person interview compared to $2159.40 per virtual interview [1, 3] . Several pre-COVID-19 pandemic studies found web-based interviews as a potential solution to these problems [4, 5] .

A number of studies offer insight into what features are valuable in the virtual interview process [4] [5] [6] . This includes the importance of pre-recorded informational videos, virtual tours, informal social events that are informative, providing applicants the opportunity to ask questions and creating a welcoming environment for the applicant. However, these studies relate to single smaller residency and fellowship programs or disciplines where applicant priorities are different. For virtual interviews to be successful, institutions need data to support choices that should be made in common across programs. Our study examined virtual interview features which applicants applying to 12 programs at one institution found valuable. These data will provide guidance to institutions and residency programs on what institutional resources and individual program efforts can improve the virtual recruitment process, as it may become a standard part of the residency application process.

The Institutional Review Board at University of California San Francisco (UCSF) approved the study. For the 2020-2021 residency match cycle, interviews were conducted using zoom or other similar platforms. Programs were aware that a shift to the online platform could result in technical issues and were prepared to handle any difficulties that would arise during the interview day. Eighteen residency programs at UCSF were invited to participate and 12 programs volunteered. An email invitation was sent by program directors to applicants between the rank order deadline and match day. All applicants who interviewed for any of the 12 programs received the survey via an email at the completion of 2020-2021 interview cycle (March of 2021) and received weekly reminder emails for 2 weeks. Participation was voluntary and anonymous. An online questionnaires platform (Qualtrics, Provo, UT, USA) was utilized to distribute the survey. The survey consisted of 26 questions (Supplemental material 1). Using 5-point Likert scales, applicants were asked to rate the effectiveness of 14 virtual features (Fig. 1 ) on interview day, identified from the literature and by the authors, and also asked to rate 17 items (Fig. 2 ) selected based on themes taken from preexisting literature indicating importance on their overall ranking decision. Standard statistical measures were also used to categorize applicant responses where applicable.

Twenty-one percent (303/1413) of the invited applicants from 12 residency programs completed the survey (Supplemental Material 2). Respondents comprised of 52% female and 43% were of white ethnicity. Additional demographic details of study participants are available as Table 1 .

Applicants (92%, median score, 5 1QR 5-5) reported that a very important interview feature is the ability to get a feel of the program followed by conversations with residents (91%, median score, 5 IQR 5-5) and conversations and interviews with faculty (79%, median score, 5 IQR 5-5) (Fig. 1 ). Post-interview social events (11%) and interview with department chair (9%) were reported as not important. Not all 14 features were available for every specialty. Comparisons between applicant program responses to features that were very important and not so important during interview day yielded similar viewpoints (Supplemental 3).

Applicants (55%) also identified important features that helped them understand the program was interaction with residents followed by questions answered by residents (54%) and social time with residents (48%) ( Table 2) . Sixty-four percent of applicants reported being extremely satisfied with resident engagement during interviews followed by programs creating a positive experience (62%) and conversations with faculty and residents (57%) ( Table 3) . A sample of applicants free-text responses to additional features and "best practices" they would like adopted in future virtual interviews is represented in Table 4 .

Seventy-one percent (median score 5, IQR 4-5) rated resident morale and happiness as extremely important elements on their ranking decision followed by learning climate (64%, median score 5, IQR 4-5) and overall quality of residents (51%) (Fig. 2) . Presence of the Children's hospital (31%) and cost of housing (11%) were most frequently reported as not-at-all important elements.

Out of 31 items evaluated, only five items had significantly different distributions of answers when compared between procedural and non-procedural-based specialties but none of these were highly valued features. There were only four statements that the procedural and non-procedural specialties answered significantly differently, including importance of flexibility of rotations and patient population (Supplemental Material 4). 

While virtual interviews for residency selection were considered for many years to alleviate the financial burden and time-away from medical school and work for both applicants and residency programs alike, in-person interviews were prioritized until the recent COVID-19 pandemic [1, 3, 7, 8] . Our study adds valuable information to initial studies highlighting preliminary strategies on the virtual residency selection process and identifying common features that were valued by applicants across specialties. Nearly, all applicants across specialties were interested in getting a "feel of the program," indicating the critical importance of this interview feature on interview day, whether in-person or virtual. Our findings are consistent with previous studies that showed applicants valued spending time with current residents and faculty. This shows that what applicants valued prior to the pandemic has not changed [9] [10] [11] . Thus, we recommend allocating more interactive time with current residents via pre-interview socials or informal meetings as well having informal meetings with faculty, since post-interview socials were rated as less valuable. Informal interactions can maximize applicants' subjective sense of the program that will be of benefit during virtual recruitment sessions.

Morale and happiness of current residents and program culture were highly valued features when ranking a program. This can be attributed to the impact of the pandemic leading to increased focus on wellness and avoiding burnout during residency. This is also similar to results of studies prior to virtual recruitment in which applicants reported resident satisfaction, happiness, and "friendliness" as the most important factors in choosing their future residency program [10] [11] [12] . Thus, features that maximize applicants' understanding of the residency program culture should be employed during virtual recruitment. Although there were a few features that procedural and non-procedural specialties answered differently, overall, the responses are similar across all the specialties showing that residency applicants' value common features in residency.

Features which applicants did not prioritize on their virtual interview days included tours of the hospital sites and formal faculty introductions. These features may not add much to the information applicants can gather online, and they can prolong the day. Furthermore, applicants may prefer learning about these aspects of the program firsthand by talking to their future co-residents. Applicants reported that cost of housing, salary and benefits, and distribution of rotations were not as important in their ranking decision. Factors that contribute to the educational environment have frequently outweighed these features [13] [14] [15] [16] . Hence, this information can be distributed via pre-interview day email or posted on the program website to shorten the length of the virtual interview day. This study has several limitations. Not all specialties were represented. Due to the small sample size from a single institution, this may not be representative of the entire residency applicant pool. The low response rate is a major limitation. Applicants took the survey within a limited time frame, between submission of rank lists and before match day to minimize response bias that postinterview contact can have on rank lists. Although the response rate is low compared to other published studies, this is the first study to the best of our knowledge that assesses and provides an understanding into features that applicants valued across multiple specialty programs and not a single residency or fellowship program. A larger response rate, however, would be needed to fully understand the challenges of virtual interviews and the impact it is has on ranking decisions. Furthermore, some of the features such as "Children's hospital setting" may not be as applicable to some specialties and contributed to the heterogeneous answers. Lastly, this study did not address what programs need from the interview process. Knowing what both sides want and implementing these features will make the virtual interview process more successful. Future studies should address this question.

With the growing emphasis on the high cost of in-person interviews and the burden it places on applicants with limited financial resources, virtual interviews may become a permanent part of residency recruitment even after the COVID-19 pandemic. As we continue to refine the process, our study informs residency programs and helps better understand features applicants' value in developing virtual interview days. We recommend focusing on these essential elements rather than spending valuable time on information applicants find less useful. 

Randomized evaluation of a web based interview process for urology resident selection

Financial and educational costs of the residency interview process for urology applicants

Feasibility and usability of tele-interview for medical residency interview

Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond

Successful virtual interviews: perspectives from recent surgical fellowship applicants and advice for both applicants and programs

Residency interviews in radiation oncology after COVID-19: perspectives from recently matched applicants

Perception of medical students and residents about virtual interviews for residency applications in the United States

The economic burden of residency interviews on applicants

Characteristics of the residency interview process preferred by medical student applicants

Senior medical student opinions regarding the ideal urology interview day

Emergency medicine residency applicant views on the interview day process

Emergency medicine residency selection: factors influencing candidate decisions

Applicant considerations associated with selection of an emergency medicine residency program

Factors applicants value when selecting an emergency medicine residency

Factors that influence medical student selection of an emergency medicine residency program: implications for training programs

Factors affecting medical students' selection of an internal medicine residency program