key: cord-0720424-qy7r5d7r authors: Chen, S.; Schroeder, M.; Park, T. K.; Zaworski, E.; Topolski, N.; Anishchenko, K.; Bae, E.; Stuard, W.; Zhu, C.; Quach, S.; Panjwani, S.; Parsa, S.; Patel, J. title: A Match Made in Cyberspace: Applicant Perspectives on Virtual Residency Interviews date: 2022-02-09 journal: nan DOI: 10.1101/2022.02.07.22270645 sha: 7adc02859bf9115d7104b20f2697b3c599192c4b doc_id: 720424 cord_uid: qy7r5d7r Purpose: The COVID-19 pandemic precipitated an entirely virtual 2020-2021 residency application cycle. As many specialties plan to continue virtual interviews, it is essential to evaluate what aspects worked well and what can be improved. Methods: From April to June 2021, authors surveyed participants of the 2020-2021 Match about their experiences with the virtual application cycle. Survey items asked about benefits and drawbacks of virtual interviews, the utility of social media, and thoughts regarding preference signaling mechanisms and application/interview limits. Results: Participants (n=158) represented 24 states and applied to 31 specialties. The majority (73.1%) were satisfied with their experience during the 2020-2021 Match. Applicants found Twitter (78.4%) and Instagram (69.1%) to be the most helpful social media platforms. Almost all applicants (98.1%) believed that the virtual setting increased financial access to interviews. The majority (68.6%) indicated that residency programs should continue to provide virtual interviews. However, most applicants (73.9%) felt that the virtual setting allowed more students to hoard interviews. Many (66.1%) also felt their medical school did not provide adequate electronic equipment for conducting virtual interviews. While many applicants (56.9%) did not support a cap on the number of applications one can submit, most supported a limit on the number of interviews one can attend (62.7%) as well as a mechanism to signal genuine interest in a program (59.7%). Conclusions: Our study highlights that, across geography and specialty, many applicants would like virtual interviews to continue and offers insight into how medical schools and residency programs can best support applicants in the virtual environment. The virtual setting also provides an opportunity to evaluate mechanisms to address the congestion from the increasingly high number of applications residency programs receive, with most applicants supporting interview caps and preference signaling. The COVID-19 pandemic caused a paradigm shift in the residency application process, also called "The Match." 1 The Match normally consists of residency programs interviewing eligible candidates in their final year of undergraduate medical training prior to both applicants and programs submitting ranked preferences. 2 These interviews previously occurred in-person and were an opportunity for students to tour an institution and meet with current residents and faculty. In addition, many programs offered "second looks" where students could revisit a location to help determine their rank lists. However, due to the COVID-19 pandemic, travel restrictions required all residency interviews during the 2020-2021 Match cycle be virtual on top of the suspension of in-person away rotations. 3, 4 Until now, there has been little information published on virtual residency interviews other than cost-saving analysis. 5 Because of the virtual setting, the COVID-19 pandemic posed unprecedented challenges for applicants during the 2020-2021 residency Match cycle along with continuing to necessitate virtual interviews during the 2021-2022 application cycle. The exclusively virtual format of the 2020-2021 Match cycle dramatically altered the landscape of residency applications. The switch to virtual interviewing sparked a surge in social media presence. [6] [7] [8] [9] For applicants, social media was used to directly connect with current residents and faculty, learn more about programs, and rank programs. 10 Residency programs' use of social media also increased for nearly all specialties. 6 Additionally, residency programs offered a variety of virtual programming prior to, during, and after the interview day. [6] [7] [8] [9] Pre-interview programming was largely advertised on social media, email lists, and program websites. While program websites were available to all prospective applicants to provide information about a given residency, this was not always the case for social media and email lists. 11 Many organizations held virtual open houses, enabling applicants to interact more closely with programs while connecting with program directors and current residents. 12,13 On interview days, programming consisted of formal interviews, social hours with residents, virtual tours, and other engagement. Post-interview programming mainly consisted of virtual second looks. 14, 15 Beyond social media and programming, the Match process has always required a large commitment of money and resources for applicants, especially when conducted in-person. Given that students on average accumulate over $200,000 of debt during medical school, there has been ongoing discussion on how to mitigate costs for students. 16, 17 Aside from tuition, one of the most significant cost burdens to students is residency interviews, which cost students approximately $5,000 on average prior to the pandemic. Moreover, for some specialties, the average interview cost was over $8,000, with some students reported costs up to $30,000. Interview-related travel expenses make up a large part of these costs. 18 Given this, several studies have been conducted to investigate potential cost saving alternatives to the traditional residency interview process prior to the COVID-19 pandemic which found that utilizing video interviews for residency and fellowship programs could be a cost-saving alternative. 5, [19] [20] [21] [22] The 2020-2021 application cycle confirmed this. Transitioning to virtual interview platforms was estimated to save candidates nearly $6,000. 23 However, despite the potential financial benefits, virtual interviewing may exacerbate inequities that exist for those without high-quality equipment and high-speed internet access due to the phenomenon of "digital redlining." 24-26 This is problematic given that most strategies for success in virtual interviewing suggest that applicants have a good internet connection as well as a quality laptop and web camera. 3 Given that virtual interviews present both benefits and drawbacks, we examined how a cross section of 2020-2021 Match applicants viewed this development. This study offers unique perspectives from the first cohort of applicants to participate in virtual interviews across multiple medical schools and specialty interests. These include attitudes regarding social media platforms, programming methods, program websites, access to resources necessary for virtual interviews, evaluation of fit, overall satisfaction with the virtual platform, and thoughts on how to best proceed with future cycles. A 30-question survey instrument was developed by consulting members of a national medical student committee focused on undergraduate medical education, 2020-2021 residency applicants, and survey experts in addition to reviewing surveys conducted by the National Resident Matching Program and specific specialties. Questions utilized either a 5-point Likert scale or free text boxes to assess applicant demographics, opinions on virtual interviews, availability of technology and resources, evaluation of fit, and thoughts on future cycles (Appendix A). Five M.D. and/or Ph.D. experts in research and survey studies pre-tested our survey. These individuals read through the data collection tool to assess that the questions posed would answer our research questions and ensured that the wording minimized bias. A pilot study was then conducted with approximately 20 fourth-year medical students via our REDCap survey tool prior to wide distribution. All applicants in the 2020-2021 Match process were eligible to take the survey, including but not limited to senior medical students, international medical graduates, and interns applying to PGY-2 spots regardless of their Match outcome. The survey was electronically distributed to 40,403 medical student members of a national medical student organization with support from the organization's staff. Approximately 11,243 students received the correspondence and of those less than one fourth were eligible for the survey. Therefore, we estimate that the total eligible number of students that were invited to All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9, 2022. ; https://doi.org/10.1101/2022.02.07.22270645 doi: medRxiv preprint participate in the survey would be less than 2,800. In addition, grassroots efforts were utilized to promote the survey through various social media platforms, including medical school class email lists, Facebook groups, and GroupMe groups. It is not possible to determine the exact numbers of students reached through these methods, and it is likely that there was an overlap with the students included in the communication from the national organization. With a total of 158 responses, we conservatively estimate our response rate to be approximately 5%. Participation was voluntary, and results were collected anonymously. Survey responses were collected from 4/27/2021 through 6/20/2021. Descriptive statistics, including mean, standard deviation, and 95% confidence intervals, were conducted for the data using a Python computing package (https://github.com/melmschr/CMEproject/blob/main/survey_analysis.ipynb). Likert categories of "Agree" and "Strongly Agree", "Disagree" and "Strongly Disagree", "Helpful" and "Extremely Helpful", and "Unhelpful" and "Extremely Unhelpful" were grouped together for analysis. Survey takers were asked seven free-response questions about their interview experience. Two independent readers analyzed these responses for common themes. Disagreements were resolved by discussion between the two analysts. The Vanderbilt Human Research Protections Program granted IRB exemption status. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Within the 158 survey respondents a wide variety of degrees pursued, genders, ethnicities, geographic regions, Match status, and specialty were represented. While approximately 70% of respondents were pursuing an MD, the remaining participants included DO and dual degree graduates. Similar diversity was captured in regard to gender and ethnicity. Additionally, the survey was not limited to responses from students that successfully matched, with almost 4% of respondents reporting that they did not match. 24 states were represented among the survey respondents with the majority from the southern region. This survey also captured medical students applying into a wide range of specialties. Applicants to 31 specialties were represented in our dataset including 8 combined specialty programs. The majority (78.2%) of respondents applied to non-surgical specialties. Table 2 : Applicant perspective on virtual interviews All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The virtual setting allowed me to attend more interviews because of reduced financial constraints. My school provided adequate coaching and/or guidance on how to prepare for virtual interviews. (12. 4-24.9) 18.0, (11.9- We asked applicants about their concerns regarding virtual interviews. Results are shown in Table 2 . The most commonly held concern, by 73.9% of respondents, is that the virtual setting allowed more students to "hoard" interviews. Ranking residency programs was also a concern, with 62.4% agreeing that the virtual setting made the ranking process challenging. Furthermore, assessing residency programs was felt to be difficult, with 51.6% of respondents feeling the lack of in-person away rotations limited their ability to evaluate programs and only 40.1% feeling that they could effectively evaluate program "fit" with virtual interviews. Similarly, 36.9% of respondents felt it was difficult "to be myself" when interacting virtually with programs. Finally, 48.7% felt that the virtual setting decreased their ability to show genuine interest in a program. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. When asked about financing interviews, 98.1% of respondents agreed that the virtual setting increased financial access to interviews, and 77.1% agreed that they were able to attend more interviews because of reduced financial constraints. When asked about access to resources, an overwhelming majority (85.2%) agreed that they had the necessary resources to complete residency interviews. More specifically, when asked about resources provided by their institutions, 59.8% agreed that their school provided adequate physical space for virtual interviews while 63.3% agreed that their school provided adequate coaching or guidance on preparing for virtual interviews. In contrast, only 21.2% of respondents thought their school provided adequate electronic equipment (web cameras, lights, etc.) for virtual interviews, while 66.1% did not. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. We also surveyed applicants on the helpfulness of various resources provided by residency programs. Results are shown in Table 3 . Residency programs utilized a variety of social media platforms to showcase their program. The residency applicants were most likely to view Twitter (78.4%) as a helpful resource, followed by Instagram (69.1%), Facebook (6.3%), Tik Tok (5.9%), and LinkedIn (5.3%). All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9, 2022. ; https://doi.org/10.1101/2022.02.07.22270645 doi: medRxiv preprint In terms of programming, the majority of students felt that pre-interview programming such as virtual away rotations or information sessions (73.4%), virtual social hours (77.3%), and residency open houses (65.3%) was helpful to evaluate programs. Interview day programming was also well-received by students who felt that interview day presentations by program director/chair (84.4%), formal interviews with faculty (87.7%) and residents (90.1%), and informal breakout rooms (81.3%) helped assess fit with a program. Additionally, many students found informal conversations with other applicants (52.1%) as well as social media accounts of residents and faculty to be helpful (53.8%). On the other hand, virtual second looks were deemed to be helpful by only a minority of applicants (38.2%). Each aspect of program websites was found helpful by most applicants. Information about the area (89.5%), post-residency placement (83.0%), and resident benefits (77.0%) were most frequently identified as helpful. Applicants should only be permitted to APPLY to a limited number of programs. Applicants should have a mechanism to indicate genuine interest in a program, commonly referred to as a 'token system'. Residency programs should continue to provide virtual interviews as an option even when in-person interviews are offered. 11.8, (6.7-16.9) 19.6, Lastly, we asked applicants about their opinions on how future application cycles should be conducted. A minority of respondents (28.1%) agreed that there should be a limit to the number of programs to which one can apply. In contrast, a majority (62.7%) thought there should be a limit to the number of programs at which one can interview. A majority (59.7%) also agreed with having a "token system" or preference signaling mechanism to indicate genuine interest in the program. Overall, most of the respondents (73.1%) were satisfied with their experience during the 2020-2021 residency application cycle, and most (68.6%) agreed that residency programs should continue to provide virtual interviews as an option even when in-person interviews are again possible. Five common themes from the qualitative analysis were identified (Figure 1 ). Many applicants found it difficult to determine compatibility, or "fit", with a program due to the lack of casual, inperson interactions. In addition, applicants could accept more interview invitations because of fewer geographical, financial, and time constraints. One applicant mentioned they attended two interviews on the same day for two institutions located across the country, which would not have been possible in-person. Resident interactions served as the most important interview day component for determining program fit within the virtual setting. However, the virtual setting hindered applicants' ability to evaluate interactions among faculty, residents, and other hospital staff. In terms of resources provided to students, medical schools ranged from providing students with interview kits with laptop stands and ring lights to providing nothing. There was consensus that applicants would have accepted fewer interviews due to financial, geographic, and time constraints had interviews been conducted in-person. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9, 2022. ; https://doi.org/10.1101/2022.02.07.22270645 doi: medRxiv preprint Thematic analysis of survey commentary. Five major themes were identified based on five freeresponse questions over number of interview invites, impact of technology innovations, components of the interview day that best allowed evaluation of fit, difficulties faced when determining fit, and the resources the medical schools provided for virtual interview. Our study provides important insights into the perspectives of residency applicants nationwide across various specialties who were interviewed virtually during the 2020-2021 Match. Multiple studies evaluated the use of virtual interviews for residency applicants prior to 2020-2021, but the evidence was limited due to the dearth of virtual interviews offered at that time. 3 However, a more recent study that evaluated the perspective of 2020-2021 residency applicants (n=104) found that their cohort of medical students "prefer in-person interviews over remote interviews." 27 Conversely, our study found that applicants were satisfied with remote interviews and wanted them to continue, even when in-person interviews become possible. This is in agreement with a 2019-2020 study before the COVID-19 pandemic where the majority of medical students favored having a virtual interview mainly due to financial considerations. 28 In light of the pandemic, social media usage has grown significantly. 29 Residency programs have followed suit by expanding their presence online. 6, 10, [30] [31] [32] This includes developing new accounts on Facebook, Twitter, Instagram, and other platforms, as well as posting informational YouTube videos online. To date, few studies have evaluated the utility of these platforms for applicants. One study published pre-Match found that students do not believe Twitter or Instagram presence is important when evaluating residency programs. 27 In contrast, our study demonstrated that a large majority of respondents found these two platforms helpful, while LinkedIn and TikTok were seen as unhelpful. Along with the usefulness of social media, our results agree with a prior study that indicated program websites as a whole were helpful to applicants. 27 Our study further evaluated the helpfulness of individual aspects of program websites, finding the most helpful to be information about the area. Applicants likely placed high value on this element due to their inability to travel to program sites. In terms of programming organized by residency programs, information presented on interview day was overwhelmingly found to be the most helpful. This included presentations by program directors and department chairs and formal interviews with faculty and residents. 33 This has not yet been assessed in previous literature. In addition to changing the amount of time required for interviews, the virtual setting also altered the financial cost and technological requirements of interviewing. While finances remained a concern for many, most applicants felt they had the resources to complete their virtual residency interviews successfully. Such resources included adequate physical space, coaching, and guidance on how to conduct virtual interviews. The one exception was that most applicants' schools did not provide electronic equipment such as web cameras or lights despite this being All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9, 2022. ; https://doi.org/10.1101/2022.02.07.22270645 doi: medRxiv preprint recognized as one of the necessary strategies to having a successful virtual interview. 3 As virtual interviews continue, this could be an opportunity for schools to combat inequities by providing their lower income and disadvantaged students with the technology required to succeed in this setting. This is critical in the age of "digital redlining," where marginalized groups are already excluded from accessing adequate broadband coverage and thus may face difficulties within the virtual setting. [24] [25] [26] While our participants were generally satisfied with the virtual interview process, our study also found that the abrupt adoption of virtual interviews raised significant concerns among residency applicants. Our respondents expressed concerns about virtual interviews consistent with previous studies. 27 A previous study noted students' concerns about their ability to fully assess programs in the virtual setting. Our study showed that only 40.1% agreed that virtual interviews allowed one to effectively evaluate "fit" within a program, and 62.4% agreed that the virtual setting made the ranking process more challenging. However, the biggest concern with virtual interviews was that it would allow students to "hoard" interviews. In our study, only a minority of applicants thought there should be a limit to the number of programs applied. In contrast, the majority of the applicants also thought there should be a limit to the number of interviews each applicant can attend. Emergency medicine, ophthalmology, and psychiatry residencies implemented maximum interview caps for 2021-2022. 33,34 If medical students continue to agree that interview limits are a benefit, we may see these limits implemented by additional specialties. While many programs experienced an increase of applicants, this was a trend even before the pandemic. 35 As a method to possibly address this congestion, a majority of respondents thought there should be a preference signaling mechanism in place. This was in line with most applicants' thinking that the virtual setting made it more difficult to show genuine interest in a program. Intriguingly, otolaryngology residencies pioneered a signaling preference system in 2020-2021. 36 An informal survey showed that the majority of the applicants were satisfied with the signaling preference. 36 Following suit, residencies for Internal Medicine, Dermatology, General Surgery will be utilizing a signaling preference for their respective specialty's supplemental ERAS. 37 It will be interesting to note if this preference signaling continues to be well-received by applicants and if additional specialties in the future will adopt this method. Furthermore, support for the study of both signaling methods and virtual interviews was recommended in the Coalition for Physician Accountability's Undergraduate Medical Education-Graduate Medical Education Review Committee's 2021 report that called for significant changes to the residency application process. 38 From our results, we have developed recommendations for medical schools, residency programs, and ERAS. Medical schools should endeavor to provide their students with all resources necessary for virtual interviews including web cameras and lights. Residency programs should [1] utilize social media platforms to reach applicants (particularly Instagram and Twitter), [2] All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9, 2022. ; https://doi.org/10.1101/2022.02.07.22270645 doi: medRxiv preprint highlight and expand upon information about location on their websites, [3] invest time in interview day programming [4] research the financial value and efficacy of virtual interviews, and [5] further evaluate the usefulness of the components of virtual interview days. Additionally, ERAS should work with specialty societies to evaluate the benefits and consequences of instituting a limit to the number of interviews that applicants may accept to lessen interview "hoarding". Finally, ERAS as an institution should continue to evaluate the efficacy of the nascent components of the application process such as supplemental applications and signaling mechanisms. Our study has several limitations. Given that survey participation was completely voluntary, there was potential for response bias towards applicants with particularly strong feelings about their Match experience. We were also able to only collect responses from a small fraction of all Match participants. However, unlike many other studies on this topic that are specialty-or institution-specific, the diversity of respondents allows us to identify common themes across geographic location and specialty. Nevertheless, more research with larger sample sizes is needed to better understand student's perspectives. Our study demonstrates that across specialty and geography, applicants support virtual interviews continuing even when in-person interviews are again possible. Additionally, our results indicate that residencies can help applicants better assess program "fit" through robust social media use, especially on the Instagram and Twitter platforms, and by elaborating upon information about the area on their websites. Medical schools can better support applicants by providing electronic equipment essential for successful virtual interviewing. Finally, the virtual setting allows for the implementation and evaluation of mechanisms that address the congestion resulting from the increasingly high number of applications residency programs receive, as most Match participants support interview limits and preference signaling mechanisms. Potential Implications of COVID-19 for the 2020-2021 Residency Application Cycle The Current State of the Residency Match Preparing for the interviewing process during Coronavirus disease-19 pandemic: Virtual interviewing experiences of applicants and interviewers, a systematic review Remote interviews for medical residency selection during the initial COVID-19 crisis: a national survey Using Video-Conference Interviews in the Residency Application Process Adaptations in anesthesiology residency programs amid the COVID-19 pandemic: virtual approaches to applicant recruitment Orthopaedic Surgery Residency Program Social Media Presence During the COVID-19 Pandemic Showcasing Your Cardiothoracic Training Program in the Virtual Era Analyzing the Proliferation of Social Media Use Among Orthopaedic Surgery Residency Programs #ENT: Otolaryngology Residency Programs Create Social Media Platforms to Connect With Applicants During COVID-19 Pandemic COVID-19 and the Radiology Match: A Residency Program's Survival Guide to the Virtual Interview Season CP/Medical_Students/IMShowcase21.aspx?hkey=8dc199fb-f458-4d69-aa12-078dd157dc71 Urology Residency Applications in the COVID-19 Era No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 9 Time and Financial Costs for Students Participating in the National Residency Matching Program (the Match©): 2015 to 2020 Virtual Surgical Fellowship Recruitment During COVID-19 and Its Implications for Resident/Fellow Recruitment in the Future Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process Avoiding the Virtual Pitfall: Identifying and Mitigating Biases in Graduate Medical Education Videoconference Interviews Digital Redlining: How Major American Communication Companies Are Controlling Who Gets Broadband Access or Not Seifi A, Mirahmadizadeh A, Eslami V. Perception of medical students and residents about virtual interviews for residency applications in the United States A rise in social media utilization by U.S. neurology residency programs in the era of COVID-19 The Coalition for Physician Accountability's Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC) The authors would like to thank Joseph Camerano, Rijul Asri, Ryan Chiu, Samuel Neher, Norman Farr, Matthew McEchron, Brian Drolet, Christine Ford, and Shane McGoey, and the other individuals who assisted with the development and distribution of this survey. All rights reserved. No reuse allowed without permission.(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.