key: cord-0682622-5hsckxee
authors: Brueggeman, David A.; Via, Garrhett G.; Froehle, Andrew W.; Krishnamurthy, Anil B.
title: Virtual Interviews in the Era of COVID-19: Expectations and Perceptions of Orthopaedic Surgery Residency Candidates and Program Directors
date: 2021-08-09
journal: JB JS Open Access
DOI: 10.2106/jbjs.oa.21.00034
sha: ac4f6767cfd2284b04b8668f16ee244dd9b58f34
doc_id: 682622
cord_uid: 5hsckxee

Orthopaedic surgery residency program directors (PDs) and candidates consider interviews to be central to the application process. In-person interviews are typical, but virtual interviews present a potentially appealing alternative. Candidate and PD expectations and perceptions of virtual interviews during the 2020/2021 orthopaedic surgery application cycle were assessed. METHODS: Candidates and PDs were surveyed electronically. Questions covered pre–virtual-interview and post–virtual-interview expectations and perceptions, and past in-person experiences (PDs and reapplicants) on the relative importance of application components, ability to assess fit, interview costs, and preferred interview mode. Identical questions allowed between-group comparisons. RESULTS: Responses included n = 29 PDs and n = 99 candidates. PDs reported diminished ability to assess candidate fit; social, clinical, and surgical skills; and genuine interest in the virtual context (each p ≤ 0.01). They placed greater importance on research and less on the interview in the virtual vs. in-person context (each p = 0.02). Most candidates (78%) reported fair/good ability to demonstrate potential and were better able to assess research opportunities than expected (p < 0.01). Candidates expected virtual interviews to increase the importance of research, transcripts, and recommendations (for each, p ≤ 0.02) and decrease the importance of the interview itself (p < 0.01). Compared with PDs, candidates overvalued research, United States Medical Licensing Examination scores, transcripts, and recommendations (each p ≤ 0.02) and may have slightly undervalued the virtual interview (p = 0.08). Most candidates (81%) and PDs (79%) preferred in-person interviews, despite both groups reporting monetary savings. CONCLUSIONS: Despite cost savings associated with virtual interviews, orthopaedic surgery residency PDs and candidates identified reduced abilities to assess candidate or program fit and displayed a preference for in-person interviews.

and broader in impact than any previous natural disaster, the greatest effect of the COVID-19 pandemic on undergraduate medical education was the widespread shift to virtual interactions and temporary removal of medical students from hospitals in 2020 [1] [2] [3] [4] [5] [6] [7] . As the Association of American Medical Colleges (AAMC) recommended, traditional in-person residency candidate interviews were suspended in favor of virtual interviews in the 2020/2021 cycle.

In transitioning from the traditional in-person format, residency programs tried to maximize the effectiveness of virtual interviews despite navigating largely uncharted territory. According to the AAMC, components of the interview process which programs sought to replicate virtually included faculty and panel interviews, group activities, resident interaction, role-playing activities, and facility tours. Virtual "happy hours" with residents, video tours of facilities, and increased supplemental information about the program and city were also proposed to maximize the experience for both candidates and programs 8 . In light of these novel developments, the orthopaedic literature is in need of studies assessing the impact of COVID-19 and virtual interviews on the application process.

Here, we report the results of a survey of US orthopaedic surgery candidates and program directors (PDs) during the 2020/ 2021 residency application cycle. Our primary purpose was to determine whether, on the basis of recent experiences, orthopaedic surgery candidates and PDs preferred a specific format for residency interviews. Secondary outcomes included assessments of PD and candidate perceptions of the importance of each component of the application package/process in virtual vs. in-person contexts. In addition, pre-virtual-interview expectations and immediate post-virtual-interview perceptions of PD ability to assess candidate fit, and vice versa, were addressed. Our primary hypothesis was that a higher proportion of candidates and PDs would prefer in-person interviews vs. virtual interviews. The null hypothesis was that neither group would express a preference.

T he PD and 2 research residents at a hybrid (academiccommunity) orthopaedic surgery residency program developed and distributed 2 anonymous, electronic surveys to orthopaedic surgery PDs and 2020/2021 candidates using Qualtrics XM software (Qualtrics International). Estimated time to complete either survey was 5 to 10 minutes. The local Institutional Review Board (IRB) determined that the study was exempt from IRB oversight. Full surveys are presented in Appendices 1, (PDs), and 2, (candidates). After initial distribution, 2 reminders were delivered 3 and 4 weeks later.

Potential candidate respondents were recruited by email from the list of 506 applicants to the authors' residency program. An assumption of the study is thus that the available recruitment pool is representative of the total population of 1,289 candidates in the 2020/2021 cycle. After excluding this study's home institution, a list of 196 potential PD respondents was compiled using the direct email addresses for primary PDs representing actively interviewing programs accredited by the Accreditation Council for Graduate Medical Education.

A screening question excluded any potential respondents who had not participated in virtual interviews during 2020/2021. Candidates and PDs provided basic information related to the application and interview process. Each survey presented candidates and PDs with the same sets of questions on the importance of different application components in the context of in-person and virtual interviews, rated using a Likert scale. Components included research, leadership, USMLE Step scores, medical school transcripts, letters of recommendation, and the interview itself.

Candidates were asked to rate their ability to assess program qualities in 2 ways: (1) expected ability before virtual interviews and (2) after completing virtual interviews. Qualities included program culture, city culture, research opportunities, facilities, and curriculum/structure. Candidates were asked to rate their ability to demonstrate their potential fit to programs. Reapplicants were asked additional questions regarding previous in-person experiences.

PDs were asked to rate their ability to assess candidate qualities in 3 ways: (1) perceived ability from past in-person experiences, (2) expected ability before virtual interviews, and (3) perceived ability after completing virtual interviews. Candidate qualities included competitiveness, program fit, social skills, clinical skills, surgical skills, and genuine interest in the program. The PD survey included questions about the ability to demonstrate program strengths and additional preparatory considerations for virtual interviews. All respondents were asked for their preferred interview modality and could provide free-text commentary.

Statistical analysis was performed in SAS 9.4 (SAS Institute), with significance set to a = 0.05. Response frequencies were calculated for each survey question within each cohort. For questions where statistical hypothesis testing was performed, numerical values were assigned to response levels and data were treated as ordinal. For importance of application components, values were 1 = not at all, 2 = slightly, 3 = moderately, 4 = very, and 5 = extremely. For ability to assess program/candidate attributes, values were 1 = could not assess, 2 = poor, 3 = fair, 4 = good, and 5 = excellent. Wilcoxon rank-sum tests were used to compare candidates and PDs for responses on application component importance. Within candidates, questions on application component importance, and preinterview/postinterview responses on ability to assess programs, were compared using Wilcoxon signed-rank tests. Within PDs, questions on application component importance were compared using Wilcoxon signed-rank tests, and responses about ability to assess candidate fit were compared between in-person, pre-virtualinterview, and post-virtual-interview contexts using Friedman tests.

Reapplicant data were analyzed independently as a sensitivity analysis of between-group comparisons, treating reapplicants as a separate group (Kruskal-Wallis tests). The hypothesis was that reapplicants would align more closely with PDs than first-time applicants.

C ompleted surveys were submitted by n = 29 PDs (15% response rate) and n = 99 candidates (20% response rate) who participated in virtual interviews. Among candidates, n = 12 were reapplicants. The sensitivity analysis found their responses did not differ from first-time candidates, so only pooled candidate results are presented.

Most PDs identified as heading university-based programs (69%). Fewer represented community (21%) or hybrid (10%) programs. Most programs accepted 4 or fewer residents per year (2: 7%, 3: 17%, 4: 35%, 5: 17%, 6: 17%, and 8 or more: 7%). Nearly all programs (97%) hosted somewhat or significantly fewer in-person rotations in 2020, and only 31% hosted virtual rotations. Most PDs (59%) interviewed 76 to 100 candidates in 2020/2021 (50 or fewer: 3%; 51 to 75: 14%; 101 to 125: 17%; and 126 or more: 7%), and most (72%) reported their numbers were the same as previous years (somewhat fewer: 7% and somewhat more: 21%). Virtual interviews resulted in 52% of PDs holding more interview days, but 48% held the same number. Roughly half (55%) of PDs invested in new/updated technology to support virtual interviews, but 83% reported spending less/significantly less money to support interviews compared with previous cycles (the same: 14% and more: 3%). Fig. 1 presents response distributions for importance of application components. PDs placed greater importance on research (S = 67.5, p = 0.02) and less on the interview (S = -67.5, p = 0.02) in the virtual context vs. in-person. Fig. 2 presents response distributions for ability to assess candidate fit. The ability to assess competitiveness was similar for inperson experiences, pre-virtual-interview expectations, and post-virtual-interview observations (x 2 = 1.08, p = 0.58). All remaining categories of ability to assess fit in virtual contexts were diminished vs. in-person (for each, x 2 ‡ 8.60, p £ 0.01). Overall, PDs' ability to demonstrate program strengths virtually was fair (38% ± 9.0%) or good (52% ± 9.3%), but the majority (79% ± 7.6%) stated they preferred in-person interviews (Fig. 3) . A representative sample of PD free-text responses is presented in Table I .

Most candidates (75%) applied to 51 to 125 programs (51 to 75: 15%; 76 to 100: 38%; and 101 to 125: 22%). For 51%, this was the same number they would have applied to if interviews were in-person; 48% applied to some/significantly more because of virtual interviews. Lack of travel influenced the choice to apply to more programs for 46% of candidates; 53% would have applied to the same number regardless of travel.

Just 16% of candidates applied to residencies in specialties other than orthopaedic surgery. Most candidates completed 2 to 3 rotations (62%) among home, away, and virtual (1 or none: 17%; 4 or more: 21%).

Compared with in-person, candidates expected that in virtual interviews, programs would place greater importance on candidate research (S = 599.0, p = 0.02), transcripts (S = 595.5, p = 0.01), and letters of recommendation (S = 1,189.5, p < 0.01), while placing less emphasis on the virtual interview itself (S = 21,191.0, p < 0.01) (Fig. 1) . Candidate expectations of the in-person context were broadly in line with PD; however, candidates tended to place greater importance on research (W = 23.71, p < 0.01) and USMLE

Step scores (W = 32.21, p < 0.01). In the virtual context, candidates overvalued research (W = 20.46, p < 0.01), USMLE Step scores (W = 27.77, p < 0.01), transcripts (W = 5.64, p = 0.02), and letters of recommendation (W = 8.93, p < 0.01) relative to PDs, and there was a trend toward candidates undervaluing the virtual interview itself (W = 3.02, p = 0.08) (Fig. 1 ).

Candidates reported being better able to assess program research opportunities than they expected before virtual interviews (S = 822.0, p < 0.01), with trends toward betterthan-expected abilities to assess program culture (S = 529.5, p = 0.06), as well as program curriculum and structure (S = 491.5, p = 0.06). Overall, candidates felt virtual interviews provided them with fair (36% ± 4.8%) to good (42% ± 5.0%) ability to demonstrate their potential to programs (although 20% ± 4.0% reported poor or no such ability). Accordingly, 81% ± 3.9% of respondents stated they preferred in-person interviews (Fig. 3) . A representative sample of candidate free-text responses is presented in (Table II) .

T his survey provides insight into expectations and perceptions of orthopaedic surgery residency candidates and PDs immediately after the virtual interview experience during the COVID-19 pandemic. Results indicate increased application burden, cost savings, reduced abilities to assess candidate/program fit, the importance of away rotations, and the emergence of "big fish" applicants. The orthopaedic surgery residency application has remained intensely competitive, with 81% of allopathic seniors successfully matching in 2021. Moreover, it has become increasingly expensive: 72% of applicants reported borrowing money to finance application costs, and costs have risen over 50% in the past 5 years 9,10 . The changes imposed by the COVID-19 pandemic are likely attributable to increasing an already intense competition, although virtual interviews and emerging technologies may be promising avenues to reduce costs. 1, 9 In 2018, the average candidate submitted 85.7 applications, representing a 59% increase over a 10-year period. This increase has challenged programs with evaluating ;124 applications per position and drove application fees to nearly $2,000 and total application costs to $8,205 per candidate 1,2,10,11 . Although the American Orthopaedic Association Council of Ordinal text responses were assigned numerical values as described in the text and as shown in the vertical axis labels. Boxes represent the interquartile range (25th to 75th percentiles), lines within the boxes are the medians, and the whiskers represent the 10th and 90th percentiles. Where 1 or more of the elements is not present visually on the plot, it means that 2 or more elements shared the same value. For example, for PD-IP responses in Fig. 1-A, the median, 25th percentile, and 75th percentile of the distribution all had the value of 3 and are thus shown as a line rather than a box. Each panel represents a single application component: Fig. 1-A How important is research? Fig. 1-B How important is leadership? Fig. 1 Program director perceptions of the ability to assess different candidate attributed under different interview conditions. Black boxes are for previous experiences of in-person interviews, gray boxes represent expected ability to assess before holding virtual interviews, and white boxes represent perceived ability to assess after holding virtual interviews. Ordinal text responses were assigned numerical values as described in the text and as shown in the vertical axis labels. Boxes represent the interquartile range (25th to 75th percentiles), lines within the boxes are the medians, and the whiskers represent the 10th and 90th percentiles. Where 1 or more of the elements is not present visually on the plot, it means that 2 or more elements shared the same value.

Each panel represents a single candidate attribute: Fig. 2 -A Ability to assess candidate competitiveness. Fig. 2 -B Ability to assess candidate fit for program. 

Residency Directors recommended for the 2020/2021 application cycle that allopathic candidates with an USMLE Step 1 score of >235 or £235 apply to no more than 40 or 60 programs, respectively, the majority of respondents here (79%) applied to more than 75 programs. Not surprisingly, 48% of candidates applied to more programs than they would have in a typical year and 52% of PDs hosted more interviews this year. Previous studies have reported theoretical virtual interview savings of more than $6,300 per can-didate, and our study supports this general finding (92% of reapplicants spent less money this year) 9,10,12 . Similarly, 83% of PDs spent less money during the most recent application cycle.

As summarized by the 2018 NRMP survey of PDs, interactions with faculty and interpersonal skills were critical aspects to the ranking of candidates, and these were best demonstrated with in-person interactions 13 . Likewise, candidates find that facility tours and informal interactions with residents and faculty are essential in assessing a program's structure and culture 14 . Our study revealed that PDs felt their ability to determine a candidate's competitiveness in a virtual format was nearly the same as in-person. However, more subjective traits such as social skills, candidate interest, clinical and surgical skills, and fit for their program were felt to be better evaluated in the in-person context. Candidates were found to undervalue the importance placed on the interview relative to PDs in both the virtual and inperson context. Inversely, candidates tended to overvalue USMLE Step scores, research, transcripts, and letters of recommendation compared with PDs. Candidates further reported that more objective measures such as a program's research opportunities were better assessed in the virtual format, compared with program culture and structure. Clearly, mismatches of perception exist that may continue to diverge without appropriate mentorship and evolving literature.

Consistent with a previous study on video conferencing during the orthopaedic adult reconstruction fellowship application, a majority of PDs and candidates (90% and 78%, respectively) found that virtual interviews allowed them fair to good ability to demonstrate their programs' strengths or their potential as a candidate 15 . Video

Pro in-person d Virtual interviews are certainly not as good as audition rotations and interviews. Neutral d Ideally perhaps, we could incorporate both, but the time involved is ridiculously expensive for the program. d I would like us to incorporate virtual interviews as round 1 and then do few second interviews in-person. d My strong preference would be for programs to submit and "lock" their rank lists 2 weeks prior to when the applicants have a deadline. Applicants could visit locations of interest without the idea or preference that it'll impact how they rank.

d Going forward, we will offer both in-person opportunities to come visit our city and hospital as well as keep the virtual interview. d The process went much better than I anticipated. I think the virtual interview process should stay. d We were still able to weed out those candidates that you don't want to rank. 

reporting each interview type preference.

Virtual Interviews in the Era of COVID-19 conferencing used in application cycles of other specialties has been deemed an acceptable means for interviewing candidates by faculty; however, the results have been more mixed for candidates 14 . The commentary provided by PDs in our study supports a more neutral to favorable view of virtual interviews relative to candidates (Tables I and II). A perceived disadvantage faced by both programs and candidates this year was the inability to participate in away rotations, except in limited circumstances. Surprisingly similar to a previous average of 2.4 away rotations, we found that 62% of candidates completed 2 to 3 orthopaedic rotations (including home and virtual away rotations) 2 . Away rotations are generally regarded by candidates as a way to impress programs with their genuine interest in the program, work ethic, social skills, and characteristics such as "grit" in a way not relatable through the Electronic Residency Application Service (ERAS) application 2, 16 . Recent studies (and anecdotally, our institution) have demonstrated that up to 50% of matched residents were the product of either their home program or away rotations and that applicants who completed 2 or more away rotations had a 10% greater chance of matching than peers completing fewer than 2 away rotations 1,2,12 . Although away rotations come at a financial costs to candidates, a previous study found that PDs value away rotations for finding "good fit" residents and 87% of PDs felt completing an away rotation increased applicants' competitiveness at their program 10, 16 .

A lack of away rotations may have limited candidates' ability to demonstrate their "grit" and overcome weaker objective metrics in the 2020/2021 cycle 2 . Anecdotally, our study supported this concern from candidates that their peers with the best objective metrics received most interview invitations. These "big fish" applicants, in theory, left fewer opportunities for the remainder of candidates. The virtual format this year likely drove this phenomenon by permitting applicants to complete more interviews, given the lack of time and travel constraints. Despite these concerns from candidates, the overall 2021 match statistics were largely in accordance with previous trends. Future studies should further explore this concern by focusing on characteristics of matched candidates and number of unmatched positions in the 2020/2021 application cycle. In addition, studies reevaluating similar metrics could provide valuable insight when the most recently matched cohort graduates in 5 to 6 years.

This study has several limitations which should be considered when interpreting the results. As a survey study, the results are subject to response bias. Response rates were approximately 15% for PDs (29/196) and 20% (99/506) for candidates (8% of 1,289 total applicants nationwide), which raises the issue of failing to capture a "neutral" set of respondents. Although other studies have used multicenter collaboration to achieve higher response rates, the authors believe that a reasonable cross-section of applicants was solicited, given the time constraint imposed by the upcoming match 16 . However, this study's data only offer insights into a very complex process. The results do not offer specific (or necessarily generalizable) suggestions or prescriptions d Something to prevent hoarding (should be done). It is unfair that a top 5 program for me would rather interview someone who will not even rank them.

d After being sent home from clinical rotations for 2 months, I decided to apply to a back-up program other than orthopaedics. I couldn't imagine the thought of not matching/sitting at home for a year.

d Interviewed at same program virtually and in-person. More awkward virtually for that program.

d The biggest con is the inability to meet the residents and faculty in-person and get a literal feel.

d Virtual socials ("meet and greets") seem minimally effective from an applicant perspective. d I think virtual interviews offer a lot of benefits that I believe will continue even after the pandemic. d I would implore the leadership to keep interviews virtual in subsequent years. A strong application cap would limit the total number of applications crossing the desks of selection committees.

for improving the process of future application cycles. They should therefore be interpreted in the context of current health guidelines and safe practices, as well as with more comprehensive survey studies. Moreover, the ability for candidates to assess programs and for PDs to assess candidates during interviews is inherently error prone, and accuracy of fit can only be determined after several years of residency. Thus, the data presented here on abilities to assess candidates and programs are specific to the a priori context of the interview process and should not be interpreted as post hoc evaluations of candidate-program fit.

Limitations aside, this is to the best of our knowledge the first study to assess the immediate expectations and perceptions of candidates and PDs regarding virtual interviews in the 2020/2021 orthopaedic surgery residency application cycle. PDs reported diminished ability to assess fit in the virtual context across most categories. Highlighting a large mismatch of perceptions, candidates tended to overvalue noninterview portions of the application process relative to PDs across all domains. Despite reported cost savings, candidates and PDs stated an overall preference for in-person interviews.

The current state of the residency match

The impact of COVID-19 on the orthopaedic surgery residency application process

COVID-19 implications on clinical clerkships and the residency application process for medical students

Proposed changes to the 2021 residency application process in the wake of COVID-19

The evolving impact of COVID-19 on medical student orthopedic education: perspectives from medical students in different phases of the curriculum

The importance of mentorship and sponsorship for thoracic surgery residency applicants during the coronavirus disease 2019 (COVID-19) pandemic

Effect of the COVID-19 pandemic on medical student career perceptions: a national survey study

A how-to guide: virtual interviews in the era of social distancing

Cost analysis of medical students applying to orthopaedic surgery residency: implications for the 2020 to 2021 application cycle during COVID-19

Current trends in orthopaedic surgery residency applications and Match rates

The cost of getting into orthopedic residency: analysis of applicant demographics, expenditures, and the value of away rotations

Otolaryngology residency interviews in a socially distanced world: strategies to recruit and assess applicants

Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond

Videoconference interviews for an adult reconstruction fellowship: lessons learned

Orthopaedic surgery residency: perspectives of applicants and program directors on medical student away rotations

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A292 and http://links.lww. com/JBJSOA/A293). This content was not copy-edited or verified by JBJS. n