key: cord-0903259-ohgcv8xr
authors: Singh, Naileshni; DeMesa, Charles; Pritzlaff, Scott; Jung, Michael; Green, Charlene
title: Implementation of Virtual Multiple Mini-Interviews for Fellowship Recruitment
date: 2021-04-19
journal: Pain Med
DOI: 10.1093/pm/pnab141
sha: a4b5bda1784d4193e879647ebeaaec09849f3323
doc_id: 903259
cord_uid: ohgcv8xr

INTRODUCTION: The SARS-CoV Disease (COVID-19) pandemic has upended health care systems and one of the casualties has been the trainee recruitment process since social distancing and travel restrictions make an in-person experience improbable. At the University of California, Davis (UCD), our Pain Division transitioned our internally validated multiple mini-interview (MMI) process to a virtual environment METHODS: Applicants signed a confidentiality agreement prior to their interviews and were invited to watch a series of videos orienting them to the process and to the program itself. All faculty raters interviewed candidates using a total of 6 non-medical MMI scenarios with corresponding questions and scoring rubrics through the Zoom platform. Applicants were then welcomed to voluntary informal conversations with the current fellow trainees and faculty. An optional survey was sent to the applicants post-interview to assess their overall satisfaction with the virtual process. RESULTS: The survey analyzed the following using a 5 point Likert scale: Overall Satisfaction, Video Overview Program, Interview Day Details, Video Tour, Web-Based Interviews, Process was Fair, and finally a question on Informed Decision regarding receiving sufficient information to formulate their rank list. All respondents (80% response rate) reported being either satisfied or very satisfied with each of the aspects of the interview process detailed above. CONCLUSIONS: While technical difficulties and confidentiality issues are of concern when offering an entirely web based recruitment, our group was able to transition traditional in-person MMI to a virtual platform using a similar structure which was well-received by applicants

The SARS-CoV Disease (COVID-19) pandemic has upended health care systems and societies for over a year now. Not only did medical organizations change their policies and procedures, residency and fellowship programs initiated a variety of novel changes to meet the both the patient care needs and Accreditation Council for Graduate Medical Education (ACGME) training requirements for resident and fellow level trainees. The recruitment of new trainees is one of the necessary yearly activities that was transitioned to a virtual format at many programs.

There are multiple risks and limitations to a virtual interview format including: security of the online encounter, inability for an applicant to get "a feel" for the program, and potential awkwardness of the interaction. From a program's point of view, it can be more difficult to assess a candidate's motivation to possibly join the program. Prior to the pandemic, the act of taking time off and traveling to interview was in itself a self-selecting process. With virtual interviews, the constraints of time, travel, and monetary expense have been removed and candidates are able to interview less selectively. This may result in less reliable indications of interest from applicants. However, with residents in anesthesiology and acute care specialties being at the forefront of clinical demands during the pandemic (even re-deployed to care for COVID-19 patients), online recruitment was considered the safest and most viable option.

The University of California, Davis (UCD) Pain Medicine Division adapted its multiple miniinterview (MMI) format for recruiting fellows, usually completed in person, to an entirely virtual Official Journal of the American Academy of Pain Medicine 4 interaction for the 2020 interview season. MMI is a validated technique that uses multiple standardized encounters and scoring rubrics to assess candidates in a controlled setting.

Research shows that MMI may have some advantages over traditional or panel interviews. [4] The goal is to limit bias, improve objectivity, ensure fairness, and select candidates with strong communication skills, emotional intelligence, and problem solving through the use of multiple assessments or stations and graders. MMI has been shown to be fair, efficient, reliable, valid, and well received by trainees and interviewers alike for programs at all levels and types of training such as undergraduate, medical and post graduate. [5] [6] [7] [8] MMI is able to predict strong performance on examinations, clinical reasoning, and performance. [7, 9, 10 ] A well designed MMI may limit racial, gender, socioeconomic level, and other biases in recruitment but more research needs to be done in achieving this particular goal consistently. [11] [12] [13] However, noncognitive skills such as communication and professionalism, which are critical in the medical fields and particularly in pain medicine, can be assessed effectively with MMI. [12, 14, 15] Internal UCD data has shown that the MMI process dramatically decreases bias for gender, primary specialty, and time of interview (in person interviews were done in two shifts) with consistent scoring between faculty raters. Therefore, even in the virtual space, this model of recruitment seemed the appropriate one to use (vs. traditional or panel interviews). As one of the first pain fellowship programs to utilize MMI, including its use in a virtual format, we describe our procedures for an effective and relatively seamless interview season. [1] [2] [3] 

Official Journal of the American Academy of Pain Medicine After systematic review of 220 applicants, 30 applicants were asked to interview with the program using an automated scheduling tool. Review criteria included Likert-scale ratings of research experience, leadership and volunteer experience, personal statement, letters of recommendation, medical licensing examination scores, and contributions to diversity, equity, and inclusion. The applicants were sent technical instructions for the interview day and an electronic agreement so that applicants could attest to the confidentiality of the process. The fellowship program director created three short introductory videos (1) Program Overview (2) Interview Day Details and (3) Video Tour. These were shared with applicants prior to their interview date with the goal of "putting a face to the program" and offering initial introductions to foster questions during the interview day. Lastly, the trained faculty raters were blinded (except the program director) to the applicant's primary specialty, current training program, and their application to limit bias.

Prior to participation in MMI, UCD faculty undergo training on implicit bias through selfawareness, a yearly one-hour intensive faculty rater training, and direct observation and behavioral assessment training. Shared information technology (IT) troubleshooting and training of the Zoom platform (Zoom Video Communications, San Jose, CA) was conducted by Faculty and current Fellows using mock interviews. Through simulated interviews, the timing and rotation of interviews was tested as well as adjustments to the video interface visual display, mute/unmute functions, screen share capabilities and other controls. Advanced

Official Journal of the American Academy of Pain Medicine 6 preparation helped ensure the best chances for live interview success. Further contingencies were made in the event of lost connections using the co-host and alternative host features as well as the program director providing phone contact information for all interviewers and interviewees. Technical support staff was available for each of the interview days.

The Zoom platform was used to invite applicants in 20 minute increments to interview 1:1 with three faculty members. This was the platform most utilized by our health system and was deemed secure enough for use. The Zoom interface has been described as an effective televideo platform for trainee and fellowship interviews. [2, 3] The faculty used unique Zoom links to conduct interviews which the candidates exited upon completion of the interview.

Each applicant was interviewed using a total of six non-medical scenarios developed by an After the interviews were completed, applicants were invited to meet current fellows for 1 hour without faculty present using a Zoom session where the fellows answered questions and described the fellowship experience. This virtual "happy hour" format was previously described as an effective modality for trainees and candidates to interact. [2] Then, an informal 1:1 optional session was held with each faculty member so that questions about the program could be asked by the applicants since they were not given an opportunity to ask any questions during the timed scenarios. These faculty sessions were facilitated by using Zoom breakout rooms to allow for privacy. The breakout rooms were timed (<5min) to allow for equitable distribution of faculty and candidate time. Post interview anonymous optional feedback surveys were sent to the applicants and did not affect the MMI rating. It was emphasized to the candidates that the surveys were anonymous and would not influence their match ranking.

This survey was determined to be an exempted study by the UCD Institutional Review Board (IRB, Net ID 1724907-1).

Post-interview surveys were completed for 24 of 30 respondents (80% response rate). Ratings 

Positives of this virtual format include safe and socially distanced format, reduced travel for applicants and programs (beneficial from both a financial, staffing, and COVID safety standpoint), less time off or use of vacation time, a shorter interview day, and ability to rewatch the videos. The virtual format was probably also favorable to home departments due to redeployment of residents for COVID related activities, allowing less time off for the interview trail. We noted no occurrences of "Zoom bombing" or other major technical setbacks and had effective implementation of our current scenarios in this format with minimal changes. The majority of those interviewed were very satisfied or satisfied with the MMI process and interview day, indicating that virtual was a good proxy for in-person interviews while being cost effective to the applicant.

Despite its successful implementation, we identified several drawbacks to virtual MMI. In some cases, the Zoom links did not work or participants were unexpectedly dropped from sessions.

Additionally, it is challenging to predict the technology competency for applicants and faculty participants in advance. Even as most operations are online due to the pandemic, there are Official Journal of the American Academy of Pain Medicine many platforms currently being used from between departments and institutions. It can be overwhelming to be familiar with various web-based meeting software and/or have knowledge of advanced functions (breakout rooms, claiming host, etc.). We experienced a variety of logistical and technical issues including applicant confusion about interview start times due to different time zones, audio lag issues, background distractions, or timing delays due to interviews running late. Another concern raised by faculty was the inability to ensure that the session was not being recorded by candidates. This could lead to dissemination of the interview scenarios thus compromising the objective nature of the MMI sessions. We attempted to mitigate this concern by having candidates electronically sign an "interview participant agreement and confidentiality verification" prior to their interview date and disabling their ability to become the "host", which would allow the ability to "record" the session. This above mentioned document outlined and explained the basic structure of MMI and each applicant attests that he or she will not record or share scenario details. As we progressed through the interview season, it became clear that having an information technology specialist or designated administrative support staff member (who is not interviewing candidates) available and involved on the day of the interviews was preferred. This individual can respond rapidly to unforeseen technical challenges and assist as needed. 

With the pandemic continuing to slowly smolder, a 2021 virtual interview season is almost guaranteed. Our goals for the upcoming year include minor refinements of the process and mastering the online platform that we have committed to use. We plan on expanding prerecorded, informational content showcasing the fellowship program and medical center that candidates will be able to access before and after the interview day. Overall, our findings show that applicants were satisfied with the interview process and that the MMI format continues to be effective in a virtual environment. Please rate your overall satisfaction with your interview.

Please rate the online video Program Overview made available to you in preparation for your interview.

Please rate the online video Interview Day Details which described the significance, structure and format of the Multiple Mini Interviews to aid in your interveiw day experience.

Please rate the online Video Tour of UC Davis Pain Fellowship Program.

In response to COVID-19, we decided to provide for Web-based interviews to maintain safety which is our highest priority. How satisfied were you with your web-based interviews? I was satisfied the UC Davis interview process was fair and the interactions throught the day provided me with the necessary information for compartive purposes in order to create a rank order list for pain fellowship match. Thinking about your overall interview experience, would you agree that you acquired the information needed to make an informed decision for ranking the UC Davis Pain Fellowship Program?

Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied

Virtual Multiple Mini-Interview during the COVID-19 Pandemic

A How-To Guide: Virtual Interviews in the Era of Social Distancing

Initial Experience with a Virtual Platform for Advanced Gastrointestinal Minimally Invasive Surgery Fellowship Interviews

What does the multiple mini interview have to offer over the panel interview?

The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions

Multiple mini-interviews: opinions of candidates and interviewers

The Multiple Mini-Interview (MMI) for student selection in health professions training -a systematic review

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Can Multiple Mini-Interviews Predict Academic Performance of Dental Students? A Two-Year Follow-Up

Predictive validity of the multiple mini-interview for selecting medical trainees

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

Exploring sociodemographic subgroup differences in multiple miniinterview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

Medical School Applicant Characteristics Associated With Performance in Multiple Mini-Interviews Versus Traditional Interviews: A Multi-Institutional Study

Relationship between Student Performances in Non

Cognitive Skills in Multiple Mini Interview and Integrated Practical Examination

How Medical School Applicant Race, Ethnicity, and Socioeconomic Status Relate to Multiple Mini-Interview-Based Admissions Outcomes: Findings From One Medical School