key: cord-0716042-2on3kdss authors: Yolanda, Sophie; Wisnu, Wismandari; Wahjudi, James Marcus; Findyartini, Ardi title: Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic date: 2020-10-28 journal: Korean J Med Educ DOI: 10.3946/kjme.2020.175 sha: 87f2cdded5d01721d2db35bb1314a7ee8334a6f9 doc_id: 716042 cord_uid: 2on3kdss The coronavirus disease 2019 pandemic has affected many aspects of life, including medical education. The selection of students using multiple mini-interviews (MMI) should be adapted promptly as internet-based MMI (iMMI) in the current setting. The current transition from MMI to iMMI is supported by experience in conducting conventional MMI in previous years; hence, the station and assessor preparation could be completed without significant issues. The greater challenges to the use of iMMI are due to technical issues, including poor internet connections and adaptations necessary for assessing candidates’ nonverbal expressions. This paper aims to describe the experience of conducting iMMI in the current pandemic situation, with further highlights on contextual challenges in a limited-resource setting. The coronavirus disease 2019 (COVID-19) pandemic, which began in February 2020, has necessitated adaptations in numerous aspects of life around the world, including in medical education. Medical schools should formulate and enact strategies to move learning and assessment fully online, defer other face-to-face and hands-on activities to later dates, prevent undergraduate medical students from conducting their clinical rotations (due to safety reasons), enrich the curriculum with COVID-19 subjects, and attempt to support medical and healthcare professionals working on the front line [1, 2] . Amid uncertainties and challenges, especially in assuring that medical students achieve clinical competencies, medical schools must ensure that the strategies being implemented still facilitate high-quality learning and consider all stakeholders, including current and future students. Multiple mini-interviews (MMI) have been used extensively around the world for conducting admission to various majors and levels of study [3] [4] [5] . Participants (candidates) are required to complete various activities at each station, varying from completing a certain task, conducting a role play based on a given scenario, making decisions, interacting with other participants or the interviewer, or answering questions from the interviewer as in a standard interview session [3, 4, 6] . In general, the reported reliability of MMI has ranged from moderate to high, with a Cronbach's α of 0.69-0.98 and a G coefficient reaching 0.81 on generalizability analysis [4] . In the Faculty of Medicine, Universitas Indonesia, this method has been applied as part of the undergraduate international-class medical student admissions process since 2013. This method has been an integral part of the process, along with the standard admission written examination, which allows assessors to evaluate applicants' non-cognitive attributes. The similar concept of conducting MMI through online videoconferencing platforms was conducted at the University of Sydney in 2011, as studied by Tiller et al. [7] . No significant difference in scores equivalence was found between conventional MMI and internet-based MMI (iMMI) [7] . The reliability coefficient demonstrated by generalizability analysis also showed satisfactory results [7] . Regarding satisfaction with iMMI, both interviewers and candidates rated it positively, showing acceptability among participants. This method is also considered cost-friendly, giving institutions significant budget savings. Based on these results, iMMI is considered an important innovation that can be readily applied by other institutions [7] . In this article, we aim to share our experience in preparing and conducting our version of iMMI as part of the medical student admissions process at the Faculty of Medicine, Universitas Indonesia, during the COVID-19 pandemic. This alternative was indeed practical and necessary, especially in this pandemic situation, which requires that people minimize social interactionconducting conventional in-person interviews is not possible. We also aim to describe challenges in conducting our version of iMMI-given limited resourcesespecially regarding internet bandwidth in Indonesia, compared with other countries in Asia [8] . Preparations for this iMMI were made over approximately 1 week, from the decision to adopt iMMI to the day of the interviews. The preparations focused primarily on revising scenarios, preparing technical implementations, selecting and informing interviewers, and informing candidates about the change of plan. Approximately 2 days before the interview dates, notifications were sent via email and online messaging platforms to interviewers and candidates. Candidates were also contacted directly through online messaging platforms and invited to join a group for the coordination process. They were told to prepare a desktop or laptop computer with the online videoconferencing software installed and were not allowed to use phones or tablets during the iMMI. The camera angle needed to be adjusted so that the upper half of candidates could be seen on-camera during the interview. Candidates were also required to have a stable internet connection for the interview. During the application process, candidates were asked to send a photo ID with their application; this photo ID was also used to identify the candidate during the briefing and interview process. No significant problems were raised by the candidates. Interviewers were contacted individually through online messaging platforms to obtain consent and to iMMI Example scenario You are a third-year high school student who has been having trouble completing your scientific paper. The scientific paper is a pre-requisite to graduate from high school. Your supervisor has been contacting you for the past 6 months and you did not respond properly since you feel that you are really behind the schedule. You think that you will only update your supervisor when there is progress on your project. However, you have been stuck and you do not know what to do. After discussing with your academic advisor, you finally decided to come to your supervisor, and you want to explain what has been happening with your scientific paper. You are meeting with your supervisor to explain your reason for not updating him/her on your scientific paper for the past 6 months. Note: An actor will be playing the role of the supervisor. You are a first-year medical student and you have a year-end project writing a thesis proposal. The thesis proposal is a pre-requisite to pass the first year of medical school. You usually meet with your supervisor every month to update her on your progress. The last meeting was conducted a month ago, just before the COVID-19 situation emerged. Due to the COVID-19 situation, this month's meeting will be held via an online application. During the duration of your study from home, you have been having trouble concentrating and writing your thesis proposal. You are now going into the monthly meeting with your supervisor via the online application. Table 1 . Each station was set up using the break-out room feature on an online videoconferencing platform, in which each break-out room consisted of the interviewer (except in the writing station), the candidate, and a co-host displaying the scenario and countdown timer for the station. The co-host station was manned by the information and technology (IT) department, which also provided technical support as needed. Each station had an interview duration of 7 minutes, with an additional 1-2 minutes for station transition, during which the candidates were moved to another break-out room/ station and the interviewers scored the candidates. All interview processes were recorded; the participation of the interviewers and candidates was considered consent for the recording. The writing station was not attended by an interviewer. To ensure that the short essay written was the candidate's own work, in addition to recording the candidate at the writing station, we developed a form for the candidates to write their short essay, with a countdown timer in which the form closes after 7 minutes. The short essays were then assessed by faculty, who also comprised the iMMI committee. The most compatible online assessment system available to our institution at the time was our objective structured clinical examination (OSCE) scoring system, in which the candidates are scored from 0 to 3 instead of 1 to 4. We modified the scoring system to the OSCE system (0: unsatisfactory, 1: satisfactory, 2: good, 3: excellent). The global rating score was also adapted to the OSCE system, which consists of "did not pass" (previously "no"), "borderline" (previously "not sure"), "pass", and "superior" (previously "yes"). To avoid technical problems, a simulation was conducted prior to the iMMI, using three stations. The simulation was held using the online videoconferencing platform that would be used during the actual interview (Zoom). Three faculty acted as the interviewers, three alumni acted as the candidates, and the IT department served as the co-host displaying the scenarios. The simulation was conducted in real-time, in which the candidates tried to answer the scenarios and complete the form prepared for the writing station. The candidates were also moved from one station/break-out room to the next. Potential pitfalls were identified. Only the event coordinator and several IT department staff were present in the faculty on the interview day. With social distancing and pandemic safety protocols in place, the event coordinator and the IT department staff accessed the interview from the faculty's computer lab, which normally can accommodate 120 computers. Thirty computers were utilized for the co-hosts at each station, displaying the scenario and countdown timer for the station. Interviewers and interviewees accessed the meeting remotely using their personal computers. One hour before the iMMI, two briefing sessions were conducted simultaneously for both the interviewers and candidates. The briefings were held in two break-out rooms on Zoom. The briefing explained the iMMI mechanism, stations, and timing, with an emphasis on the expectations for the candidates and the scenarios for the interviewers. During the briefing, interviewers were informed of their assigned stations and were given the password to access the stations' scenario. They were also briefed on how to access and complete the online assessment system. This is relatively similar to MMI, although more technical training for the assessors would be beneficial. Constructed with the assumption that face-to-face activity is possible, there is an actor in the station. Modified for the COVID-19 situation, no actor is in the station. The interviewer also plays this role if necessary. There are six evaluation stations and all stations have interviewers. There are six evaluation stations and no interviewer is present at the writing station. Scoring 1 to 4 global rating scale of no, not sure, and yes 0 to 3 global rating scale of did not pass, borderline, pass, and superior Simulation No simulation was conducted as interviewers were already familiar with the conventional MMI process. -A simulation is conducted 1 day before the interview day to prevent technical problems. -Only three stations are set for the simulation. Briefing -Face-to-face briefing -After the briefing, interviewers at the same station discuss and agree upon the scoring at the station. -Before the interview day, notifications are sent to interviewers and candidates via e-mail and online messaging platforms. address any technical problems that might arise. Personnel from the IT department manned a computer lab in which the computers served as the co-host in each station. The timers on the co-host computer proved highly useful in helping the interviewer and the candidate adhere to the allocated time for each station, as there were technical difficulties in moving the candidates to the next break-out room. The differences between iMMI and MMI are summarized in Table 2 . The preparation stage posed only minimal difficulties, as there was sufficient time to plan and coordinate the changes to be applied to the iMMI. In terms of cost, the iMMI required less budget allocation compared to MMI due to significant cost reduction for transportation and consumption. The implementation stage, however, posed myriad problems, such as inadequate human resources, poor internet connectivity, and the faculty's limited technological expertise. The briefing session for the interviewers did not begin on-time, as some interviewers experienced problems in joining Zoom. Two interviewers also were suddenly unable to participate in this interview; thus, last-minute substitutions needed to be made. Due to these disruptions, the interview session was delayed for approximately 1 hour. The most significant problem during the interview was poor internet connectivity, both for the interviewers and the candidates. No disconnection problems occurred with the interviewers, but two candidates were disconnected completely from the beginning of the interview for more than 15 minutes, causing them to miss two stations. The assessments for these two candidates were annulled and make-up sessions were conducted on a different day with a different set of scenarios. There were also issues regarding station change. Several times, the station change took more than 2 minutes because the IT department had to move 20-30 candidates at once to the next designated break-out room. This issue delayed the entire interview processeach session lasted for approximately 90 minutes when they should have lasted no longer than 1 hour. Although the interviews lasted longer than anticipated, these delays did not significantly disrupt the interview process. This paper aims to elaborate the preparation and implementation of iMMI, which was conducted in response to the needs of student selection during the COVID-19 pandemic in a medical school in Indonesia. The school has been conducting face-to-face MMI for the past 6 years. iMMI was conducted in different settings, both as the original student selection method [7] and as an adaptation during the pandemic [9] . Our report supports the feasibility of iMMI implementation, despite challenges with internet connectivity and familiarity with the use of videoconferencing technology for selection purposes. The cost efficiency of iMMI also supported the feasibility, although further technical preparations and the internet bandwidth might pose additional cost. We learned that more detailed technical preparation and a pilot study for such a high-stakes selection assessment should be conducted. This should involve more technical training for the assessors and the IT support team to prevent delays due to unanticipated technical glitches. Overall, because Indonesia has systemic challenges in internet connectivity, the decision to conduct iMMI in our setting was subject to numerous considerations, including a contingency plan to overcome technical problems. The recommended internet bandwidth for the online videoconferencing service that we used (Zoom) is 1.5 Mbps (up/down). This bandwidth is well within the capability of Indonesia's internet connection, although stability differs widely. In the future, the recommended bandwidth information should be sent to all candidates and interviewers, along with guidelines on how to check the internet bandwidth using a speed test, in order to minimize technical problems during the interview. We realized that, despite the technical challenges, iMMI was the best possible way to conduct MMI for student selection while still ensuring implementation of COVID-19 transmission-prevention measures [9] . The station blueprint for iMMI was the same as for the Sophie Yolanda, et al : Adaptation of internet-based multiple mini interview 287 "conventional" MMI that we had conducted before the COVID-19 pandemic. This ensured that the MMIs had comparable content. Several adjustments were made for iMMI in our current setting-the interviewer also became an actor performing roleplay at the communication skills station, no interviewer/supervisor was present at the writing station, and the use of Zoom for technical meetings and the iMMI itself was optimized. The adjustments implemented did not jeopardize the validity of the iMMI stations. The total number of stations and the duration of each station were within an acceptable range of best practices for MMI [4] . Some stations also utilized the COVID-19 pandemic as contextual material, which made the candidates' experience at the iMMI stations more relevant to the current global challenges. The assessors previously conducted conventional MMI; therefore, they were well-trained in assessing MMI. However, as mentioned previously, it became apparent that the transition to iMMI required further planning to prepare the assessors to use the iMMI system [10] . In addition to the assessors' readiness, the iMMI requires adequate preparedness by the support and IT staff, who play significant roles in ensuring the "smoothness" of iMMI implementation. While conventional MMI also requires such technical support, the support of IT staff is more crucial for iMMI. The IT staff must be able to navigate the online platform being used and to manage hardware, network, and software issues simultaneously. To assure the reliability of assessments in similar stations across iMMI circuits, the assessors were allowed to discuss the relevant scenario, questions, and scoring to reach a similar understanding, which is key to ensuring the reliability of the MMI [4] . One challenge during iMMI was preparing the assessors and candidates to complete the virtual interview at each station. While assessors seemed able to assess the candidates' critical thinking, reasoning, judgments, and verbal expressions, it was quite difficult to observe non-verbal expressions and the candidates' comfort in a new environment, which was a reported challenge in conducting iMMI [9] . Nevertheless, given evidence that MMI has good fairness and does not favor candidates with certain cultural, gender, or socio-economic backgrounds [4, 6] [7, 10] . In addition, the limitations of iMMI in assessing non-verbal expressions and stress tolerance in a new environment require further adaptations [9] . The adaptation level of assessors and candidates in using virtual interviews should be assessed; this must be a consideration in using iMMI. In a limited-resource setting, especially regarding internet access and quality, technical issues due to network problems may reduce iMMI's advantages. Therefore, whether or not iMMI becomes common practice after the pandemic depends on multiple factors. It is more important that the MMI format being used is valid, reliable, fair, and feasible. We are fully aware that this report has several limitations. First, despite the validity of iMMI, as evidenced by the brief evaluation and feedback received from the assessors, candidates, and organizers during separate debriefings following iMMI, this report is not In conclusion, we have described the experience of conducting iMMI in a limited-resource setting. All aspects of the preparation process-the scenario, stations, scoring system, simulation before interview day, briefing for candidates and assessors, and all technical supportshould be detailed in advance to anticipate and mitigate risks during the iMMI. Because internet connections can pose a great challenge in the current setting, a contingency plan for internet connectivity problems should be discussed in advance and enacted promptly. Despite the challenges and limitations, the Faculty of Medicine at Universitas Indonesia has succeeded in conducting its first iMMI while preserving its validity and reliability. Future evaluation is necessary to reassess the format for future MMI. Response and lessons learnt managing the COVID-19 crisis by School of Medicine 5 Ways medical students can assist during the COVID-19 pandemic Multiple mini-interviews: same concept, different approaches Adaptation of internet-based multiple mini interview The multiple mini-interview (MMI) for student selection in health professions training: a systematic review Medical school selection criteria and the prediction of academic performance The utility of multiple mini-interviews: experience of a medical school Internet-based multiple mini-interviews for candidate selection for graduate entry programmes Virtual multiple mini-interview during the COVID-19 pandemic COVID 19: designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape All authors provided final approval of the version to be published.