key: cord-0876724-oaw57vtp
authors: Nusanti, Syntia; , Dearaini; Bani, Anna Puspitasari; Kartasasmita, Arief S.; Muhammad Ichsan, Andi; Virgana, Rova; Anggraini, Neni; Rahayu, Tri; Irfani, Irawati; Edwar, Lukman; Rifada, Maula; Yudantha, Anggun Rama; Ismail, Muhammad Abrar; Komaratih, Evelyn; Wildan, Arief; Knoch, Andrew M. H.
title: Delivering a modified continuous objective structured clinical examination for ophthalmology residents through a hybrid online method
date: 2021-11-30
journal: Korean J Med Educ
DOI: 10.3946/kjme.2021.210
sha: 3ae5b96d65019ad3e7f1dcea5703c74e453aedaa
doc_id: 876724
cord_uid: oaw57vtp

Since coronavirus disease 2019 was declared a global pandemic by the World Health Organization, it has become a challenging situation to continue medical education, including in Indonesia. The situation prohibited face-to-face (direct) educational activities in clinical settings, therefore also postponing examinations involving especially procedural skills. Adaptations were urgently needed to maintain the delivery of high-stake examinations to sustain the number of ophthalmology graduates and the continuation of eye health service. Objective structured clinical examination (OSCE) has been one of our widely used method to assess clinical competencies for ophthalmology residents, and is the one method that involves gatherings, close contact of examiners, examinees and patients, therefore the most difficult to adjust. Pandemic challenges brought technical changes in our delivering the OSCE to online, maximizing digital platforms of meetings, while still concerned to guarding the safety of candidates, patients and staffs. OSCE scenarios were also made as timely efficient as possible by changing continuous station models to a cascade one. The purpose of this article is to document our experience in conducting a feasible and reproducible OSCE in this pandemic era filled with limitations.

The ophthalmology residency program in Indonesia refers to the principles and guidelines of the International Council of Ophthalmology. The main purpose of our program is to establish qualified ophthalmologists in academic and professional scopes to serve the community. Therefore, residents in training are required to pass a national high-stake board examination arranged by the Indonesian College of Ophthalmology (KIKMI) in order to become board-certified ophthalmologists.

KIKMI has adapted the use of high-stake examinations to assess knowledge and clinical skill competencies of ophthalmology residents in their final year, and its outcome as a tool to assess program quality and accountability of each residency training center [1] . Among the modes of examinations, OSCE in our experience was the most difficult to implement due to its flow complexity, the necessity for direct interaction, and need for a well-thought scenario to create a reliable setting for assessing comprehensive clinical competencies. This article is intended to document our experience in preparing and conducting an online national examination for clinical skill assessments in a pandemic era for ophthalmology residents in Indonesia, using a continuous station model while prioritizing the safety of all participants affiliated in the examination.

In response to pandemic, the Indonesian government has limited traveling and prohibited large gatherings. Third, several studies also advised to minimize the number of stations in one circuit and also to use a cascade method instead of circuit, to reduce contact between examinees [2, 3] . Finally, we implemented digital-based scoring method instead of the usual paper-based.

The pre-pandemic OSCE that consisted of 12 stations in circuit method with dissociated clinical scenarios and tasks were adjusted to a cascade method that reduced the time needed for mobilization between rooms/station ( Fig. 1 ). The number of stations itself were reduced to five without breaks in-between, and the clinical scenarios were connected between the now virtual "stations". Given 10 minutes for each station, each examinee will finish in 50 minutes (maximum 60 minutes). Standardized patients were limited to only one station to minimize human resources, while historytaking or doctor-patient communication skills were simulated by the committee through Zoom.

Institutions were to prepare an examination room and another two isolation rooms for pre-and postexamination, with seating arrangements distanced 1 meter apart. One examination room was to accommodate a maximum of four examinees. These rooms were equipped with multiple stationery cameras to monitor the room. The examination room had additional mobile audiovisual devices and computer for communication and displaying questions. Each camera was linked to individualized Zoom accounts according to its function, and all were supervised by the national committee.

The pre-pandemic objective structured clinical examination (OSCE) (top picture) design needed 12 examination rooms for 12 active stations and 2 break stations, with participants moving from station to station (2 minutes for moving in-between); hence an extra of 24 minutes for mobilization only. The new arrangement (bottom picture) where stations are relatively virtual only required 1 examination room with the participant moving only between tables of equipment within the room. Examination room is only limited to examinee, local committee with mobile phone acting as mobile camera, and one standardized patient for ultasonography station.

The breakout room feature from Zoom was a very useful tool for conducting this parallel design of exam; it allowed us to create separate rooms within one Zoom meeting, therefore several examination rooms could exist simultaneously whilst easily being monitored by the national committee. Each breakout room was dedicated for one examination room from participating institution.

It had been arranged that the examiners and examinee in one breakout room must not come from the same institutions to maintain objectivity. (Fig. 2) . Therefore, we required all institutions to maximally accommodate a reliable internet connection to prevent interruptions during the examination; however, unstable internet connection was unfortunately the most inevitable factor.

After refining, all participating institutions were informed on logistics obligated to be provided in the examination room. Multiple stationery cameras were intended for monitoring and should reveal the whole room from different angles. We also required two smartphones that will serve as mobile cameras and audios to function in (1) shooting close-ups of resident's performance and as (2) microphones. All cameras Unstable internet connection had to be corrected to prevent delays during examination day. individually via e-mail to obtain consent and a signed integration pact. Previous paper-based assessment rubrics for OSCE were converted into digital scoring systems using Google forms. Each rubric had three assessment points and were scored from 0 to 3, plus a GRS consisted of "fail", "borderline", "pass", and "superior". To prevent scoring breach, the links to these digital forms were only distributed 1-2 minutes before the beginning of each station on examination day, one form for each station.

Proper evaluation is needed in order to establish more efficient examinations in the future. We composed anonymous feedback questionnaires for all examination participants including examinees, examiners, and head of ophthalmology resident programs from each institution (Appendices 1-3). These questionnaires were distributed shortly after the examination to reduce recall bias. 

Considering the large number of participants and to reduce the duration of participants being in the physical examination room, the examination time needs to be very efficient. Therefore, instead of the previously unassociated OSCE case contents with one scenario for one station, the case itself has been modified into one overall case but with different tasks for each station that runs in accord to the patient's progression of disease.

Creating a continuous linked one-patient scenario. The scenario and tasks were compiled into one PowerPoint presentation (Microsoft Corp., Redmond, USA) and

shown slide-by-slide operated by the central PIC from national committee. All tasks had been adjusted to assess both knowledge and dexterity of the examinee. An example of the continuous scenario is as seen in Fig. 3 .

After completing all OSCE stations, the examinees left the examination room to proceed to the postexamination isolation room. Examiners were required to submit a completely filled digital assessment form, which will be cross-checked by the central PIC, before allowed to proceed to the next examinee. At the end, all assessment forms from all examiners were compiled into one Excel sheet (Microsoft Corp.); data will be analyzed using a borderline regression method to set the passing grade for each station. machine, blurred microscope lens) and time management were considered as source of stress. There was also internet problem which caused interruption in video interpretation. Overall, residents were grateful to take part in the examination. They also appreciated our effort to conduct modified OSCE in the exceptional circumstances during pandemic era.

In this study, our online method was shown to be comparable to the pre-pandemic offline method in terms of exam's objectives (procedural skills and knowledge), passing results, and scoring aspects, except for technical preparation and the need for painstaking supervision to each participating institution. Ten out of 70 examinees were declared "failed" to pass the exam. This number showed no significant difference between pre and pandemic OSCE, where the number of passing examinees were around 80%-90%. In this study which involves a high-stake examination, validity and reliability are maintained since the content of exam still assesses procedural skill exams representing the curriculum and adequate number of stations in one circuit, unchanged regulated scoring rubrics, also the use of trained examiners [4] . It is also more cost-effective since the expense for travel to a centralized exam location is not needed, therefore favoring the travel limitation, and technically feasible to be conduct.

As we wanted to keep aspects from conventional OSCE mechanism, a modified face-to-face combined with virtual format was our best option. It was quite a challenge to modify the examination mechanism in short time with a lot to consider and there were limited studies around online clinical assessment for high-stake examination. However, it was recognized as effective in several studies. For example, Blythe et al. [5] held virtual OSCE in which nine medical students participated and there were no significant concerns regarding exam delivery and metrics. Hytönen et al. [6] also successfully implemented modified online OSCE in four dental institutes of Finland, and had positive feedbacks from participating examinees. Despite the similarities, those studies did not assess clinical procedural skills that might be challenging to execute virtually.

We used the blueprint for conventional OSCE to internet connection was also an issue. This resulted in low quality of images and sound shared online also created a disruption in accessing digital scoring platform [7] . Another study in Indonesia also shared similar problem regarding internet connection while conducting internet-based multiple mini-interviews in medical school, which resulted in delayed in the entire process [8] . Aside from technical difficulty and internet disruptions, we also had difficulty in recruiting examiners who are relatively familiar with the use of video conference technology.

Despite of these challenges, we believe that modified OSCE was the best way to perform high-stake examinations during pandemic era. It also required less budget due to huge cost reduction for transport, accommodation, and massive human resources in traditional setting. We realized this modified online examination has several limitations. Compared to traditional OSCE setting, we cannot fully observe and assess the resident's performance. Although we tried our best to maintain secrecy regarding the stations' objective, some things could go beyond our control and thus compromising fairness and objectivity.

To conclude, we have described the experience of implementing modified continuous online OSCE examination in ophthalmology residency setting in Indonesia.

It is feasible to complement the traditional OSCE with 

High-stakes testing: Oxford Bibliographies Online Datasets

Letter to Editor: recommendations for safer management of holding OSCE during COVID-19 outbreak

Conducting a high-stakes OSCE in a COVID-19 environment

The objective structured clinical examination (OSCE): AMEE guide no. 81. Part I: an historical and theoretical perspective

Undertaking a high stakes virtual OSCE

Modification of national OSCE due to COVID-19: implementation and students' feedback

Implementing an online OSCE during the COVID-19 pandemic

Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic