key: cord-0960990-ef3ykrrt
authors: Sihombing, Aaron.T.; Taher, A.; Rodjani, A.; Mochtar, C.A.; Hakim, L.; Daryanto, B.; Danarto, H.R.; Umbas, R.
title: Assessing the online objective structured clinical examinations in urology qualifying exam for urology residents in Indonesia during COVID-19 time
date: 2021-03-24
journal: MethodsX
DOI: 10.1016/j.mex.2021.101316
sha: 56107c0cd0dff3b8bb5af4f5c9a94229766b7b47
doc_id: 960990
cord_uid: ef3ykrrt

This project aimed to assess the Online National Board of Urology Objective Structured Clinical Examination (OSCE) feasibility in evaluating candidates simultaneously from five urology training centers in Indonesia during the COVID-19 pandemic. Data were collected from two online OSCE simulation trials and the Online National Board of Urology OSCE. A self-administered questionnaire was used to assess examiners and candidates’ perception. The average final score of the Online OSCE was compared to previous face-to-face OSCE results. All candidates and examiners (100%) heard and saw clearly the audio-visual in both OSCE simulation trials. None of the candidates had a failing score on the mock exam from all stations. There was a statistically significant difference between the online OSCE and December 2019 face-to-face OSCE. The Online National Board Urology OSCE was feasible and comparable to face-to-face OSCE in evaluating urology candidates. It may be beneficial for the future OSCE method in the medical education system. • Objective Structured Clinical Examination (OSCE) which assesses a broad range of urology candidates’ high-level clinical skills, is a more valid and reliable assessment instrument than the traditional oral examination; • The Online National Board of Urology OSCE method can help evaluate urology candidates, especially during the unprecedented COVID-19 pandemic.

Material and methods

The key components of the Indonesian urology board OSCE are the faculty evaluators, the urology candidates, the communication media, the standard clinical case scenarios, exam invigilators, timekeeper, the room stations to observe and evaluate the candidates, and the rotation of the candidates to each station.

The Online OSCE was designed to accommodate and synchronize the timing of all those components with the easiest and commonly used application programs for the faculty members, the urology candidates, exam invigilators, and timekeeper. The selected application programs were ZOOM, WhatsApp, and TeamViewer.

Two Online OSCE simulations were planned before conducting the definitive Indonesian urology board OSCE. The first online OSCE simulation was designed to assess whether the sound and visual from the computer could be heard and seen clearly by the faculty members and the urology candidates. Also, it assessed the effective online communication between the faculty members and urology candidates, the internet connection used by each urology training center, the easiness of filling and sending the score to the convener, the synchronization of candidate station rotation in five cities, and the communication between exam invigilator, faculty member, and ICU supervisory board member. We defined communication as the relational process of creating and interpreting messages that elicit a response [5] .

Supervised by the ICU, the convener was responsible for the course of the online OSCE event by managing the online communication between examiner, candidate, local supervisor, and data collector ( Fig. 1 ) . The convener managed and responded to any communication problems that occurred, adjusted time, and synchronized each city's candidate rotation.

Nine clinical case stations and one rest station were planned with 18 faculty evaluators (4 faculty evaluators from Jakarta, 4 faculty evaluators from Bandung, 4 faculty evaluators from Surabaya, 3 faculty evaluators from Jogjakarta, and 3 faculty evaluators from Malang). Two faculty evaluators from different training centers were appointed in each station. 10 urology candidates (3 candidates from Bandung, 2 candidates from Jakarta, 2 candidates from Surabaya, 2 candidates from Jogjakarta, and 1 candidate from Malang). 10 exam invigilators (2 for each training center) and one timekeeper who accompanied the convener were involved in the first online OSCE simulation.

The convener controlled the Zoom application program and the separation of the examination station utilizing the breakout rooms. There were 9 breakout rooms for the clinical case and 1 breakout room for communicating with the invigilator. Nine computers were placed in 9 different rooms in each of the 5-training centers. Each computer was tagged with electronic identification containing the city name and the station number (e.g., city A room 1) for recognition by the host to select and to place the corresponding station number and city in the online Zoom breakout room. The faculty evaluator's computer was identified by the faculty evaluator's name. ( Fig. 2 ) Each breakout room Nine short clinical cases consisting of simple questions about abnormalities in the radiographic image were constructed by ICU as the message that would be sent, interpreted, and elicit responses to faculty evaluators and urology candidates. These short clinical cases that consist of radiographic images also assessed the audio-visual and online communication effectiveness between the faculty evaluators and urology candidates. The time was set in four minutes for communicating these short and simple questions in each station and two minutes added time in between stations for preparing candidates for the next station. If there were a communication problem, it would not finish on time and the checklist would be sent empty. On the day of the online OSCE simulation, the faculty evaluators were given the answers for the radiographic abnormalities. The cases were presented using Microsoft Word or Microsoft PowerPoint utilizing the ZOOM shared screen application to the urology candidates. The questions were then asked by one of the two faculty evaluators from a different location to the urology candidates in their training center to assess the audio and communication from the evaluators to the candidates. An evaluation checklist that assessed the candidate's ability to see the radiographic abnormalities was filled by both examiners to assess the interpretation of message and response by the urology candidates and then communicate it to both faculty evaluators. The checklist was filled independently by both examiners without both knowing what other faculty evaluators from the same station filled to assess that the communication from the urology candidates was received and interpreted clearly by both faculty evaluators . This agreement between examiners on the checklist was used to evaluate the audio and message from candidates whether it could be clearly heard and interpreted by both faculty evaluators from a different location.

To synchronize the rotation of 10 urology candidates located in 5 different cities to each station, one timekeeper shared the allocated time using TeamViewer to the exam invigilator in each city and broadcasted the time reminder using ZOOM broadcast application to the faculty evaluators. The timekeeper and the 10-exam invigilator communicated using WhatsApp video call and Zoom application. A pre-determined starting station location of each urology candidate in each city was constructed. The list of station locations was sent to every exam invigilator in each city which will guide the urology candidates' station location. The direction of movement of each urology candidate was from a lower number station to the next higher number station. A faculty evaluator and a urology candidate questioner using rating scales (from 1 to 10, Fig. 3 ) were developed to evaluate the perception of the overall clarity and online OSCE feasibility from the first simulation [6] . A question The audio could be heard clearly 8.5 2

The radiographic image could be interpreted (clear visualization)

It is comfortable looking at the computer screen during the examination period (does not have eyestrain fatigue)

Instruction/sign from exam invigilator is understand clearly 7.0 5

Briefing time before examination is sufficient 7.5 6

It easy to rotate to each station 8.0 about preference in using Microsoft Word or Microsoft PowerPoint and comparison to face-to-face OSCE was added to the questionnaire The second online OSCE simulation was designed to refine the online OSCE using data from the first simulation, to assess the allocated time for the OSCE faculty evaluator to fill and send checklist score to the convener using WhatsApp, and to assess candidate comprehension to the clinical case that was presented using shared-screen Zoom application. 9 mock exams that resemble the actual exam were designed to evaluate any online communication problems that could disturb candidate comprehension to the clinical case exam. The faculty evaluator checklist was designed for simplicity and easiness for filling and sending the result to the convener using the WhatsApp application program. A self-administered questionnaire for the faculty evaluator and the urology candidate was designed to evaluate faculty evaluator and urology candidate perception of the second simulation online OSCE.

Actual OSCE will be assessed by comparing the mean online OSCE score of each station to the 5 previous face-to-face OSCE. A self-administered questionnaire about urology candidates' perception and comparison to the face-to-face OSCE and faculty evaluators' perception of the clarity of instructions at each station, adequacy of time allocation for each station, and degree of fatigue from online OSCE were asked to be completed at the end of the examination.

"METHOD VALIDATION"

Eighteen faculty evaluators, 10 exam invigilators, and 1 timekeeper participated in the first online OSCE simulation. However, of 10 urology candidates, only 8 urology candidates could participate in the simulation. Two urology candidates from Surabaya that were prepared for the first online OSCE simulation were isolated at home because of COVID-19 exposure. From 9 clinical case stations, 64 complete checklist answers (91%) (128 checklist answers from each faculty evaluator) from expected 70 checklist answers were sent to the convener. There were six uncomplete checklist answers because of the disturbance of internet connection causing a slow shared screen process which caused time to run out and failure of online audio that disturbed the effective communication. From 128 complete checklist answers, all faculty evaluators (100%) sent the same checklist answer.

We found no problem in synchronizing the rotation of urology candidates' stations in each city. The exam invigilators could effectively supervise urology candidates and communicate with the conveners. All of the urology candidates that participated in the first simulation completed the self-administered questionnaire ( It is easy to recognize correct candidate rotation to the station 7.0 Table 3 The mean score of urology candidates.

Mean score (First round) ( n = 9) Mean Score (second round) ( n = 8) Of the 15 faculty evaluators, there were 10 (66.67%) evaluators who preferred using Microsoft PowerPoint, and 5 (33.33%) evaluators felt no difference. From the first online simulation, we found some problems mainly in the form of internet connection stability that disturbed effective communication from conveners to exam invigilators which hindered instruction from exam invigilators to urology candidates. Adjustment and backup plan were made and the results of the second online OSCE were as follows: 18 faculty evaluators, 17 urology candidates, 10 exam invigilators, and 1 timekeeper participated in the second online OSCE. There were 2 rounds (each round has 4 rotations) in the second online OSCE simulation. Based on the first simulation, all the exam question was presented utilizing Microsoft PowerPoint. The first round consisted of 9 urology candidates (2 candidates from Bandung, 3 from Jakarta, 2 from Surabaya, 2 from Jogjakarta) and the second round consisted of 8 urology candidates (1 candidate from Bandung, 4 from Jakarta, 2 from Surabaya, and 1 from Malang). We removed the rest station and added additional time for changing between stations (from 2 min to 3 min). If there is a disturbance in audio or visual in one station, the rotation will stop until the disturbance is repaired. The rotation will be continued after the repair is complete.

In the first round, there were 34 (89%) checklist answers that were sent to the convener (there was a misunderstanding about when to begin the examination at the first and the fourth station because the faculty forgot to send the checklist answer). 3 internet connection problems that occurred could be solved by stopping the rotation. The rotation then was continued after the problems were solved. On the second round, 100% checklist answers could be sent to the convener without a problem. There was only a minimal internet connection problem that could be solved without consuming candidate time. The mean score of the mock exam from the first and the second round is presented in Table 3 .

From Table 3 , we found that there were none of the average scores below 50. Compared to the face-to-face OSCE, 4(23.5%) of the urology candidate felt less comfortable, 6(35.3%) felt no difference, 6(35.3%) felt comfortable and 1 (5.9%) felt more comfortable. There were 17 urology candidates (100%) filled the self-administered questionnaire [ Table 4 , Graphic 3 ] .

Compared to the first simulation, there were none of the candidates chose a category below 5. Of 18 faculty evaluators that participate in the first online OSCE simulation 18 (100%) completed the self-administered questionnaire ( Table 5 , Graphic 4 ), 18 (100%) faculty evaluators who participated in the first online OSCE simulation completed the self-administered questionnaire) It easy to rotate to each station 8.0 It is easy to recognize correct candidate rotation to the station 8.0 From this second online simulation, we found that adjustment that was made could solve problems that happened during the first simulation. Also, the solution to stop the rotation when there was any audio-visual problem in a room could manage the examination time and all the candidates could be examined with the given amount of time. None of the urology candidates and the faculty evaluators scored below category 5 in the self-administered survey. it means that there was an increase of good perception from them. The ICU concluded that Online OSCE could proceed. We proceeded to the definitive OSCE one week later.

Definitive online OSCE was conducted simultaneously and synchronized in 5 cities. It was joined by 27 urology candidates, 18 examiners, 10 local supervisors, and a convener supervised by the ICU through online communication. There were 9 stations of examination. The actual OSCE was done in 3 sessions, each session enrolled by 9 candidates. The fourth session was done for a candidate which was still recuperating at home due to an emergency operation several days earlier. ( Table 6 ) , and 5 candidates score 80 or above in all station (data not shown) There was a statistically significant difference in the average score; however, the difference was only found in December 2019 (face-to-face OSCE). While from three other face-to-face OSCEs (June 2018, December 2018, and July 2019), there were no statistically significant differences ( Fig. 4 ) .

The impact of COVID-19 did not only affect Urology services but also training in Urology. Most of the residents were involved in a smaller number of cases because of prioritizing elective cases for operation [5 , 6] . At the same time, the authority of the Urology training program must keep the wellbeing of their residents as the utmost priority [7] . One affected aspect of the Urology training was the qualifying exam for final year residents which has been postponed in some countries with severe infection rate [8 , 9] . According to the ICU rule, the final year residents should pass the surgical skill and MCQ exam to be eligible for the national OSCE. Those prerequisite exams could be done in their respective centers, supervised by ICU. However, the national OSCE should be held together in a designated place, which means that all the candidates and the examiners must come and stay at least one night.

The plan to have an OSCE at the end of June this year in Indonesia was disrupted due to the COVID-19 pandemic which struck our country in early March 2020 and soon thereafter widespread to all the provinces [4 , 10] . Although there is no lockdown policy, there are numerous reports about getting infected during traveling by mass transportation hence most of the people are reluctant to travel [11] . With the uncertainty of the pandemic to end, it was challenging to set the date if we choose to postpone. Another viable option is doing an on-line OSCE with one condition: it should be done simultaneously to the candidates in 5 centers while the examiners stay in their respective cities. Considering that, before the COVID-19 pandemic most people worldwide have familiarized with webbased learning, virtual meetings, live streaming video, conference, and more face-to-face activities have changed to the online system especially telemedicine as a preferred method since the COVID-19 pandemic was declared; hence the ICU decided to assess the feasibility of online OSCE [12 -14] .

Results of our online OSCE preliminary trials showed that online OSCE was feasible, although it had some problems. The results of the mock exam in our second preliminary trial also showed that candidates could comprehend the OSCE question in each station. From the survey result, adjustments and refinements were made by the ICU. The ICU then decided to proceed with an on-line version of OSCE and was finally done in the last week of June 2020.

Our results showed that online OSCE (June 2020) had a comparable result with 3 previous face-toface OSCE results from the period of June 2018, December 2018, and July 2019 ( Table 2 ). In the faceto-face OSCE in December 2019 that taken by twenty candidates, the mean score was 75 (SD ± 4.28) which was different from the three previous face-to-face OSCE and online OSCE. We could not explain why there was a difference in the final scores in December 2019.

The online NBU OSCE used regular ZOOM and WhatsApp platforms which were affordable and familiar for all training centers. ZOOM and WhatsApp become familiar during the COVID-19 pandemic because they are increasingly used for telemedicine and education [12 , 14] . During our first and second trial, some technical problem occurred during the process were solved quickly and did not affect much of the timeline and exam results. However, despite the huge efforts by the convener and his team as well as local supervisors, some technical problems still occurred.

Based on a post-examination survey to the candidates and reports from the examiners, critical images such as plain X -ray, CT-Scan, and MRI in four stations (pediatric, oncology, trauma & reconstruction, and stone) as well as a urodynamic chart in female/functional station could be fairly assessed although the candidates could not ZOOM-in the images as in the face-to-face setting. Moreover, 19% of the candidates scored 80 or above in every station, similar to our previous faceto-face OSCE. Additionally, by doing it online, all the doctors were staying in their respective cities and still serving their hospitals which highly needs them during this unprecedented situation and minimizing their risk of being infected from traveling.

The limitation of this study was our inability to compare the candidates' impression between the face-to-face OSCE and the virtual one since all the candidates were first takers. With the advancement in networking technology and familiarities of using online-based communication in the future, formal analysis comparison between face-to-face and virtual examination in the environment, anxiety test, and satisfaction test should be part of further comprehensive research.

This was our first experience doing an online National Board of Urology OSCE simultaneously for 5 centers with virtual examiners and this was the first report in its kind to date based on authors' knowledge.

The Online National Board of Urology OSCE method was shown feasible, affordable, and comparable to the face-to-face National Board Urology OSCE in evaluating urology candidates.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

To complete the Urology residency training program in Indonesia and become a licensed Indonesian urologist, a urology resident candidate must first pass a standard National Board of Urology examination. Since 2010 the Indonesian urology board examination comprises two parts: surgical skill examination and knowledge examination. The knowledge examination has two parts. The first part is a theoretical based written examination, delivered as multiple-choice questions (MCQ). The second part which assesses a broad range of high-level clinical skills, including data gathering, patient management, and communication and counseling skills, delivered as Objective Structured Clinical Examination (OSCE) replacing the traditional oral examination as a more valid and reliable assessment instrument [1 , 2] .

The National Board of Urology (NBU) OSCE, which is held biannually in June and December, consisting of 9 stations: andrology, benign prostatic hyperplasia (BPH), urinary tract stone disease, oncology, pediatric urology, female/functional urology, trauma and reconstruction, general urology/infection and a short assessment. The duration of each station is 10 min. Two examiners selected from the faculty members, both are from different institutions, are appointed to observe and evaluate each urology candidate in each station.

The NBU-OSCE location is transitory and it is designated by the Indonesian College of Urology (ICU). A convener from the faculty members is selected by the ICU in every OSCE implementation to manage the NBU OSCE. Eligible urology candidates and qualified examiners who are spread throughout different training centers in Indonesia then travel to the designated NBU OSCE location.

In early March 2020, coronavirus disease 2019 (COVID-19) was confirmed in Indonesia [3] . At the end of March 2020, the Indonesian government declared large-scale social restriction, stopped all faceto-face education activities, emphasized the need to stay at home for all Indonesian citizens and a regional quarantine that limited the population movement from one city to another 4. The COVID-19 pandemic heavily affected the high stake Indonesian urology board OSCE implementation, which was planned in June. This unprecedented situation drives the ICU to explore the possibility of shifting faceto-face OSCE to the online OSCE to overcome the limitation caused by the COVID-19 pandemic. The ICU embraced this challenge for ensuring eligible urology resident candidates could complete their training program and graduate without delay.

This study aimed to assess the use of the Online NBU OSCE in evaluating twenty-seven urology candidates simultaneously from five urology training centers in five different cities in Indonesia.

When are residents treated as doctors under the law?

OSCE performance among Quebec urology residents: a retrospective study from

Evaluation of the effectiveness of postgraduate general medicine training by objective structured clinical examination-pilot study and reflection on the experiences of Kaohsiung Medical University Hospital

COVID-19) pandemic: World Health Organization (WHO)

A First Look at Communication Theory

Research Methods in Education

Impact of the COVID-19 pandemic on urology residency training in Italy

Prioritising urological surgery in the COVID-19 era: a global reflection on guidelines

Well-being and education of urology residents during the COVID-19 pandemic: results of an American national survey

Adapting urology residency training in the COVID-19 era

The impact of COVID-19 on European health care and urology trainees

Positive correlation between general public knowledge and attitudes regarding COVID-19 outbreak 1 month after first cases reported in Indonesia

COVID-19 and Indonesia

Transforming ophthalmic education into virtual learning during COVID-19 pandemic: a global perspective

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.