key: cord-0906211-gmjchl09 authors: Looi, Jeffrey CL; Maguire, Paul A; Bonner, Daniel; Reay, Rebecca E; Finlay, Angus JF; Keightley, Philip; Tedeschi, Michael; Wardle, Claire; Kramer, David title: Final-year medical student Psychiatry and Addiction Medicine synchronous summative tele-assessments during a COVID-19 Delta-variant stay-at-home lockdown date: 2022-02-28 journal: Australas Psychiatry DOI: 10.1177/10398562221077885 sha: e7adf31a7e874d2b6b840292271688d9cb460668 doc_id: 906211 cord_uid: gmjchl09 OBJECTIVE: We describe the planning, process and evaluation of final-year Psychiatry and Addiction Medicine summative assessments in a four-year graduate medical degree program, during a COVID-19 Delta-variant public health stay-at-home lockdown. CONCLUSIONS: We conducted separate written and clinical synchronous (real-time simultaneous) tele-assessments. We used online assessment technology with students, examiners and simulated patients, all in different physical locations. Medical students’ examination performance showed a good range. This was comparable to other discipline stations, and performance in previous years. There was no differential performance of students through the day of the assessments. "In years to come, Harry would never quite remember how he had managed to get through his exams when he half expected Voldemort to come bursting through the door at any moment." -J.K. Rowling T here has been a need for continuous innovation in medical student teaching and assessment during the COVID-19 pandemic. 1 The limitation of physical attendance led to innovations in synchronous and asynchronous distance education. 1 There has been increasing evidence that synchronous distance education is effective for components of medical education. 1 Crucial bedside and community-based clinical teaching has mostly been maintained in Australia. Due to pandemic disruptions, final-year summative assessments have required reconfiguration with considerable variation in approaches. 2 We focus here on final-year synchronous distance tele-assessment necessitated by rapidly changing pandemic public health measures. Australia experienced a wave of SARS-CoV-2 Delta-variant public health lockdowns from June 24 th 2021 that led to public health stay-at-home lockdowns in NSW, ACT and Victoria. 3 Our four-year graduate medical program was subject to a COVID-19 Delta-variant lockdown from August 12 th 2021 until October 29 th 2021. This included restrictions on travel, except for essential workers; mandatory face masks; social distancing; and gathering limits. Our final-year summative assessments for Psychiatry and Addiction Medicine were conducted before the release from stay-at-home lockdown on 20-21 October 2021. Since the beginning of the pandemic, the 110 students in our 4 th year program attended 6-week rotations in Psychiatry and Addiction Medicine, with 3-week placements in acute and general settings. Even with the public health restrictions, students were able to complete their on-site clinical placements and a preparation for internship term. Psychiatry and Addiction Medicine rotations form one of four total terms (each of 6 weeks) in Women's and Children's Health, Acute Care, Medicine and Surgery. Students were also provided with online interactive casebased teaching and didactic lectures. These medical student teaching arrangements were similar to the measures adopted internationally. 1 Considering the pandemic public health measures, we recommended that Psychiatry and Addiction Medicine summative assessments be conducted via online synchronous tele-assessments. Written summative assessments were conducted via an online platform (WATTLE 4 ) with (Proctorio 5 ) online invigilation. Summative clinical assessments were conducted entirely via a commercially licensed videoconferencing platform (Zoom 6 ). Students, simulated patients and examiners participated remotely from their own locations. We note that the principles of tele-assessment are applicable to different online platforms. The summative clinical assessments comprised two Objective Structured Clinical Examination (OSCE) tasks: 1. viewing an online-presented clinical vignette video and presenting (in 10 min) a mental state examination (MSE), differential diagnoses and a management plan to the remote examiner via video-conferencing; and 2. via video-conferencing, interviewing a simulated patient (for approximately 20 min) to obtain a relevant history while being observed by the examiner/s via video-conferencing (i.e. a conference session with 3-4 participants). Examiners completed paper assessment forms for scoring of the MSE and role-play OSCE. We describe and analyse the process and evaluation of the final summative examinations in Psychiatry and Addiction Medicine. Based on our previous experience, we hypothesised that remote online student assessments would show comparable student assessment performance to previous face-to-face and online assessments. We descriptively analyse the planning, procedures and the assessment performance of the students. Due to the rapidity of events, we were not able to survey students or examiners regarding their views on the process. The Medical School had maintained online assessment and videoconferencing capability to allow for the prevailing pandemic public health measures. From August 12 th until October 29 th 2021, our Medical School was under COVID-19 public health stay-at-home orders. Accordingly, we enacted online summative tele-assessments for all participants. Based on feedback from last year's remote assessment, we planned for 5-minute breaks between consecutive student assessments. 7 Students read the single-page examination instructions on an examiner video-conference sharedscreen for a maximum of five minutes, during a total observed assessment period of 20 minutes. Students had the option to commence the interview as soon as they wished. This obviated the need for examiners to move students through the individual pages of instructions that proved problematic in 2020. 7 There were 2-minute breaks between students for the mental state examination OSCE to reduce examiner fatigue. Spare examination slots were included at strategic points in the OSCE timetables to cater for potential connectivity or other technical problems. Students were briefed on the assessment formats as soon as possible prior to the formal summative assessments. Written summative tele-assessments for 110 students were successfully conducted via multiple-choice and extended matching questions that were administered online across all final-year disciplines, Psychiatry and Addiction Medicine, Women's and Children's Health, Acute Care, Medicine and Surgery, as well as Population Health and Professionalism. We were able to conduct the summative clinical OSCE tele-assessments of 110 students on two consecutive days, 20-21 October 2021. The Psychiatry and Addiction Medicine summative tele-assessments were intercalated with the concurrent final-year summative tele-assessments for Women's and Children's Health, Acute Care, Medicine and Surgery. This involved examining from 9.00am to 4.30pm for the MSE OSCEs on Day 1. Examinations ran from 8.30am to 5.00pm for the roleplay OSCEs on Day 2. There were 9-10 lead examiners on each day. The main technical difficulties related to stability of the internet connections for the tele-assessments. Flexible timing and addition of breaks between consecutive students allowed adjustments. It will be important to have back-up simulated patients and examiners for the future. Some unexpected challenges to the online assessment were external noise (construction and neighbours) and unexpected interruptions by others (humans and pets!). We describe de-identified data (Figures 1-6 ). Australasian Psychiatry 0(0) As seen in Figure 1 , the mark distribution of the combined MCQ and extended matching R-Type questions demonstrated a range of scores, which was slightly right-skewed. There was mean of 71, standard deviation of 6 and Cronbach's alpha of 0.796. A pass mark of 54 or greater was set, resulting in 3 failures for the overall assessment. Assessment performance scores of students according to the clinical discipline, Figure 2 , shows the distribution of performance of students in Psychiatry and Addiction Medicine is commensurate with the other disciplines. However, the smaller number of Addiction Medicine questions has resulted in a larger display box of scores, with minimal whiskers, in the statistical summary. From Figure 3 , the clinical OSCE summative assessments showed a range of scores, again, slightly right-skewed. There was a mean of 70, standard deviation of 7.8 and Cronbach's alpha of 0.719. The OSCE scores were evaluated by number of passes for each of the 12 OSCEs (2 per discipline), resulting in 6 overall failures. The performance scores of students across the OSCEs showed a similar distribution across all the disciplines (see Figure 4 ). The MSE OSCE student performance results are plotted in Figure 5 (a), showing that there was differentiation of performance scores. There was a mean of 75, standard deviation of 16 and Cronbach's alpha of 0.89 ( Figure 5(b) ). There were 6 failures and 22 higher-level performances. The statistical summary of morning (AM) versus afternoon (PM) student performance shows no significant variation ( Figure 5(c) ). The role-play OSCE results are plotted in Figure 6(a) , showing there was differentiation of performance scores. There was a mean of 75, standard deviation of 16 and Cronbach's alpha of 0.76 (Figure 6(b) ). There were 7 failures and 19 higher-level performances. The statistical summary of morning (AM) versus afternoon (PM) student performance shows no significant variation (Figure 6(c) ). The rapid imposition of a stay-at-home lockdown necessitated final-year medical student summative tele-assessments, with the additional requirement that all participants interacted remotely online. We specifically added breaks between students in the clinical OSCEs that reduced examiner/ simulated patient fatigue and improved flexibility. We were able to intercalate with other final-year assessments and complete the assessments within reasonable working hours. Due to rapid changes, we were not able to formally survey students and examiners to obtain feedback on the processes. Anecdotal feedback from examiners was that familiarity with the format and planned breaks reduced fatigue. However, as can be seen from the results reported, we have been able to present more data on objective student performance than in 2020. In 2022, we will seek to incorporate more user feedback to improve assessments, which are currently planned for a hybrid face-to-face and remote model. We are planning for further research on the formal comparison of tele-assessment performance with face-to-face assessments. During the first year of the pandemic, Khoo et al. 8 at the University of Queensland Medical School also described shifting to online written summative assessments with invigilation. They substituted student interview of real patients with simulated patient examinations in 2020. Internationally, other medical schools have had different assessments, for example, open-book examinations in the context of cancellation of OSCEs in the UK. 1 In the 2003 SARS pandemic, telephone viva voce and audio conferences were used for examinations, while current computing, software, and internet technology allowed video and simulated patient assessments. 9 We used a range of technologies, such as laptop computers, videoconferencing cameras, online videoconferencing platforms, invigilation software, wireless technology and the Australian National Broadband Network. The emergence of new variants of SARS-CoV-2 continues (e.g. Omicron), even as Australia has reached COVID-19 vaccination rates of over 90% for people aged 16 or older. The future of the pandemic remains uncertain. However, our final-year summative synchronous written and clinical OSCE tele-assessments were conducted successfully in relation to student assessment performance. Online written summative MCQ and extended matching question tele-assessments may be usefully retained in the future. The unexpected stresstesting of our remote assessment capacity has yielded ongoing useful lessons (see Table 1 ). However, tele-assessments are not necessarily suitable for a range of OSCE tasks within and outside our discipline. Synchronous tele-assessment has been able to assess competence in our disciplines with a high degree of confidence. Nonetheless, it will be preferable to return to face-to-face clinical OSCEs whenever possible. Medical education challenges and innovations during COVID-19 pandemic Medical students and COVID-19: the need for pandemic preparedness COVID-19 vaccination daily rollout update Proctorio: securing the integrity of your online assessments Conduct and evaluation of final-year medical student summative assessments in psychiatry and addiction medicine during COVID-19: an Australian University medical school experience Teaching medical students remotely during a pandemic -what can psychiatry offer? Medical education, pre-and post-pandemic Era: a review article The authors report no conflict of interest. The authors alone are responsible for the content and writing of the paper. The authors received no financial support for the research, authorship, or publication of this article. Jeffrey CL Looi  https://orcid.org/0000-0003-3351-6911 Rebecca E Reay  https://orcid.org/0000-0001-9497-5842 Angus JF Finlay  https://orcid.org/0000-0001-9505-9758 Philip Keightley  https://orcid.org/0000-0001-9753-3097