key: cord-0890342-qy36b3dc authors: Emin, Elif Iliria; Emin, Ece; Bimpis, Alexios; Pierides, Michael; Dedeilia, Aikaterini; Javed, Zibad; Rallis, Kathrine-Sofia; Saeed, Ferha; Theophilou, Georgios; Karkanevatos, Apostolos; Kitapcioglu, Dilek; Aksoy, Mehmet Emin; Papalois, Apostolos; Sideris, Michail title: Teaching and Assessment of Medical Students During Complex Multifactorial Team-Based Tasks: The “Virtual on Call” Case Study date: 2022-05-05 journal: Adv Med Educ Pract DOI: 10.2147/amep.s357514 sha: 0dff61015cdbfcec98dcd4684f8d95e9f0b06956 doc_id: 890342 cord_uid: qy36b3dc ABSTRACT: Although considerable efforts have been made to incorporate simulation-based learning (SBL) in undergraduate medical education, to date, most of the medical school curricula still focus on pure knowledge or individual assessment of objective structured clinical examination skills (OSCE). To this end, we designed a case study named “iG4 (integrated generation 4) virtual on-call (iVOC)”. We aimed to simulate an on-call shift in a high-fidelity virtual hospital setting in order to assess delegates’ team-based performance on tasks related to patient handovers (prioritisation, team allocation). METHODS: A total of 41 clinical year medical students were split into 3 cohorts, each of which included 3 groups of 4 or 5 people. The groups consisted of a structured mix of educational and cultural backgrounds of students to achieve homogeneity. Each performing group received the handover for 5 patients in the virtual hospital and had to identify and deal with the acutely unwell ones within 15 minutes. We used TEAM(TM) tool to assess team-based performances. RESULTS: The mean handover performance was 5.44/10 ± 2.24 which was the lowest across any performance marker. The overall global performance across any team was 6.64/10 ± 2.11. The first rotating team’s global performance for each cycle was 6.44/10 ± 2.01, for the second 7.89/10 ± 2.09 and for the third 6.78/10 ± 1.64 (p = 0.099 between groups). CONCLUSION: This is one of the first reported, high-fidelity, globally reproducible SBL settings to assess the capacity of students to work as part of a multinational team, highlighting several aspects that need to be addressed during undergraduate studies. Medical schools should consider similar efforts with the aim to incorporate assessment frameworks for individual performances of students as part of a team, which can be a stepping-stone for enhancing safety in clinical practice. assessments of medical students' performance as part of a team is yet to be formally introduced in most medical school curricula. This could potentially be delivered in a form of team-based OSCE assessments 2,3 using validated tools. 4 Being a safe and efficient junior doctor can be challenging, and often necessitates the application of a melange of nontechnical and technical skills with applied knowledge. 1, 5 Junior doctors on call is a classic example where an individual (doctor) must work as part of a wider multidisciplinary team (MDT) under stressful circumstances. 6 This requires the application of complex non-technical skills which can include closed loop communication with other junior or senior doctors, leadership, task-based focus with effective prioritisation, situational awareness etc. 2 One of the most critical steps in this chain is the effective delivery of multidisciplinary handover which is critical to maintain patient safety. During the handover process, junior doctors identify the applicable tasks to be done, distribute them across the team and prioritise them in a timely manner. Based on that, they apply their clinical and managerial knowledge to accomplish a huge variety of tasks and deliver safe care for their patients. Increasing workload, communication gaps, lack of staffing, diversities in both educational and cultural backgrounds of the staff, as well as stress during a busy shift can affect overall healthcare provision. Hence, many final year medical students complain that they do not feel confident or prepared for on call shifts and feel nervous during them. To this end, we designed an innovative study named "iG4 (integrated generation 4 [7] [8] [9] ) virtual on-call (iVOC)". Using high fidelity SBL facilities of a virtual hospital environment, the iVOC study aspired to simulate a real on-call shift. Participants were clinical year medical students from different educational and cultural backgrounds. We primarily aimed to design a reproducible virtual on call model based on SBL. Our secondary aims were to measure team-based performance in order to gauge the ability of clinical-year medical students to prioritise the tasks generated after handover, including patient reviews according to clinical urgency. Essential Skills in the Management of Surgical Cases -ESMSC Marathon Course 10 is a 3-day international multifaceted surgical course aimed at the undergraduate level. ESMSC combines a mix of technical and non-technical skills modules with basic and applied surgical science workshops. Technical skills modules involve high fidelity in vivo swine model simulation (anaesthetised pig), as well as lower fidelity ex vivo or dry lab modules. Basic Science workshops involve essential elements that the future surgeon should be proficient at, i.e. fluid resuscitation or Arterial Blood Gases Interpretation. Applied Surgical Science modules involve small group teaching in a case-based setting. Non-technical skills settings vary from low fidelity Simulation-Based Learning (SBL) to more advanced group-based high fidelity human factors scenarios. 2, 3 The course concept and curriculum have been described in the literature previously; 5,10 the course concept has evolved throughout time using certain evidence and expert experience 7 to facilitate the vision of holistic education. 8, 9, 11 Apart from the non-technical skills core, none of the other aforementioned modules are directly related to this current study. iVOC is a pilot novel study, which aims to expand the ESMSC curriculum in the direction of team-based non-technical skills education. This is a case study which ran at the Virtual Hospital Setting of the Centre of Advanced Simulation and Education (CASE, Acibadem University Istanbul, Turkey) which offers a fully equipped hospital allowing high fidelity simulation. The unique advantage of this setting is that it allows simultaneous SBL scenarios to run, using advanced technology, achieving the highest possible fidelity. The rest of the ESMSC modules (unrelated to this study), took place at the in vivo and dry lab simulation facility of the same lab. We followed the standard operating procedures (SOP) of the CASE lab regarding the SBL standards of this study. ESMSC invites applications from several universities of the UK and the EU as well as other global medical schools including those in Turkey and India. Delegates are selected via a competitive process which includes submission of their CV and statement of interest. Each series of the course is attended on average by 40 delegates. Delegates should at least be at their clinical years of training and beyond or equivalent. This study involves students and not patients, hence there is no direct patient or public involvement. The course participants were split in 9 groups of 4 or 5 people. We mixed students from different years of studies (experience) and educational background (diversity) to achieve equality (knowledge, clinical experience) across different groups and for it to be representative of an international healthcare service system such as the National Health Service (NHS) in the UK. For this study, we matched 3 groups in each cohort and ran 3 cycles of scenarios. Hence, we had 3 cohorts consisting of 3 groups and each cohort attended 3 cycles of scenarios. During each cycle, all teams performed once and were observed twice; this means 1 group was performing, and 2 groups were observing through 9 cameras in a special control room designed for this purpose ( Figure 1 ). Each performing group was given a handover sheet which consisted of 5 scenarios (Appendix 1); 2 non-urgent and 3 which required urgent attention. Each patient had a location in the virtual hospital. The students in each performing group were meant to interact and identify the 3 cases that required input and split their capacity to deal effectively and simultaneously with all emergencies within 15 minutes. The other 2 groups were able to observe the performing group only whilst in the simulation rooms, but not during the handover process. As stated, each cohort included 3 teams performing. For the first team that performed (1st cycle of the cohort) we designed relatively familiar or common scenarios. The second team performing had identical concept scenarios but slightly more demanding to manage (increasing difficulty). The last team would have observed the 2 previous groups, hence we decided to exponentially increase the level of difficulty in the scenarios. In total we had 3 cycles/cohort; each cycle was designed to be gradually more challenging to account for the experience that teams were gaining through observation. We used the Team Emergency Assessment Measure (TEAM) to assess students during the handover and during each scenario. TEAM involves 11 domains and a global score (Table 1) . TEAM is a validated tool which was not only deemed appropriate for the circumstances but is also accepted as the gold standard tool to assess team performance as stated in a recent systematic review. 4 Students could ask for help from seniors or nurses through the internal communication system throughout the cycles. Outcomes were defined by the TEAM tool, and this mainly included individual performance in a team-based setting; for the purpose of simplicity, we focused on global scores only. Two dedicated assessors were allocated in each room (where the scenarios took place) and two during the handover observing how students identify and allocate workload. All assessors were independent, and scenarios were video Table 1 Team Emergency Assessment Measure -TEAM™ The team leader let the team know what was expected of them through direction and commands The team leader maintained a global perspective The team communicated effectively The team worked together to complete the tasks in a timely manner The team acted with composure and control The team morale was positive The team adapted to changing situations The team monitored and reassessed the situation The team anticipated potential actions The team prioritised tasks The team followed approved standards and guidelines Global Score recorded; a third independent assessor marked each of the cycles. Each assessor filled out the TEAM form with a free space for feedback and had a dedicated code so we could go back and discuss any discrepancies. Data were transferred in an Excel file, following harmonisation with video recording results. During each cohort there were 3 rotating teams; one performing and 2 teams observing and collating feedback which was communicated during the debriefing session after each cycle (15 minutes). Feedback sessions were together with assessors who discussed the assessment and provided feedback in the presence of all 3 teams. Each team was performing once per cycle and observing twice. The handover process was the only part of the performance which was not observed by the other two teams. This was to ensure we captured a baseline performance of the delegates in allocating and organising their handover tasks. We used inferential univariate statistics to describe data on IBM SPSS for Macintosh version 26. We assessed normality of data using Shapiro-Wilk test. P-value less than 0.05 was considered as statistically significant. Data summary is provided in Table 2 which summarises each TEAM item score across the different rotating teams. The mean global (team) score for handover performance was 5.44/10 ± 2.24; although overall lower than any other element of performance this did not reach a statistically significant level. The overall global (team) score for baseline performance across any team was 6.64/10 ± 2.11. The overall global score for performance for the first rotating team of each cycle was 6.44/10 ± 2.01, for the second rotating team was 7.89/10 ± 2.09, and for the third rotating team 6.78/10 ± 1.64. Comparison across the rotating teams did not show any statistically significant difference (p = 0.099). We did not compare handover performance across rotating teams as this was the only process not observed by the other 2 teams during each cycle. We found baseline handover performance to be poor; this has been the case for all the cohorts who had a similar baseline score. Handover performance was remarkably lower for all elements compared with the rest of team performance for each scenario. This potentially indicates a lack of hierarchical thinking and communication gaps which can be attributed to either language barrier and/or mix of educational backgrounds. Handover requires a mix of background clinical knowledge with non-technical skills, which results in essentially a more complex-to-teach process, but is still vital to maintain safety in clinical practice. Providing direct observation of every part of the cycle (apart from handover) resulted in an improved performance between the first and the second rotating team; however, this was not the case for the third rotating team whose performance was still better than the first one, but worse than the second. This can be attributed to the fact that the third cycle had a disproportionally difficult cohort of scenarios. Overall, there is improvement in performance via peer observation only, which did not reach statistically significant results because of sample limitations. From the TEAM items, we noticed a slightly lower performance trend on team leader skills (items 1 and 2). Observing the inter-item (TEAM tool) differences can essentially point out the direction of the teaching 462 interventions needed to establish effective composite skills i.e. handover or teamwork, which are the foundation of safety in modern medicine. There is patchy knowledge and understanding of handovers and relevant pitfalls that can lead to serious incidents. 12 Although it is evident that the root cause of most serious incidents are down to communication, 13 it is more challenging to identify the stage of communication at which an error occurs, e.g. during handover. 14 Recent research suggests that handover may in fact play a significant role in malpractice claims. 15 However, there is a gap in the evidence for this, which may lead to handover and its critical role in providing safe clinical practice being overlooked. Handover is the bridge of communication amongst allied healthcare providers. It is vital that it is performed with no error by avoiding lack of information and clarity, which can be hindered in such dynamic hospital settings. Unfortunately, there are studies showing that handover is performed poorly; this is associated with morbidity and mortality. 16 This has sparked interest in the field of handover. There have been reports of studies, particularly questionnaires, on understanding the views and perceptions of trainees on the importance of handover. 17 Some have made recommendations such as examining the approach to handover of healthcare professionals, with others identifying other potential ways to deliver handover such as effective electronic systems, devoted times for handover and writing good summaries for patients. 18, 19 However, such research is focused more on the delivery of handover, i.e. symptomatic treatment of poor handover delivery, rather than attempting to explore the underlying pitfalls in handover and identifying ways to overcome this. Other studies where handover interventions were made and assessed for outcomes have mostly demonstrated poor methodology and inability to repeat them. 20 Nowadays, handover teaching in undergraduate medical education consists of revised templates and mnemonics to guide students. 21 There are also simulation-based learning opportunities, however, these focus on single-patient based scenarios without the complex interplay of having multiple patients to hand over, and other factors such as the involvement of multidisciplinary team members from different backgrounds. 22 This is one of the first reported high-fidelity simulation-based learning studies to assess the capacity of students to work as a team. It has been performed in a centre of excellence (Centre for Advanced Simulation and Education) which provided a virtual hospital environment. This is a major strength, because this environment allows direct observation of how students from different educational backgrounds, whose native language may not be English and/or their degree may not be delivered in English, working together in a gold standard high fidelity setting. In simple terms, a virtual hospital, makes the environment real, allowing in-depth observation of the students' performance. At the same time, iVOC study provides a blueprint of developing and refining an easily reproducible setting, globally adaptable, as most interventions described in the study can be adapted in lower fidelity setting which are provided by most medical schools. Finally, our students represented educational backgrounds of popular health systems including the NHS, Turkey (Acibadem University) as well as Cyprus, Greece and Asia (India). We recognise a series of limitations; although we used an international cohort, this is still a single centre study. Firstly, the number of participants is limited to reach statistically significant results, however the iVOC concept is clearly showing some useful trends in our results, enough to draw safe conclusions on our root cause analysis for students' performance. Secondly, although the clinical scenarios were meticulously designed to reflect the study purpose, they were not validated formally. Finally, we did not introduce inter-observer variability measurements as this would complicate the analysis process in a small cohort of numbers. Our primary aim was to simulate an on-call shift using a high-fidelity virtual hospital setting and assess students' team performance of tasks related to patient handovers (prioritisation, team allocation) during a busy handover. This would serve as a stepping-stone to develop in a high-fidelity setting, a reproducible OSCE module. This will introduce a ground-breaking shift in the mindset of "individual-based performance OSCE" to a "team-based performance". Medical schools should consider standardisation of similar efforts to result in pilot team-based OSCEs. Those teambased OSCEs can be more representative of everyday practice and can potentially provide the students with skills that prepare them for safe practice, especially in multi-cultural healthcare systems. In two of our previously published studies 2,3 we discussed some alternative low-cost models for how group-based OSCEs can be introduced as part of the medical school assessments. Modern medicine is shifting continuously to teamwork-led practice, and this should also be reflected in undergraduate assessments. Such initiatives can provide a global assessment framework, which can be a stepping-stone for enhancing safety in clinical practice. It could also result in smooth integration of healthcare practitioners from abroad, who may be new or not familiar with the healthcare system that they plan to work in (for instance the NHS). Center of Advanced Simulation and Education, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey. The ESMSC course is compatible with the current 3R principles for animal-model simulation (refinement, replacement, and reduction). Ethical approval was granted and met directive 63/2010, PD 56/April 2013, according to the local ethics committee. The license reference number is 4857/15-09-2017, MS, AP et al. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. All the participants have signed a consent form to participate in the study. Developing a novel framework for non-technical skills learning strategies for undergraduates: a systematic review Advancing medical students' non-technical skills in a group-based setting Team performance training for medical students: low vs high fidelity simulation Measuring the teamwork performance of teams in crisis situations: a systematic review of assessment tools and their measurement properties Developing a novel international undergraduate surgical masterclass during a financial crisis: our 4-year experience What challenges did junior doctors face while working during the COVID-19 pandemic? A qualitative study Student views on a novel holistic surgical education curriculum (iG4): a multi-national survey in a changing landscape A novel multi-faceted course blueprint to support outcome-based holistic surgical education: the integrated generation 4 model (iG4) Retrospective qualitative study evaluating the application of IG4 curriculum: an adaptable concept for holistic surgical education Developing an international combined applied surgical science and wet lab simulation course as an undergraduate teaching model ABC of surgical teaching: time to consider a global blueprint for holistic education Clinical handover incident reporting in one UK general hospital Standardised handover process with checklist improves quality and safety of care in the postanaesthesia care unit: the postanaesthesia team handover trial Residents report on adverse events and their causes Resident handoff training: initial evaluation of a novel method Improving measurement in clinical handover. Qual Saf Health Care Clinical handover practices among healthcare practitioners in acute care services: a qualitative study Handover among junior doctors: a quality improvement study Perspectives of clinical handover processes: a multi-site survey across different health professionals educational interventions to improve handover in health care: an updated systematic review Teaching handoff communication to nursing students: a teaching intervention and lessons learned the modified, multi-patient observed simulated handoff experience (m-oshe): assessment and feedback for entering residents on handoff performance peer-reviewed, open access journal that aims to present and publish research on Medical Education covering medical, dental, nursing and allied health care professional education. The journal covers undergraduate education, postgraduate training and continuing medical education including emerging trends and innovative models linking education, research, and health care services. The manuscript management system is completely online and includes a very quick and fair peer-review system All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work. Dr Aikaterini Dedeilia reports grants from World Health Organization -Health Workforce Department funding for research on the COVID-19 effects on health workforce education, outside the submitted work. The authors report no conflicts of interest in this work.