key: cord-1003354-tmzy6gie authors: Herman, Anastasha; Joseph, Janso Padickakudi; Ghosh, Samit; Lloyd, Simon K; Lakhani, Raj; Cartwright, Samuel; Orban, Nara title: National, virtual otolaryngology training day in the United Kingdom during the COVID-19 pandemic: results of a pilot survey date: 2020-10-19 journal: J Surg Educ DOI: 10.1016/j.jsurg.2020.10.013 sha: 62ddf431f034dd2cbb0a234014406057c9a213da doc_id: 1003354 cord_uid: tmzy6gie The COVID-19 pandemic has forced an unprecedented worldwide change in our daily lives, particularly in healthcare. In response to the lockdown measure it has brought on, some deaneries cancelled planned teaching days, whilst others had delivered remote smaller scale teaching sessions electronically. Due to significant regional variation, the National Association of Programme Directors in ENT (NAPDENT) set up a national educational programme, matched to the Intercollegiate Surgical Curriculum Programme (ISCP). This is delivered virtually and for free. This pilot survey evaluates participants’ learning experience of the first NAPDENT National Otology Training Day, which was held on 20 April 2020 via Zoom. This is a pilot survey evaluating the first online, national educational programme delivered by the National Association of Programme Directors in ENT (NAPDENT) in the United Kingdom. In the times of the COVID-19 pandemic, our programme has rapidly learnt from this initial experience and continues to build on lessons learnt for future editions. It is feasible to augment the national ENT curriculum with a collaborative, online teaching programme. Regional buy-in is required to make this work. This modality allows for access to subject experts across the country. Synchronous delivery allows for participant interaction, while an online repository allows for asynchronous viewing when work or personal commitments do not allow live attendance. The key to participant engagement is adequate structure, appropriate breaks and opportunities for interaction. Even once the COVID-19 pandemic has passed, this teaching modality will remain as an opportunity to educate ENT surgeons of the future. The COVID-19 pandemic has forced an unprecedented worldwide change in our daily lives, particularly in healthcare. The UK introduced lockdown measures on 23 March 2020. A joint policy statement from the Royal Surgical Colleges issued on 13 March 2020 recommended non-essential activities to be postponed, including educational courses to either be postponed or delivered via video call or webinar. Health education bodies across England, Wales and Scotland oversee training and educational delivery to postgraduate healthcare professionals in the UK. Local delivery is coordinated by regions or 'deaneries', with each medical and surgical specialty led by a Training Programme Director (TPD). Within ENT higher surgical training an educational programme is most frequently delivered as a monthly face-to-face 'teaching day' including mainly didactic materials, with occasional workshop and simulation held at a central location within the local deanery. Trainees within the deanery use study leave on these specific days to travel to the specified location. In the pre-COVID 19 era, these regional teaching days were always face-to-face. There is approximately 450 ENT Specialty Trainees Year 3 to 8 (ST3 to ST8) in the UK, who have to maintain 70% teaching day attendance within an academic year. National-level teaching was delivered at annual subspecialty conferences and biennial British Academic Conference of Otolaryngology (BACO) held at various locations across the country. Further opportunities were also provided by the annual Live International Otolaryngology Network (LION) global broadcast of surgical demonstrations with a focus in Otology and Neurotology. In response to the COVID-19 pandemic, deaneries had varied approaches to education delivery. Some cancelled planned teaching days, whilst others had delivered remote smaller scale teaching sessions electronically. Due to significant regional variation, the National Association of Programme Directors in ENT (NAPDENT) set up a national educational programme, matched to the Intercollegiate Surgical Curriculum Programme (ISCP). This is delivered virtually and for free. This pilot survey evaluates participants' learning experience of the first NAPDENT National Otology Training Day, which was held on 20 April 2020 via Zoom (Zoom Video Communications Inc., 2020) The NAPDENT National Otology Training Day was advertised five days prior to all ENT TPDs in the UK and on the Association of Otolaryngologists in Training's (AOT) website and mailing list of about 1500 recipients, which included trainees and consultants as well as staff grades, associate specialists and specialty doctors (SAS doctors). Participants were required to register in advance to receive Zoom meeting login details. The questionnaire was sent to all registered participants after the session. Participants had two weeks to complete the survey. The pilot questionnaire was divided into four sections -demographics, pre-session, session content and post-session. There were 60 questions in total, which consisted of single-answer multiple choice, four-point interval scale responses and free-text. The 'session content' questions were multiplied by five to reflect each of the five sessions delivered. All non-free text questions were compulsory. A Google Form online link was sent to all the registered email addresses of participants. The information was collected anonymously with no incentives offered to participants. The four-point interval scale was used. This was preferred to the traditional 5-point Likert scale because the primary objective was to capture direction (positive or negative) of responses, rather than their intensity level of agreement of disagreement. The four-point scale also eliminates the neutral response option, which reduces ambivalence and forces participants to present their true feelings regarding the subject. The questions were initially designed based on the Kirkpatrick model of learning evaluation, which includes four outcome levels: satisfaction, learning value, behavioural change and organisational change or patient outcome. These were reflected mainly in the latter half of the questionnaire within sections 'session content' and 'post session'. The former half of the survey focused on attendees' demographics, devices used and level of preparation involved prior to viewing the session. All questions were peer reviewed by four ENT consultants. Demographics 230 people registered for the session. There was fluctuation in attendance during the five-hour session with a minimum of 170 and a maximum 250 people logged on at any one time. 120 participants (52%) responded to the questionnaire. Individuals from England, Scotland, Northern Ireland and the Republic of Ireland attended. There were no participants from Wales. 40% of participants were from North West and Mersey deanery (see Figure 1 ). Most participants (62%; n=75) were specialist trainees in ENT, but there was a varied audience including foundation trainees, core trainees, clinical fellows, speciality and associate specialist doctors and consultants (see Table 1 ). Communication for TPDs was the most influential source of advertisement (54%, n=65). Pre-session 97% (n=116) of respondents found the registration process easy. 68% (n=82) watched the online session from home, whilst the rest watched from work ( Figure 2 ). There were different types of devices used: 64% (n=77) laptops, 20% (n=24) mobile phones and 16% (n=19) desktop computers. 79% (n=95) did not research the speakers or topics prior beforehand. 89% (n=107) planned to stay for the whole session, with 80% of those managed to watch until the end. 36 respondents provided free text comments on reasons behind not watching the session in entirety -58% (n=21) stated work commitments, 36% (n=13) already had personal commitments scheduled due to short notice of the session and technical issues were stated by the remaining 6% (n=2). There were five sessions in total. The majority of participants perceived the duration of these to be "perfect" (71%). However, more than 30% participants perceived session 2 and 5 to be 'just over' or 'too long' (see Figure 3 ). Interestingly, it was session 1 that had the longest actual duration. Further 4-point interval scale questions relating to delivery, content and relevance of the topics presented in each session were also asked in the questionnaire (see Figure 4 ). Session 2 and 5 received an overall more negative scale response compared to the other sessions, which suggested that actual time duration is a secondary factor. Figure 4 outlined the responses to each session's content and overall rating. Questions from participants during the session were asked using the chat function with a nominated moderator who presented them to the speakers. An average of 18 questions were asked during each session. These were either answered verbally by the presenting speaker or generated further discussion within the forum involving both consultants and trainees. Figure 9 demonstrated that 83% (n=99) found it a more comfortable means to ask questions compared to a face-to-face lecture setting. A majority of respondents also felt that it was easy to use (93%, n=112), their questions were answered adequately (63%, n=75) and it was moderated well (91%, n=109). A vast majority of participants found it easier to attend and concentrate in this format compared to face-to-face sessions (see Figures 5 and 6 ). 91% of respondents suggested the ideal frequency of scheduled breaks in a typical four-hour session should be every 45 to 90 mins (see Figure 7) . Overall, participants regarded the virtual training day to have an overwhelmingly positive learning value (see Figure 8 ). Figure 9 demonstrated positive evaluation of the chat function as a tool for interaction with the speakers and other participants. The COVID-19 pandemic has brought about profound medical education adaptations that have shifted the classroom into the virtual world. This newly-created, national venture was the result of a national collaboration of ENT training programme directors. It aimed to deliver aspects of the national ISCP curriculum utilising online learning. The speciality-specific (knowledge) content of the programme was rated highly. The national reach of the training day allowed for collaboration and access to experts from around the country. During the sessions, participants were able to ask questions via a chat function. A nominated moderator fielded and relayed questions to the speaker. These interactive modalities were viewed as easy to use, adequate and non-threatening ways to engage with the live content, as shown in Figure 9 . The session was developed as a mixture of instructional and case-based learning. Synchronous content delivery with experts allowed for collaborative learning, where learners received information and were able to communicate with each other simultaneously. Lessons learnt regarding the use of the new online teaching modality have been profound. In keeping with previously described components of e-learning content -development, delivery, management and standardisation -each of these posed challenges and opportunities. Development of the speciality-specific content had to be adapted to the new virtual environment, and flexible enough to cater to live interaction. The session was designed for junior registrars (ST3-5), but remote attendance enabled other training and non-training grade doctors to attend. Investment into an institutional subscription to Zoom Pro eliminated the meeting duration limit and increased the potential attendees to up to 500 people. The teaching session was recorded and made available on YouTube with links available on the AOT website for free, so learners could attend asynchronously if required, or revisit material in the future. The videos were edited accordingly prior to being uploaded by removing any patient data to maintain confidentiality. They were also bookmarked based on the topics presented to allow more targeted viewing. This investment into information technology infrastructure is vital to store the material in an accessible online repository, supporting short-and long-term educational goals. Standardisation of a new learning material is vital to promote its compatibility and usability across different computer systems. Therefore, the use of free existing established software, such as Zoom, was preferred as evidenced by the participants' positive responses to ease of online registration, variability of devices used and ease of access from both home and the workplace. Another aspect that has become apparent in this endeavour is the crucial role of advertising for sessions of online learning. Advertisement from TPDs was shown to be the commonest source of communication about the session, indicating that regional leadership is vital for the success of this national scheme. This may be useful as well to address the perceived absence or low number of trainees from certain deaneries (see Figure 1 ). Interestingly, trainees from Republic of Ireland (4.2%, n=5) joined the session because of the advertisement from AOT and 'word of mouth' as their TPD was not included in the initial communication to all UK ENT TPDs. Other methods of communication, such as advertisement on social media, will also need to be considered for future sessions. The latest systematic review on e-learning in medical education by Regmi et al identified several factors, categorised into enablers and barriers, influencing the enhancement of lifelong learning. They found that e-learning was more effective than traditional learning when it could deliver stimulating integrated content, facilitate learning acquisition and integration into practice, provide flexibility to address short and long-term educational needs, as well as encourage learning autonomy by improving engagement and retention. Factors in e-learning that may pose a barrier include poor motivation and expectation, resource intensive, unsuitability for some disciplines and a relatively high IT skills requirement for both facilitators and participants. Interactive online education provides a novel intersect of behaviourist, cognitivist, constructivist and connectivist approaches to learning . An underlying appreciation of different learning theories can help educators tailor online teaching to appropriate audiences. The most significant element of this multimodal learning approach is its flexibility to evolve in pace with learning needs and technological advances. A majority of respondents did not research the topics or speakers prior to the session. There may be several factors affecting this, including the short advertisement notice of only five days, perceived familiarity of topics, unfamiliarity of the session format (eg. didactic or interactive) or the training level at which topics were presented and personal or work commitments, amongst other factors. In relation to this, some respondents commented on the 'free text' section about the need for presession materials, in the form of MCQs, summary of topics and panel discussion cases. An active trend in current medical education is the concept of a flipped classroom, which is a type of blended learning where the instructional lecture-type information is viewed at home and the teacher has a more facilitator role in case-based discussions in the classroom. Taking all of these factors on board, future sessions will aim to provide relevant pre-session materials that will support a wider variety of learning styles. This will also set participants' expectations in order to boost their learning motivation, which is one of the key determinants of e-learning success identified by Regmi et al. Remote attendance of online sessions provided greater flexibility on where participants can attend from, which was reflected by the overwhelming positive response on this as demonstrated in Figure 5 . However, Figure 6 showed that 44% of respondents selected 'somewhat agree', rather than 'strongly agree' and 23% expressed disagreement that this format allowed for better concentration level. This may be explained by Figure 2 , which showed 68% viewed it from home, where access to a quiet space or lack of unforeseen distractions might not be as readily available as in a purpose-built lecture theatre, for example. Data on perceived duration of individual presentations, shown in Figure 3 , and responses on ideal frequency of scheduled breaks, shown in Figure 7 , will inform us on the design of future sessions in order to maximise concentration and learning value obtained. The data suggested that in a typical four to five hour online teaching session, each session should last for 40 to 60 minutes, with a scheduled break after one or two sessions. Figure 4 demonstrated that a higher negative scale rating in slide design, delivery style and content relevance impacted poorly on the overall rating of a session as shown in Session 2. Addressing these aspects during session preparation may help speakers in the future to improve audience engagement and retention of materials delivered. The learning value of the virtual training day was evaluated using questions based on the Kirkpatrick model of learning evaluation. Feedback from participants indicates that behavioural change may have been affected by the session, resulting in Kirkpatrick Level 3 training. (Figure 8 ). The programme was rated highly overall not only due to its high quality specialty-specific content, but also because it was delivered through the new technology-enhanced learning modality, which allows for greater collaboration, provides repository of shared knowledge and increases accessibility whilst maintaining participants' engagement. Table 2 summarised our key recommendations. The survey questions were peer reviewed independently by four ENT consultants to ensure its effectiveness in measuring the intended outcomes of the study. However, given the wider context of operating within an emerging pandemic, we were not able to further pilot the questions to a smaller focused group of participants. After reviewing the results, we acknowledged that there was a high number of questions and some questions measured the same outcome. The former aspect might have introduced an element of fatigue and indifference towards the later stages of the questionnaire. Therefore, we would recommend future iterations to be more streamlined. We were also unable to quantify how many people registered but did not watch the session as it was proven very difficult to keep a record on the number of participants throughout the session because people were logging in and out. Moreover, there may be more than one registered participant using one device and those that did not watch the session or only parts of it would be less likely to fill in the questionnaire. Given the relatively short five days advertisement period, the range of 170 to 250 attendees logged on at a given time could be considered as a success. Extrapolating from the 62% figure of the questionnaire responders who were specialty trainees, our session likely have reached more than a third of all specialty trainees across the UK. We acknowledged that this number could still be improved by continuing to outline clear learning objectives for each session and allowing longer advertisement period to allow trainees to make the necessary arrangements to attend virtually. Another aspect of online delivery that was not measured in this survey was faculty experience, which will be included in our future sessions. It is equally important for them to have an overall positive outlook. Lastly, as we planned for future training days covering other ENT topics, we would also like to recommend a formal method to track learner's attendance in their portfolio, such as in the form of awarding Continuing Professional Development (CPD) points. This pilot survey evaluates the first online, national educational programme delivered by the National Association of Programme Directors in ENT (NAPDENT) in the United Kingdom. In the times of the COVID-19 pandemic, our programme has rapidly learnt from this initial experience and continues to build on lessons learnt for future editions. It is feasible to augment the national ENT curriculum with a collaborative, online teaching programme. Regional buy-in is required to make this work. This modality allows for access to subject experts across the country. Synchronous delivery allows for participant interaction, while an online repository allows for asynchronous viewing when work or personal commitments do not allow live attendance. The key to participant engagement is adequate structure, appropriate breaks and opportunities for interaction. Even once the COVID-19 pandemic has passed, this teaching modality will remain as an opportunity to educate ENT surgeons of the future. It is possible to deliver high-quality, specialist education on a national level using existing online technology 2. Live, interactive delivery allows for a collaborative learning experience Joint policy statement from the Royal Surgical Colleges A technique for the measurement of attitudes. Archives of psychology Three good reasons NOT to use five and seven point Likert items The effects of the neutral response option on the extremeness of participant responses Great Ideas Revisited: Revisiting Kirkpatrick's Four-Level Model Medical Education Adaptations: Really Good Stuff for educational transition during a pandemic The impact of e-learning in medical education. Academic medicine Learning at home during COVID-19: A multi-institutional virtual learning collaboration Association of Otolaryngologists in Training ENT Education. National Training Day Playlist A systematic review of the factors-enablers and barriers-affecting elearning in health sciences education. BMC medical education Constructivism: Comparing Critical Features from an Instructional Design Perspective. Performance Improvement Quarterly Connectivism: A knowledge learning theory for the digital age Theories and Frameworks for Online Education: Seeking an Integrated Model Inverting the classroom: A gateway to creating an inclusive learning environment