key: cord-0849622-dowd6lkk authors: Dave, Manas; Dixon, Carly; Patel, Neil title: An educational evaluation of learner experiences in dentistry open-book examinations date: 2021-08-27 journal: Br Dent J DOI: 10.1038/s41415-021-3258-7 sha: 8d1e78e5fcd00301bbb75f8605a3ddd017a2ea36 doc_id: 849622 cord_uid: dowd6lkk Introduction The COVID-19 pandemic resulted in substantial global disruptions, with many universities restructuring their examinations to open-book assessments. In this format, learners are allowed to use revision notes, textbooks and other permitted material during their examination. Traditionally, dentistry is assessed using closed-book examinations where the use of such materials is prohibited. Learners will have limited or no prior experience of open-book examinations in dentistry, hence this educational evaluation aimed to explore their experiences and assess suitability for continuity of this examination. Methods A retrospective anonymous survey was conducted on all undergraduate dentistry and oral health sciences learners on completion of their examination and once they received their results. The questionnaire was designed to contain both qualitative and quantitative data items, validated through previous published studies. All authors independently analysed the data through tallying quantitative parameters and thematically analysing qualitative free text. Results In total, 210 learners participated in the survey, with 52.4% strongly agreeing a preference for open-book examinations. In addition, 42.9% of learners either agreed or strongly agreed that less preparation was required for open-book examinations; however, 80.5% reported learning new information. Furthermore, 85.2% disagreed or strongly disagreed that open-book examinations were stressful. Thematic analysis highlighted four core themes: stress, memorisation of facts, motivation to learn and unfairness. Conclusions There were several learning points for both educators and learners that can be actioned for future open-book examinations, including tailoring exam questions to foster application of knowledge rather than resource-searching skills, intended learning objective mapping against curriculum requirements and learner preparation. SUPPLEMENTARY INFORMATION: Zusatzmaterial online: Zu diesem Beitrag sind unter 10.1038/s41415-021-3258-7 für autorisierte Leser zusätzliche Dateien abrufbar. The coronavirus disease 2019 (COVID-19) pandemic resulted in social distancing policies, restriction on travel and cancellation of faceto-face teaching in March 2020 in the United Kingdom (UK) and numerous countries globally. Due to these restrictions, many universities restructured their programmes for delivery of online classroom learning and remote assessments. Traditional assessments, which are often closed book in nature, are undertaken in test venues where invigilators would monitor learners to ensure there is no academic malpractice such as cheating using notes. This same level of monitoring would not be possible for online examinations, hence numerous universities restructured their examination format to open-book examinations. 1, 2 Open-book examinations allow learners to refer to their notes, textbooks and other permitted materials when answering questions. The time duration of the examination is often increased to account for the use of revision notes, information technology issues, disability requirements and time zones for international learners. The main focus of open-book examinations is to test higher-order thinking skills such as application of knowledge, evaluation and critical thinking. In closed-book examinations, learners are not permitted to have any supporting material such as revision notes or textbooks. Closed-book examinations can test the same knowledge as open-book examinations; however, traditionally, have tested more factual learning and memorisation rather than application of knowledge. 3, 4 There are numerous studies published comparing test scores between open-and closed-book examinations; however, there is limited evidence on learner perspectives for the delivery of this examination format. Moreover, to the best of the authors' knowledge, there are no studies comparing open-and closed-book examinations in dentistry to date. Due to the COVID-19 disruptions, during the 2019-2020 academic year, all outstanding examinations in this UK university dental faculty were undertaken in an online openbook format. This educational evaluation explores learner experiences of openbook examinations in dentistry during the COVID-19 pandemic to receive feedback on the examination process, learner satisfaction and the suitability for continuity of this examination format in the future. This retrospective survey (hereafter referred to as study) was deemed an educational evaluation through an internal university ethics decision tool and liaison with the university ethics committee ; therefore, ethical approval was not required. This study involved an anonymous online survey undertaken in a single university in the UK, with informed consent implied through participation. Potential coercion was minimised through anonymisation of the survey. Learners were informed of the study through e-mail and were informed that participation was voluntary, as well as that the results would be shared wholly with educators in the division and made available to the public (through publication). The methods protocol fulfilled the internal university ethics requirements and was discussed within the dental school management team before dissemination. An anonymous 14-item questionnaire was designed which contained both quantitative (n = 12) and qualitative (n = 2) data items based on previous studies (see online supplementary information). 5,6 A phenomenological approach was adapted to ensure richness of data, and to ensure the relevant parameters were evaluated to understand learner experiences and barriers to effective engagement with the open-book examination format. The questionnaire was distributed to all undergraduate dentistry learners in years 1-5 and BSc oral health sciences learners in years 1-3 who had sat their examinations and received their results over a five-week period to maximise participation. The questionnaire was distributed through Blackboard on Select Survey, which allowed verification of learner status and only allowed learners to complete one entry to prevent duplicates. The first ten learners that responded to the survey were taken as the test pilot. Any changes could be made to the online questionnaire; however, this was not needed. The learner responses were exported to Microsoft Office Excel (Washington, United States of America) for analysis. The free text in response to qualitative questions was analysed by thematic content analysis using a constant comparative method to identify emerging themes. To enhance data synthesis and interpretation, emerging themes were discussed frequently between the team and categories refined. 7, 8 Relationships and patterns were explored across categories. All authors independently analysed the data by tallying quantitative parameters and thematically analysing the data as described above. Any discrepancies were resolved through discussion and planned involvement of an experienced university researcher; however, this was not required. This educational evaluation aimed to determine the following: 1. To understand the belief and attitudes of learners towards open-book examinations with a focus on learner preference, perception of examination preparation and access to materials, revision strategies, learner stress and learner perception of fairness 2. To inform educators of the main improvements required to enhance learner experience and satisfaction in open-book examinations in dentistry. In total, 210 learners participated in the survey ( Table 1 summarises the response rate). These included both learners from the UK and international learners. The response rate was only 25% for fifth-year dentistry learners. This is likely to have occurred as learners had graduated and were no longer checking their university e-mails or Blackboard portal. All results were amalgamated together across both dentistry and BSc oral health sciences from all year groups unless otherwise stated. The open-book examinations which learners undertook were as follows: short-answer paper, multiple-choice question, anatomy extendedmatching item and, for fifth-year dentistry and third-year BSc learners, clinical cases. The results of the quantitative analysis are summarised in Table 2 and positive outcomes of 'agree/strongly agree' are presented in Figure 1 . Overall, 52.4% of all learners strongly agreed a preference for open-book examinations in comparison to conventional closed-book examinations. Only 5.7% of learners strongly disagreed with this examination format. Furthermore, 42.9% of learners either agreed or strongly agreed that less preparation was required for open-book examinations. Despite this, 80.5% of learners reported learning new information. It is of note that across all learners, 36.2% either disagreed or strongly disagreed that less preparation was required for open book examinations. Also of importance, 6.7% of all learners disagreed or strongly disagreed with having a suitable working space during the openbook examination.. In addition, 85.2% disagreed or strongly disagreed that open-book examinations were stressful and 62.4% agreed or strongly agreed that they were easier compared to conventional closed-book examinations. Nonetheless, 62% agreed or strongly agreed that open-book examinations were fair. Nearly half (49.5%) of learners indicated they would prefer an open-book examination in the future, 12.4% indicated a preference for closed-book examinations and 38.1% a combination of both examination formats. Four core themes were identified: stress, memorisation of facts, motivation to learn and unfairness. The word 'stress' , 'stresses' or 'stressed' was used by 70 learners to describe open-book examinations. Sixty-six learners reported open-book examinations resulted in less stress while four learners reported open-book examinations increasing stress. Learners also mentioned that reduced stress from openbook examinations resulted in increased happiness and improved mental wellbeing, which allowed them to perform to the best of their ability. Some learners reported that closed-book examinations gave them anxiety and panic attacks; however, this was not experienced during open-book examinations. A disadvantage of open-book examinations was highlighted by learners who stated the lack of stress made them feel like there were not any rewards when they got their results. Learners cited they felt reduced pressure because they did not feel they had to memorise facts, they anticipated the questions they would be asked (notably as they would also do for a closed book examination), the increased time to undertake the actual examinations and access to resources such as revision notes. One learner reported they 'enjoyed my revision a lot more, focusing on note making to help myself understand content' . There were mixed responses by learners regarding why open-book examinations were more stressful than closed-book examinations. One learner referred to the increased examination time (learners felt they were stressed for longer), the ambiguity of some questions to determine how to answer it and personal problems of completing the examination at home. One learner stated that they felt 'you will be marked more harshly as examiners believe you "have all the answers"' . Forty-five learners mentioned the term 'memory' , 'memorisation' or such a variant. The majority of learners reported that during closed-book examinations, they would have memorised facts and statistics in anticipation of exam questions; however, would forget this soon after the examination had finished: 'I didn't spend a long time trying to memorise random facts, only to be forgotten after the exam'. During revision for open-book examinations, learners stated they did not commit content to memory to the best of their ability (in comparison to learning the content thoroughly during closed-book examinations) as they knew they had access to resources during the examination: 'Because you know you can use your notes, you're less likely to fully revise and memorise the content because there isn't that stress of not knowing it' . Motivation to learn Some learners reported that they were not motivated to study for their examinations and expressed concern on preparedness in subsequent years of their degree: '[...] when I knew the exams were open book I stopped revising because that actually demotivated me as I couldn't see the point of revising if we have access to our books' and 'open book exams can make you feel less prepared for the following year and the rest of your degree'. Moreover, some learners reported concerns on their level of knowledge in their future careers: 'there was no incentive to actually learn some of the dates/facts that we will actually need to know for daily life as a dentist' . Other learners reported that they stopped revision for the examinations only to realise they needed to retain the content for the following academic year and spent their summer holidays revising. Following release of results, some learners reported that the sense of achievement was lower with open-book examinations in comparison to closed-book examinations: 'there isn't the same gratification and feeling of achievement after an open book exam, as less hard work is required' . However, other learners reported that reliance on notes during the examination motivated them to amend their revision strategy by producing more complete notes to help them gain a better understanding of the content. This in turn increased their confidence and motivated them to gain a better understanding of the course content. Learners reported challenges with a suitable working space and internet connection with examinations conducted at home, which put them at a disadvantage compared to their peers. Other learners reported learners colluding and working together and some having more access to notes and resources, which put them at an unfair advantage. The majority of concerns were related to learners not getting recognition for the level of work they put in: 'People who haven't worked as hard as others can potentially do better' . The COVID-19 pandemic has caused a substantial impact in the delivery and assessment of dental education, requiring universities to be responsive to continued changes in government policy and restrictions. This study has illustrated that the adapted open-book examination format of 2019-2020 was generally well received by learners. Of particular interest is the fact that this examination style led to an improvement in learner wellbeing, through reductions in stress and (in some learners) its associated symptoms such as panic attacks, and perceived happiness against a background of COVID-19 uncertainty. Managing learner wellbeing is a high-priority task for higher education institutions. 9 A crossover study compared open-book and closed-book examinations between pharmacy learners and discovered there was no difference in learning approach for both examination formats. Moreover, learners preferred openbook examinations which were associated with lower anxiety levels, less need for memorisation, greater scope for analytical thinking and problem-solving, and less effort for preparation. 3 Our evaluation showed that numerous learners reported the reduced stress of open-book examinations, their increased confidence and subsequently that they performed better, which has similarly been reported in other studies. 3, 10 Broyles 11 conducted a non-randomised trial with medical learners sitting both open-and closed-book examinations. They discovered that over 80% of learners reported open-book examinations made them less anxious, less stressed and more comfortable. Interestingly, learners felt there may be increased pressure to perform well due to the examination format, which mirrors the results reported in this study. The requirement to undertake open-book assessments has enabled educators to review in more detail their role in higher education within dentistry. Open-book examinations have been reported to encourage a greater depth of knowledge and reinforcement of learning key concepts. 12 However, it is important to evaluate the long-term impact of open-book assessments on candidate knowledge retention for progression into future academic years and provision of clinical care within dentistry. Learners may progress to postgraduate studies and may be illequipped to adapt to closed-booked assessments, which are still very much used in postgraduate education assessment. Lipener 13 conducted a randomised control trial for physicians between open-and closedbook examinations, but restricted the use of resources to ensure all participants had access to the same information. The resource they used was commonly used in practice; therefore, the examination was a more accurate simulation of clinical practice. Such concepts can be adapted to dentistry; for instance, using certain dental guidelines (for example, British National Formulary). However, with learners undertaking examinations remotely, this may be difficult to achieve without appropriate proctoring. It was interesting to see learners reporting that their peers were colluding and working together, which is against university policy. The emergence of open-book examinations raises concerns of malpractice; proactive and robust policies, including quality-assured measures of malpractice detection (for example, computer audio and visual recording during the examination), need to be implemented by educators to maintain fairness. The delivery of online assessments in the 'new normal' has enabled reflection on the content of questions and assessment methods to ensure that types of examination questions are suited to either open-book or closed-book formats. Bloom's taxonomy (Fig. 2) is a classification which contains six categories of cognitive skills ranging from lower-order to higher-order skills. 14 The latter requires deeper learning and a higher level of cognitive processing. At the lower end of the classification pyramid are knowledge, comprehension and application which are cognitive skills tested generally by closed-book examinations. The higher levels of the pyramid involve analysis, evaluation and synthesis. These involve critical thinking, creating a novel product from a specific situation (such as a management/treatment plan) and evaluative skills such as evidence-based medicine/ dentistry. 15 Open-book assessments (among other assessment methods) allow educators the opportunity to assess learners' higher-order skills of understanding the content; however, this requires appropriately worded questions that are planned to test analysis, synthesis and evaluation. Examples of closed-book examination questions and their adaptation to the open-book format are provided in Table 3 . The questions in the closed-book examination test knowledge, understanding and application in Bloom's taxonomy by requiring learners to demonstrate recall of knowledge, correct understanding of the dose and delivery, and its application in a practical situation. In contrast, the open-book examinations focus on more open-ended questions to test the learner's situational response in evaluating the scenario, analysing key information and creating a treatment plan. The wording of the questions focuses on assessing the higher cognitive skills of the learners. The knowledge, understanding and application are foundation skills required at a minimum to understand the questions asked. The University of Newcastle, Australia has produced this helpful guide for developing questions for open-book exams, which educators may find a useful resource. 16 In the academic year of 2019-2020, finalyear dental learners who undertook open-book examinations also undertook online clinical vivas to enable a mixed model approach. This was to ensure final-year learners were assessed by two different assessment methods, akin to traditional final examinations which have a clinical and written component. In the viva examinations, candidates' higherlevel thinking was explored around three clinical cases on a recorded video conference with two examiners, enabling the open-book examination to act as an adjunct rather than a replacement to closed-book examinations. This approach enabled mapping of intended learning outcomes (ILOs) to ensure candidates were able to demonstrate well-rounded understanding and also to provide quality assurances for each individual candidate. The recordings of clinical vivas enabled external examiners to review for quality assurance, demonstrating the importance of recorded clinical examinations in future years. Further evaluation of examiners' and candidates' experience of, and performance in, virtual examinations is required. With changes to assessment models, it is important to ensure learners are clearly informed of new assessment methods and supportive learning material to tailor revision and mock assessment, which was provided to all learners via Blackboard. Communication to the academic staff on changes in assessment methods and revalidation of the question banks for open-book examinations is key for longterm success of the examination platform. Our experiences of open-book examinations demonstrated reduced stress and anxiety for learners, enabling all learners to safely (with respect to COVID-19 infection) undertake remote examinations regardless of international location. However, learners reported that access to revision material meant they did not learn the material as thoroughly as they would have done for closed-book examinations. Educators need to devise methods (for example, limiting access to certain types of resources or limiting the examination time) to ensure learners are still learning the content to the best of their ability and not completely relying on their revision material, which would make the examination a resource-searching exercise rather than a test of knowledge. Nonetheless, open-book examinations allow learners to be examined against ILOs and progress in their dental education. This educational evaluation was designed to evaluate the open-book examination format compared to traditional closed-book examinations. As the results are from a single institution on a cohort of learners where educators had limited time to amend the examination format due to the unprecedented disruptions caused by COVID-19, it is likely that the limited planning may have resulted in several limitations reported by the learners. The results are less generalisable to year five dentistry learners given their low response rate. Learners received their results and then conducted the survey, hence there is the potential for learners to have completed the survey based on the examination outcome rather than as a true account of learner examination experiences. The response rates were less than 100%, and as learners completed the survey after their examination results, there is the potential for ascertainment bias, whereby learners who were dissatisfied with their grades may be more likely to respond to the survey. Learners were aware of the educators involved in the survey, and while it was completely anonymous, there is the potential for participant bias whereby learners provide answers they believe would benefit the research team. Nonetheless, in the wider literature, there is limited evidence examining the role of open-and closed-book examinations in dentistry, and this evaluation provides further support for future directed research. The COVID-19 impact on education is substantial, not only on finances for universities and educational establishments, but also on learner experiences. Dental education continues to evolve in the current climate and educators must seek to evaluate new models of assessment, including remote open-book assessment and virtual platforms instead of faceto-face assessments. Sharing best practice across dental schools is key to enable robust evidencebased examinations, which is integral to ensure learners are able to demonstrate learning outcomes to graduate as safe beginners and begin their professional career. Adaptation to Open-Book Online Examination During the COVID-19 Pandemic Open-book examinations Study on the impact of open and closed book formative examinations on pharmacy students' performance, perception, and learning approach Influence of open-and closed-book tests on medical students' learning approaches Exploring the relationship between experienced students' preference for open-and closed-book examinations, approaches to learning and achievement Student Perspectives of Open Book versus Closed Book Examinations a Case Study in Satellite Communication Prevention of type 2 diabetes in British Bangladeshis: qualitative study of community, religious, and professional perspectives Beginning Qualitative Research: A Philosophic and Practical Guide Is the health and wellbeing of university students associated with their academic performance? Cross sectional findings from the United Kingdom The major functions of the open-book examination at the university level: A factor analytic study Open book tests: assessment of academic learning in clerkships Test-enhanced" focused self-directed learning after the teaching modules in biochemistry Effect of Access to an Electronic Medical Resource on Performance Characteristics of a Certification Examination: A Randomized Controlled Trial Taxonomy of Educational Objectives: The Classification of Educational Goals Bloom's taxonomy of cognitive learning objectives A guide for academics -open book exams Preparing for practice A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives Selected results of this study have previously been published in the British Dental Journal. Ethical approval was not sought after as the article involves data gathered for the purpose of service evaluation in an education institution. None declared. Closed book What is the first-line drug given to a patient having an anaphylactic shock? 2.What is the correct dose of this drug? 3.What is the correct method of delivery? Your patient becomes unwell following administration of a 3-gram amoxicillin sachet. They become short of breath, develop swelling of the lips and a blotchy rash. 1.Describe your method of assessing and managing this patient 2.How would you determine if your management is working? 3.What changes would you make to your practice to prevent this type of episode occurring again? Correction to: Cause for applauseThe original article can be found online at https://www.nature.com/articles/s41415-021-3294-3Journal's correction note: News article Br Dent J 2021; 231: 78.When this article was originally published, the image caption incorrectly identified the dentist featured as Dr Neil Sikka. The journal apologises for any inconvenience caused.