key: cord-0969969-ln4u0dec authors: Liu, Chin-Sheng; Wang, Yuan-Min; Lin, Hsiu-Na title: An 8-year retrospective survey of assessment in postgraduate dental training in complicated tooth extraction competency date: 2020-12-24 journal: J Dent Sci DOI: 10.1016/j.jds.2020.11.006 sha: ffc3a43a35f7bb7e1ac056f16bd897536c5a3094 doc_id: 969969 cord_uid: ln4u0dec BACKGROUND/PURPOSE: Direct observation of procedural skills (DOPS) has been increasingly used in health education in recent years. This study evaluated the effect of education and trainees’ perception of assessment on the clinical skills of postgraduate dental trainees in complicated tooth extraction. MATERIALS AND METHODS: This study was conducted as a retrospective survey among postgraduate dental trainees learning complicated tooth extraction in Taipei and Linkou Chang Gung Memorial Hospital from 2012 to 2019. Practical skills were assessed using DOPS by trainees and faculty members. Each clinical case included a complicated extraction of a permanent tooth. RESULTS: A total of 69 participants (26 men and 43 women, average age = 26.49 years, range = 24–34 years) were included in this study. Within the survey cohort, faculty assessments scored significantly higher than did trainees’ self-assessments in each complicated tooth extraction procedure, with no difference between both sexes. The higher-performing trainees tended to underrate their performance much more than did the lower-performing trainees. More than 40% of the trainees evaluated themselves as having “poor capability” in some invasive surgical procedures, even though their actual performance was not lower than that of those who evaluated themselves as having adequate or good capability. CONCLUSION: Self-assessment skills should be developed with more practice and experience. We hope that these findings can guide the planning of faculty development programs for clinical instructors, particularly the new cohort of faculty who will succeed the rapidly retiring boomer generation in the next 10 years. Since the outbreak of severe acute respiratory syndrome (SARS) in Taiwan from May 5 to June 4 in 2003, 1 residency programs in the country have mostly ignored generalized medical and dental education. Students have long been perceived as disliking their school experiences; therefore, publications discussing student feedback have continually mentioned that change is necessary. 2 Recent graduates generally consider themselves unprepared for independent clinical practice despite having likely undergone adequate clinical training. 2, 3 To resolve this problem in medical educational programs for postgraduates, the Taiwanese Department of Health (DOH) funded a nationwide project, "the Postgraduate Year Training Program" (PGY), in 2003. Moreover, the Taiwan Dental Association started a 2-year postgraduate training program in 2010. The training program implemented by the DOH comprises 2-year courses in operative dentistry, endodontics, prosthodontics, periodontics, oral and maxillofacial surgery, pediatric dentistry, and orthodontics. In addition, a 36-h basic curriculum, including medical ethics and related laws, evidence-based medicine, infection control, and dental care quality, was also enforced into this training program. The principal objective of the 2-year course is to improve postgraduate dental trainees' psychomotor abilities by using a standard technique to render these abilities more germane to clinical practice. It is hoped that trainees will be able to achieve their desired goals due to their enhanced capacity to self-evaluate their psychomotor abilities. These goals include the abilities to perform lifelong learning and provide excellent patient care. 4 There are great differences in the content, concepts and tools of the curriculum in dental education around the world. Traditionally, in Taiwan, trainees learn the tooth extraction technique during their internship in a hospital in the sixth year of dental school, and no phantom-head exercises on the tooth extraction training are conducted in a patient simulation laboratory at dental schools. In this environment, students tend to become passive learners without seeking refinement in clinical practice. 5 Most of the PGY trainees have graduated from dental school within 1 or 2 years and are still somewhat unfamiliar with clinical skills and procedures. The clinical curriculum of the PGY training course generally comprises theory presentations, course manuals, and demonstrations of clinical procedures. During the course, clinical instructors evaluate pre-clinical assignments and provide verbal feedback or active instructions. Dental education should comprise theoretical and practical learning for postgraduate dental trainees to develop the ability to treat patients clinically. For dental residents, the practical courses in the first 2 years of postgraduate study represent a novel experience in their education, and they must learn to self-assess. 5 Selfassessment is a basic and indispensable skill for dental professionals to continually improve their competence during their careers. 4 It is frequently the case that an individual's level of selfconfidence is not justified by their ability level. Considerable academic effort has been exerted to understand why this is the case in diverse fields, including clinical, cognitive, personality, organizational, and social psychology. 6 Education, practice, and experience are necessary to accurately and rationally undertake assessment. 7,8 Self-assessment ability improves and become more accurate with practice. Numerous scholars assert that dental curricula require both clinical relevance and a scientific basis to enhance social responsibility. 9 Scott opines that an ideal dental training program should produce dentists who are principled and who can perform general dentistry services in accordance with the reasonable expectations of the public that they work for. 10 Additionally, they must commit to professional and educational development over their lifelong careers. Manogue lists other desirable features of newly qualified dentists and determines that following qualification, trainees have to be both life-long researchers and competent practitioners. 11 Because dental education guides dental students through the transition into oral physicians, it should teach students to have a holistic vision and offer courses relevant to oral health, irrespective of any medical school affiliation. 12 Direct observation of procedural skills (DOPS) has been increasingly used in health education in recent years. 13, 14 In the oral surgery training course, DOPS was used to evaluate the preclinical and clinical skills of trainees. This study aimed to evaluate trainees' self-assessment of complicated tooth extraction skills and compared it with faculty assessment. In addition, to the best of our knowledge, this type of comparative investigation in postgraduate dental education has not been previously conducted in Taiwan. This study was conducted as a retrospective survey among postgraduate dental trainees learning complicated tooth extraction in Taipei and Linkou Chang Gung Memorial Hospital from 2012 to 2019. Every trainee received a 2month mandatory course in the first year in the department of oral and maxillofacial surgery. The practical skill assessment was performed using DOPS in accordance with the listed requirements of the 2-year postgraduate dental training program sponsored by the DOH in Taiwan since 2010. Each clinical case included a complicated extraction of a permanent tooth, with the involvement of advanced surgical procedures (e.g., flap reflection, removal of the covering bone, tooth sectioning, removal of the tooth fragment, and wound suturing). Two DOPS checklists were modified for use in selfassessment and clinical faculty evaluation (Fig. 1) . To set the qualification and standardization of grading outcomes, all of the 7 designated faculty members joined the cultivation camp held by the DOH for the clinical instructors of postgraduate dental training programs. All certified clinical instructors were required to meet the education and experience requirements established by the DOH. The faculty member to trainee ratio for supervision of complicated tooth extraction was 1:1. Trainees were also formally instructed on grading for each step of extraction procedures. Before the procedure started, trainees were asked to complete a form to assess their expected performance. Faculty members evaluated each item during each stage of tooth extraction. Following evaluations were recorded: 1) examination score (points awarded by the faculty) and 2) trainee self-assessment (self-assessed points). At the end of the 2-month training period in the department of oral and maxillofacial surgery in the first year, charting forms were collected, and these data were manually digitized into a spreadsheet. The relevance of all data was calculated and statistically analyzed using descriptive and inferential statistics. To analyze the differences between faculty assessments of trainees and trainees' own self-assessments, paired ttesting was conducted; for identifying significant differences between genders, independent-samples t-testing was employed. A P value of <0.05 was considered statistically significant. By using mean faculty assessments as a criterion for comparing trainees' mean self-assessments, self-assessment accuracy was determined. Next, the traineefaculty (T-F) gap was calculated as the difference between mean self-assessments and mean faculty assessments, with a T-F gap score of 0.0 indicating no difference between the trainee and faculty assessments and a positive gap score indicating more favorable trainee self-assessment compared with faculty assessment. However, because positive and negative gap scores may offset each other, leading the mean gap to approach 0.0, the absolute T-F gap scores were also calculated. When calculating absolute T-F gap scores, no positive or negative scores are considered; rather, only absolute differences, which can be considered the selfassessment accuracy, are reported. The scores were graded at three levels ( Fig. 1) : poor capability (score: 1e3), adequate capability (score: 4e6), and good capability (score: 7e9). The relevance between Technical assessment in postgraduate dental training 3 + MODEL each level of self-assessment and faculty assessment was also compared. A total of 69 participants (26 men and 43 women, average age Z 26.49 years, range Z 24e34 years) completed the 2month training course in the first year in the department of oral and maxillofacial surgery from 2012 to 2019. Approximately 62.32% of the participants were women. The female dentists among the survey cohort tended to be younger than the male dentists, with respective mean ages of 25.84 and 27.35 years. Mean scores and standard deviations for each group (trainees' self-assessments and faculty assessments) are presented in Table 1 . Within the survey cohort, the faculty assessments scored significantly higher than did the trainees' self-assessments in each procedure. For absolute T-F gap scores, overestimates and underestimates did not average out as they did for the T-F gap scores; thus, absolute T-F gap scores were higher than their corresponding T-F gap scores. Here, negative gap scores indicated lower trainee self-assessment compared with faculty assessment in the exercise. Table 2 presents the correlation between the aforementioned scores and both sexes of the trainees. The faculty assessments scored significantly higher than did the trainees' self-assessments in each procedure in both sexes. However, no significant difference was observed between both sexes in each procedure. Because good self-evaluated trainees were among the minority in all training items (<6%), trainees were divided into group 1 (poor self-evaluation) and group 2 (adequate or good self-evaluation) for comparison. More than 40% of the trainees evaluated themselves as having "poor capability" in some invasive surgical procedures (flap reflection, removal of the covering bone, and tooth sectioning) and also obtained lower average self-evaluation scores. This finding indicated that many trainees were unconfident in these procedures. However, the actual performance of the trainees with poor self-evaluation was not lower than that of the trainees with adequate or good self-evaluation. Although no statistical significance was observed in faculty assessments between the two groups, average scores were slightly higher in the poor self-evaluation trainee group. On the other hand, some steps in the procedure (e.g., gauze packing and postoperative wound management) had a lower percentage of the trainees evaluating themselves as having poor capability, indicating higher average self-evaluation scores (Fig. 2 ). Teaching is considered inseparable from learning. Teaching generally involves traditional lectures, which provide students with information while allowing few chances to interact. The optimal dental educational facility would empower students to obtain the required clinical, interpersonal, and theoretical competencies and provide them with an insight into the "clinical experiences" that they will likely encounter following graduation. Trainees' physical and psychological health and social and mental well-being must be maintained and enhanced, and their moral principles and cultural values should be fortified. Personal and occupational development should be provided in all higher education environments. Furthermore, extraordinary opportunities to volunteer, collaborate, and perform community service should be provided to develop desirable values. In addition, to avoid the personal and professional isolation that frequently affects dental practitioners, dental trainees should learn to collaborate and communicate effectively with health professionals as well as their colleagues. 2 Table 1 Descriptive statistical measures for variables and t-tests comparing mean differences between scores of trainee selfassessments and faculty assessments (N Z 69). The statistical significance of the difference between trainee self-assessments and faculty assessments in all 13 DOPS items is also presented. Differences between faculty score and self-assessment (T-F gap) are statistically significant in all items. Technical assessment in postgraduate dental training 5 The PGY dental residents in this study revealed they can still greatly improve their accuracy of self-assessment. According to a review article in 2016, 15 some studies by other scholars have reported similar conclusions. The reason that the trainees were prone to perform an incorrect evaluation warrants investigation. Low self-assessment may be related to self-confidence of the trainees. 16 Notably, overall, the trainees tended to provide lower self-assessment scores than faculty assessment in our present study. However, this result is just the opposite of the results of other studies. 15, 17 Experimental results were probably affected by cultural differences. However, because surgical extraction-related research remains rare, this potential effect of cultural differences on the current experiment design cannot be determined. Surgical extractions are one of the more advanced and aggressive dental treatments. Students relatively lacked relevant experience in dental school; 18e20 therefore, they were less confident about it. 18, 21, 22 Trainees' preclinical performance during their undergraduate years may be correlated with selfassessment accuracy because of many reasons. With practice, trainees' self-assessment accuracy may improve alongside their hand skills. 5, 15 At all stages of dental education, the confidence of students is crucial. Self-confidence strongly influences graduate education selection. 23 Because clinical exposure helps build confidence, 24e26 the disparate reports of confidence in the literature may reflect differences in experience working with patients. Self-assessment is often an external expression of self-confidence. 6e8, 15 Assisting the trainees understand their shortcomings and enhancing their self-confidence will indirectly enhance their self-assessment ability. Because 'feedback' is a major component of DOPS, the trainees can understand their shortcomings in surgical procedures immediately after teachers' timely feedback. This also helps improve self-confidence of the trainees. 13, 27 Our findings could help the dental trainees' scope for improving their self-assessment accuracy based on the statistical significance between the trainees' selfassessments and faculty assessments in each procedure. In addition, high performers tended to underestimate their expected performance much more than low performers. Personality traits may explain this finding. Highperformance students may perform well because they are highly critical and hold themselves to high standards, whereas low-performance students may be lackadaisical and not focus painstakingly on minutiae. Successful students have been discovered to assess themselves in comparison with their past performance or their potential and thus underestimate themselves. 5,28e30 Such results are similar to that obtained in the present study. The self-assessment ability correlated with clinical performance; as trainees perform with more confidence, their self-assessment accuracy improves. 5, 15, 17 Early operating experience and clinical exposure to some surgical steps help foster trainees' clinical confidence. 4, 17, 19, 25 In this study, some steps (e.g., gauze packing and postoperative wound management, etc.) had a lower percentage of the trainees evaluating themselves as having "poor capability", indicating higher average self-evaluation scores. These steps were usually considered non-invasive surgical procedures, and trainees mostly had some experiences of early clinical exposure during their clerkship and internship in dental school. On the other hand, more than 40% of the trainees evaluated themselves as having "poor capability" Figure 2 Trainees were divided into group 1 (poor self-evaluated) and group 2 (adequate or good self-evaluated) for comparison. The number in the grid represents the ratio of the total number of trainees in each group. Graphical representation of DOPS marks for assessment scores (mean value), showing that the actual performance of poor self-evaluated trainees was not lower than that of adequate or good self-evaluated trainees. No statistical significance in faculty assessments was observed between the two groups in each procedure. in some invasive surgical procedures, including flap reflection, removal of the covering bone and tooth sectioning. New graduates lack confidence in these surgical procedures. Further developments in the areas of clinical supervision and teaching in these surgical procedures were considered. Oral surgery has some of the most serious complications in dentistry; this strains faculty members and trainees involved. However, some dental programs struggle to provide students with an adequate number of surgical extraction cases. Such inadequacy can hinder the output of graduates confident in their dental extraction abilities. The application of the DOPS method appeared to help to improve clinical skills among the trainees, and it is recommended that dental education centers employ the DOPS method to assess trainees' clinical skills in conjunction with other methods to help promote their learning. 13, 14 Traditionally, phantom head mannequins or pig's heads was utilized for pre-clinical training. 24 With advances in technology, three-dimensional (3D) and virtual reality (VR) preclinical training can be the next-generation training methods. All these teaching aids will help dental trainees augment and enrich their learning experience, thereby developing self-confidence in clinical practice. 31 As the requirement for trainees' self-assessment becomes more common, trainees will become more skilled in assessing their performance. However, our challenge as dental educators is not to apply self-assessment as an endpoint but rather to encourage dental trainees to be selfdirected in the self-assessment process so that they can evaluate their knowledge against valid measures within their area of expertise. Other crucial concerns that must be emphasized in future studies are as follows: (1) the necessity of assessment and feedback in DOPS training programs; (2) the clinical relevance and importance of communication and empathetic abilities and dental training; and (3) requesting that hospitals allocate more equipment and human and financial resources to dental training. This study is one of the few studies to evaluate the competency of complicated tooth extraction using DOPS. A limitation of this study may be the low number of the trainees who participated in the dental PGY courses during 2012e2019. In addition, complete standardization is difficult because complicated tooth extraction is dependent on a number of unknowns, such as the cooperation and anxiety levels and status of the patient, experience of the trainee, local anatomical parameters, and labor factors. 18, 19, 24 Competency training must ensure trainees can perform any operation required in dentistepatient interaction as well as educate them holistically. In future studies, larger samples should be evaluated, and a prospective cohort design should be applied for reducing error and bias. We will continue to contribute to improve the 2-year postgraduate dental training program, and we believe that this project will drastically change the behavior of trainees and substantially improve the quality of dental care in Taiwan. In conclusion, within the survey cohort, these postgraduate dental trainees had room for improving their selfassessment accuracy in complicated tooth extractions. Furthermore, this survey highlighted that more than 40% of the trainees were unconfident with some invasive surgical procedures, including flap reflection, covering bone removal, and tooth sectioning. Self-assessment skills should be developed with more practice and experience. We hope these findings can guide the planning of faculty development programmes for clinical instructors, particularly the new cohort of faculty who will succeed the rapidly retiring boomer generation in the next 10 years. The authors have no conflicts of interest relevant to this article. The academic environment: the students' perspective Facing the challenges of competency-based assessment of postgraduate dental training: Longitudinal Evaluation of Performance (LEP) Dental students' ability to assess their performance in a preclinical restorative course: comparison of students' and faculty members' assessments Dental students' ability to evaluate themselves in fixed prosthodontics Flawed self-assessment: implications for health, education, and the workplace Evaluating clinical competence in the health professions Medical student selfassessment of performance on an obstetrics and gynecology clerkship Dental education at the crossroads: a report by the Institute of Medicine Dental education in Europe: the challenges of variety Managing the curriculum e for a change Why dentists should be called oral physicians now Influences on the implementation of Mini-CEX and DOPS for postgraduate medical trainees' learning: a grounded theory study Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review A systematic review of the use of self-assessment in preclinical and clinical dental education Dental student selfassessment of a medical history competency developed by oral and maxillofacial surgery faculty The relationship between dental students' assessment ability and preclinical and academic performance in operative dentistry Management of impacted wisdom teeth: teaching of undergraduate students in UK dental schools Oral surgery undergraduate teaching and experience in the United Kingdom: a national survey Assessing competency in dentoalveolar surgery: a 3-year study of cumulative experience in the undergraduate curriculum Undergraduate training as preparation for vocational training in England: a survey of vocational dental practitioners' and their trainers' views An assessment of the views held by recent graduates on their undergraduate course Perceived sources of stress among Greek dental students Student perceptions of exodontia competency assessment in a UK dental school A national survey of UK final year students'opinion of undergraduate oral surgery teaching Clinical competency exercises: some student perceptions The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory Dental students' selfassessment of preclinical examinations Self-assessment in medical practice Medical students' clinical self-assessments: comparisons with external measures of performance and the students' self-assessments of overall performance and effort Use of peer assessment in tooth extraction competency This study would not have been possible without the support of many people. The author would like to thank Dr. Yang-Ming Chang for their guidance and enthusiastic encouragement with respect for this work.