key: cord-0819068-fitvp6to authors: O'Connor, Paul; Lydon, Sinéad; Mongan, Orla; Byrne, Dara title: Longitudinal survey of self-reported level of entrustment across the first year of clinical practice date: 2022-01-20 journal: HRB Open Res DOI: 10.12688/hrbopenres.13487.1 sha: 21d317a1a574504bbb891fc69ee1b4ee3513de2b doc_id: 819068 cord_uid: fitvp6to Background. Newly graduated medical students often report that they lack the skills required to care for patients, and feel unprepared for clinical practice. However, little is known about when, and if, they acquire these skills in practice. The aim of this study was to assess self-reported level of entrustment in, and frequency of performance of, the seven Entrustable Professional Activities (EPAs) comprising the EPA framework for interns in Ireland. EPAs describe discrete activities that are essential to a particular profession. Methods. Self-report survey data were collected from doctors in the Republic of Ireland during their first year of clinical practice at four different time points during 2020/21. Results. Response rates to the survey varied from 73.3% (126/172) at Time 1 to 25.6% (44/172) at Time 4. After three months, the respondents reported that they could execute all of the EPAs, inclusive of 12 essential procedural skills, with indirect supervision. As the year progressed there was an increase in the proportion of respondents reporting that they performed the EPAs at least once a week. However, the proportion of respondents performing five of the essential procedural skills (e.g. nasogastric tube insertion) remained low across all time points. Conclusion. Consideration should be given as to how to better prepare medical students to execute these EPAs, how the interns can be better supported during the first quarter of internship. The findings from this research are positive. However, there is an urgent need to carry out formal assessments of entrustability, rather than relying on self-report. It has consistently been found that junior doctors lack the skills required to care for patients, and feel unprepared for clinical practice 1, 2 . The training and support received by junior doctors is variable, and dependent on the setting and clinical teams in which they are placed 3 . Moreover, junior doctors continue to the next stages of training, not based upon a determination of whether they have developed the necessary competencies, but rather as a result of the time they have spent in the role 4 . The issues associated with a time-based approach to training has led to an interest in a Competency Based Medical Education (CBME) for the training of junior doctors. CBME is concerned with outcomes of the education experience, independent of time spent in education 5 . A particular approach to CBME that has received interest within the context of the training of junior doctors is Entrustable Professional Activities (EPAs) 6 . EPAs are an effort to bridge the gap between the theory of competencies and practical clinical work 7 . EPAs are units of professional practice that capture essential competencies in which trainees must become proficient 8 . EPAs describe discrete activities that can be entrusted to trainees, are essential to the profession, and encapsulate one or more core competencies 9 . EPAs use observable work descriptors (e.g., clerk a patient) as opposed the person-descriptors (e.g., good communicator) 9 . EPAs provide supervisors with a framework to support the delegation of activities to trainees 8 . EPAs have potential for improving the efficiency of training of junior doctors, and ensure that they perform only the activities they have been deemed safe to perform 10 . In Ireland, a framework of seven EPAs (see Table 1 ) has been developed for the intern year (the first year of clinical practice after graduation in the Republic of Ireland). This EPA framework was developed through an iterative eight-stage consensus building with key stakeholders (see O'Dowd et al. 4 for a description of this process). Levels of entrustment of trainees for each EPA are made on a scale of 1 to 5: 1. Observation but no execution, even with direct supervision. 2. Execution with direct, proactive supervision. 3. Execution with reactive supervision, i.e., on request and quickly available. 4. Supervision at a distance and/or post hoc (regarded as the threshold for competent independent practice). 5. Supervision provided by the trainee to more junior colleagues 4 . The desirable minimum level of entrustability in these EPAs for interns is to be able to execute with reactive supervision (level 3), or supervision at a distance and/or post hoc (level 4). The rationale for the need to reach this level of entrustability is that interns tend to work autonomously with limited support from more senior doctors-particularly during night and weekend shifts 11,12 . Lack of supervision has been identified as a contributor to more than half of errors made by interns that 'played on their mind' 13 . This issue of limited supervision is further compounded by the fact that junior doctors are also often unwilling to seek guidance and clinical support from seniors 12,14 . Therefore, ideally, interns should be able to execute these EPAs with indirect supervision. The aim of this study was to assess self-reported level of entrustment within, and frequency of performance of the seven The doctor can request appropriate and interpret basic diagnostic laboratory and radiological investigations. EPA 3: Perform essential procedural skills. By the end of internship, the doctor demonstrates confident and is skilled in performing 12 essential clinical procedures: (1) hand hygiene; (2) venepuncture; (3) peripheral intravenous cannulation; (4) blood cultures from a peripheral vein; (5) arterial blood gas sampling; (6) electrocardiogram; (7) nasogastric tube insertion; (8) urinary catheter insertion; (9) preparation, reconstitution, dilution and administration of IV drugs; (10) blood sampling & blood cultures from central line and tunelled line; (11) sterile field set up; and (12) sterile glove application. EPA 4: Manage the work of in-patient care. The doctor can manage their daily workload to prioritise, delegate tasks, advance patient flow, and deliver patient centred care. EPA 5: Prescribe and monitor drugs and fluid. The doctor can prescribe sagely in compliance with legal requirements in both a hospital and community setting, and in an elective and emergency setting. EPA 6: Recognise and manage the deteriorating/ acutely unwell patient. The doctor can identify and respond to the acutely unwell patient appropriately. EPA 7: Handover and discharge a patient. The doctor can handover and receive the handover of a clinical case to/from colleagues and manage the discharge of a patient competently. Entrustable Professional Activities (EPAs) comprising the EPA framework for interns in Ireland at four time points in the intern year. This will allow an indication of when, and if, interns are able to execute these EPAs without the need for direct supervision. The study is reported in accordance with the Consensus-Based Checklist for Reporting of Survey Studies (CROSS) 15 . The intern year is the first year of postgraduate clinical practice for doctors in Ireland. Each intern is attached to one of six national intern training networks (ITN) and rotates through four clinical attachments, each of three-month duration, within the hospital group associated with their ITN. This study was carried out in one ITN. This ITN had one main 708-bed hub hospital and 8 smaller affiliated hospitals varying in size from 72 to 359 beds. Ethical approval was received from the Ethics Board Chairperson of Galway University Hospital (ref CA 2241). All participants provided written informed consent prior to completion of the survey. Two annonymous survey instruments were developed (see Underlying data 16 ). In survey A participants were asked to indicate their level of entrustability in each of the seven EPAs in the EPA framework for Irish interns 4 . For EPA 3 'perform essential procedural skills', participants were asked to indicate their level of entrustability with which they perform the 12 associated procedural skills (see Table 1 ). The participants provided an entrustability rating from 1 (observation but no execution, even with direct supervision) to 4 (supervision at a distance and/or post hoc). Level 5 was not included as this is level of supervision is not carried out by the interns. Participants were also asked to provide their sex, age, and whether they were a graduate of an Irish or non-Irish medical school. A self-generated identification code was also included (birth date, middle initial, number of older siblings) to track participation. Survey B is the same as Survey A except that the participants were also asked if they perform the EPA/clinical procedure: every shift, every two or three shifts, once a week, less than once a week, or not performed during this rotation. Interns from one ITN (n=172) were asked to complete Survey A immediately prior to starting internship in May 2020 (Time 1). Interns from the same ITN were asked to complete Survey B in August 2020 (Time 2), November 2020 (Time 3), and May 2021 (Time 4). A link to the online questionnaire was sent by email from the intern coordinator, with a reminder a week later, and a final reminder two weeks later. The participants were also given the opportunity to enter into a draw for an Apple© iPad mini at Times 3 and 4. The analysis was carried out using IBM SPSS version 22. The data sets of responses for each of the four times the surveys were distributed are available as in the Underlying data 16 . No adjustments were made for missing data. Participants Table 2 provides and overview of characteristics of the participants on each of the occasions that the survey was completed. A total of 0.6% of the data was missing. Table 3 shows the median level of self-reported entrustability for each of the six EPAs and the 12 procedural skills in EPA 3. Table 3 also shows the percentage of participants whose self-reported level of entrustability was level 3 (execution with reactive supervision, i.e., on request and quickly available), or level 4 (supervision at a distance and/or post hoc). This is an important distinction as this means the intern does not believe they require direct supervision in performing these skills. Table 4 shows the percentage of respondents who reported that they perform the six EPAs and the 12 clinical skills in EPA 3 at least once a week during the three time points at which they were working in the hospital. The Spearman's Rho correlation between self-reported entrustability and frequency of performance for all of the EPAs across all three time points (expect for EPA2) was 0.33 (p<.01). The Spearman's Rho correlation between self-reported entrustability and frequency of performance for the 12 clinical skills across all three time points was 0.42 (p<.01). These can be considered to be moderate correlations. Medical training has traditionally comprised a time-based apprenticeship model both in the Republic of Ireland as well as internationally. As such, doctors advance to the next stages of training based on time, and not competence. EPAs have been promoted as an approach to support competency-based assessments 4-6,9,10 and improve patient safety. The purpose of our study was to assess self-reported level of entrustment within, and frequency of performance of, the seven Entrustable Professional Activities (EPAs) comprising the EPA framework for interns in Ireland in order to identify when, and if, interns are able to, execute them without the need for direct supervision. Prior to starting internship, the respondents reported a need for direct supervision to carry out many of the EPAs, and the more advanced procedures in EPA 3. This finding is consistent with other studies that have found that high percentages of newly graduated medical students report feeling under-prepared to begin working in a hospital 11 , and variability in their confidence to perform specific clinical skills 1 . A national survey of junior doctors in Ireland found that only 51% believed that medical school prepared them well for their intern year 17 . However, even after only three months of clinical practice there was a large change in in the self-reported level of entrustability, with a much greater proportion of interns reporting that they could complete many of the EPAs and clinical skills with either reactive supervision, or supervision at a distance as compared to the baseline assessment. This is certainly a positive finding, and suggests that internship is effective in increasing the level of competence of junior doctors. However, there may be a cost to patients in terms of compromising safety, the efficiency of the health service (e.g. the need to repeat procedures 18 , ordering unnecessary tests 19 ) as well as to the interns themselves (e.g. stress or burnout 13 ) in terms of this largely on-the-job approach to skill development. These costs are worth considering, and may help to justify the resources required to establish a competency-based approach to education and training. There is also need to consider what level of entrustability is required for newly graduated medical students as this has implications for by the training and supervision required by interns. This is an issue that has been addressed by the UK General Medical Council for practical procedures and skills 20 . It is suggested that this should also be considered within the broader context of the EPAs. The interns reported carrying out the majority of the EPAs and clinical procedures at least once a week, with an increase in the frequency of performance during the year. It could be postulated that as interns complete the EPAs and clinical procedures more, they then become more confident in their ability-as indicated by the moderate correlation between self-reported entrustability and frequency of performance. However, there were five clinical procedures (nasogastric tube insertion; urinary catheter insertion; preparation, reconstitution, dilution and administration of IV drugs; blood sampling and blood cultures from central line and tunnelled line; and sterile field set-up) that, even by the third time point, the majority of the interns had not carried out in the past week. This information should be considered in terms of how many repetitions are required to become competent in a procedure. For example, it was found that it took junior doctors between 19 and 146 repetitions of peripheral venous cannulation procedures in the clinical environment to reach an acceptable level of performance 21 . Therefore, despite their confidence in their abilities, it may be that the intern are not receiving sufficient exposure to these clinical procedures in order to reach competency. Particularly for procedures for which the interns receive limited exposure, there is a need to give additional opportunities for them to practice and receive feedback on these skills. It is suggested that simulation-based education provides a mechanism for interns to become competent, in a safe learning environment. The Irish intern year EPAs provide a framework for medical schools to design teaching and assessment to better prepare newly graduated medical students for internship and clinical practice. A higher level of preparedness of medical students at the beginning of internship would have a number of benefits. The newly graduated students could be formally assessed to ensure they are competent to start internship, it would reduce the need for 'boot camps'and on-the-job learning of these EPAs. Given that most learning occurred during the first three months of training, it is suggested that extra support should be available for interns during this time period. A particular approach used in the ITN that was the subject of our research is a 'buddy' intern programme. In this programme, 'buddy' interns are recruited from a pool of intern volunteers who have just completed the intern year. The 'buddy' interns work alongside the new interns for the first month and provide support and mentoring, skills training, and work night shifts. There was strong support for this programme from the participating newly graduated doctors 22 . There is a need to assess entrustability, beyond self-report. It has been suggested that there is general tendency for doctors to be over confident in their ability to complete particular clinical tasks 23 . Therefore, we cannot draw conclusions about actual levels of entrustability as the interns' competence has not been formally assessed. Such assessment will allow the identification of those interns that are not able to perform these without direct supervision, and provide them with extra training required. Approaches to the assessment of EPAs that have particular potential include the use of simulation and portfolios 6,24,25 . However, there is a need for research on how such tools can be used-to include a consideration of the feasibility of these assessments being carried out by busy clinical supervisors 10 . The main limitation of the research reported in this assignment is that the data is based on self-report. As discussed above, there may be a tendency for the interns to be over-confident in the ability to execute the EPAs. So, these findings should be considered with that in mind. There are also a number of other limitations that should be acknowledged. The research was carried out during the COVID-19 pandemic. Therefore, the findings may not be the same as if the data was collected during a more typical intern training year. The response rate although high at baseline, dropped considerable on each occasion that the survey was distributed-corresponding with a time in which there was a COVID-19 surge in Ireland. Finally, only respondents from one Irish ITN were surveyed. After three months of clinical practice, almost all of the interns believed they could execute all of the EPAs in the Irish intern EPA framework without the need for direct supervision. This is certainly a positive finding. However, consideration should be given as to how to better prepare medical students to execute these EPAs as an integral part of their undergraduate training, how the interns can be better supported during the first quarter of internship, and there is an urgent need to carry out formal assessments of entrustability, rather than relying on self-report. Adopting this type of competency based approach will benefit the interns (as they will know they have the competencies required to perform the job), their supervisors (who know they can trust the interns), and most importantly their patients-the main beneficiaries of well-trained interns. Underlying data Zenodo: An evaluation of self-reported level of entrustment across the first year of clinical practice. https://doi.org/10.5281/ zenodo.5825638 16 . This project contains the following underlying data: If applicable, is the statistical analysis and its interpretation appropriate? Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Partly Preparing new doctors for clinical practice: an evaluation of pre-internship training Prepared for practice? Interns' experiences of undergraduate clinical skills training in Ireland A mixed-methods examination of the nature and frequency of medical error among junior doctors A comparison of the teamwork attitudes and knowledge of Irish surgeons and U.S Naval aviators A consensus-based Checklist for Reporting of Survey Studies (CROSS) An evaluation of self-reported level of entrustment across the first year of clinical practice You Training Counts: Trainee experiences of Clinical Learning Environments in Ireland An evaluation of the use of deliberate practice and simulation to train interns in requesting blood products Targeted ordering of investigations reduces costs of treatment for surgical inpatients The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method Creating a supportive buddy system for newly appointed interns-the'nanny'intern programme Are we right when we're certain? overconfidence in medicine What we measure … and what we should measure in medical education Developing entrustable professional activities as the basis for assessment of competence in an internal medicine residency: a feasibility study PubMed Abstract | Publisher Full Text Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Partly Are sufficient details of methods and analysis provided to allow replication by others? Yes The authors analysed self-reported levels of entrustment in interns in their first year of clinical practice in one national intern training network in Ireland. Interns were asked to self-assess their performance in the seven Entrustable Professional Activities (EPAs) in the EPA framework for interns in Ireland (clerk a patient, request and interpret basic investigations, perform essential procedural skills, manage the work of in-patient care, Prescribe and monitor drugs and fluid, recognize and manage the deteriorating/acutely unwell patient and handover and discharge a patient.) They found that a high percentage of participants feel confident to execute the Irish first year EPAs with either reactive supervision or supervision at a distance after only 3 months of training. The authors conclude that internship is effective in increasing the level of competence of junior doctors. This is an important finding, however, the database are self-reported and may be prone to bias 1 . It may be important to confirm the results of these self-assessments by assessments of supervisors. No competing interests were disclosed. Reviewer Report 08 February 2022 https://doi.org/10.21956/hrbopenres.14708.r31303 © 2022 McVeigh T. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Cancer Genetics Unit, Royal Marsden NHS Foundation Trust, London, UKThe authors have outlined a study in which interns in a single Irish intern training network were surveyed at 4 time points over the course of their intern year and asked to self-assess their entrustability in each of the 7 Entrustable Professional Activities (EPAs) outlined in the EPA framework for Irish interns -as well as providing specific entrustability ratings for 12 procedural skills under EPA3. Participants were also asked to comment on the frequency at which they performed these skills in clinical practice.The authors demonstrate a moderate correlation between self-assessed entrustability and the frequency at which the EPAs were performed.Somewhat surprisingly, some of the core tasks (urinary catheterisation, were performed weekly by only a minority of interns, even by the last time point, IV drug administration, NG insertion, etc), and yet median self-reported entrustability for all such tasks was 4. As the authors point out, confidence does not necessarily correspond to competence, and those respondents rating their skills highly without sufficient practice may be unaware of their developmental needs in this area.The attrition of respondents across the four time points is also a significant limitation of the study, with just over a quarter of potential participants responding to the final survey -despite the best efforts of the authors to increase participation with the addition of a prize draw for later surveys. This attrition limits the strength of conclusions that can be drawn from this study -it is possible that those participants who continued to respond are biased towards the more confident.The survey also collected some basic demographic data including age, gender, and whether training was in Ireland or in another country, but no analyses including these data were undertaken. Response rates in the group <24 reduced over time, while those in the group 25-34 increased -I wonder if this reflects "aging up" of a number of respondents into the next category rather than discordant trends across age groups.An important finding from this study is the low rate of self-reported entrustability in core skills at the end of undergraduate medical training, in line with previous work showing that interns feel underprepared for clinical practice after medical training. The authors have suggested useful strategies to address this and support these medics in the transition from student to doctor.Ultimately, entrustment judgements should be informed by a number of pieces of evidence. It would be interesting to determine if the self-assessment of the participants corresponded with the judgements of their supervisors or peers. Are all the source data underlying the results available to ensure full reproducibility? Yes Competing Interests: I worked with the authors a number of years ago, but have not been involved in any of their work/research related to Entrustable Professional Activities, this has not affected my ability to provide an impartial review for this particular article. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.