key: cord-342592-sf3iv8t4 authors: Loftus, Tyler J.; Filiberto, Amanda C.; Upchurch, Gilbert R.; Hall, David J.; Mira, Juan C.; Taylor, Janice; Shaw, Christiana M.; Tan, Sanda A.; Sarosi, George A. title: Performance Improvement With Implementation of a Surgical Skills Curriculum date: 2020-09-01 journal: J Surg Educ DOI: 10.1016/j.jsurg.2020.08.030 sha: doc_id: 342592 cord_uid: sf3iv8t4 OBJECTIVE: To assess the efficacy of an intern surgical skills curriculum involving a boot camp for core open and laparoscopic skills, self-guided practice with positive and negative incentives, and semiannual performance evaluations. DESIGN: Longitudinal cohort study. SETTING: Academic tertiary care center. PARTICIPANTS: Intervention group (n = 15): residents who completed the intern surgical skills curriculum and had performance evaluations in fall of intern year, spring of intern year, and fall of second year. Control group (n = 8): second-year residents who were 1 year ahead of the intervention group in the same residency program, did not participate in the curriculum, and had performance evaluations in fall of second year. RESULTS: In fall of second year of residency, the intervention group had better performance (presented as median values with interquartile ranges) than the control group on one-hand ties (left hand: 9.1 [6.3-10.1] vs 14.6 [13.5-15.4] seconds, p = 0.007; right hand: 8.7 [8.5-9.6] vs 11.5 [9.9-16.8] seconds, p = 0.039). The intervention group also had better performance on all open suturing skills, including mattress suturing (vertical: 33.4 [30.0-40.0] vs 55.8 [50.0-67.6] seconds, p = 0.001; horizontal: 28.7 [27.3-39.9] vs 52.7 [40.7-57.8] seconds, p = 0.003), and a water-filled glove clamp, divide, and ligate task (28.0 [25.0-31.0] vs 59.1 [53.0-93.0] seconds, p < 0.001). Finally, the intervention group had better performance on all laparoscopic skills, including peg transfer (66.0 [59.0-82.0] vs 95.2 [87.5-101.5] seconds, p = 0.018), circle cut (82.0 [69.0-124.0] seconds vs 191.8 [155.5-231.5] seconds, p = 0.002), and intracorporeal suturing (195.0 [117.0-200.0] seconds vs 359.5 [269.0-450.0] seconds, p = 0.002). CONCLUSIONS: Implementation of a comprehensive surgical skills curriculum was associated with improved performance on core open and laparoscopic skills. Further research is needed to understand and optimize motivational factors for deliberate practice and surgical skill acquisition. RESULTS: In fall of second year of residency, the intervention group had better performance (presented as median values with interquartile ranges) than the control group on one-hand ties (left hand: 9.1 [6.3-10.1] vs 14.6 [13.5-15.4 ] seconds, p = 0.007; right hand: 8.7 [8.5-9 .6] vs 11.5 [9.9-16.8] In the United States alone, more than 15 million inpatient surgeries and 48 million outpatient surgeries are performed annually. 1, 2 Major complications occur in more than 15% of all inpatient surgeries, increasing costs by as much as $11,000 per complication. 3, 4 Technical errors account for more than two-thirds of all potentially preventable major complications. 4 Technical skill is highly variable among surgeons; better skills are associated with better patient outcomes. [5] [6] [7] Therefore, skill acquisition and optimization is a vital aspect of surgical training. Since implementation of residency duty-hour restrictions and Medicare and Medicaid guidelines requiring greater degrees of attending supervision than required in previous eras, graduating residents perform fewer cases and have alarmingly low confidence in their operative abilities. [8] [9] [10] Lack of confidence among surgical trainees is worse in academic centers, where more than half of all surgery residents are trained. Residents in academic programs have unique opportunities to manage complex patients and participate in NIH-funded research, and they perform well on American Board of Surgery Examinations, but are less satisfied with their operative experience and less confident in their ability to operate independently compared with residents in community programs. [10] [11] [12] [13] Confidence allows individuals to have a positive perception of their abilities and has important implications for job satisfaction and performance. In addition, the COVID-19 pandemic substantially decreased elective operative volumes, supporting the premise that surgical training cannot depend exclusively on cases performed in the operating room. Given residents' low case volumes and low confidence in operative abilities, optimizing simulation-based surgical skills training may be beneficial. Prior studies have demonstrated that interns who participated in a surgical boot camp in medical school have better technical skills than interns who did not, as well as increased confidence in their technical skills. 14-16 Early mastery of core surgical skills has the potential to reduce technical errors and the cognitive load of performing manual dexterity tasks, allowing residents to focus on intraoperative decision-making. It seems prudent to encourage early mastery of core skills during intern year so that residents can earn increasing levels of trust and autonomy throughout the remainder of their training. Interactive instruction, deliberate practice, and benchmarking progress are important factors in technical skill acquisition. 17, 18 The authors incorporated these principles in a surgical skills curriculum in which interns participate in a surgical skills boot camp are incentivized toward selfguided deliberate practice, and compete in semiannual performance assessments. The purpose of this study was to assess the curriculum's efficacy with the hypothesis that interns in the intervention group would develop better surgical skills than second-year residents in the control group. This retrospective longitudinal cohort study included general surgery interns (n = 15, intervention group) who completed a comprehensive, longitudinal surgical skills curriculum, and a control group of second-year general surgery residents (n = 8) who were 1 year ahead of the intervention group in the same residency program and did not participate in the curriculum. The primary outcome was performance on 8 core surgical skills after completing approximately 17 months of residency. This study was performed at the University of Florida Health, an academic tertiary care center. To protect the identity of involved residents, specific date ranges are not provided. All data were collected within 3 years of publication date. The University of Florida Health Institutional Review Board approved this study (IRB #202000929). The surgical skills curriculum had 3 core components: (1) an intern surgical skills boot camp, (2) ongoing selfguided deliberate practice, and (3) competitive semiannual surgical skill performance assessments, as illustrated in Figure 1 . Each component of the curriculum focused on the following 8 core surgical skills: (1) twohand ties (left-and right-handed with free ties and tie on a pass), (2) one-hand ties (left-and right-handed), (3) vertical mattress suturing, (4) horizontal mattress suturing, (5) a water-filled glove clamp, divide, and suture ligate task, (6) laparoscopic peg transfer, (7) laparoscopic circle cut, and (8) laparoscopic intracorporeal suturing. These skills were chosen by consensus among the faculty and senior residents that developed the curriculum to (1) include open and laparoscopic skills that should be in the armamentarium of a junior resident, and (2) to promote efficacy and efficiency with both left-and righthanded techniques. Interns and second-year residents were provided with descriptions of important concepts and testing competencies for each skill, which are listed FIGURE 1. Conceptual design of the surgical skills curriculum. Interns completed a surgical skills boot camp during orientation activities immediately prior to beginning clinical work. Boot camp focused on 5 core open skills and 3 core laparoscopic skills. Interns were provided with all materials and instructions necessary to perform self-guided practice to achieve baseline and goal competencies composed of skill-specific quality standards (e.g., no broken sutures or air knots) and cutoff times. Baseline and goal competencies were associated with positive incentives for earning pager-free cases and negative incentives for matriculation. Twice per year, interns and second-year residents competed in Surgical Olympics with official judging, timing, and recording of performance metrics. in Supplemental Figure 1 . Briefly, the curriculum specified that attempts on knot tying tasks were disqualified for broken suture or air knots, attempts on the clamp, divide, and suture ligate task were disqualified for any leakage of water following release of the clamps, attempts on the laparoscopic peg transfer task were disqualified for dropping a peg inside or outside of the field of vision, and attempts on the laparoscopic circle cut task were disqualified for cuts that were 5 mm or more outside of the guideline. Interns completed a one-week technical skill boot camp during the final week of orientation activities, immediately prior to beginning clinical work. Boot camp activities were led by senior residents and attending surgeons. Interns were provided with all materials necessary to perform all 5 open core surgical skills. These materials are listed in Supplemental Figure 2 . Interns were invited to keep all materials so that they could use them for selfguided practice following boot camp. Laparoscopic skill stations were consistently available to residents at 2 inhospital locations throughout the study period, with 4 stations in a simulation center and 1 station in a call room. Boot camp sessions consisted of didactics, listed in Supplemental Figure 2 , as well as demonstrations of each skill by senior residents and attending surgeons, and opportunities for interns to practice skills under the supervision and instruction of senior residents and attending surgeons. In comparison, the control group had a single, brief open skills demonstration and handson practice session during intern orientation activities, and had no formal didactics or simulation training for laparoscopic skills. Interns were encouraged to practice core surgical skills and achieve baseline and goal competencies, as listed in Supplemental Figure 1 . Competencies had 2 components: (1) quality standards that were specific to each skill (e.g., broken suture, air knot, leak, peg drop, or cuts outside margins) and (2) cutoff times for task completion. Baseline and goal competency cutoff times had to be achieved on 2 separate attempts while meeting skill-specific quality standards. For laparoscopic skills, goal competency times were set to be similar to expert-derived performance times from the Fundamentals of Laparoscopic Surgery course; baseline competency times were set at twice the goal time. The authors are unaware of any standardized, expert-derived performance times for the open skills used in the intern skills curriculum. Therefore, for open skills, goal competency times were set several seconds slower than times achieved by the 7-year-old daughter of one of the authors; baseline competency times were set at twice the goal time. Although this approach was unconventional, the authors observed that competency times derived by this method were reasonably ambitious and potentially motivating for interns and second-year residents. For categorical interns, achievement of all baseline competencies was required to matriculate to the second year of residency. For each goal competency achieved, the intern was rewarded with a case as "primary surgeon" while his or her pager and duties were managed by a senior resident. Achievement of all goal competencies earned a pager-free case per week for the remainder of the academic year. For categorical second-year residents, achievement of all goal competencies was required to matriculate to third year of residency. Interns and secondyear residents could request testing sessions through the surgical education office at the earliest availability of a senior resident or attending surgeon, who would judge and time the session. A senior resident made a standing offer to have one-on-one practice sessions with any intern or second-year resident at their earliest availability. Twice per academic year, once in fall and once in spring, all interns and second-year surgical residents competed in Surgical Olympics. To standardize the experience for trainees, all 8 core technical skills were judged and timed by a senior resident or attending surgeon that helped lead boot camp activities including demonstrations of the skills, or had one-on-one training on the skill station that they were timing from a senior resident that led boot camp activities. Two attempts were allowed within a 7-minute time limit. Attempts that exceeded the time limit or did not meet quality specifications for that skill (e.g., broken suture, air knot, leak on glove cut, or drop on peg transfer) were disqualified and recorded as 3 times the goal competency time. For each skill, each competitor's fastest time that met quality specifications was compared with times by other competitors, using second attempts as tiebreakers. The fastest time was awarded 5 points, the second fastest time was awarded 3 points, and the third fastest time was awarded 1 point. Total points for each competitor were compared to determine gold, silver, and bronze medal winners. Medalists were recognized by the residency Program Director or Associate Program Director at Departmental Grand Rounds. Curriculum efficacy was assessed by comparing median times for completing core surgical skills between the intervention group (i.e., residents who completed the intern surgical skills curriculum) and control group (i.e., second-year residents who were 1 year ahead of the intervention group in the same residency program and did not participate in the curriculum, which began after they finished their intern year). The intervention group competed in Surgical Olympics at 3 time points: fall of intern year, spring of intern year, and fall of second year; the control group competed in Surgical Olympics once, in fall of second year. To facilitate direct comparisons between intervention and control groups that were independent of time spent in residency, the primary outcomes were median times for completing core surgical skills in fall of second year. Secondary outcomes were median times for completing core surgical skills within the intervention group in fall of intern year versus fall of second year. The curriculum was developed and implemented as an administrative chief resident initiative with oversight from the residency Program Director and administrative support from the Surgery Education Office. Faculty and senior resident judges participated enthusiastically without financial incentive; their efforts were recognized at Departmental Grand Rounds after Surgical Olympics awards ceremonies and in Department of Surgery Announcements made by the Chair of Surgery. One year after implementation of the surgical skills curriculum, the residency program expanded the administrative chief resident contingent from 1 to 2 residents, with the expectation that chief residents would sustain the curriculum moving forward, supported by faculty and senior resident volunteers that are passionate about fostering the development of highly skilled surgeons. Statistical analysis was performed in SPSS version 23 (IBM, Armonk, NY). Results are presented as median values with interquartile ranges. Median values were compared by the nonparametric Mann-Whitney test with significance set at a = 0.05. In the intervention group, 7 residents (6 categorical residents and 1 preliminary resident) from the original group of 15 interns entered second year of general surgery residency; the control group was composed of 8 residents (6 categorical residents and 2 preliminary residents) in their second year of general surgery residency. Complete data was available for each of these residents. All values in the dataset were included in all analyses. Results for two-hand ties are illustrated in Figure 2 .0] seconds). In both groups, median times to complete two-hand free ties were similar to or higher than those of the 7-year old whose times were used to establish goal competency cutoff times (left-hand free tie: 14.7 seconds, right-hand free tie: 14.4 seconds). Results for one-hand ties are illustrated in Figure 3 . The intervention group demonstrated significant improvements over time and outperformed the control group. For left-hand ties, the intervention group improved over a one-year period (from 14.1 [11.5-16 .5] to 9.1 [6.3-10.1] seconds, p = 0.021), achieving significantly faster times than the control group (14.6 [13.5-15.4 ] seconds, p = 0.007). For right-hand ties, the intervention group improved over a 1-year period (from 13.0 [10.0-14.0] to 8.7 [8.5-9.6 ] seconds, p = 0.004), achieving significantly faster times than the control group (11.5 [9.9-16.8 ] seconds, p = 0.039). Results for open suturing skills are illustrated in Results for laparoscopic skills are illustrated in Figure 5 . The intervention group demonstrated modest improvements over time and outperformed the control group. These findings demonstrate that the implementation of a comprehensive surgical skills curriculum was associated with improved performance on core open and laparoscopic skills. Residents that completed the curriculum demonstrated significant performance improvements for multiple skills over a 1-year period, as should be expected. The primary outcome of interest was surgical skill performance early in the second year of residency, compared between residents who completed the FIGURE 2. Two-hand ties. The intervention group demonstrated modest improvements over time and narrowly outperformed the control group, though none of these differences were statistically significant. Green regions represent goal competencies, determined by skill-specific quality standards (e.g., no broken sutures or air knots) and cutoff times; yellow regions are baseline competencies; red regions are below baseline competencies. Results are presented as median values, boxes representing interquartile range, and whiskers representing maximum and minimum values. PGY2, postgraduate year 2 residents. curriculum versus a control group of second-year residents who did not complete the curriculum, but trained in the same residency program. For one-hand ties, open suturing, and laparoscopic skills, residents who completed the curriculum outperformed residents who did not complete the curriculum. The intervention group demonstrated skills regression from spring of intern year to fall of second year of residency for two-hand ties and laparoscopic skills, but still outperformed the control group on laparoscopic skills. Collectively, results from this study suggest that implementation of a comprehensive surgical skills curriculum was associated with improved performance on core open and laparoscopic skills. It is difficult to compare findings from this study with previous findings from other studies because the design, implementation, and assessment of the surgical skills curriculum presented here is unique in several ways. There is no consensus definition of technical competence within surgical education literature, and so baseline competency and goal times were established using times achieved by the 7-year-old daughter of one of the investigators and by expert-derived performance times from the Fundamentals of Laparoscopic Surgery course. 19 Further, many studies of structured surgical skills curricula involve "surgical boot camp," which has been shown to increase the confidence and competence of medical students entering their surgical internships, but few previous studies make objective performance assessments, which hinders assessment of the efficacy of the curriculum itself. 20 There are even fewer studies that evaluate implementation of a structured curriculum with pre-and post-training assessments. 21, 22 Therefore, the present study may have value both in its novelty and in the design of the surgical skills curriculum itself, which appears to be effective. Longitudinal surgical skills curricula implemented early in surgical training have the potential to build confidence in technical skills, promote patient safety, and increase self-awareness in technical abilities. 23 Developing confidence in performing technical skills is an important aspect of surgical training, especially in an era in which many graduating residents have insufficient confidence in their ability to operate independently. 9, 10 Prior studies have reported that surgical skills curricula for medical students and residents are associated with trainees having greater confidence in their technical skills. [14] [15] [16] To ensure patient safety, it seems prudent to provide trainees with opportunities to improve their technical skills and develop confidence in these skills prior to performing operations on real patients in an operating room. Once trainees are serving as the first assistant or primary surgeon in the operating room, it is important that they have accurate perceptions of their technical abilities. Surgical skills curricula may improve trainee self-awareness regarding their technical abilities. Several studies have demonstrated that surgeons can accurately rate their own operative skills, with improvement in the accuracy of their ratings as their training and experience progresses. 24, 25 Building trainee confidence in technical skills, ensuring patient safety in the operating room, and increasing trainee self-awareness in their technical abilities all hinge on 1 essential component of surgical skills curricula: practice. Studies of human performance consistently demonstrate that practice is better than no practice, and that the quality of practice affects skill acquisition and performance. 17, 18 High-quality practice involves identifying a goal, maintaining full concentration and effort in practicing toward the goal, improving practice habits based on immediate and formative feedback, and repeating this process over time until the goal is achieved. 26 These principles are summarized in a quote that is credited by some to Vince Lombardi, former coach of the Green Bay Packers football team: "Practice does not make perfect. Only perfect practice makes perfect." This appears to be true in surgery. High-volume surgeons often have better outcomes than low-volume surgeons, and there is substantial variability in outcomes, even among high-volume surgeons. 5, 27, 28 Practice habits may account for variability in outcomes among surgeons. 29 Associations between surgical skill and patient outcomes support this hypothesis. 6 Therefore, encouraging high-quality practice should be a major objective of surgical training and surgical skills curricula. This study was limited by its single-institution design and small cohort sizes. Although the single-institution design limits cohort sizes and generalizability to other residency programs, it may also provide greater internal validity by ensuring that the control group had similar training-related operative experiences, which was important for this pilot study. Senior resident and attending surgeon judges were not assigned to the same station for each evaluation session, which may have caused inter-observer variability. In addition, judges were not blinded to trainee identity, introducing a possible risk for bias. There was also no way to control for the impact of operative experiences gained by each resident over time, which could influence improvements in technical skill performance over time. This study was not designed FIGURE 5 . Laparoscopic skills. The intervention group demonstrated modest improvements over time that were not statistically significant, and outperformed the control group. Green regions represent goal competencies, determined by skill-specific quality standards (e.g., no peg drops, cuts outside margins, broken sutures, or air knots) and cutoff times; yellow regions are baseline competencies; red regions are below baseline competencies. Results are presented as median values, boxes representing interquartile range, and whiskers representing maximum and minimum values. PGY2, postgraduate year 2 residents. to assess trainee confidence or motivational factors for improvement. A survey assessing participant factors such as demographics, confidence, training behaviors, and career plans may help to further elucidate factors that influence performance improvement among surgical residents. Implementation of a comprehensive surgical skills curriculum was associated with improved performance on core open and laparoscopic skills. This curriculum involved a boot camp for core open and laparoscopic skills, self-guided practice with positive and negative incentives, and semiannual performance evaluations; it remains unclear whether these elements have causal relationships with technical skill performance. Further research is needed to understand and optimize motivational factors for deliberate practice and surgical skill acquisition, and to generalize findings from this study to other training environments. Ambulatory surgery data from hospitals and ambulatory surgery centers: United States Profile of inpatient operating room procedures in US hospitals in 2007 Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program Complications in surgical patients Variations in morbidity after radical prostatectomy Surgical skill and complication rates after bariatric surgery Association of surgical skill assessment with clinical outcomes in cancer surgery The general surgery chief resident operative experience: 23 years of national ACGME case logs Our trainees' confidence: results from a national survey of 4136 US general surgery residents Graduating general surgery resident operative confidence: perspective from a national survey Military and academic programs outperform community programs on the American Board of Surgery Examinations Resident operative experience at independent academic medical centers-a comparison to the national cohort Effect of program type on the training experiences of 248 university, community, and US military-based general surgery residencies Surgical practical skills learning curriculum: implementation and interns' confidence perceptions Senior medical student "Boot Camp": can result in increased self-confidence before starting surgery internships The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies Deliberate practice and acquisition of expert performance: a general overview Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains Assessing technical competence in surgical trainees: a systematic review Medical school surgical boot camps: a systematic review Evaluation of a technical skills training program in surgical residents Use of collapsible box trainer as a module for resident education Trauma surgeon personality and job satisfaction: results from a national survey Resident selfassessment of operative performance Self-assessment of performance among surgical trainees during simulated procedures in a simulated operating theater Grit: The Power of Passion and Perseverance Hospital and surgeon procedure volume as predictors of outcome following rectal cancer resection Surgeon volume as an indicator of outcomes after carotid endarterectomy: an effect independent of specialty practice and hospital volume Deliberate practice enhances quality of laparoscopic surgical performance in a randomized controlled trial: from arrested development to expert performance SUPPLEMENTARY INFORMATION Supplementary material associated with this article can be found in the online version at doi:10.1016/j The authors thank Ashley J. Beavers Counts, MHA, CHES, and other members of the Surgery Education Office at the University of Florida Health Department of Surgery for supporting the surgical skills curriculum described in this study.