key: cord-0811127-dgpkiqgu
authors: Ferriss, J. Stuart; Frost, Anja S.; Heinzman, Alison Brooks; Tsai, Rita; Patterson, Danielle; Patzkowsky, Kristen; MLIS, Jaime Blanck; Bienstock, Jessica L.
title: Systematic Review of Intraoperative Assessment Tools in Minimally Invasive Gynecologic Surgery
date: 2020-10-18
journal: J Minim Invasive Gynecol
DOI: 10.1016/j.jmig.2020.10.007
sha: 3705b2bf0e69a6532d521dfaa97bd8de5e09a959
doc_id: 811127
cord_uid: dgpkiqgu

Objective : To collect, summarize, and evaluate the currently available intraoperative rating tools used in abdominal minimally invasive gynecologic surgery (MIGS). Data Sources : Medline, Embase, and Scopus databases from January 1, 2000 to May 12, 2020 Methods of Study Selection : A systematic search strategy was designed and executed. Published studies evaluating an assessment tool in abdominal MIGS cases were included. Studies focused on simulation, reviews, and abstracts without a published manuscript were excluded. Risk of bias and methodologic quality were assessed for each study. Tabulation : Disparate study methods prevented quantitative synthesis of the data. Integration and Results : 10 studies were included in the analysis. Tools were grouped into global (n=4) and procedure specific assessments (n=6). Most studies evaluated small numbers of surgeons and lacked a comparison group to evaluate the effectiveness of the tool. All studies demonstrated content validity, at least one dimension of reliability, and 2 have external validity. The intraoperative procedure specific tools have been more thoroughly evaluated compared to the global scales. Conclusions : Procedure specific intraoperative assessment tools for MIGS cases are more thoroughly evaluated compared to global tools; however, poor-quality studies and borderline reliability limit their use. Well designed, controlled studies evaluating the effectiveness of intraoperative assessment tools in MIGS are needed.

Studies focused on simulation, reviews, and abstracts without a published manuscript were 28 excluded. Risk of bias and methodologic quality were assessed for each study. 29

Tabulation Disparate study methods prevented quantitative synthesis of the data. 30

Integration and Results 10 studies were included in the analysis. Tools were grouped into 31 global (n=4) and procedure specific assessments (n=6). Most studies evaluated small numbers 32 of surgeons and lacked a comparison group to evaluate the effectiveness of the tool. All studies 33 demonstrated content validity, at least one dimension of reliability, and 2 have external 34 validity. The intraoperative procedure specific tools have been more thoroughly evaluated 35 compared to the global scales. 36

Conclusions Procedure specific intraoperative assessment tools for MIGS cases are more 37 thoroughly evaluated compared to global tools; however, poor-quality studies and borderline 38 reliability limit their use. Well designed, controlled studies evaluating the effectiveness of 39 intraoperative assessment tools in MIGS are needed. 40

A major focus of postgraduate education in gynecology is the attainment of specialized 42 skills in pelvic surgery. The assessment of this core mission varies among programs and is often 43 performed in a manner that is not standardized and subject to bias.(1) Residents and fellows 44

seek to achieve procedural mastery and rely on repetitive practice (case volume), good surgical 45 coaching, and feedback. With the effects of the global pandemic on health systems now 46 evident, some institutions face fluctuating case volumes and pressures to conserve personal 47 protective equipment -which can affect the number of learners allowed per case.(2, 3) More 48 than ever, surgical coaching and feedback are essential to allow learners to make each case 49 count. Faculty and learners need tools to help make the development of surgical skill more 50 efficient. 51

Objective evaluation tools in minimally invasive gynecologic surgery (MIGS) have been 52 described for a variety of settings and procedures. These assessment scales can provide 53 learners with useful, timely feedback which can be integrated rapidly into their practice. Most 54 tools have been developed for simulations, with fewer used intraoperatively.(1) Simulation is a 55 valuable and cost effective way of providing skill acquisition and assessment in a low-stakes 56 environment. However, a recent survey of obstetrics and gynecology residents noted that few 57 found simulation exercises to be valuable to their learning.(4) Perioperative assessment tools 58 may prove more useful as they function as a catalyst for a meaningful debriefing discussion 59 once the case is finished. ( 

Search results were loaded into Covidence for processing and duplicates were 78 removed. (7) The software was configured to present abstracts to all reviewers until each 79 abstract had been screened by 2 authors for inclusion or exclusion. Conflicting results were 80 resolved by JSF. Articles describing the use of an evaluation tool focused on a minimally invasive 81 abdominal gynecologic procedure in the intraoperative or perioperative setting were included. 82

Reviews, commentaries, abstracts without a manuscript, and articles focused on simulation or 83 minor procedures (e.g. hysteroscopy) were excluded from the review. Following abstract 84 screening, the full text of each remaining study was obtained for further review. Full text 85 screening was completed in a similar manner with each paper screened by 2 authors for 86 inclusion and any resulting conflicts resolved by JSF. 87

Data were collected from the included studies and recorded in an electronic 89 spreadsheet. Collected data included name of the first author, year of publication, whether the 90 tool was a global evaluation or procedure specific, the name of the evaluation tool, the type of 91 learner, the number of learners evaluated, and any reported measures of validity and reliability. 92

Intraclass correlation (ICC) of .80 was used as an acceptable threshold for reliability as values 93 above this number are unlikely to be significantly different.(8) Differences in opinion regarding 94 the extracted data were resolved by consensus. The quality of the data and the risk of bias for 95 each included study was evaluated with the Newcastle-Ottawa Quality Assessment Scale for 96 cohort studies.(9) Due to differences in study design and methodology, meta-analyses of the 97 data were not possible. However, data were grouped and summarized for tools with 98 overlapping characteristics. 99 100

The search returned 3016 unique citations and 2967 were rejected based on title and 102 abstract screening ( Figure 1 ). Full text was obtained for 51 citations including 2 additional 103 citations which were found through hand searching references of the included papers. 41 full 104 text papers were excluded for the reasons listed in Figure 1 . After review and consensus, 10 105 papers met our inclusion criteria and were included in the analysis ( Table 1) . Four of the studies 106 used a global assessment tool, and the remaining 6 used a tool evaluated during a specific 107 minimally invasive procedure: salpingectomy (2), supra-cervical hysterectomy, total 108 laparoscopic hysterectomy (2), and robotic hysterectomy. The risk of bias and quality 109 assessment using the Newcastle-Ottawa Scale found that all study designs had a risk of bias and 110 were poor quality given the lack of meaningful comparison groups. evaluated. The CAT-LSH score successfully discriminated between the three groups of surgeons 184 (construct validity), regardless of observer (intra-operative or blinded). However, the score of 185 the intra-operative observer was consistently higher compared to the blinded ones. Inter-rater 186 reliability was acceptable (ICC .85) for the two blinded raters, but below threshold for the intra-187 operative rater (ICC .75). Intra-rater reliability was not reported. 188 between the three groups of surgeons. Inter-rater reliability was evaluated for each element of 194 the tool and all fell below threshold (ICC range .28 -.75). Intra-rater reliability was not 195 reported. 196 In summary, 6 studies evaluating 5 different intraoperative assessment tools during 197 specific MIGS procedures were identified and evaluated. Most studies were small with a 198 median of 21 (range 14-52) participants. The study designs were noted to be susceptible to bias 199 and were judged to be of poor quality given the lack of comparison groups. The study by 200

Oestergaard is a notable exception, as this study included a comparison group despite the small 201 sample size. Additionally, this study applied the OSA-LS tool in a setting unrelated to prior work, 202 providing evidence of external validity. Currently, the OSA-LS intraoperative assessment tool 203 has the most data supporting its use among those evaluated. learners. This type of feedback is known to be essential to a successful surgical training 213 program.(20) To fulfill their promise, these tools need to be feasible (not time intensive or 214 complicated to use), have validity (content, construct, and external), and also have acceptable 215 levels of reliability for the same rater (intra-rater) and among different raters (inter-rater). (1) 216 Further, if they are going to be used in the educational setting, there should be evidence that 217 these tools are both effective as a teaching aid and also acceptable to learners. 218

Tools to assess surgical skills that are designed to be used intra-operatively or 219 immediately following a surgical procedure are part of a larger framework of surgical training. Adherence to best practices, such as requiring more than one reviewer for every abstract and 228 full text review, helps insure against user error. This review is also limited by the quality of 229 evidence from the included studies. Finally, the heterogenous methods used in these studies 230 prevented quantitative synthesis of the data. 231

Based on this review, the current menu of available intraoperative assessment tools is 232 small and the quality of evidence supporting their use is poor. The quality of the data is a 233 reflection of the early evolution of these tools, more so than any shortcomings of the included 234 studies. Each assessment tool must undergo a series of evaluations before it can be declared 235 both useful and effective. Therefore, additional work in larger populations is needed to further 236 characterize and refine the assessments discussed here. Most notably, well designed studies 237 with appropriate comparison groups are currently lacking. This work will be necessary before 238 these assessments can be used in high-stakes ( rating index of technical skills, LSI = laparoscopic skills index, OSA-LS = objective structured 361 assessment of laparoscopic salpingectomy, GERT = generic error rating tool, OSA-TLH = 362 objective structured assessment of total laparoscopic hysterectomy, CAT-LSH = competence 363 assessment tool for laparoscopic supracervical hysterectomy, Intra-op = intraoperative, RHAS = 364 robotic hysterectomy assessment score 365

Surgical skills assessment tools in gynecology

Peabody's Paradox: 249 Balancing Patient Care and Medical Education in a Pandemic

Adjusting to the new 252 reality: Evaluation of early practice pattern adaptations to the COVID-19 pandemic

Motivation to access laparoscopic skills training: 255 Results of a Canadian survey of obstetrics and gynecology residents

Immediate Surgical Skills Feedback 258 in the Operating Room Using "SurF

The PRISMA 261 statement for reporting systematic reviews and meta-analyses of studies that evaluate 262 healthcare interventions: explanation and elaboration

Quantifying test-retest reliability using the intraclass correlation coefficient and 266 the SEM

Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses 2012

Surgical Skills 271 Feedback and myTIPreport: Is There Construct Validity?

Interactive voice response to 274 assess residents' laparoscopic skills: an instrument validation study

Comparing self-assessment of laparoscopic technical skills with expert opinion 277 for gynecological surgeons in an operative setting

Reliability Study of the Laparoscopic Skills Index (LSI): a new 279 measure of gynaecologic laparoscopic surgical skills

Objective 281 assessment of surgical competence in gynaecological laparoscopy: development and validation 282 of a procedure-specific rating scale

Can both 284 residents and chief physicians assess surgical skills?

The Generic Error Rating 286 Tool: A Novel Approach to Assessment of Performance and Surgical Education in Gynecologic 287 Laparoscopy

Objective assessment of total 289 laparoscopic hysterectomy: Development and validation of a feasible rating scale for formative 290 and summative feedback

Surgical Competency for Robot-Assisted Hysterectomy: Development and Validation of a 296

Robotic Hysterectomy Assessment Score (RHAS)

Standardized training programmes for 298 advanced laparoscopic gynaecological surgery

Development of an objective 302 structured assessment of technical skills for obstetric and gynecology residents

Preventing Error in the Operating Room: 305 Five Teaching Strategies for High-Stakes Learning

A computer vision technique 307 for automated assessment of surgical performance using surgeons' console-feed videos

Crowdsourcing to Assess Surgical Skill