key: cord-0880795-zwa5ui2u authors: Aziz, Hassan; Khan, Sara; Rocque, Brittany; Javed, Muhammad Usman; Sullivan, Maura E.; Cooper, Jeffrey T. title: Selecting the Next Generation of Surgeons: General Surgery Program Directors and Coordinators Perspective on USMLE Changes and Holistic Approach date: 2021-07-31 journal: World J Surg DOI: 10.1007/s00268-021-06261-7 sha: 6bf0264f93b6e3b9f0b14ee443fcba9f7b2338e4 doc_id: 880795 cord_uid: zwa5ui2u INTRODUCTION: The United States Medical Licensing Examination (USMLE) was designed as a universal assessment tool for states to determine physician’s medical licensure's candidacy. Recent changes in the USMLE exam have changed the way future surgical residency candidate applications will be reviewed. The survey aimed to assess the effect of changes in USMLE exams—USMLE Step 1 pass/fail, complete dissolution of USMLE clinical skills exam, and the role of holistic review in future surgical residency candidacy selection. METHODS: An anonymous online survey was created and distributed to general surgery program directors and coordinators across the USA. The survey aimed to assess attitudes toward changes to USMLE exams and the potential changes with a holistic review of candidate applications. RESULTS: The response rate was 63.7%. Most program directors and coordinators disagree with changing USMLE Step 1 to a pass/fail scoring system. The majority felt that contacts, the medical school's name, and performance in clinical electives and sub-internships would hold more significance. They also believe that a holistic review of application will decrease socioeconomic discrepancies and promote a more diverse and inclusive resident cohort. CONCLUSION: Step 2 clinical knowledge (CK) will gain more importance in future residency matches because of the change in the scoring system of Step 1. The medical school's name, personal contacts, and clinical performance in rotations will hold more significance. On January 26, 2021, the Federation of State Medical Boards (FSMB), and NBME (National Board of Medical Examiners), co-sponsors of the United States Medical Licensing ExaminationÒ (USMLEÒ), announced that they are discontinuing the work to relaunch a modified Step 2 Clinical Skills examination (Step 2 CS) [1] . A year earlier, on February 12, 2020, the FSMB and the NBME had announced changes to the USMLE Step 1, which will now be a pass/fail exam rather than a score-based exam [2] . These changes will inevitably alter the metrics that are submitted to residency programs by prospective surgical trainees. For years, programs have used the USMLE Step 1 scores to filter applicants to offer an interview [3] . Aside from its intended use of determining medical license eligibility, the USMLE exams have evolved into a screening tool for residency program applications. Many surgical residency programs set target scores to select competitive candidates [4] . Surgical program directors (PDs) also use these scores to formulate rank lists and differentiate between applicants. Scores not only reflect an applicant's basic medical knowledge but have also been correlated with the academic performance during residency and serve as the best objective data available at the time of candidate review [5] . This magnitude change carries important ramifications for surgical residencies that rely on the value of Step 1 scores and Step 2 CS as a measure of medical knowledge and clinical skills. Secondly, there has been an emphasis on a holistic approach toward resident selection. The holistic approach emphasizes the importance of considering every applicant. It provides operational guidance for developing mission-driven, diversity-oriented processes; and encourages applying a balanced approach to assess the experiences, attributes, competencies, and academic metrics of each candidate [6, 7] . This study aimed to survey the program directors and program coordinators of all General Surgery residency programs regarding their attitudes toward the changes in USMLE exams and holistic approach for resident selection. An anonymous 24-question multiple-choice survey was developed using Qualtrics, an online survey platform [8] . All general surgery residency program directors and coordinators in the USA were invited to participate. We developed the survey questions after a comprehensive review of current literature and validated instruments. The survey queried their program location, training program type, and survey participants' years of experience associated with residency education and leadership. The questions were related to attitudes toward USMLE changes-Step 1 being a pass/fail test, cancellation of Step 2 CS, the significance of Step 2 CK, and Step 3 after these changes. We also assessed the importance of the medical school's name, personal contacts, electives, and clinical experience once the changes in USMLE are implemented. The questions also evaluated whether programs employ a holistic review of applications and their opinions on a holistic approach for resident candidacy selection. There was a free text option available for program directors or coordinators who had any concerns about the holistic approach. The questions were converted to an online format and beta tested to ensure brevity, clarity, relevance, and consistent interpretation. The survey was then refined, incorporating feedback from the beta-tested survey. The survey was preceded by a statement explaining that the survey was for research purposes and that data would be de-identified before analysis. A definition also preceded the survey for terms used in the survey to make the questions clearer. The holistic review was defined as mission-aligned admissions or selection processes that consider applicants' experiences, attributes, and academic metrics and the value an applicant would contribute to learning, practice, and teaching. ''Personal contacts'' were defined as mentors or academic acquaintances a candidate had built due to clinical work. Past ''non-academic struggles'' were defined as life adversities or stressful situations such as personal health issues, being a single parent, death of any family member, supporting a family financially at a younger age, lack of resources, financial restraints, lack of support systems, or previous health problems. We wanted to assess if program directors would consider such personal stressors in the applicants' lives and the mechanism by which they responded or recovered from those challenges. There were no incentives or disincentives to participate. This study was deemed exempt by the Institutional Review Board. A link to the survey was emailed to 310 general surgery program directors (PDs) and program coordinators (PCs). The emails of PDs and PCs were obtained from the Association of Program Directors in Surgery website (www.apds.org) [8, 9] . Responses were captured anonymously. The questions asked in the survey are reported in Appendix A1. Descriptive statistical analysis was performed to summarize the data. A response rate was calculated by the number of responses from a program (either by PD or PC) divided by the total number of programs we approached. The survey was sent in April 2021. We sent a total of 4 reminders regarding survey completion, which were sent one week apart. The primary purpose was to assess program directors' and coordinators' viewpoints regarding the recent USMLE exam changes and evaluate their opinions on resident applicants' holistic review. We did two sub-analyses: 1-to assess if there was any difference in opinions of the program coordinators compared to the program directors and 2-to assess if there is any difference in opinion between program directors with five or more years of experience compared to program directors with 1-4 years of experience. The data were reported as proportions. The Chi-square test was utilized to assess any difference in responses between the groups. A p value B 0.05 was considered statistically significant. Out of 310 general surgery programs emailed, 40 programs declined to participate, and 30 programs had older or invalid emails. A total of 185 responses were collected-119 from the program directors and 66 from the program coordinators. The overall response rate was 63.7% (185/ 290). Table 1 highlights the demographics of our respondents. 70.9% of the program directors did not agree with the change of USMLE Step 1 to a pass/fail exam. 86.2% of the program directors felt that USMLE Step 2 will gain more significance and will now be utilized instead of Step 1 for granting residency interviews and ranking candidates. 80.9% of the respondents stated that residency programs should make USMLE Step 2 CK scores a mandatory requirement to submit their application. We found mixed results regarding the discontinuation of Step 2 CS: 37.8% of the program directors were neutral about this change while 30.2% agreed with it. 40.2% of program directors felt that some clinical skill competency assessment is required. Regarding USMLE Step 3, 30.6% of the program directors felt that Step 3 would have a more critical role in the resident selection process, while 42.9% felt its role would remain unchanged. 74.9% of respondents stated that resident selection will now be more challenging because of the recent changes in the USMLE structure. International medical graduates 48.7% of the program directors stated that matching international medical graduates (IMGs) into general surgery programs will be more challenging with the USMLE exams' changes. However, 20.6% disagreed with this, while the remainder (30.7%) reported a neutral opinion. 42.5% of the program directors stated that the significance of personal statements would increase. 85.4% agreed that the reputation of the candidates' medical school would gain more importance. 85.4% of the responding program directors agreed that an applicant's clinical rotations and electives would hold more significance than before. Table 2 highlights the responses of program directors in the survey. Regarding a holistic approach for residency candidates, 45.2% of the program directors stated that their program is moving toward a holistic approach. In comparison, 15.1% of the programs will continue with their current practice for resident selection. Interestingly, 39.7% of the programs remain undecided about how they will be selecting residents due to the recent changes in USMLE exams. 43.3% of the responding PDs stated that they believed a holistic review would decrease socioeconomic disparities among applicants. 46.6% agreed that a holistic review of applicants would promote diversity and inclusion in their program. When asked about the significant anticipated outcomes from a holistic review, 33.6% of respondents felt a more fitting resident cohort would be recruited, 27.5% stated that it would lead to an increase in recruitment of people of color and ethnic minorities. In comparison, 20.4% said that this would allow them to find and recruit candidates with past, non-academic struggles. Only 18.3% of the respondents felt that applicants who have failed any component of USMLE have more success in the holistic review than the traditional match. Figure 1 highlights the free text comments of program directors and coordinators. Overall, the opinions of PDs and PCS were similar. However, some differences did exist in terms of the Step 3 will carry more importance than before compared to 16.1% (p 0.01). Table 3 highlights the comparison of opinions between the PDs and PCs. PDs with less than five years' experience vs. PDs with five or more years of experience Program directors with C 5 years of experience were more likely to disagree with the USMLE Step 1 change to passfail than PD with \ 5 years of experience (81.3% vs. 73.3%; p 0.01). PD \ 5 years of experience were more likely to agree that removal of Step 2 CS is a good idea (46.7% vs. 37.3%; p0.01), that the name of the medical school will matter even more than before (83.4% vs. Responses are recorded in percentages *Indicates a response greater than 50% World J Surg 67.5%; p0.01) than compared to the PDs C 5 years' experience. Table 4 highlights these outcomes. We found that most surgical program directors and coordinators disagreed with the transition of USMLE Step 1 to a pass/fail exam. Our survey results indicate that they anticipate a more difficult resident selection process, with more emphasis on Step 2 scores than before. The USMLE exams have evolved into a screening tool for residency programs across many specialties, and many surgical residency programs set target scores to select competitive interviewees [9] . Surgical program directors also use these scores to formulate rank lists and to differentiate between applicants [10] . Specifically, there has been a heavy reliance on the USMLE Step 1 score in the selection process. In 2019, InCUS (Invitational Conference on USMLE Scoring) suggested changing USMLE Step 1 to pass/fail to reduce the adverse impact of the current overemphasis on USMLE performance in residency screening [11] . The survey showed that many assistant/associate deans and course/clerkship directors support this change. In contrast, only 26% of the PDs and PCs agreed with changing USMLE Step 1 to a pass-fail exam. With USMLE Step 1 becoming a pass/fail exam, the significance of Step 2 CK scores will increase and play a more integral role in selecting surgical residency candidates. This is evident by our survey results, as most of the program coordinators and directors believe that USMLE Step 2 CK's significance will change and become more critical in the future. Other organizations and surveys have shown similar findings and concerns that these changes hinder an objective applicant evaluation [12] [13] [14] . In a recent survey of urology program directors, 84.6% felt that the USMLE Step 1 change would make it more challenging to compare applicants objectively and place a great emphasis on Step 2 CK scores [15] . In our survey, the respondents believe that personal contacts, medical schools' names, surgical clerkship performance, and additional clinical rotations would all play a more prominent role in the selection criteria. The schools that will be best prepared to handle the increased emphasis on Step 2 CK are many of the top medical schools that require students to take Step 1 and Step 2 CK after completing their core clinical rotations during their third year, rather than at the beginning of their second year. Our findings are similar to a recent survey of otolaryngology program directors, which showed that recent changes in USMLE exams are anticipated to increase the significance of core clerkship grades and elective rotations at respective institutions, Alpha Omega Alpha and other awards, and [16] . They also anticipate that this change will negatively impact international medical graduates, Osteopathic Medicine Doctors, and underrepresented students [16] . We found that 42% of surgery programs are moving from a traditional approach to a holistic approach for their upcoming residency selection process. An increasing emphasis is being placed on the diversification of residency training to reflect an expanding, diverse patient population [17, 18] . Incorporating a holistic review process, providing a broader assessment of applicants can play a pivotal role in increasing the proportion of women and underrepresented students involved in the general surgery recruitment process. Nehemia et al. have shown that after implementing a holistic review in 2017, a statistically significant increase in the proportion of women and underrepresented students ranked in their program compared to their prior ''traditional'' approach [19] . Our respondents believe that the pass/fail scoring change may encourage more students to apply to general surgery, which is traditionally considered to be ''competitive.'' Those concerned with obtaining prohibitively low scores may now build a more well-rounded application and focus on Step 2 CK, predominantly centered on clinical medicine instead of its basic science-focused counterpart, Step 1. However, they believe that because of a lack of World J Surg more objective data, applicants' names of medical school, personal statements, and contacts may play a more critical role in the match than before. By eliminating a core objective measurement of achievement such as Step 1, many students lose an opportunity to distinguish themselves from students graduating from more prestigious schools. Similarly, letters of recommendation are also prone to inequity as students may not have access to prominent letter writers, especially in the setting of unaffordable visiting rotations or have since been canceled. Our study found that additional clinical rotations would now play a more prominent role in the selection criteria. COVID-19 has not only severely affected our patients but also our trainees and learners [8, 20, 21] . With the current pandemic, many medical students nationally and internationally have been prevented from completing away rotations. Secondly, many students have not been able to participate in extracurricular activities and experiences during this time or build ''contacts'' all of which will gain more significance in a holistic approach. The effect of change in USMLE and lack of opportunities because of stay-at-home have a huge implications for future applicants [8] . Hence, we believe that the timeline of implementation of these changes should be reconsidered given the possible compounded disadvantages it may give to underrepresented students and international medical students due to the pandemic. We chose to survey the program coordinators in our study along with the program directors. We analyzed their opinion in comparison with the program directors and did not find any difference in opinion. We do understand that program directors make the final decision in resident selection. However, there is no denying that program coordinators play an essential role in residency programs and are well-versed in the thought process that governs a particular residency program. Similarly, we compared PDs who have been program directors for five years or more with those who had less than five years of experience. We found that the overall outcomes were similar. However, there were some points where PDs with more than five years were more decisive in their approval, such as the change of USMLE Step 1 to pass/ fail, while there were some topics where the PDs with less than five years of experience had a more robust approval, such as removal of Step 2 CS and the significance of the name of the medical school. The findings of our study should be interpreted in the context of a possible response bias. Nearly one-third of recipients did not respond to the survey request. PDs and PCs with stronger opinions on the current changes may have been more likely not to participate in the study. The initial setup of the questions may also have produced some of the results seen in the study. Future studies should focus on capturing objective general surgery match data after the implementation of USMLE to evaluate its ramifications better [13] . While our data are subjective, they represent the opinions of those directly involved with the residency selection process. As the evaluation of surgical candidates changes, this survey could provide great insight for program directors and coordinators as they embark on this journey with a great sense of insecurity and lack of norm. Another limitation of the study is that only electronic recruitment methods via e-mails were employed. Had other methods such as social media, postal, or telephone surveys been employed, maybe those non-respondents who did not have up-to-date electronic contact information could have been adequately recruited. Step 2 CK is anticipated to gain more importance in future general surgery residency matches because of the change in the scoring system of Step 1. The applicant's medical school reputation, personal contacts, and clinical performance in rotations will hold greater significance. Many PDs and PCs believe that a holistic review of applicants will increase the recruitment of students of color and minorities and candidates with non-academic struggles. United States Medical Licensing Examination, www.usmle.org/announcements A plea to reassess the role of United States medical licensing examination step 1 scores in residency selection Federation of State Medical Boards of the United States and the National Board of Medical Examiners. 2014USMLE bulletin of information United States Medical Licensing Examination Selection criteria for residency: results of a national program directors survey National Resident Matching Program, Data Release and Research Committee. Results of the 2012 NRMP Program Director Survey Does US medical licensing examination step 1 score really matter in surgical residency match outcomes (and should it) Effect of COVID-19 on surgical training across the United States: a national survey of general surgery residents Is USMLE Step 1 score a valid predictor of success in surgical residency? Associate program directors in surgery: a select group of surgical educators United States Medical Licensing Examination. Summary report and preliminary recommendations from the invitational conference on USMLE Scoring (InCUS) The USMLE step 1 pass/fail reporting proposal: the APDR position The change of USMLE step 1 to pass/fail: perspectives of the surgery program director Characterizing the effect of pass/fail US medical licensing examination step 1 scoring in neurosurgery: program directors' perspectives USMLE step 1 scoring changes and the urology residency application process: program directors' perspectives The effects of pass/fail USMLE step 1 scoring on the otolaryngology residency application process A renewed call for a more equitable and holistic review of residency applications in the era of COVID-19 Dermatology resident selection: Shifting toward holistic review? Looking beyond the numbers: increasing diversity and inclusion through holistic review in general surgery recruitment Review of COVID-19 outcomes in surgical patients Effects of coronavirus disease 2019 on solid organ transplantation Author contributions HA, SK, MUJ, and MS contributed to study conception and design, SK and HA contributed to acquisition of data, SK, HA, BR, and JC contributed to analysis and interpretation of data, HA, BR, MUJ, MS, and JC contributed to drafting of the manuscript, and BR, MS, and JC contributed to critical revision.