key: cord-1042380-8mj2lzhm
authors: Pontell, Matthew E.; Makhoul, Alan T.; Kumar, Nishant Ganesh; Drolet, Brian C.
title: The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director
date: 2020-07-10
journal: J Surg Educ
DOI: 10.1016/j.jsurg.2020.06.034
sha: 570f5bfc8af07f6626dc1384134ad1ef70904b11
doc_id: 1042380
cord_uid: 8mj2lzhm

OBJECTIVE: This study sought to evaluate the perspectives of surgical program directors regarding the change of USMLE Step 1 to pass/fail grading. DESIGN: Validated electronic survey. SETTING: Vanderbilt University Medical Center Department of Plastic Surgery. PARTICIPANTS: Program directors of all ACMGE-accredited General Surgery, Integrated Vascular Surgery, Integrated Thoracic Surgery, and Integrated Plastic Surgery residency programs. RESULTS: The overall response rate was 55.5%. Most PDs (78.1%) disagreed with the scoring change. Only 19.6% believe this change will improve medical student well-being. For 63.5% of PDs, medical school pedigree will become more important, and 52.7% believe it will place international medical graduates at a disadvantage. Only 6.2% believe Step 2 CK should also be pass/fail, while 88.7% will increase the weight of Step 2 CK and 88.4% will now require Step 2 CK score submission with the electronic residency application service. CONCLUSIONS: While well-intentioned, changing USMLE Step 1 to pass/fail may have unintended consequences and may disadvantage certain groups of applicants. The emphasis on Step 1, and resulting test-taking apprehension, will likely shift to Step 2 CK. Proponents of equitable evaluation should direct their efforts toward increasing, not decreasing, the number of objective measures available for student assessment.

The 3-part US Medical Licensing Examination (USMLE) was designed as a universal assessment tool for states to determine candidacy for medical licensure. [1] [2] [3] Although licensing outcomes are binary, the Step 1, Step 2 CK, and Step 3 exams all report 3-digit scores. 4 The Step 1 exam is taken during medical school, before residency applications are submitted, and is used to assess understanding of the "scientific principles required for maintenance of competence through lifelong learning." 3 Teams of scientists, educators, and clinicians ensure the validity of Step 1 questions, 2 and the accompanying scores are critical to the surgical residency application process. [5] [6] [7] [8] [9] Aside from its intended use, the Step 1 examination has evolved into a screening tool for residency programs across many specialties, 1,10-14 and many surgical residency programs set target scores to select competitive interviewees. [5] [6] [7] [8] [9] Surgical program directors (PDs) also use these scores to formulate rank lists 13, 14 and to differentiate between applicants. 13, 15, 16 Scores not only reflect an applicant's basic medical knowledge, but have also been correlated with academic performance during residency across surgical and nonsurgical fields. 5, [17] [18] [19] [20] [21] [22] [23] [24] As many US medical schools have transitioned to pass/ fail preclinical grading systems, the numeric scoring of the USMLE Step 1 has been challenged. 25 In addition, the test is known to cause substantial anxiety and has been implicated in fostering racial and demographic disparities. 14, 26 To address these concerns, the Invitational Conference on USMLE Scoring (InCUS) was held in March of 2019. Intent on facilitating "broader, systemwide changes to improve the transition from undergraduate to graduate medical education." One year later, in February 2020, the Federation of State Medical Boards and National Board of Medical Examiners made the decision to change USMLE Step 1 scoring to pass or fail. 27, 28 A change of this magnitude carries important ramifications for surgical residencies that rely on the value of Step 1 scores. A recent national survey of all residency PDs demonstrated that the majority did not believe the change to pass/fail grading was a good idea. 29 However, the viewpoints of surgical PDs have not been presented. The aim of this study was to survey PDs of all General Surgery and General Surgery-affiliated integrated residency programs regarding their attitudes toward the change in USMLE Step 1 scoring.

After receiving exemption from the Vanderbilt Medical Center University Institutional Review Board (IRB# 200355) an electronic survey was developed in the Research Electronic Data Capture (REDCap) program (Nashville, TN). The survey was validated through prepilot and pilot testing prior to distribution. Internal validity was assessed (Cronbach's alpha = 0.87).

The American Council for Graduate Medical Education (ACGME) listing of accredited residency programs was queried for all active General Surgery residency programs, as well as all Integrated Thoracic, Vascular and Plastic Surgery residencies. This list yielded a total of 506 programs (28 Integrated Thoracic, 63 Integrated Vascular, 82 Integrated Plastic, 333 General Surgery). Contact information was unavailable for five General Surgery PDs and one Integrated Plastic Surgery PD. One PD declined to participate and 2 General Surgery PDs each directed 2 separate residency programs. In total, 497 unique PDs were surveyed (28 Integrated Thoracic, 62 Integrated Vascular, 81 Integrated Plastic, and 326 General Surgery).

The 19-item survey consisted of 8 questions regarding the change of Step 1 scoring, 4 questions regarding changes made as a result, 5 demographic questions, and 1 free response section. Four rounds of survey distribution proceeded over the course of 8 weeks and responses were recorded anonymously. Responses were analyzed with descriptive statistics and all values are listed as discrete numbers, means § standard deviation (SD) or percentages with 95% or 99.9% confidence intervals (CI). Statistical significance in response plurality was determined by nonoverlapping confidence intervals.

Of the 497 unique PDs surveyed, 276 responded (response rate 55.5%). Individual response rates were as follows: General Surgery 54.3% (177/326), Integrated Plastic Surgery 65.4% (53/81), Integrated Vascular Surgery 56.5% (35/62), and Integrated Thoracic Surgery 39.3% (11/28) . Overall, the mean PD age was 50.7 § 7.9 years and respondents were mostly male (78% male, 21% female, 1% nonbinary). Average PD tenure was 6.6 § 5.26 years. Programs were located in the Northeast (27%), South (33%), Midwest (25%), and West (15%). Overall, 78.1% of PDs did not believe changing USMLE Step 1 to pass/fail was a good idea. Eighty-eight percent believe that this change will make it more difficult to objectively compare residency applicants, and 85.4% believe it will make applicant screening more arduous. The majority (52.7%) believe this change will place International Medical Graduates (IMGs) at a disadvantage. Most (53.6%) do not believe the change will decrease socioeconomic disparities in the application process. PDs were largely neutral regarding its effect on basic science knowledge. Only 19.6% believe this change will improve medical student well-being. While most PDs are neutral regarding the addition of a supplementary application, 88.4% will now require applicants to submit Step 2 CK scores with the electronic residency application service (ERAS). This contrasts with only 5% (25/497) of PDs who already do this. The majority of respondents (88.7%) will increase the weight of USMLE Step 2 CK in the application process and only 6.2% believe Step 2 CK should also be pass/fail. Finally, 63.5% of PDs agree that the importance of which medical school an applicant attends will become more important during the screening process (Table 1) .

Responses were compared among PD age, gender, tenure, program region and program specialty ( Table 2 , Appendices A-D). Responses did not differ between demographic groups. The free response rate was 32% (88/276) and responses focused on disapproval of the change, citing Step 1 as a valuable piece of objective data. Many respondents were concerned with the shift of testing burden onto Step 2 CK and the negative impact on IMGs, Osteopathic applicants, those from less well-known medical schools and those from lower socioeconomic backgrounds.

Over 90% of US medical graduates pass the USMLE Step 1 on the first attempt. 3, 25 As medical schools transition to pass/fail preclinical grading systems, 3,25 students have fewer objective measures by which to differentiate themselves from their peers. The Step 1 scoring change was proposed to mitigate USMLE-based applicant screening and encourage a more balanced assessment of interviewees. 27 In 2019, the InCUS conducted a large survey to obtain feedback on their 3 core recommendations. 27 Contrary to most assistant/associated deans and course/ clerkship directors, only 26% of residency PDs agreed that a scoring change would reduce USMLE-mediated residency screening and selection practices. 30 Our study suggests that surgical PDs have concerns regarding the impact of this change. Most (78.1%) PDs disagree with this change and 88.3% believe it will make it more difficult to objectively compare applicants.

In the field of General Surgery, USMLE Step 1 scores not only aid in applicant selection and ranking, 31 but have also been correlated with resident academic performance. Some studies have portrayed Step 1 scores as predictors of resident matriculation, clinical and academic performance during General Surgery residency, 4 and even manual dexterity. 5, 32 The test is also associated with performance on the American Board of Surgery In-Service Training Examination (ABSITE) more so than Step 2 CK or Step 3-as well as performance on the American Board of Surgery Qualifying Examination (QE). 5, 15, [33] [34] [35] [36] [37] [38] [39] As long as QE performance is considered by the ACGME during program accreditation, the correlation with USMLE performance will be considered by PDs. Integrated Plastic Surgery, 7 Vascular Surgery, 6 and Thoracic Surgery 8,9 residency programs all report using Step 1 scores to assist in applicant selection as well, and a recent study demonstrated that 81% of Plastic Surgery PDs disagree with the scoring change. 40 Interestingly,

Step scores have even been associated with physician professionalism ratings and risk for state board disciplinary action. 41 Although the 3-digit score has proven useful in many ways, it has also been criticized as a catalyst for medical student burnout and depression. 2 While the pass/fail change was aimed at mitigating the psychological consequences of high-pressure testing, only 19.6% of surgery PDs believe the change will improve student wellness. Most expect

Step 2 CK will ultimately replace Step 1 with regards to applicant screening, simply shifting the test-taking anxiety to a different exam. Furthermore, only 6.2% felt that Step 2 CK scores should also be changed to pass/fail, further emphasizing the importance of objective measures during consideration for surgical training positions. 

In a positive light, the pass/fail scoring change may encourage more students to apply for traditionally "competitive" training positions. Those concerned with obtaining prohibitively low scores may now have the opportunity to build a more well-rounded application and focus on the Step 2 CK, which is predominantly centered on clinical medicine as opposed to its basic science counterpart, Step 1.

Surgery PDs expressed concerns that pass/fail exam scoring may worsen socioeconomic status (SES) disparities in the application process. In the United States, SES and race are intimately linked, 42 and those from lower SES backgrounds may opt to attend a less prestigious medical school for financial reasons. Without objective measures of achievement like Step 1 scores, these students lose an opportunity to distinguish themselves from students graduating from more prestigious schools. The results of this study support that concern, as 63.5% of respondents agree that medical school pedigree will become more important. This is particularly important for IMGs as foreign medical schools are uniformly less well-known than US schools, and IMGs rely on Step scores to distinguish themselves from their US counterparts. Most (52.7%) surgery PDs reported that the Step 1 scoring change will place IMGs at further disadvantage. With the decline of measures for objective assessment, the importance of visiting rotations-an out-of-pocket applicant expense 43,44 -will likely increase. Other subjective measures may also become more important, such as the Medical Student Performance Evaluation and letters of recommendation. Letters of recommendation are also prone to inequity as students may not have access to prominent letter writers, especially in the setting of visiting rotations that were unaffordable or have since been cancelled due to COVID-19. Indeed, only 13.4% of surgery PDs believe the scoring change will decrease socioeconomic disparities in the application process. Taken together, these findings raise concern that the change may exacerbate disadvantages that applicants from lower socioeconomic backgrounds already face.

Limitations of this study include the use of psychometric data. This study reports PD opinions that are constitutionally biased (i.e., perceptions). Future studies should focus on capturing objective data after the change of USMLE Step 1 scoring to pass/fail. While our data is subjective, it represents the opinions of those personally involved in the residency selection process. We hope this study helps both PDs and applicants understand the potential implications of this change in medical education. The Coalition for Physician Accountability, whose goal is to assist in the transition from undergraduate to graduate medical education, will play a critical role in managing the effects of the scoring change along with other hardships that occur during this transition, all of which will be augmented by the ongoing viral pandemic. 45 

The USMLE and surgical residency application processes are far from perfect. A student's cognitive abilities cannot be represented by a single number, and yet, a scarcity of objective data has likely contributed to overemphasis of Step 1 scores. With the change to pass/ fail scoring, the emphasis on Step 1, and associated testtaking apprehension, will likely shift to Step 2 CK. This change begins the long-awaited evolution of the transition from undergraduate to graduate medical education; however, it is just the commencement and further refinements are needed. Proponents of equitable applicant evaluation should direct their efforts towards increasing, not decreasing, the number of objective measures available for comprehensive assessment. 

National Resident Matching Program Data Release and Research Committee

A plea to reassess the role of United States medical licensing examination step 1 scores in residency selection

Federation of State Medical Boards of the United States and the National Board of Medical Examiners. 2014USMLE bulletin of information

United States Medical Licensing Examination

The predictive value of general surgery application data for future resident performance

Integrated vascular surgery resident selection: a survey of integrated vascular surgery residency program directors

Resident selection protocols in plastic surgery: a national survey of plastic surgery program directors

Integrated thoracic surgery residency: current status and future evolution

Integrated cardiothoracic surgery: developing a successful residency application

Selection criteria for residency: Results of a national program directors survey

National Resident Matching Program, Data Release and Research Committee

Does US medical licensing examination step 1 score really matter in surgical residency match outcomes (and should it)

Is there declining interest in general surgery training?

Racial bias in using USMLE Step 1 scores to grant internal medicine residency interviews

Predictive value of general surgery application data for future resident performance

Orthopaedic resident-selection criteria

Relationship between performance on part I of the American Board of Orthopaedic surgery certifying examination and scores on USMLE steps 1 and 2

USMLE step 1 and 2 scores correlate with family medicine clinical and examination scores

USMLE step 1 scores as a significant predictor of future board passage in pediatrics

Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents

Do U.S. medical licensure examination step 1 scores correlate with council on resident education in obstetrics and gynecology in-training examination scores and American Board of obstetrics and gynecology written examination performance

Do scores of the USMLE step 1 and OITE correlate with the ABOS part I certifying examination?: a multicenter study

The utility of pre-residency standardized tests for anesthesiology resident selection: the place of United States Medical Licensing Examination scores

Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated internal medicine residency training program

Core entrustable professional activities for entering residency

US graduate medical education

United States Medical Licensing Examination. Summary report and preliminary recommendations from the invitational conference on USMLE Scoring (InCUS)

The change to pass/fail scoring for Step 1 in the context of COVID-19: implications for the transition to residency process

Epublished ahead of print

Objective measures needed-program directors' perspectives on a pass/fail USMLE Step 1

Score Reporting Preliminary Recommendations: Summary of InCUS Public comment

Importance of academic achievements and candidate performance during interview in the ranking process

Correlation of manual dexterity with USMLE scores and medical student class rank

Association between licensing examination scores and resource use and quality of care in primary care practice

Association between licensure examination scores and practice in primary care

Setting a fair performance standard for physicians' quality of patient care

Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries

The relationship between licensing examination performance and the outcomes of care by international medical graduates

BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice

Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study

Pass/fail USMLE step 1 scoring: a plastic surgery program director survey

Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards

Understanding associations between race, socioeconomic status and health: patterns and prospects

Away rotations and matching in integrated plastic surgery residency: Applicant and program director perspectives

Away Rotations in Plastic and Reconstructive Surgery: A Survey of Program Directors

Accreditation Council for Graduate Medical Education. Final Report and Recommendations Submitted by The Coalition for Physician 2 Accountability's Work Group on Learner Transitions from Medical Schools to Residency 3 Programs in 2020