key: cord-0016780-dlctzq80 authors: Mun, Frederick; Scott, Alyssa R.; Cui, David; Chisty, Alia; Hennrikus, William L.; Hennrikus, Eileen F. title: Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey date: 2021-04-16 journal: Medicine (Baltimore) DOI: 10.1097/md.0000000000025284 sha: 7c879bd0dd563b89baf1ed8d59697a6fd6bf508b doc_id: 16780 cord_uid: dlctzq80 1. internal medicine residency program directors’ perceptions on the change of Step 1 to a pass/fail exam, and 2. the impact on other factors considered for internal medicine residency selection. A validated REDCap survey was sent to 548 program directors at active Accreditation Council for Graduate Medical Education internal medicine residency programs. Contact information from the American Medical Association's Fellowship and Residency Electronic Interactive Database was used. Contact information from the American Medical Association's Fellowship and Residency Electronic Interactive Database was used. The survey had 123 respondents (22.4%). Most internal medicine program directors do not support the pass/fail change. A greater importance will be placed on Step 2 Clinical Knowledge exam, personal knowledge of the applicant, clerkship grades, and audition electives. Allopathic students from less highly regarded medical schools, as well as osteopathic and international students, will be disadvantaged. About half believe that schools should adopt a graded pre-clinical curriculum (51.2%) and that there should be residency application caps (54.5%). Internal medicine program directors mostly disagree with the pass/fail Step 1 transition. Residency programs will need to reevaluate how applicants are evaluated. Other factors, such as Step 2 Clinical Knowledge score, personal knowledge of the applicant, grades in clerkships, and audition rotations will now be emphasized more heavily. On February 12, 2020, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME) announced that score reporting for Step 1 of the United States Medical Licensing Examination (USMLE) would change from the three-digit numeric score to reporting a pass/fail outcome-starting no earlier than January 2022. At this time, Step 1 is one of the most important factors used to screen applicants by internal medicine residency programs, due to the high volume of applicants. [1] In 2020, internal medicine programs received the most number of applications among all specialties, with 13,118 applicants competing for 8697 positions. [1, 2] The number of applicants is projected to continue to grow in future years. [2] With the absence of a scored Step 1, internal medicine residency programs will need to utilize other methods to screen applicants for an interview. The purpose of this study was to investigate the perceptions among internal medicine residency program directors (PDs) on the change of Step 1 from a graded to a pass/fail examination. We also aimed to investigate how the change would impact the other factors normally considered for internal medicine residency selection. This study was a multi-center, cross-sectional, 27-item electronic research survey. The College of Medicine's Institutional Review Board granted exempt status (ID#: STUDY00015130). We obtained publicly available contact information for PDs at all active Accreditation Council for Graduate Medical Education (ACGME) internal medicine residency programs through the American Medical Association's Fellowship and Residency Electronic Interactive Database (FREIDA Online). We identified and contacted 554 U.S. internal medicine residency PDs. A survey invitation email with the appropriate informed consent information was sent to each PD. Submission of the survey indicated the respondents' consent to participate. The electronic survey was distributed on July 8, 2020. Follow-up emails were sent 2 and 4 weeks after the initial submission. The survey was closed on August 5, 2020. This anonymous survey was developed, housed, and distributed through REDCap TM . The survey was developed using criteria analyzed in the 2018 National Resident Matching Program (NRMP) PD survey, which investigated the factors involved in selecting applicants to interview. [1] Our survey questions used single-answer multiple-choice, multiple-answer multiple-choice, a Likert scale ranging from 1 (strongly disagree or least important) to 5 (strongly agree or most important), and free text formats. Prior to distribution, the questions were pretested and tested with subsets of medical students and physicians, then adjusted and readjusted for validation. It was then sent to PDs of a different specialty before final revisions were made. In total, the survey underwent five revisions prior to distribution. All PDs were contacted at verified email addresses, and responses were recorded anonymously. All 554 U.S. internal medicine residency PDs were contacted via email. We collected the PDs' answers from the anonymous survey and conducted statistical analysis to determine their perceptions of USMLE Step 1 changing from a scored to pass/fail examination. Responses were analyzed with descriptive statistics and percentages. We defined statistical significance (P < .01) by nonoverlapping 99.9% confidence intervals. About 554 programs were contacted, and 548 programs were included in this study due to invalid email accounts. The survey had 123 respondents, for a response rate of 22.4%. 58 programs responded after the first email, 47 after the follow-up email, and 18 programs after the final email. The survey is shown in Supplement 1, http://links.lww.com/MD2/A46. Step 1? The majority of PDs do not support the change of Step 1 to pass/fail (74.0%), and feel that the pass/fail transition was not transparent (78.9%). Yet, only 39.0% of PDs think that a graded Step 1 adequately measures an applicant's ability to succeed in internal medicine. See Table 1 . Step 1 impact internal medicine residency programs? A majority feel that the pass/fail Step 1 will make it harder for residency programs to select which applicants to interview (68.3%), and that the pass/fail Step 1 will not allow the match process to be fair and meritocratic (59.3%). See Table 1. 3.2.3. How will the pass/fail change impact the importance of the factors reviewed by internal medicine residency programs when assessing applicants? A majority of PDs believe that the pass/fail Step 1 exam result will be less important as a selection factor (71.5%). They believe that a greater importance will be assigned to Step 3.2.4. How will various applicant groups be affected by the change to a pass/fail Step 1? A majority of PDs believe that MD students who attend highly regarded medical schools will have an advantage (55.3%) and that international medical graduates (82.9%), MD students who do not attend highly regarded medical schools (63.4%), and osteopathic (DO) students (52.0%) will be at a disadvantage. See Table 1. 3.2.5. How will changing to a pass/fail Step 1 affect medical students interested in internal medicine? A majority of PDs do not think that changing the Step 1 to pass/fail would have any effect on medical students' research interests, extracurricular/ leadership involvement, hobbies, number of away rotations or applications to other specialties. However, a majority of PDs believe that a pass/fail Step 1 will encourage students to apply to more internal medicine residency programs (56.9%). See Table 2 . A statistically significant majority of PDs also believe that there should be a cap on the number of residency applications a medical student can submit (54.5%), with suggestions ranging from 6 to 50 residency applications per student. See Table 1. 3.2.6. What are the future implications on residency applications and medical education? Because Step 1 is going to be pass/fail, about half of the PDs believe that medical schools should adopt a graded pre-clinical curriculum (51.2%). Step 1 This study reveals that most internal medicine PDs do not support the Step 1 change to pass/fail and expect the selection process to be more difficult and less fair and meritocratic. They expect Step 2 CK, clerkship grades, personal knowledge of the applicant and audition electives to rise in importance. The PDs also believe that allopathic medical students not attending highly regarded medical schools, osteopathic students and international medical graduates will be disadvantaged with a pass/fail Step 1. The majority of PDs recommend limiting the number of residency applications per medical student to < 50. The USMLE Step 1, as developed in 1992, is the first of a threepart series created for licensing eligibility. [3] However, with a number of additional conglomerate variables, the numeric scoring of this national exam, has resulted in unintended consequences. A few major changes in medical education and residency training since 1992 have forced the over-magnification of Step 1 numeric scores in assessing residency applicants. First, many medical schools have adopted a pass/fail pre-clinical curriculum. [4] Second, there has been an increase in medical school matriculants, without a corresponding increase in the number of federally funded residency positions. And third, the Match process requires PDs and medical students to submit ranking lists. [5] These changes have resulted in student fear of "not matching". As a result, the average number of residency applications has increased from single digits to 60.3 applications per medical student. [6] With fewer standardized objective criteria, and more applicants, 95% of PDs use Step 1 scores to screen and filter applicants for residency interviews with 70% of internal medicine residency programs enforcing a cutoff score. [1] There is moderate correlation between failing Step 1 or scoring in the lowest quartile and having difficulty passing the American Board of Internal Medicine. [7] Despite the limited evidence supporting a numerically scored Step 1's ability to screen for successful residents or future successful physicians, internal medicine residency programs have placed a strong emphasis on Step 1 scores. [1, [8] [9] [10] [11] Currently, internal medicine PDs use USMLE Step 2 CK scores, internal medicine clerkship grades and USMLE Step 1 scores as their top three criteria for deciding to offer applicant interviews. [11] The trickle-down effect of the emphasis placed on Step 1 scores is the unanticipated parallel curriculum which is time consuming and expensive for the students. [12] [13] [14] Because of minimal standardizations in grades, clerkship grade inflation, and lack of class rank in medical schools, PDs place a greater emphasis on Step 1 scores as one of the few objective measures to assess applicants. [1, [14] [15] [16] [17] Our study also reflected PDs' desires for concrete evaluation metrics, with a statistically significant majority of internal medicine PDs not supporting the pass/fail change, and over half of them advocating for a graded pre-clinical curriculum. The Invitational Conference on USMLE Scoring and numerous commentaries have asserted that the change to a pass/fail Step 1 may simply lead to the scored Step 2 CK becoming the new screening metric. [18] [19] [20] [21] [22] [23] A significant majority of internal medicine residency PDs in our survey agree that the Step 2 score will become more important, replacing the emphasis on Step 1. The literature suggests that there is a better correlation between Step 2 scores and clinical performance than there is between Step 1 scores and clinical performance. [24, 25] In addition, internal medicine programs will place a greater importance on personal knowledge of the applicant, grades in required clerkships, and audition elective/rotation with their own department. It seems that one of the goals of changing Step 1 to pass/fail is to develop a more holistic approach to residency applications. But interestingly, PDs do not plan to increase their emphasis on a student's research experience, involvement in the Alpha Omega Alpha Society, Gold Humanism Society, or volunteer and leadership extracurricular activities. PDs do not plan on adjusting the impact of the personal statement, or letters from nonmedicine or unrecognized faculty. In our study, a majority of internal medicine residency PDs believe that the pass/fail transition will disadvantage MD students who do not attend a highly regarded medical school, DO students, and foreign medical students. Citing school reputation and personal relationships as increasingly important in residency selection seems at odds with meritocracy and holistic consideration of applicants. The majority of PDs believe that medical students will adapt to the pass/fail transition by applying to even more residency programs. More applications, and less objective criteria, can potentially make residency admissions decisions even more difficult. More than half of internal medicine residency PDs would support a cap/limit on the number of residency applications each medical student can submit. Understanding that adaptations are necessary to adjust to the pass/fail Step 1 change, the FSMB and NBME do not plan to implement this change until sometime after January 2022. The disagreement about making Step 1 pass/fail is largely based on losing a universal standardized metric by which to expeditiously select a few candidates from thousands of applicants. The FSMB and NBME are allowing time for medical schools and graduate medical education programs to develop new metrics of student Table 2 How do you think changing to a pass/fail Step 1 will affect medical students interested in internal medicine? In conclusion, the majority of internal medicine PDs do not support the transition of USMLE Step 1 to a pass/fail scoring system. Internal medicine residency programs will have to reevaluate how they assess applicants for an interview. Other factors will now be emphasized including Step 2 CK, personal knowledge of the applicant, grades in required clerkships, and audition rotations. There is time, before Step 1 becomes pass/fail, to make adjustments to the undergraduate and graduate medical education evaluation process. Changing Step 1 to pass/fail is just the first step. Our study had several limitations. Firstly, we were unable to obtain responses from all internal medicine program directors. However, we received responses from 123 PDs (22.4%), which we believe is reasonable and representative. We sent the survey out three times because we believed that subsequent emails would not have a significant impact on response rate. Secondly, our study may be limited by sampling bias and non-response bias. PDs who felt strongly about the change in Step 1 may have been more likely to respond. Finally, some contact information provided by FREIDA Online did not belong to the PD, but instead the program's coordinator or administrative assistant. Our email requested program coordinators and administrative assistants to send the survey to the PD. Funding acquisition: Eileen F Hennrikus. Investigation: Frederick Mun National Resident Matching Program, Results and Data: 2020 Main Residency Match ® . National Resident Matching Program Objective Measures Needed -Program Directors' Perspectives on a Pass/Fail USMLE Step 1 Grading systems use by US medical schools The impact of a pass/fail step 1-a residency program director's view Should the USMLE be pass/fail? Association of American Medical Colleges The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 Examination Associations between residency selection strategies and doctor performance: a meta-analysis A multicenter analysis of the ophthalmic knowledge assessment program and American Board of Ophthalmology written qualifying examination performance USMLE step 1 scores as a significant predictor of future board passage in pediatrics Internal medicine residency program directors' screening practices and perceptions about recruitment challenges MD A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection Step 1 climate" in preclinical medical education The prevalence, causes, and consequences of experiencing a life crisis during medical school The USMLE Step 1 pass/fail reporting proposal: The APDR position Numerical versus pass/fail scoring on the USMLE: what do medical students and residents want and why? The new USMLE Step 1 paradigm The pass/fail decision for USMLE Step 1-next steps USMLE Step 1 pass/fail winners and losers Sparking change Why pass/fail step 1 is really only step 1. The Harvard Crimson Cheers and jeers as med school's Step 1 test becomes pass/fail Medicine (2021) 100:15 www.md-journal The change to pass/fail scoring for Step 1 in the context of COVID-19: implications for the transition to residency process Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination The relationship between licensing examination performance and the outcomes of care by international medical school graduates The authors of this study have no sources of funding to disclose.All authors participated in the research and preparation of the manuscript.The authors have no conflicts of interest to disclose.Data Access: All data generated/analyzed during this study are included in this article.Supplemental Digital Content is available for this article.All data generated or analyzed during this study are included in this published article [and its supplementary information files].