key: cord-0701770-gqonoqoc
authors: Bhandarkar, Archis R.; Graffeo, Christopher S.; Johnson, Jeremiah
title: Stepping Up: How United States Neurosurgery Training Programs Can Innovatively Assess Resident Applicants in a Post-Step 1 World
date: 2020-07-16
journal: World Neurosurg
DOI: 10.1016/j.wneu.2020.07.078
sha: 4094ad0bd0388c06d1afa2da6c6525a9288fd554
doc_id: 701770
cord_uid: gqonoqoc

nan

Identifying and selecting optimal candidates for neurosurgery training is among the most important and challenging activities engaged in by residency programs in the United States.

Although a range of objective and subjective metrics are routinely employed, the United States Medical Licensing Exam (USMLE) Step 1 score is among the most long-standing and widely embraced tools for initial screening and ranking of candidates. In early 2020, the organizations administering the USMLE-National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB)-announced that Step 1 grading will transition from the familiar three-digit score to Pass/Fail by 2022. 1 According to NBME and FSMB representatives, this reconfiguration was motivated by "deleterious effects of high-stakes examinations like Step 1 on medical student well-being and the potential adverse effects of such an environment on medical school learning." 2 Given the competitive nature of our specialty, the considerable length-of-training, and the mandate to maintain high-resolution, high-fidelity selection processes, we sought to outline this sea change from the perspective of neurosurgery residency programs, with an eye towards avenues for improving our collective ability to identify those individuals most likely to demonstrate diversity, empathy, and clinical excellence in their training.

Few rites-of-passage have loomed so large in the minds of neurosurgery resident applicants than the USMLE Step 1, which has traditionally provided the first yardstick by which candidates are measured. Ostensibly written to test examinees' abilities to "apply important concepts of the sciences basic to the practice of medicine" such as the biochemistry, pathology, and pharmacology relevant to multiple organ systems," the Step 1 has been widely criticized for its clinical irrelevance. However, in spite of this potential shortcoming, it has consistently ranked among the top-weighted factors in applicant selection. One analysis of neurosurgery residency match outcomes from 1990-2007 identified Step 1 score >245 as the most significant predictor of successful match, with an odds ratio >20, when compared to applicants with a score ≤245. 3 Numerous studies have sought to better characterize the relative strengths and weaknesses of Step 1 as a metric for resident applicant selection. Given the curricular heterogeneity between institutions, as well as the inherent biases of medical school grading, Step 1 does provide one of the few standardized metrics accessible to all candidates, independent of their pre-residency educational pathway. Notwithstanding, Step 1 score has also failed to predict eventual ABNS certification status, academic rank, or research productivity. 4 Indeed, the only specific achievement in neurosurgery training associated with a higher Step 1 score is a higher score on the ABNS written primary exam-an association that likely reflects test-taking acumen, rather than knowledge base or clinical competency. In addition, related studies from the radiation oncology literature identified Step 1 screening cutoffs as associated with a disproportionately negative impact on female and under-represented minority applicants-groups that are critical to improving diversity in the neurosurgery workforce. 5

Perhaps the most obvious alternative to the Step 1 score is the Step 2 Clinical Knowledge (CK) score, which will continue to be reported as a numerical three-digit score-at least for the foreseeable future. 2,6,7 Potential benefits of an increased reliance on Step 2 CK include the familiarity of the scoring system, standardization of the exam across medical schools, and significantly more clinically-focused content. To this end, related studies from the Internal Medicine literature have identified the Step 2 CK exam as the strongest predictor of multi-modal resident performance assessments. 8 Unsurprisingly, increasing emphasis on Step 2 CK after Step 1 converts to Pass/Fail has already been affirmed widely in a survey of neurosurgery program directors. Although the potential for excessively weighting a single assessment remains in play with Step 2 CK, alternative approaches such as weighted or composite scores incorporating related metrics such as the numerical NBME shelf scores from key rotations (e.g. surgery, neurology, medicine) may have a role in offsetting this bias. 6

Looking outside of neurosurgery, several programs have developed novel tools for resident applicant assessment that may prove useful in our field. One such instrument is simulation-based interviewing, in which the formal resident applicant assessment includes a component of dedicated medical evaluation, with skills such as suturing, imaging interpretation, or laboratory result assessments. Although data is limited, and not available in the specific cohort of neurosurgery resident applicants, preliminary analyses suggest that simulation-based interviewing is highly predictive of individual outcomes in related in-service training assessments, such as general surgical skills evaluations, or milestones review. 9 Using the general surgery experience as a template, one can readily imagine a parallel process for neurosurgery, with candidate activities ranging from suturing, to MRI interpretation, EVD or LP placement, bedside neurologic examination, and many more. However, expectations should be carefully tempered, as one can imagine such an unconventional, high-stakes assessment resulting in excessive performance anxiety. In turn, this runs the risk of not only biasing result, but doing so in a way that disproportionately impacts those students already lacking in resources such as access to a home program with opportunities for salient teaching, mentorship, and potentially simulation or participation in procedures.

Research accomplishments are already prominent among the factors weighted in neurosurgery resident applicant selection and the elimination of a Step 1 exam is anticipated to increase the role played by academic achievements in candidate assessment. However, the specific ways in which this will be realized remain to be seen. Candidly, one hopes that an emphasis on quality will emerge as preferred to the present preference for quantity, which is an easier metric to assess, but ultimately less reliable. This reality is emphasized by a recent study of neurosurgery residency applicants to Vanderbilt University 2006-2012, which determined that the percentage of applicants misrepresenting the number and authorship of their publications had increased from 33% to 45% during the study period. 10 A variety of existing bibliometric tools, such as h-index, or impact factor of the journals publishing the applicant's research may be of value. However, these approaches could potentially be biased against applicants without access to a home program, or whose program lacks robust scholarship at the staff or resident level. Alternatively, consideration may be given to requesting that each individual applicant submit a brief portfolio of 1-2 representative publications of their choosing, which can then be subject to more direct scrutiny by the program.

Still other iterations of the process can be imagined that parallel the "simulation-based interview," in which a journal club or study design component is incorporated into the candidate assessment exercises.

In tandem to the increasing emphasis placed on research portfolio, interpersonal assessments such as letters-of-evaluation from sub-internships are also anticipated to rise in prominence as resident applicants are assessed outside the purview of a scored Step 1 exam. In order to improve the generalizability and standardization of these tools, an argument has been proposed and recently ratified by the Society of Neurological Surgeons that residency programs should incorporate standardized letters-of-evaluation (SLOEs) in lieu of personal letters. 7 In these example SLOEs, studied during the 2018-2019 application cycle, evaluators were prompted to rate candidates on pre-specified criteria (e.g. work ethic, subspecialty knowledge, ability to work on a team, etc.) using an objective, 3-point, relativistic Likert scale (e.g. "above peers," "at level of peers," "below peers"), with opportunity for a supplemental 250-word commentary. 11 Although still vulnerable to bias, these steps would potentially elevate letters as assessment tools beyond the anecdotal value they currently contribute. Also, given the unpredictable long-term impacts that the COVID-19 pandemic may have on the existence of medical student rotations, SLOEs may be of limited utility during the initial roll-out of post-Step 1 practices.

As ratified by essentially all neurosurgery residency program directors, the transitioning of USMLE Step 1 to a Pass/Fail grading scheme by NBME and FSMB will have sweeping impacts on candidate identification and selection in our field. Although this change clearly challenges the traditional screening and ranking models developed, it also constitutes a critical opportunity for the neurosurgery community to fundamentally transform and improve resident selection for the next generation of neurosurgeons in the US, with an eye towards minimizing bias, optimizing diversity, and incorporating metrics that are significantly more predictive of inresidency and long-term performance of individuals as trainees, surgeons, and academicians.

Numerous strategies warrant consideration and, ideally, formal testing in the years ahead. Key examples include a weighted sum of Step 2CK and neurology/surgery shelf scores, simulationbased interviewing, objective assessments of publication quality in lieu of quantity, and subinternship SLOEs. Although these candidate metrics may warrant consideration, we also anticipate that a range of novel assessments will be developed and studied as neurosurgery evolves, with the ultimate goal of continuing to attract and select the strongest candidates, in spite of the shifted testing landscape.

USMLE program announces upcoming policy changes

The USMLE Step 1 Decision: An Opportunity for Medical Education and Training

United States Medical License Exam

United States Medical Licensing Exam Step I Score as a Predictor of Neurosurgical Career Beyond Residency

Overemphasis of Step 1 Scores May Affect Application Pool Diversity in Radiation Oncology

Medical Education Takes a Step in the Right Direction: Where Does That Leave Students? Jama

The Pass/Fail Decision for USMLE Step 1-Next Steps

USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency

A simulation-based selection process for trying to identify medical students who will become outstanding general surgery residents. American journal of surgery

Publication misrepresentation among neurosurgery residency applicants: an increasing problem

Council of Residency Directors in Emergency Medicine. OFFICIAL CORD STANDARDIZED LETTER OF EVALUATION (SLOE)