key: cord-0743273-le9fi2t6
authors: Quesada, Pompeyo R.; Solis, Roberto N.; Ojeaga, Macaulay; Yang, Nuen T.; Taylor, Sandra L.; Diaz, Rodney C.
title: Overemphasis of USMLE and Its Potential Impact on Diversity in Otolaryngology
date: 2021-07-20
journal: OTO Open
DOI: 10.1177/2473974x211031470
sha: bc9bb02c4e0abcd27c9714c54828a0e03d628f97
doc_id: 743273
cord_uid: le9fi2t6

OBJECTIVE: Applicant demographics during the 2019-2020 residency cycle were evaluated to determine if strict utilization of United States Medical Licensing Examination (USMLE) scores in applicant selection could lead to a restriction in diversity. STUDY DESIGN: Cross-sectional study. SETTING: Otolaryngology residency applicants to a single institution. METHODS: A total of 381 applicants were analyzed by age, gender, applicant type, race/ethnicity, USMLE scores, permanent zip code, and graduating medical school. RESULTS: Among applicants, 37% were women; 9% were ≥30 years of age; 12% were underrepresented minorities (URMs); 71% to 81% had above-average socioeconomic surrogate markers; 22% were from a top 25 US News & World Report–ranked institution; and 81% were from an institution with an otolaryngology residency program. There was no increase in applicants who identified as URM from the 2015-2020 cycles. Multivariable regression analysis showed that applicants who were international medical graduates, URMs, and ≥30 years of age had lower Step 1 and Step 2 scores (P < .05). Applicants who identified as women had a lower Step 1 score, and those from top 25 National Institutes of Health–funded institutions had a higher Step 1 score; however, there was no difference when Step 2 scores were compared (P > .05). CONCLUSION: Our data suggest that in the pre–USMLE Step 1 pass/fail setting, strict adherence to USMLE scores may lead to disproportionally low recruitment of applicants who are women, ≥30 years of age, URMs, and from institutions without an otolaryngology residency program. We must implement measures against overemphasizing the absolute values of USMLE scores for a true holistic review of applicants, specifically to prevent an overemphasis on the USMLE Step 2 score.

I n 2020, the United States Medical Licensing Examination (USMLE) announced that Step 1 will become pass/fail starting in 2022. 1 The Federation of State Medical Boards and the National Board of Medical Examiners believe that this new change in policy will help reduce the current overemphasis on USMLE performance and is an ''important first step toward facilitating broader, system-wide changes to improve the transition from undergraduate to graduate medical education.'' 1 Also in 2020, the National Resident Matching Program reported a record-high 44,959 registered applicants for 37,256 residency positions. 2 In otolaryngology, there were 505 applicants for 350 residency positions. 3 Many residency selection committees among the more competitive specialties have traditionally used the USMLE Step 1 score to filter through large applicant pools vying for limited residency positions. 4 Consequently, medical students without a ''competitive' ' Step 1 score may prematurely abandon plans to pursue a given specialty. 4, 5 This overemphasis on

Step 1 scores for resident selection may have unintended consequences of reducing diversity in competitive specialties based on gender, age, and distribution of underrepresented minorities (URMs). 6 Various studies have reported the positive effect on patients and communities as a whole when a diverse physician workforce is in place. [7] [8] [9] Otolaryngology lags behind other medical specialties in the representation of URMs and women. 10 This study evaluates the demographics of the most recent pool of residency applicants to determine whether a correlation exists between the distribution of USMLE scores and the diversity of applicants.

After institutional review board approval for this study was obtained from the University of California-Davis, we collected all Electronic Residency Application Service (ERAS) applications submitted to the university's otolaryngologyhead and neck surgery residency program for the 2019-2020 cycle. The following data were obtained: age at the time of application submission, gender, applicant type, race/ethnicity, Step 1 score, Step 2 score, permanent zip code, and name of the medical school of graduation.

Age was categorized as \30 versus 30 years; gender as man versus woman; applicant type as US allopathic, US osteopathic, or international medical graduate (IMG); and race/ ethnicity as URM versus non-URM. URM was defined as African American/Black, Hawaiian/Pacific Islander, Hispanic/ Latino, or Alaskan Native/Native American, 6 and non-URM was defined as any race/ethnicity not in the URM definition, including Caucasian, Asian, and prefer not to state. The Step 1 cutoff score was set at 240 for in-depth Step 2 univariate and multivariable regression analyses and labeled Step 1 240 .

Applicants were excluded from zip code data analysis when their reported permanent zip codes matched their current locations and the sites of their medical schools of graduation. Applicants with zip codes outside the United States or Puerto Rico were excluded. These exclusions were created to optimize the permanent zip code analysis by capturing the applicants' residences independent of their medical schools of graduation. Utilizing the American FactFinder of the US Census Bureau, we defined the following variables in a binary format based on publicly available national data: median household income ($63,179), 11 percentage below the poverty line (11.8%), 11 high school graduate .25 years of age (90%), 12 and age .25 years with a graduate or professional degree. The percentage of individuals with a graduate or professional degree per zip code were stratified in an ascendingtier grading system: 11.8% = 0, .11.8% to 23.6% = 1, .23.6% to 35.4% = 2, .35.4% = 3. 13, 14 Medical schools were classified as top 25 US News or top 25 NIH based on the 25 highest-ranked schools in the 2021 US News & World Report 15 and the 25 institutions with highest funding from the National Institutes of Health (NIH). 16 Additionally, each applicant was categorized by home ENT program (yes vs no) based on whether the medical school of graduation had an affiliation with an otolaryngology residency program. The race/ethnicity of the 2018 US population was collected from American FactFinder of the US Census Bureau. 14 The race/ethnicity of the 2019 US medical school graduates and 2015-2019 US otolaryngology applicants was collected from the National Resident Matching Program. 3, 17 Univariate statistical analyses of difference in mean Step 1 and Step 2 scores were performed with a 2-sample t tests and analysis of variance for the explanatory variables listed in Tables 1 and 2. A multivariable linear regression analysis was conducted modeling Step 1 and Step 2 scores versus predictor variables of applicant type, URM, gender, age, top 25 NIH, home ENT program, and Step 1 240. For nominal data, we performed a 2-sample Z test for the difference between proportions. All statistical tests were 2-sided and evaluated at a significance level of 0.05. Statistical analyses were conducted with R version 4.0 (R Project for Statistical Computing).

Demographic information was collected from 381 applicants who applied to the University of California-Davis during the 2019-2020 cycle. Table 1 contains mean, median, and mode of the age, Step 1, and Step 2 variables for all applicants analyzed.

Step 1 and 2 scores were 245 6 12.5 and 253 6 12 (mean 6 SD), respectively. The applicant pool comprised 352 US allopathic medical school graduates (92%), 14 US osteopathic medical school graduates (4%), and 14 IMGs (4%; Table 2 ). The typical applicant was a man (63%), \30 years of age (91%), and of non-URM status (88%). The majority of the applicants self-identified as Caucasian (50.5%), followed by Asian (29.6%), Hispanic/Latino (6.3%), African American/Black (5.8%), and Native Hawaiian/ Pacific Islander (0.5%) ( Figure 1 ). Figure 2 shows the percentage of otolaryngology applicants by race/ethnicity from 2015 to 2020. There was no significant increase in URM otolaryngology residency applicants from 2015 to 2020.

Univariate analysis ( Table 2 ) showed that Step 1 scores were significantly different by applicant type (P \ .001). Scores were significantly higher for applicants belonging to any of the following categories: top 25 NIH (P = .005), top 25 US News (P = .034), home ENT program (P = .045), 30 years of age (P = .008), or non-URM (P = .001). A total of 274 and 265 applicants were in the Step 1 and Step 2 univariate permanent zip code analysis, respectively. There were no significant differences in Step 1 scores when comparing Seventeen applicants did not report Step 2 and were excluded from analysis.

across categories of permanent zip code: household income, poverty line, high school graduates, and graduate or professional degree. Univariate analysis ( Table 2 ) showed that Step 2 scores were significantly different for applicant type (P \ .001). Scores were significantly higher in home ENT program applicants (P = .041), those 30 years of age (P = .002), and non-URM applicants (P = .016). Applicants who scored 240 on

Step 1 had a significantly higher Step 2 (P \ .001). There were no significant differences in the Step 2 scores of otolaryngology applicants when comparing means across categories for top 25 NIH, top 25 US News, gender, and permanent zip code data.

On multivariable linear regression analysis ( Table 3) for Step 1 and Step 2 as response variables, Step 1 scores were lower for IMG applicants than US allopathic applicants (P = Step 1 score

Step 2 score .017), URMs versus non-URMs (P \ .001), women versus men (P = .017), and those aged 30 versus \30 years (P = .006). Conversely, a statistically significant increase in Step 1 scores was observed in applicants who graduated from a top 25 NIH-funded medical institution (P = .015), but this did not hold true for Step 2 (P = .757). There was no significant difference in Step 1 or Step 2 scores for home ENT program applicants after multivariable analysis. Multivariable regression analysis for

Step 2 scores showed significantly lower scores in applicants who were IMG (P = .007), URM (P = .007), 30 years of age (P = .010), and scored \240 on Step 1 (P \ .001). While gender was a predictor of Step 1 performance, no difference was found for Step 2 scores (P = .502).

Otolaryngology currently faces a significant representation deficit of URMs and women, even though research in various industries, such as finance, technology, and government-run agencies, has shown that the deliberate creation of a diverse workforce selects for increased success and innovation. 18-20 Editorials in otolaryngology have emphasized the idea that diversity in otolaryngology should be sought, as the power of diversity is critical for breaking down bias, improving innovation, and preparing for success. 18, [21] [22] [23] [24] To that end, diversity is multifaceted, as it encompasses areas such as gender, age, educational background, and social background. 25 Currently, there are 155 allopathic and 36 osteopathic accredited medical schools in the United States. 26, 27 However, only 120 otolaryngology programs are listed in the 2020-2021 ERAS, 28 which leaves many students with limited access to mentorship and guidance. Our data demonstrate that we are mostly recruiting students from institutions with affiliated otolaryngology programs (81%; Table 2 ). 3 Even with several initiatives to recruit and promote diversity in the field with scholarships, lecture series, and diversity programs, 29-33 our cohort breaks down as follows: 37% women, 9% aged 30 years, 4% US osteopathic, 4% IMG, and 12% URM.

The 2018 US population of URMs was 32.1%, 14 yet only 13.5% of 2019 medical school graduates identified as such. 17 Just 5.6% of the 2018 US population 14 identified as Asian, as compared with 21.5% of 2019 medical graduates 17 and 29.6% of 2020 otolaryngology applicants (Figures 1 and 2) . Even when all URM applicants are combined (African American/ Black, Hispanic/Latino, Hawaiian/Pacific Islander, and Alaskan Native/Native American), their percentage is just 15.7% of the Asian/Caucasian 2020 otolaryngology applicant pool. Recent work by Smith et al showed that only 19% of otolaryngology programs have department-level diversity and inclusion initiatives, 31 which may expose an avenue to increase recruitment of URMs into otolaryngology. There is still significant work to be done to expand the pipeline for diverse candidates and create a sustainably diverse racioethnic field within otolaryngology that truly reflects our country's population. [29] [30] [31] [32] [33] Simply increasing the number of applicants does not guarantee an increased number of URMs obtaining residency interviews or matching into otolaryngology. The issue that we face is 2 pronged: the needed increase of URM applicants and URM matriculates in otolaryngology residency. Research in applicant trends has shown that stringent Step 1 expectations may limit future talented physicians from entering the field of otolaryngology while limiting diversification of the field. 4, 5, 29 The 2018 National Resident Matching Program's Charting the Outcomes showed that otolaryngology had an average Step 1 of 248 and Step 2 of 254, which is well above the national average. 34 The 2018 Program Director Survey highlights the importance that program directors place on Step 1 in making decisions for offering interviews and ranking applicants. 35 However, a clear link between Step 1 and resident performance has never been fully established. 36, 37 Our study demonstrates that applicants who are 30 years of age or identify as URM have lower Step 1 and Step 2 scores as compared with their younger or non-URM counterparts (Tables 2 and 3) . Given the historical value placed on Step 1 in otolaryngology, we advise against overemphasizing the absolute values of the USMLE scores to stratify applicants, as this may continue to adversely affect URMs to a greater degree. With the current COVID-19 pandemic, these disparities may be exacerbated. Students coming from institutions without affiliated otolaryngology programs will face challenges garnering letters of recommendations and valuable face time with programs, which could lead to an even greater emphasis on the USMLE. 38 In 2017 the Association of American Medical Colleges reported that for the first time, more women than men matriculated into medical school. 39 Despite that great achievement, the association noted that of the 1542 otolaryngology residents in 2017 to 2018, 40 just 558 (36.2%) were women. 41 Additionally, an estimated 14.5% of practicing otolaryngologists are women. 42 Our data highlight the need for increased recruitment of otolaryngology applicants who are women, as only 37% of applicants in our cohort are women ( Table 2) . Women in our cohort averaged slightly lower Step 1 scores than men; however, our data show that for Step 2, the mean score difference between genders did not persist ( Table 3) . Table 3 . Multiple Linear Regression Analysis of USMLE Step 1 and Step 2 Scores.

Step 1 score

Step 2 score Similar to URM applicants, our results suggest that further work is needed for equal gender representation in our applicant pool. We highlight that an overemphasis of the Step 1 score has the potential to contribute to the gender gap seen in otolaryngology resident selection. Leaders in academic medicine have begun to advocate against implicit or subconscious bias by promoting training and education to create a diverse workforce that reflects the population that we treat. [43] [44] [45] Step 1 transitions to pass/ fail, there exists the potential for applicants coming from lower-tier or newer medical schools to face a subconscious bias from resident selection committees as compared with their colleagues attending a school labeled as top 25 NIH or US News or with a home ENT program. Chatterjee et al recently explored how subconscious biases are prevalent in the residency interview trail, 46 while Hauge et al showed how a ''closed file review'' may affect how interviewers rate applicants. 47 These studies suggest that efforts are needed to limit biases based on ''educational pedigree'' and/or background. After adjusting for covariates, applicants coming from top 25 NIH schools have significantly higher Step 1 scores (3.7-point estimate); however, this is no longer significant in their Step 2 scores ( Table 3) . It is unclear why a difference exists between these groups, although it could be explained by the low percentage of URMs, women, and those aged 30 years in our cohort who come from top 25 NIH medical schools (12.7%, 39.2%, 10.1%, respectively). Therefore, we propose that subconscious bias training be implemented for all application reviewers and interviewers, while subconscious bias-prone variables such as educational pedigree be omitted from application review. Additionally, we propose that closed file interviews be undertaken to limit bias even before the beginning of the interview, with the goal of promoting diversity of selection, especially because 78.7% of URMs, 77.9% of women, and 77.1% of those aged 30 years in our data come from medical schools not labeled top 25 NIH.

Area-based socioeconomic research has used zip code data to evaluate outcomes in health disparities. 48 Using similar zip code methods, we utilized surrogate markers for socioeconomic status to compare the influence on USMLE examinations on performance and found no significant difference in any of the variables (Table 2) ; however, we excluded multiple applicants as described in the methods. The exclusion limited our power for an in-depth area-based socioeconomic analysis in our cohort and could be the reason for lack of significance. Interestingly, the majority of applicants in the analysis reported living in a permanent zip code above the national mean or median for household income (71%), were high school graduates .25 years of age (80%), had a graduate or professional degree (81%), and fell below the national mean in poverty line (73%). Further investigations are needed to evaluate how socioeconomic background affects performance in the USMLE and diversity in otolaryngology, as this could help validate increased support to students who come from backgrounds limited in funding, education, and preparation. 49, 50 This study has several limitations and strengths that should be considered when evaluating the data. All variables were self-reported by the applicants in their ERAS application, including race/ethnicity, permanent zip code, and gender. Additionally, applicants had the option prefer not to state, which could have affected the analysis of this study. Our study captured 75.4% of all otolaryngology residency applicants during the 2019-2020 cycle; thus, there is a possibility that the demographics of the applicants not reviewed could affect our results and not be representative of the entirety of the otolaryngology applicant pool for this cycle. However, given the robust significance of our data and the affinity of our data to those seen in other residency applicant years, 10, 34 we trust that our applicant cohort demonstrates an acceptable portrayal of the otolaryngology applicant pool as a whole.

With the change to a pass/fail Step 1, the question is whether there will be an inevitable shift in emphasis to the absolute value of the Step 2 score in differentiating applicants. 51 In our data 100% of the applicants passed Step 1, and while the scoring changes may remove the emphasis in that factor, it leaves Step 2 as the sole objective variable for consideration. Even though our multivariable regression analysis showed that applicants who scored 240 on Step 1 had a significantly higher Step 2 score, we advise against the future overemphasis of Step 2 in applicant selection as a surrogate for Step 1 ( Table 3) . Specifically, our data demonstrate that the currently underrepresented groups in otolaryngology would continue to be at a significant disadvantage if an overemphasis on Step 2 were to develop. Furthermore, using other metrics as filtering tools, such Alpha Omega Alpha and Gold Humanism Honor Society, holds intrinsic limitations due to the lack of standardization in criteria for nomination, selection, and even availability of honor society chapters at institutions. Using publication numbers or impact factor of published scholarly work would likely negatively affect applicants with limited access to research opportunities, as such opportunities and funding are often concentrated at major academic centers with a plentitude of resources and faculty dedicated to scholarly activity. Therefore, we highly encourage a more holistic review of applicants to assess motivation, passion, academic rigor, personality, and other skills irrespective of factors external to their control, as this may encourage URMs to consider otolaryngology and it may increase the number of URMs being interviewed and matched. 4, 5, 29, 34, 35 As Step 1 moves to pass/fail, otolaryngology residency selection committees-and graduate medical education as a whole-should take clear preventative measures to avert the unwarranted overemphasis of Step 2 scores in filtering applicants.

Utilization of the Step 1 score absolute value in resident selection may lead to disproportionally low recruitment of applicants who are women, 30 years of age, URM, socioeconomically disadvantaged, or from institutions without an affiliated otolaryngology residency program while holding true for Step 2 for applicants who are URM and 30 years of age. Therefore, overemphasis of USMLE scores may perpetuate the critical lack of diversity in our field.

National Resident Matching Program. Thousands of resident applicants celebrate NRMP match results. Press release

National Resident Matching Program. Results and Data: 2020 Main Residency Match. National Resident Matching Program

A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection

The state of the otolaryngology match: a review of applicant trends, ''impossible'' qualifications, and implications. Otolaryngol Head Neck Surg

Overemphasis of Step 1 scores may affect application pool diversity in radiation oncology

Race, gender, and partnership in the patient-physician relationship

Patient-physician relationships and racial disparities in the quality of health care

The association of doctorpatient race concordance with health services utilization

The state of diversity based on race, ethnicity, and sex in otolaryngology in 2016

US Census Bureau. High school completion rate is highest in US history

High school completion rate is highest in US history

The best medical schools for research, ranked. US News & World Report. Accessed

OTHDH&fy=2019&state=&ic=&fm=&orgid=&distr=&rfa= &om=n&pid= 17. Association of American Medical Colleges. 2019 facts: enrollment, graduates, and MD-PhD data

Letter from the editor: why we are better when we are diverse. ENTtoday

Why diverse teams are smarter

Do pro-diversity policies improve corporate innovation?

Letter from the editor: otolaryngology must address

addresses bias and diversity in otolaryngology. ENTtoday

Recognizing diversity is essential for delivering quality, affordable health care

Female otolaryngologists pursuing research careers need more support, researchers conclude. ENTtoday

Workforce diversity: a key to improve productivity

Accredited US programs

American Association of Colleges of Osteopathic Medicine. US colleges of osteopathic medicine

Diversity in otolaryngology residency programs: a survey of otolaryngology program directors

How otolaryngology programs are working to create a more diverse workforce. ENTtoday

Diversity in academic otolaryngology: an update and recommendations for moving from words to action

Acting internship: Hilary A. Brodie, MD, PhD Endowment for Diversity in Medical Student Otolaryngology Visiting Elective Clerkship

National Resident Matching Program. Charting Outcomes in the Match: US Allopathic Seniors. National Resident Matching Program

National Resident Matching Program. Data Release and Research Committee: Results of the 2018 NRMP Program Director Survey. National Resident Matching Program

What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States Medical Licensing Examination Step 1 scores

Is USMLE Step 1 score a valid predictor of success in surgical residency?

Otolaryngology residency application during the SARS-CoV-2 (COVID-19) pandemic. Otolaryngol Head Neck Surg

More women than men enrolled in US medical schools in 2017. Association of American Medical Colleges

Women were majority of US medical school applicants

Association of American Medical Colleges. Table B3: number of active residents, by type of medical school, GME specialty, and sex

Women in otolaryngology

Who gets chosen for your residency and who gets overlooked? Implicit bias in medical education

Unconscious bias in academic medicine: overcoming the prejudices we don't know we have. Association of American Medical Colleges

Let's face it: we are biased, and it should not be that way

US medical school applicant experiences of bias on the interview trail

Evaluating resident candidates: does closed file review impact faculty ratings?

Evaluating area-based socioeconomic status indicators for monitoring disparities within health care systems: results from a primary care network

The consequences of structural racism on MCAT scores and medical school admissions

Examining demographics, prior academic performance, and United States Medical Licensing Examination scores

USMLE Step 1 is now pass/fail-who benefits from this big change? Forbes

Association of American Medical Colleges. ERAS statistics

We thank Lisa Yee-Isbell from the Department of Otolaryngology-Head and Neck Surgery at the University of California, Davis, for her support in data extraction, as without her this research would not have been possible.Competing interests: None.

Funding source: This project was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 TR001860.

Pompeyo R. Quesada, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Roberto N. Solis, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Macaulay Ojeaga, substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Nuen T. Yang, substantial contributions to analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Sandra L. Taylor, substantial contributions to analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved; Rodney C. Diaz: substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; acknowledgment of drafting the article or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.