key: cord-0754082-30zw0owf authors: Pletcher, Steven D.; Chang, C.W. David; Thorne, Marc C.; Malekzadeh, Sonya title: The Otolaryngology Residency Program Preference Signaling Experience date: 2021-10-05 journal: Acad Med DOI: 10.1097/acm.0000000000004441 sha: 0c0a5c227a89fbfe62b90a8b1d3c3710982fa9cb doc_id: 754082 cord_uid: 30zw0owf PROBLEM: In the 2021 residency application cycle, the average otolaryngology applicant applied to more than half of programs. Increasing application numbers make it difficult for applicants to stand out to programs of interest and for programs to identify applicants with sincere interest. APPROACH: As part of the 2021 Match, otolaryngology applicants could participate in a preference signaling process, signaling up to 5 programs of particular interest at the time of application submission. Programs received a list of applicants who submitted signals to consider during interview offer deliberations. Applicants and program directors completed surveys to evaluate the signaling process and assess the impact of signals on interview offers. OUTCOMES: All otolaryngology residency programs participated in the signaling process. In total, 611 students submitted applications for otolaryngology residency programs, 559 applicants submitted a Match list including an otolaryngology program, and 558 applicants participated in the signaling process. The survey response rate was 42% for applicants (n = 233) and 52% for program directors (n = 62). The rate of receiving an interview offer was significantly higher from signaled programs (58%) than from both nonsignaled programs (14%; P < .001) and the comparative nonsignal program (23%; P < .001) (i.e., the program an applicant would have signaled given a sixth signal). This impact was consistent across the spectrum of applicant competitiveness. Applicants (178, 77%) and program directors (53, 91%) strongly favored continuing the program. NEXT STEPS: Many specialties face high residency application numbers. Programs have difficulty identifying applicants with sincere interest, and applicants face limited opportunities to identify programs of particular interest. Applicants to these specialties may benefit from a preference signaling process like that in otolaryngology. Additional evaluation is needed to determine the impact of signals across racial and demographic lines and to validate these early outcomes. All otolaryngology residency programs participated in the signaling process. In total, 611 students submitted applications for otolaryngology residency programs, 559 applicants submitted a Match list including an otolaryngology program, and 558 applicants participated in the signaling process. The survey response rate was 42% for applicants (n = 233) and 52% for program directors (n = 62). The rate of receiving an interview offer was significantly higher from signaled programs (58%) than from both nonsignaled programs (14%; P < .001) and the comparative nonsignal program (23%; P < .001) (i.e., the program an applicant would have signaled given a sixth signal). This impact was consistent across the spectrum of applicant competitiveness. Applicants (178, 77%) and program directors (53, 91%) strongly favored continuing the program. Many specialties face high residency application numbers. Programs have difficulty identifying applicants with sincere interest, and applicants face limited opportunities to identify programs of particular interest. Applicants to these specialties may benefit from a preference signaling process like that in otolaryngology. Additional evaluation is needed to determine the impact of signals across racial and demographic lines and to validate these early outcomes. The average applicant to Otolaryngology-Head and Neck Surgery in the 2021 residency application cycle applied to more than 50% of otolaryngology programs nationwide, submitting 77 applications, 1 a 34% increase over 5 years. 2 This surge has made it difficult for residency programs to holistically review applications and has limited opportunities for applicants to stand out to programs of particular interest. The residency application process typically offers multiple opportunities for applicants to informally signal their interest in a program: visiting rotations, advocacy from faculty mentors, self-advocacy, and participation in departmental events. The fidelity of these informal signals, however, is highly variable. Furthermore, access to mentors, the cost of away rotations, imbalances in social capital, awareness of opportunities, and the implicit biases of faculty who send and receive these informal signals likely exacerbate inequities in the application process. In addition, the COVID-19 pandemic affected the residency application and recruitment process. 3 Visiting rotations were discontinued for most students, and virtual interviews raised the specter of interview hoarding. Without having to travel, a small group of applicants could complete a disproportionate number of the available interviews, crowding out other applicants, particularly those from nontraditional backgrounds. As a result, programs could be left with unfilled slots despite an overabundance of qualified applicants. A survey of students applying to otolaryngology programs in the 2021 residency application cycle revealed that applicants felt less confident in their ability to match given these COVID-19-related changes, and they identified difficulty expressing interest in specific programs as a contributing factor. 4 In the spring of 2020, the Otolaryngology Program Directors Organization (OPDO) discussed signaling as a means to mitigate these concerns and committed to implementing a formal signaling process for the 2021 Match. A formal preference signaling process provides all applicants with access to a known and stable quantity of signals. Through this process, at the time of application submission, students send a signal to indicate to a defined number of residency programs their particular interest in those programs. Such signals allow students to stand out to their favored programs and allow programs to receive a list of highly interested applicants. To our knowledge, this approach has not been used previously in the residency application process. Yet, articles advocating for signaling exist in the otolaryngology literature, [5] [6] [7] and the methodology, rationale, and results of preference signaling for graduates of economics PhD programs applying for faculty positions have been described. 8 An OPDO working group, comprising the 4 authors, drove the establishment of a signaling process. In the spring of 2020, we held a series of meetings and webinars to engage stakeholders in the development and implementation of a signaling process. We included students, program directors, and specialty societies, such as the Society of University Otolaryngologists and the Association of Academic Departments of Otolaryngology Otolaryngology Chairs Organization. Additional discussions with the Association of American Medical Colleges, the Electronic Residency Application Service (ERAS), and the National Resident Matching Program also took place. In hindsight, establishing consensus across stakeholders proved to be the most challenging hurdle to successful implementation of our signaling process. Stakeholders had to accept this change and the inherent risks of implementing a "never before in medicine" process. These discussions, however, also provided a critical opportunity to refine our proposal and create an educational ecosystem that accepted this signaling process. Based on stakeholder input, a review of the limited literature, and robust discussions within the OPDO Council, we opted to move forward with 5 signals per applicant. Additional refinements included the following: applicants could not signal their home program nor programs where they completed an in-person clinical rotation and rules were developed around the confidentiality of signaling information. 9 The creation of a signaling website (http://entsignaling. org) allowed for the collection of ongoing input from all stakeholders and for easy access to signaling information. Next, we had to demonstrate our capacity to receive and distribute signals. Otolaryngology-Head and Neck Surgery is a small field with approximately 130 residency programs, 350 residency positions, and fewer than 600 applicants in the Match. This provided a significant advantage in implementation. We had a manageable amount of data, so a few individuals could receive the signals from applicants, manually organize them for distribution, and send them to the appropriate programs. Because there is no mechanism within ERAS to run a signaling process, our goal was to maintain control of the signaling system and keep the data confidential within OPDO and its affiliated organizations (Society of University Otolaryngologists and Association of Academic Departments of Otolaryngology), using existing staff from these organizations to run the process. We designed an opt-out process for residency programs to prevent the inadvertent exclusion of programs and an opt-in process for applicants, a necessity for those who chose to send signals. Before receiving signals, programs were required to sign a code of conduct focused on signal confidentiality (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B194). Applicants who were unaware of the signaling process could be at a significant disadvantage, so we pursued multiple outreach approaches. Following the student-focused webinars, each of which had more than 200 attendees, we held similar webinars for program directors and student advisors and encouraged these groups to confirm that their applicants were aware of the signaling process. We engaged the robust online community and discussion boards in otolaryngology including Otomatch.com and Headmirror.com and worked with the Association of American Medical Colleges to publicize the process among medical school deans. Finally, a page was added to the ERAS website informing otolaryngology applicants of the signaling process and directing them to the OPDO signaling website (see above) for more information and to find the portal for signal submission. We established a clear timeline with the goal of distributing signals on the day that applications became available to programs. This included a nearly 2-week window for applicants to submit their signals (September 28 to October 9, 2020) and an 11-day window for us to process those signals (October 10 to 20, 2020). During the submission window, applicants used a secure survey on the OPDO signaling website (see above) to select up to 5 residency programs to signal. We used the same program list as ERAS. Applicants were also asked to enter their name and ERAS number and to select their home program or indicate the lack of a home program. An automated confirmation of signals was sent to applicants upon submission. We then exported the data from the secure survey into an Excel spreadsheet (Microsoft Corporation, Redmond, Washington) and, on October 21, 2020, sent each program a list with the names and ERAS numbers of the applicants who sent signals to that program. Programs were encouraged to cross-check names and ERAS numbers and report any concerns. None were reported. To assess the signaling process, we distributed electronic surveys to applicants during Match week in March 2021 and to residency programs shortly after. Along with items about general attitudes about the signaling process, we asked applicants to identify the program they would have signaled had they been provided a sixth signal (which we called the comparative nonsignal program). Additional survey questions asked applicants for the number of applications they submitted and the number of interviews they received from signaled, nonsignaled, and the comparative nonsignal program. We asked program directors how they used the signals during the application review process. Students and program directors were also asked whether the signaling process should continue, and they were given space for unstructured comments. We compared interview offer rates (self-reported number of interviews/ self-reported number of applications) using paired t tests (Excel), and the lead author (S.D.P.) analyzed the unstructured comments using a 6-phase reflexive thematic analysis. 10 Here, we report a post hoc analysis of de-identified survey data collected for quality assurance/ quality improvement purposes, which the University of California, San Francisco Institutional Review Board categorized as not human subjects research thus exempt from review. All otolaryngology residency programs attested to the code of conduct, and none opted out of the signaling process. By October 21, 2020, the date that applications were released to programs, 611 students had submitted applications to otolaryngology residency programs, 559 applicants had submitted a Match list including an otolaryngology program, and 558 applicants had participated in the signaling process. Of 119 nonmilitary otolaryngology residency programs, 118 received at least 1 signal. The number of signals received per program ranged from 0 to 71 with a mean of 22 (standard deviation 17) and a median of 16 (see Figure 1 ). The survey response rate was 42% (233/558) for applicants who participated in the signaling process and 52% (62/119) for program directors. Respondents strongly favored continuing the signaling process: 77% (178/232) of applicants and 91% (53/58) of program directors. Program directors most commonly reported using signals as a tiebreaker for similar applications and as part of an initial application review algorithm. One program required a signal to offer an applicant an interview. Thematic analysis of the unstructured comments revealed that 18% (22/122) of applicants expressed displeasure with this requirement that a signal was needed to be considered for an interview. The most common theme in applicants' comments (32%, 39/122) was a recommendation to increase the transparency and consistency of how programs use signals. Although our guidance suggested that signals should not be used when determining a rank list, 18% (11/61) of program directors reported using signals for this purpose. Applicants reported applying to a mean of 77 programs (standard deviation 21), including their 5 signaled programs. The rate of receiving an interview offer was significantly higher for signaled programs (58%, 670/1,150) compared with both nonsignaled programs (14%, 2,394/16,520; P < .001) and the comparative nonsignal program (23%, 53/230; P < .001; see Figure 2 ). To assess the impact of signaling across the spectrum of applicant competitiveness, we divided applicants into quartiles based on their overall likelihood of receiving an interview offer. Signals had a significant impact (P < .001) on interview offers across all quartiles (see Figure 3 ). We are particularly enthusiastic that signaling markedly increased applicants' ability to obtain interview offers from programs of particular interest and that this effect was present across the spectrum of applicant competitiveness. Future studies should verify this effect outside self-reported survey data and assess the impact of signals across racial and demographic lines. The magnitude of signal impact we found likely represents both an increased rate of interview offers from signaled programs and a decreased rate of interview offers from nonsignaled programs. In contrast to our 5 signals, the American Economic Association provides 2 signals for graduates of economics PhD programs applying for faculty positions, suggesting that signal scarcity preserves its value and intent. 8 Decreasing the number of signals would force applicants to narrow their list of programs of primary interest and would disincentivize signaling "dream" programs. By increasing the number of signals, the lack of a signal becomes an indication of disinterest. If enough signals are provided, signaling could have a similar impact on the application process as an application cap. The ideal number of signals then must be explored. Our data demonstrated that signaling allowed applicants to influence their likelihood of receiving an interview offer from programs of particular interest. Given the distribution of signals received across programs, we believe that signaling also improved the distribution of interview offers among applicants, which could mitigate interview hoarding. We were unable to directly measure this impact as multiple factors, including virtual interviews, influenced the potential for interview hoarding in the 2021 residency application cycle. For the 2022 residency application cycle, we continued the signaling process. We provided applicants with 4 signals, as they have the opportunity to complete 1 away rotation. In future years, without restrictions on visiting rotations, we will consider providing an additional signal for applicants without a home program. We also expanded the program code of conduct to prohibit both using signaling as a prerequisite for interview consideration and using signals in the postinterview setting, such as during rank list deliberations. While applicants appear to benefit significantly from signaling, they also bear the responsibility of targeting their signals appropriately. The timing of signaling at application submission necessitates that applicants make early program evaluation decisions with limited information. Programs can assist by providing data about historically successful applicants, and mentors can help students assess their standing within the competitive group of otolaryngology applicants. Multiple specialties have expressed interest in adopting a similar program. However, otolaryngology is not representative of all medical or surgical specialties. It is a small, competitive surgical subspecialty with a 63% Match rate and no unmatched residency slots in the 2021 cycle. While the impact of signaling may vary significantly outside of these parameters, we are optimistic that the benefits will carry over to other specialties. Incorporating a signaling option within ERAS would facilitate both wide adoption and further analysis of such a process. Figure 2 Likelihood of an applicant receiving an interview offer from an otolaryngology residency program, by type of program, 2021. Displayed are the rates of interview offers (self-reported number of interview offers/self-reported number of applications submitted) for the overall cohort of applicants, for programs that received a signal, for programs that did not receive a signal, and for comparative nonsignal programs (the program an applicant would have signaled if provided a sixth signal). Likelihood of an applicant receiving an interview offer from an otolaryngology residency program, by applicant competitiveness quartile, 2021. Applicants were divided into quartiles based on their overall likelihood of receiving an interview offer (self-reported number of interview offers/self-reported number of applications submitted). Within each quartile, the rate of receiving an interview offer from a signaled program greatly exceeded that of receiving an interview from the comparative nonsignal program (the program an applicant would have signaled if provided a sixth signal) and the cumulative group of nonsignaled programs. Funding/Support: None reported. Other disclosures: None reported. Ethical approval: This report includes a post hoc analysis of de-identified survey data collected for quality assurance/quality improvement purposes, which the University of California, San Francisco Institutional Review Board categorized as not human subjects research thus exempt from review. Results and Data: 2021 Main Residency Match Otolaryngology match trends: Considerations for proper data interpretation COVID-19's impact on the 2020-2021 resident match: A survey of otolaryngology program directors Otolaryngology match 2020-21: Survey of prospective applicants in the setting of COVID-19 A Novel approach to the National Resident Matching Program The star system A computer simulation model to analyze the application process for competitive residency programs Preference signaling in the National Resident Matching Program The job market for new economists: A market design perspective Preference signaling for the otolaryngology interview market Thematic analysis | a reflexive approach Acknowledgments: The authors thank Emily Maurer, senior administrator at the American College of Surgeons, for her invaluable contributions to the execution of the otolaryngology signaling program. They also thank the Otolaryngology Program Directors Organization and the councils of the Society of University Otolaryngologists and the Association of Academic Departments of Otolaryngology for their support of the signaling program. The authors are grateful to Otomatch.com, Headmirror.com, and the American Academy of Otolaryngology-Head and Neck Surgery for assistance in publicizing the signaling program and to all applicants who participated in its inaugural year. S. Malekzadeh is professor and vice chair for education, Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC.