key: cord-1003088-4xk4insn
authors: Shah, Neeral L.; Miller, Jennifer B.; Bilal, Mohammad; Shah, Brijen
title: Smartphone Apps in Graduate Medical Education Virtual Recruitment During the COVID-19 Pandemic
date: 2021-02-09
journal: J Med Syst
DOI: 10.1007/s10916-021-01720-z
sha: 7d378837413273ea0ae322fd0c080da227ef0fe0
doc_id: 1003088
cord_uid: 4xk4insn

The COVID-19 pandemic required a shift of graduate medical education recruitment to a virtual format. In order to share information and insight into the culture of our program with applicants, we created a smartphone app for those that were invited for an interview. By collecting the analytics of the app, we were able to follow trends in the timing of applicants downloading the app, their viewing histories, and when information was accessed. The app was mostly downloaded at the time of the interview invite or 48 h prior to the interview day. Around the interview day, applicants tended to access trainee profiles, faculty profiles, and videos about the program and community. Closer to the rank list due date, training information, the graduate medical education (GME) documents, and the diversity and wellness initiatives seemed to have more activity. This analysis from the use of a smartphone app in virtual recruitment gives insight into the use of a smartphone app by applicants, and the information that they find useful during the process.

The COVID-19 pandemic impacted several aspects of graduate medical education including moving the fellowship and residency interview season to a virtual format [1] [2] [3] . Through a nationwide video conference call between gastroenterology fellowship program directors, the authors of this article brainstormed ideas. We sought to create an appealing virtual experience and incorporate innovative techniques to ensure a user-friendly process for applicants [4] . One strategy was the development of a smartphone application (app) to share in-depth information with applicants invited to interview. We aimed to evaluate the incorporation of a smartphone app on the virtual GI fellowship interview season.

Planning involved choosing an application to design our program's smartphone app. The web-based app design program provided multiple templates for individual pages allowing customization, editing portability, and rapid update publishing. Cost was a major factor in our decision. Pricing for an app ranged from $300 to $600 per year.

During the design process, 12 fellows and 10 faculty provided suggestions as test users. Based on feedback, it was evident the app was a dynamic tool, hence, it was geared to complement the program website. To create a connection to our trainees, current fellows recorded an informal video tour, which oriented applicants to the hospital and work spaces. The applicants could view a welcome video from the program 

We used analytics of the app to track use (downloads, views, individual page traffic). This information helped us to understand when the applicants utilized the app and what pages were used most frequently. Applicants had an option to complete a pre-interview survey to assist with feedback on the functionality of the app. A post-interview survey was sent to the applicants a day prior to the rank list due date.

There were three time periods that applicants downloaded the app. About 23% (9/40) downloaded the app within 48 h of receiving the interview invite. Seventeen percent (7 of 40) downloaded the app within 10 days of receiving the invite, while, the remaining 60% (24/40) downloaded the app within the 48 h before interviewing. (Figure 1 ) Peak views of the app occurred the afternoon prior to an interview day (between 4 pm -9 pm) and immediately after the interview (between 1 pm -8 pm). The top pages of interest were: GI Fellows (599 views), GI faculty (480 views), Program Leadership (343 views), and Interview Day (182 views). (Table 1) .

Of the 40 applicants interviewed, 25 (65%) responded to the pre-interview survey. Emerging themes from the free text responses regarding the most useful aspects included videos that provided insight into the culture (32%), program training information (32%), and program director's welcome video (16%). Information that applicants felt was missing were biographical information about current fellows/faculty (32%), fellow/faculty contact information (12%), research projects (4%), and post-training career information (4%). After the final interview day, we tracked the app use until the day prior to rank lists were due. Views of the app demonstrated two peaks after completion of interviews. The first peak (seven views) occurred ten days after the last interview day. Over the two weeks prior to the rank list due date, we noted an upward linear trend of app views: two weeks prior to the due datetwo views, one week priorsix views, and the day prioreleven views.

The post-interview survey was done prior to the rank list due date. We received 15 responses (38%) from 40 applicants. Of particular interest, was the applicant direct comparison of an app to web pages. ( Table 2 ) Respondents indicated that they were 53% more likely to review the program information included on the app than on the web. Additionally, 86% found the app slightly easier or much easier to use than a web page. None of the respondents thought the web was easier to use. When comparing content, 80% found the app content of higher or slightly more value than information on the web. Reflecting on the most useful pages of the app, program information was indicated by 53%, and the GI videos, fellows' profiles, and fellow newsletters /QI projects were chosen by 13% of respondents. Eighty seven percent of respondents indicated a preference of having an app in addition to a website, but 87% were also opposed to all aspects of the interview to run solely through the app. Finally, 33% indicated that the app has no effect on their rank list, but 47% reported a moderate effect on their final rankings.

The overall themes from free text responses indicated applicants felt the app was a good gesture to the commitment of recruitment (13%), and provided easy access to the program information (13%). However, some felt the app was too small on the phone, and that there is a need for emails as a method of record keeping (7%).

Overall, applicants praised the easy usability, readily and quickly obtainable program information, and uniqueness of the app. The ability to access information on a phone, as opposed to using a web browser, was universally appealing. Program information on the app was accessed after invites were sent and within the 48 h prior to the interview day. To our knowledge, this is the first time this metric has been tracked from applicants. This timing should be considered to ensure information is up to date, and to consider pushing alerts or content prior to the interview day. As applicants envision their future workplace, personnel pages with fellows, faculty and leadership videos seemed to provide the most information, and therefore should be easily accessible to applicants prior to their interview.

Conversely, in the post-interview period, the more commonly viewed pages included training information, the graduate medical education (GME) documents, and the diversity and wellness initiatives. We assume this is due to applicants directly comparing aspects of multiple programs prior to submitting rank lists.

Finally, almost half of the applicants reported the app had a moderate effect on their rank list. We postulate that having program information readily available at the time of making rank list is helpful [5] . The interview trail can be long and it can be difficult to remember the fine details of each program. Furthermore, divisional websites and the traditional "interview day program folders" can be challenging to navigate, so having an app focused on the program provides an important resource.

By integrating new technological methods to enhance the experience, we found it is possible to carry out a successful virtual interview day. The addition of a smartphone app provides an innovative source of information for applicants. Many programs already utilize program websites [6, 7] , which may not add additional cost to the virtual process, however the accessibility to make changes and flexibility to organize information is often limited by institutional barriers and approvals. Further, we were able to provide more personal information on the app (i.e. videos of the fellows, pictures from social events) that we would not include on a publicly available webpage. While there are variations in each program's resources, our surveys show that providing applicants readily accessible and organized program information may help with rank list decisions. Despite the challenging times our graduate medical education community faces, embracing new methods such as a smartphone app can enhance the interview experience. A continual examination of this process will be necessary as some combination of virtual and face to face interviews will likely be the way recruitment is conducted in the upcoming years. Future direction may include the identification of one or a few app development programs to be used by program directors, which may provide uniformity for applicants entering this process. Moreover, based on feedback from applicants each year, future versions of the app can be updated easily to include information that will be useful to applicants.

Code availability (software application or custom code) N/A Authors' contributions NSconcept, writing, editing, JBconcept, writing, editing, MBediting, BSediting.

Data availability Available upon request.

Declarations Conflict of Interest: All authors have no financial or non-financial conflicts of interest with this manuscript.

Ethical approval This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of interest None.

The New Virtual Reality: How COVID-19 Will Affect the Gastroenterology and Hepatology Fellowship Match

Impact of COVID-19 Pandemic on Training: Global Perceptions of Gastroenterology and Hepatology Fellows in the USA

Proposed Changes to the 2021 Residency Application Process in the Wake of COVID-19

Virtual Interviews in the Era of COVID-19: A Primer for Applicants

Aspects of residency programs' web sites important to student applicants

Evaluation of Integrated Interventional Radiology Residency Websites

The Content and Accessibility of Orthopaedic Residency Program Websites

Division of Gastroenterology and Hepatology, University of