key: cord-0905214-2io4xhim
authors: Mack, Jacob A.; Morgan, Helen K.; Fitzgerald, James T.; Walford, Eric C.; Heidemann, Lauren A.
title: The Development of a Video Intervention to Improve Senior Medical Students’ Performance on Outpatient Telephone Encounters: a Delphi Analysis and Randomized Controlled Trial
date: 2021-06-21
journal: Med Sci Educ
DOI: 10.1007/s40670-021-01331-w
sha: 072a42fc0bbf95b7805e6ec37fc492507f223abd
doc_id: 905214
cord_uid: 2io4xhim

INTRODUCTION: Postgraduate trainees address outpatient telephone calls (OTCs) with little prior training. This study determines the skills necessary for OTCs and examines whether a video intervention improves medical students’ performance on simulated OTCs. MATERIALS AND METHODS: We utilized a Delphi technique to determine skills needed for OTCs and created a 9-min video teaching these skills. Senior medical students were randomized to Intervention (viewed video) and Control (did not view video) groups. Students were assessed pre-/post-intervention on simulated OTCs. The primary outcome was the between-group difference in improvement. RESULTS: The Delphi yielded 34 important skills with the highest focus on communication (n = 13) and triage (n = 6). Seventy-two students completed assessments (Control, n = 41; Intervention, n = 31). The score (mean ± SD) improved 4.3% in the Control group (62.3 ± 14.3% to 66.6 ± 25.0%) and 12.2% in the Intervention group (60.7 ± 15.2% to 72.9 ± 20.4%, p = 0.15). The effect size measured by Cohen’s d was 0.55, considered effective (> 0.33) for an educational intervention. CONCLUSIONS: This project fills a gap in OTC training. The use of the Delphi technique, intervention development based on the results, and evaluation of efficacy is a process that could be reproduced for other educational gaps. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01331-w.

New postgraduate trainees are often assigned telephone medicine responsibilities [1, 2] with little prior training [3, 4] . This may occur when covering an assigned panel of continuity clinic patients or when working after-hours (e.g., night float) [2] . Telephone encounters address a variety of patient issues such as answering questions, addressing concerns, communicating results, and discussing plans. Moreover, the importance of telemedicine ("the use of electronic information and communications technologies to provide and support health care when distance separates participants," which includes synchronous telephone medicine [5] , the focus of this study) is increasing dramatically due to the coronavirus disease 2019 (COVID-19) pandemic [6, 7] . Given the multiple challenges of remote communication, outpatient telephone calls (OTCs) have been shown to increase the potential for medical errors [1, [8] [9] [10] . As such, postgraduate trainees express discomfort with this task [11, 12] .

There is a need for telephone medicine curricula to be incorporated in medical school [13] . The importance of telephone medicine has been recognized by the Accreditation Council for Graduate Medical Education Program Requirements for Family Medicine programs, which specifies that postgraduate trainees engage in telephone visits (IV.C.4.f.) [14] and incorporate telephone medicine as a Communication milestone (C-4) [15] . Additionally, other health professions such as nursing recognize the importance of telehealth and offer formal training and telehealth credentialing [16] . However, there is a general lack of guidelines or curricula available targeting learners at the medical student level [13] . The few existing guidelines pre-date the current ubiquitous use of Electronic Medical Records (EMR) [17, 18] or are time-intensive [19] . Therefore, there is a need for an updated OTC curriculum prior to the start of postgraduate training.

One well-described way to increase preparedness for postgraduate training is through Residency Preparation Courses (RPCs), which are courses at the end of medical school intended to better prepare senior students for postgraduate expectations [20, 21] . A strategy utilized in RPCs for both education and assessment is a mock paging curriculum in which students receive pages with simulated scenarios and must communicate accordingly with either a simulated nurse (inpatient scenarios) or a simulated patient (outpatient scenario). Mock paging curricula are generally well-received by students [22] [23] [24] [25] [26] and have been shown to improve confidence [27, 28] , decrease anxiety [27] , improve clinical decision making [25, 29, 30] , improve communication skills [25, 27, 31] , and improve attitudes toward interprofessional communication and collaboration with nurses [32] .

We hypothesized that an educational intervention directed specifically at OTCs would improve student performance on mock outpatient pages (i.e., simulated OTCs). Our study addressed this question by first utilizing a Delphi technique to determine the OTC skills important to teach students. Second, we used this information to develop a brief video educational intervention. Finally, we tested the efficacy of the educational intervention with a randomized controlled trial involving senior medical students enrolled in RPCs at a single institution.

The Delphi technique is a method used to establish consensus among a group of experts for a given issue. It has been used in medical education extensively with good effect in multiple studies in both the medical and nursing education literature [33] [34] [35] . We performed a single-center, conventional, 2-round Delphi with electronic surveys created and distributed using Qualtrics survey software (Qualtrics International Inc., Provo, UT, USA).

We invited seven faculty members and eleven postgraduate trainees from six different fields (Family Medicine, Internal Medicine, Internal Medicine-Pediatrics, Obstetrics and Gynecology, Pediatrics, and Surgery). They were invited based on outpatient clinical experience, experience with trainees, and experience in transition-to-postgraduate education. Residents were invited because of their relatively recent experience transitioning from medical school to postgraduate training.

Respondents were asked to provide free-text responses to three questions regarding what an intern should know to address OTCs:

1. "What are the most important skills for interns to know when interacting with outpatients over the phone?" 2. "What are the most frequent challenges for which interns should be prepared when interacting with outpatients over the phone? What are the skills interns need to address these challenges?" 3. "What other things should be included in a video to teach interns about interacting with outpatients over the phone?"

The authors developed these questions based on their clinical experience. Similar responses were collated by one author and verified by another. Skills were categorized according to themes (e.g., triage, communication, medical knowledge, etc.). See Appendix 1 for the full survey.

The skills identified in the first round were sent via Qualtrics, and respondents were asked to rank the importance of the skills on a 4-point Likert scale ("Not at all important," "Slightly important," "Very important," or "Extremely important"). A neutral middle point was excluded to compel participants to choose either importance or unimportance and to assist in calculations for consensus [33] . The survey in the second round also included questions regarding respondents' demographics and clinical experience (Appendix 2).

Results were analyzed according to importance (important or not important) and degree of consensus (high consensus, approaching consensus, no consensus). High consensus for a skill being important was defined as ≥ 80% of respondents agreeing on a skill being "Extremely" or "Very important" [33] . Approaching consensus for a skill being important was defined as 50-79% agreeing on a skill being "Extremely" or "Very important." High consensus for a skill not being important was defined as ≥ 80% of respondents agreeing on a skill being "Not at all" or "Slightly important." Approaching consensus for a skill not being important was defined as 50-79% agreeing on a skill being "Not at all" or "Slightly important."

The skills selected by the Delphi process were incorporated into a video intervention. This 8:38-min video presented a schematic and algorithmic framework by which to address OTCs. Select images from the video are available (see Fig. 3 and Supplementary Information) . The video was a narrated Google Slides (Alphabet Inc., Mountain View, CA, USA) presentation recorded with Camtasia (TechSmith, Okemos, MI, USA).

Participants were senior medical students who were enrolled in specialty-specific RPCs (Family Medicine, Internal Medicine, Obstetrics and Gynecology, Pediatrics, or Surgery) in February and March 2019 based on their career interests. Participants were excluded from analysis if they dropped out of the RPCs at any time, or if they did not complete (e.g., due to other scheduled RPC requirements) or results were not available for both the pre-and post-intervention simulated OTCs. See Fig. 1 for further details of study population.

Students were assessed on their ability to manage OTCs based on their performance responding to simulated OTCs within a larger mock paging curriculum. The assessment used trained registered nurses acting as standardized patients to page students with simulated outpatient scenarios; students would call the standardized patient to address their concern. These simulated OTCs have been used for several years as a part of our institution's RPCs. The scenarios presented in these simulated OTCs are meant to represent outpatient scenarios common to the RPC specialty (e.g., the scenarios are different for Surgery compared to OB/GYN). Students were graded according to a previously established expert-validated rubric based on a weighted checklist for each case. They earned points for what they "Must do" and "Should do" and were penalized points for what they "Should not do" and "Must not do." These points contributed to an Overall Score which was reported as a percentage. See Appendix 3 for an example case.

Students were stratified within their respective RPC and randomized into two groups, Intervention and Control. Students were randomized using a random number generator within Google Sheets (Alphabet Inc., Mountain View, CA, USA). Participants were blinded to randomization. The Control group received "usual care," i.e., there was no placebo. The nurses (acting as standardized patients) were the outcome adjudicators and data collectors in that they evaluated the students according to the primary outcome and recorded their scores; the nurses were blinded to group assignments. The author who acted as data analyst was not blinded to group assignments; however, he did not participate in the data collection or any outcome adjudication.

Students received a total of two simulated OTCs. Both groups received Mock Page 1 over the course of 1 week and were scored by the adjudicator according to the rubric for the case. The Intervention group was then sent a link to the educational video via Qualtrics. The Qualtrics survey recorded whether a student opened the link to the video. The following week, both groups then received Mock Page 2 and were again scored. After the conclusion of the simulated OTCs, the Control group was sent the link to the educational video to view for edification (see Fig. 1 ).

After viewing the educational video, students were automatically redirected to two free-text survey questions:

1. What did you learn from this video that you did not learn previously during medical school? 2. What feedback do you have to improve this video for future students?

The primary outcome was the mean improvement in overall score pre-and post-intervention of Intervention compared to Control. The secondary outcome was the perceived value of the video intervention based on survey comments about what skills were learned.

The primary outcome was examined using an intentionto-treat analysis. Differences between groups were determined using a multivariate repeated measures analyses for significance (p ≤ 0.05) and Cohen's d for effect size. The threshold for an educational intervention to be considered effective is a Cohen's d > 0.33 [36] . The data were analyzed using JMP Pro 14.2.0 (SAS Institute, Inc., Cary, NC, USA).

Respondents included 4/7 faculty (57%) and 8/11 postgraduate trainees (73%) in four specialties (Internal Medicine, n = 5; Medicine-Pediatrics, n = 1; Obstetrics and Gynecology, n = 3; Surgery, n = 3) for a total response rate of 67%. See Table 1 for respondent characteristics. The first round of the Delphi process resulted in 144 ideas. These ideas were collated into 40 teachable skills within the categories of communication with the patient (n = 13), triage (n = 6), medical knowledge (n = 5), gathering information (n = 4), logistics of telephone encounters (n = 4), documentation/communication with other providers (n = 3), available resources (n = 3), prescribing medications (n = 1), and follow-up (n = 1). Eighteen skills were important, with high consensus. Sixteen skills were important, approaching consensus. Five skills were not important, approaching consensus. One skill was not important, with high consensus. All Table 2 ).

Eighty-nine students enrolled in the RPCs in February and March 2019. Eighty-five were stratified and randomized into Control (n = 46) and Intervention (n = 39) groups. Forty-one students in the Control group and 31 students in the Intervention group completed pre-and post-intervention assessments (Fig. 1) . Of the 31 students that completed both preand post-intervention assessments in the Intervention group, 26 (84%) accessed the link to view the video.

The scores for both groups improved; Control group scores increased from 62.3 ± SD 14.3% to 66.6 ± 25.0% and Intervention group scores increased from 60.7 ± 15.2% to 72.9 ± 20.4% (p = 0.15). The intervention had a notable effect as indicated by the medium effect size (Cohen's d = 0.55). Values are tabulated in Table 3 and scores are plotted in Fig. 2 .

Of the 77 students who accessed the video link, 60 students (78%) responded to the first free-text question and 47 (61%) responded to the second question. Students learned skills in categories that are reflective of those that emerged from the Delphi including: gathering information, communication with the patient, triage, available resources, and documentation / communication with other providers ( Table 4 ). The most common suggestion to improve the video was to include more specific case examples during the video or in follow-up material that address common situations (n = 21) (Fig. 3) . Students had differing levels of experience prior to the RPCs. Some students had little experience, as in "I've never had formal teaching in medical school on how to handle outpatient telephone encounters…" and, "I think the biggest thing is just having a framework for approach to the more remote management of outpatients because it's not something we routinely get exposure to in medical school." Another said, "It was a nice template for a type of patient encounter that we get little-to-no explicit training…during medical school." Other students had more consequential prior experiences. For example, "…I feel like I learned this in school and was fortunate to have experience practicing this during my clerkships a couple of times." Another student had experience based on their extracurricular activities: "I actually do feel like I've had a lot of practice with managing outpatients remotely in medical school as a result of my work with the Student-Run Free Clinic…I oversee all of the labs that come back for patients, create medical plans, and communicate those to patients. So luckily, I feel like I am already really familiar with the concepts in this video." 

In this study, we demonstrated an evidence-informed process by which to develop an educational intervention related to managing OTCs and proved its efficacy based on a previously validated assessment. We demonstrated that the video was effective in improving student performance responding to simulated OTCs through the level of results according to the Drs. Kirkpatrick [37] .

The Delphi technique showed utility in establishing consensus on a topic without available guidelines as an established way to "crowdsource" ideas from experts, as has been demonstrated previously in medical education literature [33, 34] . One particularly relevant example is a study by van Houwelingen et al. who used the Delphi technique to determine entrustable professional activities for nurses in the telehealth setting [35] . In our study, the skills deemed most important and with highest consensus were related to communication, followed by triage and logistics. This highlights the unique skillset involved in addressing OTCs beyond just the medical knowledge of outpatient medicine. Strengths included the variety of specialties represented, and the differing levels of experience that included early experienced clinicians as well as postgraduate trainees in order to create an educational video informed by different perspectives and that would be relevant regardless of students' intended specialty. Our study also has relevance for a scoping review which is currently under protocol by Cooper and Alexander of the Joanna Briggs Institute seeking to examine the current evidence on the "components, skills and training required for conducting initial telephone consultations [38] . This scoping review will, reciprocally, put our results in the greater context of the current state of medical and allied health professionals training.

Our intervention to improve telephone medicine skills complements those designed by other researchers. Seal et al. designed a 1-h in-person training session and measured the frequency of gathering important information and self-reported student confidence during simulated telephone calls. They found that, compared to the control group, students who had completed the training gathered important information more frequently and had greater self-reported confidence in their performance [31] . Saba et al. designed a study in which students attended a 2-h in-person training session, followed by telephone calls with real patients, and finally a telephone call with a standardized patient. They found that, compared to the control group, students who had completed the program scored better on some measured communication skills [39] . Contrasting our intervention with the work of these researchers, the development of a video intervention rather than an in-person lecture was useful as it promoted standardization (i.e., can be used in the future EMR electronic medical record, RN registered nurse, LPN licensed practical nurse, MA medical assistant, SW social worker, PharmD Doctor of Pharmacy regardless of lecturer availability or experience), flexibility of scheduling learning, and repeat viewing if desired. Further, while a considerable amount of time is required to initially develop the video, it requires no subsequent resources from educators. This also conforms with suggestions for providing self-directed learning materials [40] in response to the trial by Seale et al. [31] .

While a rapid pivot to remote learning is important (especially in the context of the COVID-19 pandemic) [7, 41] , it is essential to test the efficacy of learning interventions. There is certainly debate regarding the utility of randomized controlled trials (RCTs) in medical education [42] [43] [44] [45] , but it is generally accepted that trial design should reflect the nature of the question and should consider local Table 4 Skills that students reported learning from the video intervention that were otherwise not learned during medical school

Available resources To talk with nurses, residents, and attendings for advice about common calls and for help when needed (n = 5) To rely on ancillary staff and know how they can help (n = 2) Communication with the patient To/How to introduce themselves over the phone (n = 5)

To ask the patient if s/he is able to talk freely and whether it is a good time to talk (n = 3) To use the teach back method (n = 2) Documentation/communication with other providers What information to include in a phone note (n = 4) To route the telephone note to the patient's primary care physician or specialists (n = 3)

Ask the patient to provide vital signs to gather some objective date (n = 6) To do a full but brief History of Present Illness with patients (n = 5) To suggest that the patient send photos through the EMR (n = 3) That remote encounters are similar to in-person encounters (n = 3) To gather information from the EMR in addition to speaking with the patient (n = 2) Triage

The importance of and how to triage outpatients over the phone (n = 9) Follow-up Know how to send a patient to the ED (i.e., the logistics for doing so) Other A general structured outline and approach for handling outpatient calls, even if all of the individual concepts had previously been taught (n = 22 students) context [44, 46] . We chose our trial design based on our ability to perform the study while all students at a single institution were completing their RPCs during the same two-month period, which lessens the problem of different training experiences and asynchrony often seen in graduate medical education [43] . Additionally, e-learning such as our video intervention is particularly suitable to RCTs [46] in that it standardizes the "teacher" [43] and therefore leads to a relatively simple intervention to improve a relatively narrow problem [47] . Furthermore, all students have access to the educational video in a timely manner and hence promotes a more equitable learning experience. Student feedback uncovered a significant number of skills that had not been acquired elsewhere in medical school. Students recounted differing levels of exposure to OTCs, which illustrates the need for an educational intervention prior to the start of postgraduate training such as the one created in this study (or ones developed by other researchers [31, 39] ).

There were a number of limitations to our single center trial. Limitations of the Delphi included the number of participants; the optimum number of participants is 15-30 [33] , and the current study had 12 respondents. Further, none of the invited participants from Family Medicine or Pediatrics responded, which may limit generalizability. There were also limitations to the randomized controlled trial. First, there were a relatively small number of participants. Second, it is unknown how differing scores on mock paging cases correspond to clinical relevance or how student performance translates to performance during postgraduate training. Third, history bias was not controlled (i.e., students may have been exposed to other lectures regarding OTCs during the course of the trial); however, this was mitigated by randomizing students within each RPC group. The use of the intention-to-treat analysis led to including 5 students in the Intervention group who did not view the intervention, 4 of whom had an increase in their scores and one of whom had a decrease. This increases the probability of a Type I error (falsely rejecting the null hypothesis that our intervention has no effect on student performance). Finally, our video intervention addressed general telephone medicine skills, whereas our measurement (i.e., the grading rubric) addressed the specific skills relevant to each simulated OTC. That is, our intervention did not perfectly match our measurement. This was an intentional decision so as to make our intervention more broadly applicable to outpatient telephone calls, while still taking advantage of the expert-validated grading rubrics that our institution has used for many years. Further, the fact that our intervention was not designed to improve the specific skills graded on the rubric reduces the probability of a type II error. 

In conclusion, this study illustrated a process by which to develop an educational intervention addressing OTCs and demonstrated its efficacy. In the future, we plan to re-record the video based on student feedback and expand the study to a larger group of students.

The online version contains supplementary material available at https:// doi. org/ 10. 1007/ s40670-021-01331-w.

Patient safety in after-hours telephone medicine

Can you hold please? How internal medicine residents deal with patient telephone calls. Telephone Encounters Learning Initiative Group

Telephone management training in internal medicine residencies: a national survey of program directors

Telephone medical care administered by internal medicine residents: concerns of program directors and implications for residency training

Health and Public Policy Committee of the American College of Physicians. Policy recommendations to guide the use of telemedicine in primary care settings: an American College of Physicians position paper

Rapidly Converting to "Virtual Practices": Outpatient Care in the Era of Covid-19. NEJM Catal Innov Care Deliv n

Medical education adaptations: really good stuff for educational transition during a pandemic

Preventing communication errors in telephone medicine

Safety of telephone triage in out-of-hours care: a systematic review

Patient safety and telephone medicine : some lessons from closed claim case review

Issues affecting residents' attitudes about telephone medicine for ambulatory patients

Physician attitudes regarding telephone medicine

Improving quality and safety of telephone based delivery of care: teaching telephone consultation skills. Qual Saf Health Care

ACGME Program Requirements for Graduate Medical Education in Family Medicine

Accreditation Council for Graduate Medical Education (ACGME)

Telehealth nursing: More than just a phone call

Effectiveness of a curriculum in telephone medicine

Telephone medicine for internists

Training residents to work in a patient-centered medical home: what are the outcomes?

Is there a role for internal medicine residency preparation courses in the fourth year curriculum? A single-center experience

Core trainee boot camp, a method for improving technical and non-technical skills of novice surgical trainees. A before and after study

Mock paging and consult curriculum to prepare fourth-year medical students for medical internship

Development, organisation and implementation of a surgical skills "boot camp": SIMweek

A senior elective designed to prepare medical students for surgical residency

A Paging Curriculum to Prepare Learners for Obstetrics and Gynecology Residencies

Mock Surgical Paging Exercises Improve Medical Student Communication with Nurses, Confidence and Decrease Anxiety

Use of simulated pages to prepare medical students for internship and improve patient safety

Mock pages are a valid construct for assessment of clinical decision making and interprofessional communication

How prepared are your interns to take calls? Results of a multi-institutional study of simulated pages to prepare medical students for surgery internship

Training final year medical students in telephone communication and prioritization skills: An evaluation in the simulated environment

Attitudes toward communication and collaboration after participation in a mock page program: a pilot of an interprofessional approach to surgical residency preparation

The Delphi technique in health sciences education research

Inpatient cross-cover consensus recommendations for medical and surgical residents: a delphi analysis

Ten Cate O. Competencies required for nursing telehealth activities: A Delphi-study

Handbook in research and evaluation: a collection of principles, methods, and strategies useful in the planning, design, and evaluation of studies in education and the behavioral sciences

Evaluating training programs: the four levels

Components, skills and training for conducting initial telephone consultations in primary care: a scoping review protocol

Teaching patient-centered communication skills: a telephone follow-up curriculum for medical students

Telephone communication: a new Mini-CEX?

Medical education during pandemics: a UK perspective

Educational research and randomised trials

Getting off the "gold standard": randomized controlled trials and education research

Broadening the debate about quality in medical education research

RCT = results confounded and trivial: the perils of grand educational experiments

Is experimental research passé

Complex interventions: how "out of control" can a randomised controlled trial be?

The authors would like to thank Whitney