key: cord-0013481-x74uxdi4
authors: Daniel, Dennis A.; Poynter, Sue E.; Landrigan, Christopher P.; Czeisler, Charles A.; Burns, Jeffrey P.; Wolbrink, Traci A.
title: Pediatric Resident Engagement With an Online Critical Care Curriculum During the Intensive Care Rotation*
date: 2020-06-25
journal: Pediatr Crit Care Med
DOI: 10.1097/pcc.0000000000002477
sha: 88aeec56af20709f5d9324b009a8bc8c7af74377
doc_id: 13481
cord_uid: x74uxdi4

Residents are often assigned online learning materials as part of blended learning models, superimposed on other patient care and learning demands. Data that describe the time patterns of when residents interact with online learning materials during the ICU rotation are lacking. We describe resident engagement with assigned online curricula related to time of day and ICU clinical schedules, using website activity data. DESIGN: Prospective cohort study examining curriculum completion data and cross-referencing timestamps for pre- and posttest attempts with resident schedules to determine the hours that they accessed the curriculum and whether or not they were scheduled for clinical duty. Residents at each site were cohorted based on two differing clinical schedules—extended duration (>24 hr) versus shorter (maximum 16 hr) shifts. SETTING: Two large academic children’s hospitals. SUBJECTS: Pediatric residents rotating in the PICU from July 2013 to June 2017. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: One-hundred and fifty-seven pediatric residents participated in the study. The majority of residents (106/157; 68%) completed the curriculum, with no statistically significant association between overall curriculum completion and schedule cohort at either site. Residents made more test attempts at nighttime between 6 pm and 6 am (1,824/2,828; 64%) regardless of whether they were scheduled for clinical duty. Approximately two thirds of test attempts (1,785/2,828; 63%) occurred when residents were not scheduled to work, regardless of time of day. Forty-two percent of all test attempts (1,199/2,828) occurred between 6 pm and 6 am while off-duty, with 12% (342/2,828) occurring between midnight and 6 am. CONCLUSIONS: Residents rotating in the ICU completed online learning materials mainly during nighttime and off-duty hours, including usage between midnight and 6 am while off-duty. Increasing nighttime and off-duty workload may have implications for educational design and trainee wellness, particularly during busy, acute clinical rotations, and warrants further examination.

I n busy clinical rotations such as the ICU, patients, diagnoses, and clinical acuity vary between rotations, and limited time and competing demands are common for both trainees and faculty (1) . Supplementing the ICU rotation experience with a blended online educational curriculum may help improve knowledge and ensure consistent exposure to core content (2) , but these resources are often superimposed on the demands of patient care and other educational experiences (3) . For our ICU residents, we designed curricula that included short videos with pre-and posttests and hypothesized that residents would use these materials most frequently during breaks in patient care while on clinical duty. However, there is a lack of previously published data that describe when and to what extent resident physicians complete online materials during ICU rotations. Such understanding would help inform decisions about how to best implement educational interventions for residents rotating in the demanding, high-acuity clinical setting of the ICU.

We conducted a prospective cohort study examining junior (post-graduate year 2) resident use of online curricula during their first rotation in medical-surgical PICUs at two large academic children's hospitals (Boston Children's Hospital and Cincinnati Children's Hospital) that are similar in size, patient population, and resident scope of responsibility. This study was approved by the Institutional Review Boards at both sites. ICU resident rotation directors created individualized curricula for each site that covered core concepts in pediatric critical care medicine. Both sites also provided in-person educational experiences (bedside teaching, didactic lectures, and manikin-based simulations). Site 1 delivered in-person education between 07:30 and 08:15 every weekday, and Site 2 did so between 12:15 and 13:00 every Monday through Thursday. We administered the online curricula from July 2013 to June 2015 at Site 1 and June 2015 to June 2017 at Site 2. Neither site provided protected time for curriculum completion. Although residents were informed of the expectation to complete curricula by the end of their ICU rotation, there were no formal consequences for failing to complete. At both sites, residents were e-mailed 2 weeks before their rotation and instructed to complete the curriculum before the end of their ICU rotation, with e-mail reminders provided at rotation weeks 2, 3, and 4. Residents were excluded from the time-of-use analysis if their daily schedule data were unavailable.

Curricula were delivered on OPENPediatrics (www.openpediatrics.org), which is based at Site 1. Each lesson contained a pretest, video, and posttest. The curriculum contained 18 lessons at Site 1 and 21 lessons at Site 2. Individual lesson videos ranged in length from 4 to 36 minutes (average length 18 min). The total curriculum video duration at Site 1 was 5 hours, 12 minutes and at Site 2 was 6 hours, 50 minutes. The website required strictly linear progress; a pretest, then video, then posttest for each lesson needed to be completed before a resident could progress to the next lesson. Residents were only able to take the pretest once, but posttests could be attempted multiple times until the minimum passing score (≥80%) was achieved. The platform allowed residents to stop and restart within preor posttests, as well as within videos, if they did not complete a given item in one sitting.

During the study interval, residents worked within two different clinical schedules as part of a concurrent trial of resident physician work hours Randomized Order Safety Trial Evaluating Resident Schedules (4), where each schedule operated for 1 year of clinical rotations at each site. One schedule cohort involved daytime and nighttime work shifts limited to a maximum of 16 hours of duration, whereas the second involved traditional extended duration (>24 hr) work shifts, with daytime shifts alternating with extended duration work shifts every fourth night. On average, residents worked about 10% more hours per week on the extended duration work schedule (5) . Throughout this article, we refer to these differing schedule cohorts as "short call" and "long call".

We collected curriculum completion data for each resident and timestamps for every pre-and the first posttest attempt that occurred during the ICU rotation and in the 14 days preceding. We did not include test attempts occurring prior to the ICU rotation in the time-of-use analysis due to the significant heterogeneity in resident clinical rotations immediately prior to the ICU rotation. We only included the first posttest attempt to avoid over-representing a given time of day if a user attempted a posttest multiple times. Video viewing activity is captured only in aggregate, deidentified fashion on the platform, so individual video view timestamps were not available for specific residents. For test attempts during the ICU rotation, we cross-referenced timestamps with each resident's schedule to determine whether or not they were scheduled for clinical service in the hospital. We calculated frequencies and percentages for resident and site characteristics and compared data between cohorts and between sites using chi-square tests of independence using a significance level of 0.05. Data were analyzed using Stata/SE 13.1 (StataCorp, College Station, TX) and Microsoft Excel (Microsoft Corp., Washington, DC).

During the study, 157 residents rotated through the ICU for the first time, and 92% (144/157) accessed the curriculum. Fifty-three percent of residents (83/157) accessed the curriculum during the ICU rotation at least once while on duty. Seven residents at Site 2 accessed the curriculum during the rotation but did not have daily schedule data available for analysis. We included 137 residents (137/157; 87%) in the time-of-use analysis. There were no statistically significant differences in demographic characteristics (gender or residency track) between the two sites or between the schedule cohorts within each site.

Sixty-eight percent of residents (106/157) completed the curriculum ( Table 1) . A greater percentage of residents completed the curriculum at Site 2 (53/60; 88%) compared with Site 1 (53/84; 63%), p value of less than 0.001. There was no statistically significant association between overall curriculum completion and schedule cohort at either site.

We included 2,828 test attempts by 137 residents from the two sites in the time-of-use analysis. Of the test attempts made during clinical duty, 40% (418/1,043) occurred during daytime shifts versus 60% (625/1,043) at night. Approximately two thirds of test attempts (1,785/2,828; 63%) occurred when residents were not scheduled to work, regardless of time of day. Approximately two thirds of all test attempts (1,824/2,828; 64%) occurred during nighttime hours (between 6 pm and 6 am), regardless of work status (Fig. 1A) of all test attempts occurring between midnight and 6 am while residents were not scheduled to work.

We observed an association between time-of-use patterns and schedule cohort at Site 1 but not at Site 2. At Site 1, residents in the long call cohort used the curriculum more during nighttime hours (428/553; 77%) compared with residents in the short call cohort (427/769; 56%), p value of less than 0.001. The long call cohort also used the curriculum more during times when they were not scheduled to work (461/553; 83%) compared with the short call cohort (512/769; 67%), p value of less than 0.001. Figure 1B displays the distribution of test attempts by day of rotation, ranging from 14 days prior to the start of the rotation to 31 days after the start date. For test attempts within the ICU rotation, a greater proportion occurred in the second half of the rotation (1,679/2,828; 60%) versus the first half (1,149/2,828; 40%). Three hundred and forty-eight additional test attempts occurred prior to the start of the rotation, the majority of which (264/348; 76%) were in the 2 days immediately prior.

Using 4 years of timestamped online learning data from two large pediatric residency programs, we found that the majority of residents in our study accessed a supplemental online curriculum during the ICU rotation. However, despite designing the ICU curriculum to include short lessons that could be completed during breaks from clinical work while on duty, only half of the included residents accessed the curriculum during clinical periods, and they frequently chose to use the materials during nighttime hours and when not scheduled to work clinically. Notably, 12% of test attempts occurred between midnight and 6 am while residents were not scheduled to work. Test attempts were made immediately prior to, and throughout the rotation, with a majority occurring in the second half of the rotation.

The inconsistent association of completion rate or time of use with schedule cohort makes it less likely that the type of schedule is a main contributor to online curriculum engagement. Because both sites provided in-person educational experiences during most workdays in addition to the online curriculum, it is possible that residents were biased against completing online materials while on-duty, since other materials were already being presented during on-duty hours. Other studies have supported the notion that medical learners often prefer online learning to be supplementary to in-person learning experiences (6, 7) . The greater proportion of on-duty use at night compared with during the daytime may be a consequence of the greater volume of clinical care demands requiring resident attention during the day, including but not limited to rounds and scheduled admissions. The greater number of test attempts in the second half of the rotation may reflect residents catching up on incomplete lessons before the end of the rotation or may be related to residents focusing on getting comfortable in the clinical environment before turning their attention to self-directed learning.

Several studies have highlighted successful implementation and outcomes of online medical learning, noting improvements in knowledge (8) and perceived utility and satisfaction from clinicians and instructors (9) . However, although residents have always incorporated self-directed learning at night and during off hours, blended learning models that increase trainees' obligatory nonclinical workload outside of dedicated educational time may have a different impact. Despite our intention to provide short lessons that would provide education during breaks from patient care, residents accessed the curriculum more often during nonclinical hours. Previous reports have commented on the need to consider the distinct time constraints and serviceeducation task conflicts in graduate medical education (10) and on the risk of creating information overload when educational content is shifted to the online environment (11) . Increases in workload added to preexisting stressors of the clinical learning environment can contribute to resident physician burnout (12, 13) and sleep deprivation. Sleep deficiency is known to adversely impact resident clinical performance (14) and increases risk of physical harm, such as motor vehicle crashes (15) and needlestick/sharps injuries (16) . Therefore, program and rotation directors of busy, inpatient rotations may need to consider alternative approaches to implementing online learning, such as incorporating protected time to complete curricula; making the curriculum optional or controlling the volume of content shifted to the asynchronous, self-directed setting (11, 17, 18) . Our study has several important limitations. As this was a purely an observational study, we did not qualitatively assess resident motivations for why they accessed the curricula at the times they did nor did we formally survey participant satisfaction with the curriculum or any potential impact on their wellness. These are important areas that warrant further investigation. Additionally, not all residents completed the online curricula, and rates of curriculum completion differed between the two sites despite having identical procedures to encourage completion. This may reflect differences in willingness to engage with online curricula during clinical rotations between residents and between the two sites, which may have led to a sampling bias. Interestingly, despite the fact that OPENPediatrics (https:// www.openpediatrics.org/) is primarily based at Site 1, curriculum completion rate was lower at that site, suggesting that there was not increased pressure among residents to use the platform due to the shared institutional affiliation. Although we have no reason to suspect differences in resident roles or workload between the sites, we did not directly assess this either. Curricular length has also been described as a barrier to curriculum completion (19, 20) ; however, we observed higher completion rates by residents at the site with the longer curriculum.

Our data show that residents will engage with online learning materials during and immediately prior to their ICU rotation but do so most often at nighttime and when off-duty, with a portion of use occurring during midnight and 6 am while offduty. This may have implications for resident well-being, including sleep, personal life disruptions, and/or burnout, and warrants further examination. Program and rotation directors will need further guidance on how best to implement blended learning models in busy clinical rotations, such as the ICU.

We wish to thank the residents at Boston Children's Hospital and Cincinnati Children's Hospital for their participation in this study, as well as the administrative staff at both hospitals and at OPENPediatrics who provided coordination and support for this study. Drs. Poynter's, Landrigan's, and Czeisler's institutions received funding from the National Heart, Lung, and Blood Institute (NHLBI), and they received support for article research from the National Institutes of Health. Dr. Landrigan received funding from Midwest Hospital Association/Executive Speakers Bureau and Midwest Lighting Institute; he reports receiving grants from PatientCentered Outcomes Research Institute, consulting fees, and equity from the I-PASS Patient Safety Institute, and consulting fees from Virgin Pulse; and he has received monetary awards, honoraria, and travel reimbursement from multiple academic and professional organizations for teaching and consulting on sleep deprivation, physician performance, handoffs, and safety and has served as an expert witness in cases regarding patient safety and sleep deprivation. Drs. Landrigan and Czeisler report being principal investigators of the Randomized Order Safety Trial Evaluating Resident-Physician Schedules, which is supported by grants (U01-HL-111478 and U01-HL-111691) from the NHLBI. Dr. Czeisler serves as the incumbent of a Harvard Medical School professorship that was endowed in 2004 by Cephalon, Inc., which has been since been acquired by Teva Pharmaceutical Industries Ltd., and he is supported in part by funding from the National Institute of Occupational Safety and Health R01-OH-010300. Dr from Ganésco and Zurich Insurance, and fees for serving as a member of an advisory board from the Institute of Digital Media and Child Development and the Klarman Family Foundation, holding a number of process patents in the field of sleep and circadian rhythms (e.g., photic resetting of the human circadian pacemaker) and an equity interest in Vanda Pharmaceuticals, being the incumbent of an endowed professorship provided to Harvard University by Cephalon, receiving fees for serving as an expert on various legal and technical cases related to sleep or circadian rhythms from Casper Sleep, Comair/Delta Airlines, Complete General Construction, FedEx, Greyhound, HG Energy, Purdue Pharma, South Carolina Central Railroad, Steel Warehouse, Stric-Lan, Texas Premier Resources, and United Parcel Service, and receiving royalties from the New England Journal of Medicine, McGraw-Hill, Houghton Mifflin Harcourt/Penguin, and from Philips Respironics for the Actiwatch 2 and Actiwatch Spectrum devices. Dr. Czeisler's interests were reviewed and managed by Brigham and Women's Hospital and Partners HealthCare in accordance with their conflict of interest policies. The remaining authors have disclosed that they do not have any potential conflicts of interest.

This study was approved by the Institutional Review Boards at both participating sites prior to data collection and analysis.

For information regarding this article, E-mail: dennis.daniel@childrens.harvard.edu

Balancing service and education in residency training: A logical fallacy

Flipped classrooms in graduate medical education: A national survey of residency program directors

A systematic review of the effectiveness of flipped classrooms in medical education

ROSTERS Study Group: Design and recruitment of the randomized order safety trial evaluating resident-physician schedules (ROSTERS) study

Effects on resident work hours, sleep duration, and work experience in a randomized order safety trial evaluating resident-physician schedules (ROSTERS)

Blended learning: How can we optimise undergraduate student engagement

The effectiveness of online and blended learning: A meta-analysis of the empirical literature

Internet-based learning in the health professions: A meta-analysis

The impact of E-learning in medical education

Flipping out: Does the flipped classroom learning model work for GME?

Twelve tips for "flipping" the classroom

A narrative review on burnout experienced by medical students and residents

Effect of reducing interns' work hours on serious medical errors in intensive care units

Extended work shifts and the risk of motor vehicle crashes among interns

Resident wellness matters: Optimizing resident education and wellness through the learning environment

Effects of health care provider work hours and sleep deprivation on safety and performance

Advances in medical education and practice: Student perceptions of the flipped classroom

Impact of required versus self-directed use of virtual patient cases on clerkship performance: A mixedmethods study

Efficacy of an asynchronous electronic curriculum in emergency medicine education in the United States

Massive open online course completion rates revisited: Assessment, length and attrition