key: cord-1007103-a33q5kle authors: Beam, Elizabeth L.; Herstein, Jocelyn J.; Kupzyk, Kevin A.; Gibbs, Shawn G. title: A simulation approach to measure critical safety behaviors when evaluating training methods for respirator education in healthcare workers date: 2020-05-12 journal: Am J Infect Control DOI: 10.1016/j.ajic.2020.05.005 sha: 2ac7e5d18174824e6c2fbcb703fe48e83b669667 doc_id: 1007103 cord_uid: a33q5kle BACKGROUND: The N95 respirator is the most common safety tool used in hospitals to protect healthcare workers (HCW) from inhaling airborne particles. Focusing on HCW behavior related to respirator use is an effective route to improve HCW safety and respiratory health. METHODS: Participants were asked to perform the donning and doffing of an N95 respirator to camera. Then they were randomized to a video alone or a reflective practice intervention. After the intervention they repeated the donning and doffing to camera. A critical safety behavior scoring tool (CSBST) was developed to compare the performance of the participants over time at pre-test, post-test and one month later for follow-up. RESULTS: The reflective practice intervention group was found to have significantly higher scores on the CSBST at post-test and follow-up than the video alone group. In the reflective practice intervention group, the participants perceived they were better at performing the N95 donning and doffing than the experts scored them. CONCLUSIONS: The CSBST is a tool to measure the performance of HCWs on a specific targeted safety behaviors. The addition of a reflective practice intervention may result in a measurable and sustained improvement in the safety behaviors demonstrated when using the N95 respirator. The N95 respirator is the most common safety tool used in hospitals to protect healthcare workers (HCW) from inhaling airborne particles. This became even more evident during the COVID-19 pandemic. The challenges of maintaining an effective respiratory protection program (RPP) are noted in the literature in statewide and regional evaluations. [1] [2] [3] One of the problems is that healthcare workers do not use the respirators properly. 2 The Centers for Disease Control and Prevention (CDC) focuses on the use of N95 filtering facepiece respirators as the common tool for HCW protection from pandemic and infectious respiratory illness. 4 There is a continued need for more tolerable respirator designs in healthcare, 5 but innovation in this area is slow. An educational research focus on HCW behavior related to respirator use is an effective route to improve HCW safety and respiratory health at this time. Outside the healthcare setting, recent studies funded by the National Institute for Occupational Safety and Health (NIOSH) have analyzed training, evaluated different types of training methods, and measured persistence of learning related to respirator use for both filtering facepiece N95 respirators and dual cartridge half facemask elastomeric respirators. [6] [7] [8] Findings from a component analysis found training of respirator users improved knowledge, but there were common challenges in performance related to seal checking and avoiding facemask contact with removal. 6 When comparing training methods, video training that included illustrations, text, and demonstration was found to be a superior training method to printed materials or computer-based training with additional review questions, both of which were self-paced. 7 The work included an evaluation of the persistence of respirator donning and doffing techniques at 6 months, and performance scores became significantly worse. 8 Educational research additionally supports video review as an effective tool for learners with various levels of experience, but it emphasizes the importance of evaluation metrics or specific benchmarks. [9] [10] Behavior related to respirator use may be improved by structuring observations through specific objective measures instead of simply reviewing video of performance. 9 Effectiveness of video feedback is enhanced when a standard evaluation form is provided to the trainee to focus their concentration on specific targeted behaviors. 10 A systematic review in 2019 found that simulation studies regarding personal protective equipment (PPE) use need to focus on increasing the number of study participants and determining best training methods. 11 Recent studies have found poor doffing behaviors in isolation care using trained observers in the clinical environment. 12 Simulation has become a feasible means to evaluate PPE infection control behaviors in HCWs. [13] [14] Simulation has also been used to measure the potential for fomite-based transmission during donning and doffing maneuvers of respirators. 15 A detailed analysis of respirator-associated behaviors from video of simulated patient care scenarios was used to identify critical safety behaviors while using a respirator. 16 This study compared two N95 respirator training methods using a simulation-based approach. One group of participants completed an evaluation metric while the second group did not. The IRB approved study occurred in two locations between August 2018 and April 2019: a 700-bed tertiary care teaching hospital and a 100-bed hospital within the same hospital system. To aid in evaluation, two video cameras were set up outside of a doorway leading into a patient care room of an unoccupied section of the hospital, providing a front and side view of the study participant. A third camera was set up on the other side of the doorway giving a view from the patient room ( Figure 1 ). An isolation cart or bedside table was stationed in front of the room with hand sanitizer, gloves, and the appropriately sized N95 respirator. A wastebasket was available on both sides of the door for disposal of the respirator. A repeated performance method was used ( Figure 2 ) for two intervention groups: Video Alone (VA) or Reflective Practice (RP). All participants began by completing a demographic questionnaire which asked about role, age, gender, and years of experience. They were then instructed to "please apply and remove the respirator as if entering and exiting a patient room under airborne isolation." Study personnel started the recordings and participants performed the donning and doffing of a respirator to camera for a pre-test measure as a simulation. A randomized educational intervention followed in the VA or RP option. The participant was then asked to repeat the donning and doffing to camera immediately after the intervention for a post-test measure. Participants returned approximately one month later for follow-up measure and repeated the video-recorded donning and doffing simulation. A randomization schedule was developed before the study began. Participants were randomized through simple urn randomization, ensuring equal sample sizes. 17 The randomized educational intervention was either: 1) VA, in which participants watched the first 4 minutes and 40 seconds of a Centers for Disease Control (CDC) training video on donning and doffing an N95 respirator and performing a seal check (available at: https://www.cdc.gov/niosh/npptl/topics/respirators/disp_part/donningdoffing.html), or 2) RP, in which the participants watched the same CDC training video segment and then watched the video of their own baseline performance while self-scoring using the critical safety behaviors scoring tool (CSBST), described in Figure 3 . The self-scoring intervention employs reflection to improve professional performance. 18 This additional intervention was hypothesized to improve performance both for the post-test and one-month follow-up. The VA option took approximately 5 minutes, and the RP option took between 8-10 minutes. The authors, with backgrounds in industrial hygiene, infection control, and research design, developed the 10-point CSBST for N95 respirator use ( Figure 3 ) from previous study findings. 16 There were six measures for donning and four for doffing. Each point is equally weighted. Hand hygiene was a critical component of both donning and doffing processes, accounting for four of the ten total points. Correct strap placement and performing a proper seal check were key behaviors noted in donning. Doffing safety behaviors included limiting touch to the respirator straps for removal and gently moving the respirator from the face to the wastebasket. These CSBST items were covered well in the CDC video, but the video did not demonstrate the conduct of hand hygiene. Prior to data analysis, each video-recorded simulation was reviewed by a member of the research team and point values were assessed using the CSBST. During the video scoring process, the research team was blinded to which educational intervention was implemented for the participants. Inter-observer agreement was assured by systematically reviewing videos independently and then comparing scores until sufficient agreement was reached. Video scoring was also randomly assessed for consistency for the videos assessed by individual scorers. No discrepancies were noted. The targeted study sample size was 30 in each intervention group. A sample of 30 participants per group is considered sufficient in a pilot study for aims involving between-group differences, estimating effect sizes, and using the results to perform a power analysis for a future, fully-powered study. 19 HCWs who are required to enter airborne isolation rooms for their work were recruited from clinical and ancillary services in the hospital. The participants worked at least part-time at a hospital with an active RPP. Participants were recruited through group emails and personal contacts with managers or unit educators throughout the hospital from August to November of 2018 which resulted in a convenience sample. Each participant consented individually and could refuse to participate at any time without penalty. A cafeteria meal ticket was given to participants as compensation for each of the two study visits required to complete the research. Study personnel worked with managers and unit educators to identify a day and time for the one-month follow-up; when unavailable, study personnel reached out via email to participants directly to schedule a time for the one-month follow-up recording. The study received expedited Institutional Review Board approval (#346-18-EX) as social and behavioral science. Prior to statistical analysis, total scores were calculated on the 10-point CSBST from the expert scoring results as previously described and entered into an electronic spreadsheet Sixty-two HCWs consented to participate, but only forty-eight participants (79%) completed the study (i.e., returned for the one-month follow-up) and will be described in this report. Randomization of the participants resulted in 24 participants in each intervention group. At least 8 of those that did not complete the follow-up were no longer hospital employees for multiple reasons not captured in the study. Based on demographic data collected, participant roles included nursing (n = 33), environmental services (n = 10), respiratory therapy (n = 3), radiology (n = 1), and patient care technicians (n=1). Participants ranged in age from 24 to 66 (average=36.8) and comprised 44 (92%) females and 4 (8%) males. Years of experience averaged 10.1 (median 9.5). While participants in the VA group were significantly older than those in the RP group (40.8 years vs. 32.8 years, p=.01), there was not a significant difference in years of experience in health care (11.9 years in VA group; 8.4 years in RP group). The two groups had equal distributions of gender (92% female) and having experienced a needle stick (20.8%). The majority of participants were White/Caucasian (81.3%), with 75% White/Caucasian in the VA group and 87.5% in the RP group (p=.267). The longitudinal follow up did have some variation. The time since post-test ranged from 21 to 245 days. The median and mean were 35 days and 53.2 days respectively. Eight of the 48 participants returned more than two months later. Although the increased latency for these participants could affect any decreases in total scores, a Mann-Whitney U test for days between visits indicates that the two groups were not significantly different in time between post-test and follow-up. The total video files analyzed for this longitudinal study sample was 432, with 3 video recorded performances and 3 views for each of them. Table 1 details the percentage of critical safety behaviors performed for the completed participants at pre-test, post-test, and follow-up. The highest total score prior to the interventions was 6 out of 10. All of the perfect scores post-test were from the RP group (7 out of 24 RP participants). The total scores were averaged and plotted in Figure 4 . An independent groups t-test indicated that the VA group and the RP group were not significantly different at pre-test [t(46)=1.43, p=.161]. The RP group was significantly higher than the VA group post-test (p<.001) and at follow up (p<.001). RM-ANOVA models showed that both groups significantly improved from the pre-test score to post-test, but the RP group improved significantly more In the RP intervention group, expert scoring was compared to the self-scoring of the participant. The participants perceived they were better at performing the N95 donning and doffing than the experts scored them. The difference in scoring was significant at p=.002 using a paired samples test. This study investigated respiratory protection education in the context of HCW safety using two randomized educational interventions measured both immediately and longitudinally. The approach in this study is novel because of the healthcare worker population studied (i.e., ancillary services in addition to nurses) as noted by recent systematic reviews in this area. 15 The study was designed to be a "no lose" situation as either intervention would be practical and relatively easy to implement. The CSBST is a tool to measure the performance of HCWs on a specific targeted safety behaviors. The addition of a RP intervention made a measurable and sustained difference in demonstrated safety behavior, while only taking participants 5 minutes longer than the VA intervention. As expected, the performance quality in both groups did decrease at follow-up which provides information on skill degradation at an earlier time point than previous studies. 8 The findings build on Harber's work which found video to be the marginally better learning tool when training on respirator use. 8 Our population was specifically HCWs who had some level of confidence and previous experience using N95 respirators for patient care, so a pre-test allowed for the evaluation of typical behaviors without training. The video used in the study focused on safely using a filtering facepiece N95 respirator but did not include detailed context for the healthcare setting specifically. The CSBST used for video scoring included several metrics on hand hygiene that may have impacted the participants' reflections and ultimately their performance scores. This may indicate that scoring tools for clinical skills and other healthcare training programs should very specifically include the critical moments of hand hygiene and rationales for them. 19 Ultimately, a study design where the educational materials are directly patterned off of the scoring tool would best demonstrate if the RP intervention truly made the difference in behavior. Nevertheless, the VA alone intervention reflects the currently available materials for most hospital RPPs at this time. Specific to filtering facepiece N95 respirators, some additions to the CSBST may be warranted for future studies. Applying a respirator correctly requires personal hand contact with the face and ungloved hands to feel for air leaks. The hands should not be considered clean enough to provide patient care afterwards. Removal can be complicated by several factors, but clean hands should be used for removal, and hands should be cleaned after contact with the respirator straps. Additional considerations from the video scoring might include donning location, doffing location (e.g., performed inside or outside of patient room), hair management, and facial hair status. These data are more complex and may need to be captured as a different level of measurement than a simple point value. There were several limitations of our study. The study participants were from one Midwestern hospital network. Our sample included only 4 males and no physicians. The only data collected on behavior was based on a simulated experience and did not include a component of real-life patient care in the clinical setting. The study used an existing educational video from the CDC on the use of N95 respirators. Follow-up visits were targeted for one month after the initial visit, which limits the long-term retention conclusions which can be made from the study. Our study provides more information about the time period after training when performance decreases, expanding the work by Harber. 8 Healthcare has become more safety focused with increased attention to novel infections, pandemic, hazardous drugs, and nosocomial transmission in recent years. This study was conducted prior to the COVID-19 pandemic. The lack of critical PPE supplies internationally presented unique challenges. Unexpected educational needs arose with the pandemic including the need for training on extended use and limited reuse of N95 respirators. 4 While the content of instruction may be altered by this global experience, the educational strategies demonstrated in this study would likely remain effective. This study design could be altered to be applicable to several clinical tasks requiring good personal protection skills with the development of a similar CSBST. Further and more systematic investigation is needed with representative samples of multiple healthcare professions to determine when the knowledge loss occurs on these clinical tasks, perhaps with weekly-recorded simulation activities over a longer period. Future studies should implement and assess the feasibility of a RP intervention by examining clinical practice outcomes related to safety behaviors to determine if these behaviors in simulation are better or worse in the construct of clinical practice demands. These practical interventions and tools to measure performance may substantially improve the safety of our healthcare workforce in the care of patients with infectious respiratory pathogens as well as other clinical tasks that require use of the N95 respirator. Respiratory protection policies and practices among the health care workforce exposed to influenza in New York State: Evaluating emergency preparedness for the next pandemic Evaluation of Minnesota and Illinois Hospital Respiratory Protection Programs and Health Care Worker Respirator Use Hospital respiratory protection practices in 6 U.S. states: A public health evaluation study Recommended guidance for extended use and limited reuse of N95 filtering facepiece respirators in healthcare settings Respirator tolerance in health care workers Component analysis of respirator user training Comparison of three respirator user training methods Persistence of respirator use learning Improving the accuracy of self-assessment of practical clinical skills using video feedback: The importance of including benchmarks Video feedback in education and training: Putting learning in the picture Personal protective equipment for preventing highly infectious diseases due to contaminated body fluids in healthcare staff Personal protective equipment doffing practices of healthcare workers A method for evaluating personal protective equipment technique by healthcare workers Method for investigating nursing behaviors related to isolation care Transfer of bacteriophage MS2 and fluorescein from N95 filtering facepiece respirators to hands: Measuring fomite potential Observed variation in N95 respirator use by nurses demonstrating isolation care Properties of the urn randomization in clinical trials The Reflective Practitioner: How Professionals Think in Action. Basic Books: USA Considerations in determining sample size for pilot studies My five moments of hand hygiene: A user-centred design approach to understand, train, monitor and report hand hygiene Funding was provided by a seed grant from the University of Nebraska Medical Center College of Nursing. Special thanks to John Lowe, Shelly Schwedhelm, Mark Rupp and the many supportive managers and staff at Nebraska Medicine who have supported this project.The authors have no conflict of interest related to the submission of this manuscript.The authors have no financial relationships to disclose.