key: cord-0793165-1ukf58b9
authors: Rickard, Kelly N.Z.; Cohen, Joanna S.; Chamberlain, James M.; Ong, Hilary; Dwyer, Matthew; Perritt, Ashley; McKinley, Kenneth W.
title: Validation of “Personal Protective Equipment Conservation Strategies Tool” to Predict Consumption of N95s, Facemasks, and Gowns During Pandemic-Related Shortages
date: 2021-09-13
journal: Comput Inform Nurs
DOI: 10.1097/cin.0000000000000831
sha: 5535d0df3ff347659f7f6638832f176f3cb18a61
doc_id: 793165
cord_uid: 1ukf58b9

We sought to prospectively validate a model to predict the consumption of personal protective equipment in a pediatric emergency department during the COVID-19 pandemic. We developed the Personal Protective Equipment Conservation Strategies Tool, a Monte Carlo simulation model with input parameters defined by members of our emergency department personal protective equipment task force. Inputs include different conservation strategies that reflect dynamic reuse policies. Over the course of 4 consecutive weeks in April and May 2020, we used the model to predict the consumption of N95 respirators, facemasks, and gowns in our emergency department based on values for each input parameter. At the end of each week, we calculated the percent difference between actual consumption and predicted consumption based on model outputs. Actual consumption of personal protective equipment was within 20% of model predictions for each of the 4 consecutive weeks for N95s (range, −16.3% to 16.1%) and facemasks (range, −7.6% to 13.1%), using “maximum conservation” and “high conservation” strategies, respectively. Actual consumption of gowns was 11.8% less than predicted consumption for Week 1, gown resupply data were unavailable on Weeks 2-4. The Personal Protective Equipment Conservation Strategies Tool was prospectively validated for “maximum conservation” and “high conservation” models, with actual consumption within 20% of model predictions.

D uring the COVID-19 pandemic, healthcare workers and hospitals have experienced shortages of personal protective equipment (PPE), including gloves, face shields, masks, respirators, and gowns. 1 PPE shortages have put providers at an increased risk for infection during patient care. Healthcare workers have high rates of disease exposure, and significant numbers have been infected with severe acute respiratory syndrome coronavirus 2, which is partially attributable to an inadequate supply of PPE. [2] [3] [4] [5] [6] [7] Compared with the general population, providers in the United Kingdom and the United States have a 3-to 11-fold increased risk of a positive test, with adequacy of PPE an important factor in modulating that risk. 8 Appropriate allocation of facemasks is especially important to protect against the spread of severe acute respiratory syndrome coronavirus 2, where asymptomatic carriage is relatively common. 9, 10 Mitigating PPE supply shortages requires a multifaceted approach. The Centers for Disease Control and Prevention has recommended contingency protocols for PPE conservation, including limiting the use of some PPE, reusing supplies intended for one-time use, and using supplies beyond their recommended shelf life. 11 Other strategies include activating the Defense Production Act to maximize production and repurposing manufacturing sites to meet demand. 1, 12 Local grassroots efforts by independent designers, clinicians, and Authorship: All authors attest to meeting the four ICMJE.org authorship criteria: (1) substantial contributions to the conception or design of the work, or the acquisition, analysis, or interpretation of data for the work; (2) drafting the work or revising it critically for important intellectual content; (3) final approval of the version to be published; and (4) agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

There was no funding to support the development of the Excel-based simulation, the validation study, or the writing of this manuscript. The development of a Web application (available at https:// ppecs-tool.childrensnational.org) was generously supported by funding from the Children's National Hospital Foundation.

Children's National Hospital (on behalf of K.W.M.) and K.N.Z.R. applied for copyright registration for the source code for the Web application and the Excel-based simulation versions of the Personal Protective Equipment Conservation Strategies Tool. The tool is intended to remain free for users on a public-facing Web site. The text of the attached manuscript was not included in the filing for copyright registration. J.S.C., J.M.C., H.O., M.D., and A.P. report no conflicts of interest relevant to this article.

Author Contributions: The study was conceived by K.N.Z.R. and K.W.M. Model input parameters were defined by J.S.C., J.M.C., H.O., M.D., A.P., and K.W.M. The Excel-based simulation model was created by K.N.Z.R. All authors take responsibility for the paper as a whole.

researchers have also led to innovative solutions such as 3D printing of face shields and fashioning protective masks from scuba diving equipment. 13 Importance Recommendations for conservation from the Centers for Disease Control and Prevention, hospitals, and our emergency department (ED) are altered frequently (see Supplemental Digital Content 1, http://links.lww.com/CIN/ A128, with daily updates in PPE recommendations from COVID-19 leadership in our ED). With fluctuations in disease burden and evolving definitions of acceptable PPE reuse, governments and healthcare systems must be able to estimate the amount of PPE needed in real time. Accurate estimations of the potential demand for PPE vary depending on a range of potential pandemic scenarios, shifting policies, and utilization behaviors. To meet this demand, a number of PPE "burn rate" calculators have emerged to help estimate needs based on usage. 14,15 These resources use average consumption of PPE to estimate future use, relying on a number of assumptions around disease burden stability. Other simulation models to predict the consumption of PPE, developed concurrently to our own, generate outputs for a hospital, including inpatient and ICU patients. 16, 17 Although these models could be modified to focus exclusively on ED PPE consumption, neither could be used to predict PPE consumption based on the most restrictive strategies for PPE used in our setting. Even using the staffing-based calculations of other models, the total number of patients is a required input to predict PPE consumption. 16, 17 In our setting, stakeholders recommended restrictive PPE strategies requiring staff to reuse PPE across one or multiple shifts, independent of patient load and length of stay.

We created the PPECS-Tool (Personal Protective Equipment Conservation Strategies Tool) to compare the consumption of PPE, using various levels of conservation, during the pandemic-related shortage. We sought to prospectively validate the model over the course of 4 weeks of actual PPE consumption during spring 2020 in a large, urban pediatric ED. We hypothesized that actual use would be within ±20% of model predictions. Based on conversations with our ED PPE task force and the COVID-19 incident command team for the ED, this was the a priori threshold for the model to be clinically useful in making decisions about PPE conservation strategies and requisition requests from our hospital's central supply.

We performed this study in a large, urban, academic children's hospital ED in the United States with approximately 90 000 pediatric ED visits annually. There are 135 full-and parttime medical providers and 130 nurses and patient care technicians. The study period was 4 consecutive weeks between April 20 and May 17, 2020.

We built a set of four models, representing four levels of PPE conservation strategies, to predict daily consumption of N95 respirators, surgical facemasks, and gowns. These models represent increasingly restrictive use patterns, based on strategies considered or actually used in our clinical setting. Prepandemic, PPE was not reused between patients, consistent with our model representation of "liberal use." In the "moderate conservation" model, each healthcare worker reuses each article across every interaction with the same patient. In the "high conservation" model, each healthcare worker is assigned a limited quantity of the PPE article per shift. In the "maximum conservation" model, a subset of healthcare workers reuse a single article across multiple shifts. Each model for the PPECS-Tool uses different input parameters, as displayed in Table 1 . Users are required to provide their own estimates for input parameters based on recent patient arrival patterns or epidemiologic prediction models.

Each model is a Monte Carlo simulation that produces a probability distribution of real-world outcomes via simple calculations between multiple variables. For a specified conservation strategy, the values of all the variables included in that model are multiplied together. This calculation is repeated across many replications (default, 300). For every replication, new values are randomly generated for each variable, with uniform probability between possible minimum and maximum values, obtained by user input. These inputs are based on recent experience or historical data, excluding outliers. Each variable is independent and identically distributed.

We built the models in Microsoft Excel (v. 2013, Microsoft Corporation, Redmond, WA, USA) with Visual Basic. We performed error-checking and verified that outputs from all four models in our overall model match direct arithmetic calculations using the same inputs. We made an initial version of our model publicly available as the Excel-based simulation on April 1, 2020. 18 We incorporated stakeholder feedback for a subsequent Web application, built in JavaScript using the same formulas, available at https://ppecs-tool.childrensnational.org.

On Monday of each week, members of our ED PPE task force met to discuss current PPE supply levels and determine if changes were necessary in PPE conservation strategies. We used the task force determination to select the conservation strategy in our model and make predictions over the following week. Each Monday, we also queried aggregate patient counts from our EHR. These patient counts were used to determine the minimum and maximum number of patients presenting on a single day in the preceding week and to estimate the minimum and maximum number of patients who would be considered a "person under investigation," defined by the number of patients for whom COVID testing was sent. Staffing was based on average daily staff levels from the previous week. After entering all of the input parameters, based on data from the prior 7 days, we executed the simulation to generate a prediction for the 24-hour consumption of each article over the following 7 days.

At the end of each week, we determined the mean 24-hour consumption of each article using an in-person count of the remaining inventory (see Supplemental Digital Content 2, http://links.lww.com/CIN/A129, which displays our PPE COVID Inventory Count Guide) and by adjusting for the number of articles provided by our hospital central supply during that period. Specifically, we tracked consumption of surgical masks with elastic ear loops, the standard facemask for healthcare workers in our unit. We did not distinguish between regular and small-size N95s because prediction of different PPE subtypes was beyond the scope of our model. Predicting combined regular and small-size N95 consumption was useful for our unit, where central supply established a maximum weekly limit of total respirators based on hospital supplies and provided the distribution of N95 sizes as requested by our ED PPE task force. Central supply does not keep records on the resupply of disposable gowns to each hospital unit. Actual consumption of gowns could only be confirmed by daily, in-person counts of inventory.

We defined average daily consumption as the difference between the most recent PPE count and the count on the day the model predictions were made, minus the resupply during that period, divided by the number of days between each count. For each of the four study weeks, we compared actual consumption of PPE with model predictions from the preceding week. Although we used historical arrival rates and recent staffing as input parameters, we compared model output with PPE use in the subsequent week, as the goal of the model is to predict future consumption based on a specific conservation strategy and the best available input data. We used this basic evaluation of model accuracy because significantly underestimating or overestimating necessary supplies can have severe consequences during a period of shortage. 19 We assessed whether actual consumption fell within 20% of the predicted consumption for each item for each week in the study period.

This study was acknowledged by the institutional review board as "Not Human Subjects Research" and did not undergo institutional review board review.

After publicly posting an early version of the model, we received feedback from one user who thought every model input was necessary to predict PPE consumption, regardless of conservation strategy. To address this in the final Web application, the PPECS-Tool displays only the required inputs for the selected conservation strategy. Beginning on April 17, 2020, our hospital policy changed and visitors arriving to the ED were no longer offered a facemask from our ED supply but rather offered a cloth mask at the hospital entrance. Based on this local experience, we excluded the three user options from the final version of the model that previously allowed users to select options for distributing PPE articles to patients and support persons (bottom three rows, Table 1 ). 

For each article during each study week, the Excel-based simulation produced an estimated 24-hour consumption rate with a 95% confidence interval. Our use patterns for N95 respirators were stable from week to week, so our input parameters did not change. Model outputs for N95s, including a mean estimate of 17.2 respirators consumed per day, are displayed in Figure 1 . During weekly meetings with the ED PPE task force, they progressively estimated fewer ED facemasks would be used per healthcare worker as staff became more familiar with facemask distribution and reuse policies. Based on these model inputs, daily facemask use was predicted to be 86.1 for Week 1 and decreased over the study period to 29.2. Predicted gown use ranged between 148.6 and 185.6 per day. We were only able to compare gown predictions with actual consumption for Week 1, during which model outputs averaged 156.3 gowns used per day.

Conservation strategies for each article of PPE were stable over our 4-week study period (N95: "maximum conservation"; facemask: "high conservation"; gown: "liberal use"). Table 2 displays average daily consumption of PPE, which was within 20% of model predictions for each of 4 consecutive weeks for N95s (range, −16.3% to 16.1%) and facemasks (range, −7.6% to 13.1%). Average daily consumption of gowns was within 20% of model predictions for Week 1 (11.7% difference) but could not be assessed for Weeks 2-4 because of a lack of resupply data.

To our knowledge, the PPECS-Tool is the first prospectively validated model to predict consumption of PPE with a simulation model, based on site-specific information and FIGURE 1. Example model outputs for Excel-based simulation, including estimates that 17.2 N95 respirators will be used in the following 24 hours, with 95% confidence interval of 17.0-17.4. Ninety-five percent confidence interval is based on the number replications and not a representation of real-word precision. These outputs were generated from (1) a user selection of maximum conservation; (2) estimates for the number of healthcare workers who will need to use an N95 in the next 24 hours, that is, between 10-14 physicians, 20-24 nurses, 3-5 respiratory therapists, and 4-6 others (eg, technicians); and (3) an estimate that between 50% and 70% of these healthcare workers would bring an N95 from a prior shift. Abbreviation: HCW, healthcare worker. ©2021 Children's National Hospital & Kelly Na'amah Rickard, image of tool outputs reproduced with authors' permission. general conservation strategies. Models for "high conservation" and "maximum conservation" were validated, based on our a priori outcome of actual consumption within ±20% of model predictions over the course of 4 consecutive weeks. Models for "liberal use" and "moderate conservation" were not validated in this study. The minimal difference between model predictions and actual consumption of gowns during Week 1 of our study suggests that the "liberal use" model may be appropriate for a general estimate for PPE consumption, but there is not enough evidence to support the use of "liberal use" and "moderate conservation" models to predict PPE consumption. Because only "high conservation" and "maximum conservation" strategies were validated in our study, this model has the greatest value for decision-makers who are considering more restrictive use of PPE to maintain supply during an acute shortage. For stakeholders who are beginning to liberalize use of PPE, past use patterns may provide more reliable estimates of "liberal use" or "moderate conservation" than the outputs from our model.

In a unit with stable PPE protocols and patient population, closely tracking usage and resupply remains the most important strategy to estimate ongoing consumption of PPE. The PPECS-Tool provides an option to explore PPE consumption with different strategies or in the event of a change in resource demands, such as from a surge in the number of patients presenting. Other researchers have been successful in using Monte Carlo simulation for forecasting hospital bed demand during the pandemic, reporting up to 85%-95% accuracy. 20 The challenges in obtaining and preserving PPE supplies have shifted since the pandemic began more than 1 year ago. 21 Due to efforts to distribute facemasks to patients and family members early in the pandemic, we initially included a model option to distribute PPE to visitors. We eliminated this input from the final model since it was set to zero in each of the 4 weeks of our validation study. The availability of facemasks has greatly increased since the early days of the pandemic, and most Americans now report regular use of facemasks in public, 22 with survey data regarding mask usage supported by direct observations. 23 Although some healthcare decision-makers might still opt to distribute hospital facemasks to visitors who are already using a facial covering, which has not been our practice, and other than occasional facemasks for patients brought in by emergency medical services, no articles of PPE have been distributed to visitors out of the ED supply since mid-April 2020 (see Supplemental Digital Content 1, http://links.lww.com/CIN/A128). Since our model was initially developed in spring 2020, processes to sterilize N95 respirators have provided new opportunities to extend the use of PPE. 24 We did not update our model based on the availability of N95 sterilization processes, and stakeholders who take advantage of such processes would have to consider the impact of N95 sterilization when generating the model input for the "proportion of healthcare workers who bring the article from prior shift."

We chose to use descriptive labels for conservation strategies rather than the three levels of operational status used by the Centers for Disease Control and Prevention: conventional, contingency, and crisis. The Centers for Disease Control and Prevention made recommendations for contingency and crisis statuses that resemble the "high conservation" and "maximum conservation" strategies in our model (eg, implement extended use of PPE and implement limited reuse of PPE with extended use, respectively). 11 However, each operational status is also associated with a suite of other administrative actions, such as canceling nonemergency procedures and deferring outpatient encounters, 25 which might not be enacted uniformly across institutions and are often outside the control of staff in a particular healthcare unit or department. Still, any data-driven response to a PPE shortage requires a multifaceted response with broader interventions than simply changing PPE use patterns in a single healthcare unit. In addition to the need to dedicate resources to maintaining sufficient supplies of PPE, resources are required for consistent research and training efforts to maximize the effectiveness of PPE for providers. 26 There are several important limitations to this study. First, we assumed independence between model inputs. Prior research into the impact of ED crowding on hospitalization offers one example for how model inputs are not completely independent in the real world. A greater proportion of highacuity ED patients may be admitted to the hospital during periods of ED crowding, 27 potentially leading to model underestimates in these circumstances. Other model inputs, such as the number of healthcare workers per patient and the number of interactions with each healthcare worker, are also dependent on ED volume and might act consistent with or counter to the effect of ED crowding on hospitalization. The unpredictable impact of ED crowding on model estimates was untested during our study period. For these reasons, under circumstances of ED crowding, our independence assumption is a major limitation for the "liberal use" model, which was also not validated in the present study because of insufficient resupply data on gowns.

Model outputs are also limited by the need for each user to estimate input parameters. It was not feasible to count the number of healthcare workers who interact with each patient, and we had to estimate that model input based on our clinical experience. We selected ED volumes based on presentation rates over the preceding week, but we were not able to use a predictive algorithm to accurately forecast ED volumes as an input for the model. We expect most decision-makers will have similar limitations in their ability to select input parameters for the model. Other researchers have attempted to address the challenge of burdensome input parameters, most notably the PPE forecasting calculator developed at the University of Pennsylvania, which integrates with a model to predict COVID-related hospitalization rates. Unfortunately, their hospitalization prediction is only applicable to the period prior to peak infections. 17 Our model performs well for predicting PPE consumption with "high conservation" and "maximum conservation" strategies at our institution, but it is unknown how generalizable these results are to different pediatric EDs, adult EDs, or other healthcare settings with less restrictive access to PPE. In our ED, access to N95s was restricted to a trusted group of senior nurses, only when working in their role as charge nurses. Small allocations of facemasks and gowns were available in highly visible public areas in the ED. Larger stores of PPE were kept in a single storage room in the ED under lock and key. Units with less restrictive access to PPE might be at a greater risk from supply chain diversions, which would lead to model underestimates.

Furthermore, some input parameters may be difficult to generate in a nonacademic setting, especially for the "liberal use" and "moderate conservation" options that were not validated in the present study. All our models rely on uniform probability distributions: if a user were to include outliers in their maximum estimates for input parameters, our use of uniform distributions would cause an overestimate in the consumption of PPE. Furthermore, many decision-makers may not have access to historical data regarding the proportion of patients of specific risk levels, much less an effective way to predict that patient breakdown. We included these risk levels, namely, high risk (person under investigation: ill/requiring hospitalization), medium risk (person under investigation: not requiring hospitalization), and low risk (other patients), because they played a role in the operational use of PPE (gowns) at our institution, but this variable added greatly to the complexity of using the model in our setting even with a team of researchers dedicated to data analytics in our ED. Future efforts in the prediction of PPE consumption will consider the value of model parsimony to ensure good performance and minimize energy in input data generation.

In summary, the PPECS-Tool was prospectively validated for "maximum conservation" and "high conservation" models, with actual consumption within 20% of model predictions during each of 4 consecutive weeks in spring 2020. The accuracy of model predictions depends on the accuracy of model inputs and model assumptions.

Critical supply shortages-the need for ventilators and personal protective equipment during the Covid-19 pandemic

COVID-19 and healthcare workers: a systematic review and meta-analysis

COVID-19: protecting health-care workers. The Lancet

Clinical characteristics of 30 medical workers infected with new coronavirus pneumonia. Zhonghua Jie He He Hu Xi Za Zhi (Chinese Journal of Tuberculosis and Respiratory Diseases)

Transmission of COVID-19 to health care personnel during exposures to a hospitalized patient

Characteristics of health care personnel with COVID-19-United States

Risk of COVID-19 among front-line health-care workers and the general community: a prospective cohort study. The Lancet Public Health

Face mask use in the general population and optimal resource allocation during the COVID-19 pandemic

Screening of healthcare workers for SARS-CoV-2 highlights the role of asymptomatic carriage in COVID-19 transmission

COVID-19): optimizing supply of PPE and other equipment during shortages

Covid-19: Biden launches national plan based on "science and public health alone

Exploration of alternative supply chains and distributed manufacturing in response to COVID-19; a case study of medical face shields

COVID-19): personal protective equipment (PPE) burn rate calculator

Interim estimate of US PPE needs for COVID-19

HSyE Institute COVID hospital surge demand model V2.0 with VBA simulation

Forecasting PPE consumption during a pandemic: the case of Covid-19. Infectious Diseases (except HIV/AIDS)

Global supply-chain effects of COVID-19 control measures

A hospital surge capacity bed, equipment, and staff demand planning model

Dynamic response systems of healthcare mask production to COVID-19: a case study of Korea

COVID-19 mitigation behaviors by age group-United States

Observed face mask use at six universities-United States

Hydrogen peroxide vapor decontamination of N95 respirators for reuse

Sourcing personal protective equipment during the COVID-19 pandemic

Medical masks and respirators for the protection of healthcare workers from SARS-CoV-2 and other viruses

The impact of pediatric emergency department crowding on patient and health care system outcomes: a multicentre cohort study