key: cord-0686573-ix0f3d95
authors: Wilde, Harrison; Dennis, John M.; McGovern, Andrew P.; Vollmer, Sebastian J.; Mateen, Bilal A.
title: A national retrospective study of the association between serious operational problems and COVID-19 specific intensive care mortality risk
date: 2021-07-29
journal: PLoS One
DOI: 10.1371/journal.pone.0255377
sha: 558cf8668dd46eacceb3c36223be6c7f25e12f20
doc_id: 686573
cord_uid: ix0f3d95

OBJECTIVES: To describe the relationship between reported serious operational problems (SOPs), and mortality for patients with COVID-19 admitted to intensive care units (ICUs). DESIGN: English national retrospective cohort study. SETTING: 89 English hospital trusts (i.e. small groups of hospitals functioning as single operational units). PATIENTS: All adults with COVID-19 admitted to ICU between 2nd April and 1st December, 2020 (n = 6,737). INTERVENTIONS: N/A MAIN OUTCOMES AND MEASURES: Hospital trusts routinely submit declarations of whether they have experienced ‘serious operational problems’ in the last 24 hours (e.g. due to staffing issues, adverse weather conditions, etc.). Bayesian hierarchical models were used to estimate the association between in-hospital mortality (binary outcome) and: 1) an indicator for whether a SOP occurred on the date of a patient’s admission, and; 2) the proportion of the days in a patient’s stay that had a SOP occur within their trust. These models were adjusted for individual demographic characteristics (age, sex, ethnicity), and recorded comorbidities. RESULTS: Serious operational problems (SOPs) were common; reported in 47 trusts (52.8%) and were present for 2,701 (of 21,716; 12.4%) trust days. Overall mortality was 37.7% (2,539 deaths). Admission during a period of SOPs was associated with a substantially increased mortality; adjusted odds ratio (OR) 1.34 (95% posterior credible interval (PCI): 1.07 to 1.68). Mortality was also associated with the proportion of a patient’s admission duration that had concurrent SOPs; OR 1.47 (95% PCI: 1.10 to 1.96) for mortality where SOPs were present for 100% compared to 0% of the stay. CONCLUSION AND RELEVANCE: Serious operational problems at the trust-level are associated with a significant increase in mortality in patients with COVID-19 admitted to critical care. The link isn’t necessarily causal, but this observation justifies further research to determine if a binary indicator might be a valid prognostic marker for deteriorating quality of care.

a1111111111 a1111111111 a1111111111 a1111111111 a1111111111

The emergence of the SARS-Cov-2 pathogen [1] , and the new more transmissible variants [2] , has resulted in large numbers of people, requiring hospital admission, often to high-acuity critical care settings [3] . In the UK for example, some hospitals increased their intensive care unit capacity by over 200% at the peak of the first wave of the COVID-19 pandemic to address the increased need [4] . Despite these re-deployed resources, and even in combination with the introduction of non-pharmacological interventions to limit disease transmission [5] , many UK hospitals far exceeded the nationally-defined threshold of 85% for safe operating capacity [4] . However, the concerns about the operational strain on health systems extends beyond the more superficial capability to admit new patients, as there is also a well-established association between operational strain and individual-level patient outcomes, including those in critical care settings [6] .

Evidence on the effect of operational strain on COVID-19 patient outcomes has begun to accumulate; for example, operating above the 'safe occupancy threshold' was associated with increased risk of COVID-19 mortality in a national cohort study of English intensive care units [7] . Similar results have been reported for the use of surge capacity (i.e. any beds that are made available above and beyond the baseline capacity of a critical care unit) in the USA and Europe [8, 9] . However, there are several limitations to using the methods employed by the aforementioned studies for defining operational strain. Firstly, the thresholds evaluated (most often for bed occupancy as a proxy for strain) reflect those set at the national level, meaning they do not reflect local or hospital-level nuances and context, and so are unlikely to provide accurate capture of local-level operational strain that could impact quality of care. Secondly, occupancy-based definitions do not capture staff absence levels. Previous research has suggested an impact of nursing and medical consultant staff absence on patient mortality risk in intensive care units [10] . Importantly, staff absence rates were raised 3-fold from the baseline

The association between serious operational problems and COVID-19 specific intensive care mortality risk PLOS ONE | https://doi.org/10.1371/journal.pone.0255377 July 29, 2021 2 / 14 with the statutory remit under which it was collected. Moreover, to retain the fidelity of the data and given that there are numerous unique cases due to low prevalence in certain geographic areas over the time period we analyse, de-identification would be insufficient to comply with the minimum Office for National Statistics' disclosure principles which would come with significant risk of reidentification. Data utilised in this study were made available through an agreement between the University of Warwick and PHE. Individual requests for access to the raw data are considered directly by PHE (contact via covid19surv@phe.gov.uk). of 4% at the peak of the first wave [11] . The potential impact of absence levels and other nonoccupancy related operational strain on COVID-19 patient outcomes has not previously been evaluated, and may be of particular importance to understand drivers of the reported temporal improvements in in-hospital COVID-19 mortality over the course of the pandemic [12] . A potential solution to assessing the impact of local operational strain on outcomes is the generic 'serious operational problems' declaration (binary Yes/No flag) that English hospitals report as part of their daily situation reports (SitRep) to the national regulators [13] . These declarations are meant to reflect local-context defined operational issues in the preceding 24 hours [14] , including those that do not rise to the standard of a regulator defined 'serious incident' (e.g. exceeding specific bed occupancy thresholds) [15] . In this study, we sought to describe the pattern of these generic 'serious operational problem' declarations over the course of the COVID-19 pandemic in England, and investigate the association between these declarations and COVID-19 specific mortality risk in intensive care units across England.

Eligible individuals were identified through a recorded admission in the COVID-19 Hospitalisation in England Surveillance System (CHESS), which captures confirmed or presumed COVID-19 related intensive care unit (ICU) admissions during the pandemic (diagnostic criteria described below) [16] . Submission of data to CHESS was made mandatory for hospitals in England. The earliest date for which all relevant data is available was the 2 nd of April 2020. All individuals admitted after this date, but before the end of the 1 st of December (i.e. the first day of de-escalation of the non-pharmacological interventions following the second 'lockdown' in England) were included. Individuals were eligible (see Fig 1 for a summary) if:

1. they were between the age of 18-99 years; 2. not known to be pregnant; 3. had gender recorded, and; Diagnostic criteria. CHESS comprises both presumed and confirmed cases of COVID-19. Presumed cases are defined as individuals with clinically diagnosed COVID-19 (whom never had a positive confirmatory diagnostic test) during the study period. However, 100% of people included here had some form of positive test. A positive test could include reverse transcriptase polymerase chain reaction (PCR) of any respiratory system specimen, an antibody, or antigen-based test.

Recorded clinical features. For each individual we extracted the following information: administrative data (admitting hospital trust (trusts being groups of hospitals that function as part of a single operational unit), date of admission, first segment of postcode (called the 'outward code')), demographics (age, sex, and ethnicity), and comorbidities coded as binary indicator variables with missingness equated to absence (chronic respiratory disease including asthma; obesity; diabetes; chronic heart disease; hypertension; immunosuppression due to disease or treatment; chronic neurological disease; chronic renal disease) [16] ; the validity of this approach to coding CHESS comorbidities has been previously evaluated [17] . Although data on severity of illness at admission is not available in CHESS, a recent report from the national intensive care audit (ICNARC) showed no significant variation in mean severity score (APA-CHE-II) of those being admitted over the course of 2020 [18] .

Recorded operational data. Administrative data was accessed from 'SitReps'; the daily situation reports hospital trusts routinely submit on bed occupancy as well as including a declaration (binary Yes/No flag) of whether they have experienced 'serious operational problems' in the last 24 hours. These submissions are signed off by each trust chief executive at 11am each day, based on the data that reflects the trusts position at 8am on that day. Conceptually these 'serious operational problems' flags represent an operational issue (e.g. staffing issues, adverse weather conditions, a healthcare service in their network suffering from its own operational problems resulting in network-side effects, or even a wild animal disrupting clinical care provision [19] ), which has led to a perceived or objective deterioration in service provision. In the UK, there can be strict definitions for the latter, but only when they rise to the level of a 'serious incident' as defined by the statutory regulator [20] , which usually requires direct discernible harm to an individual (or a near miss). However, there are often circumstances where discernible harm is not detected, but local administrators are still able to identify an operational problem that compromised service provision. The guidance to trust administrators is to "use their judgement on what, other than an issue which leads to a 'serious incident' as defined by the national framework, constitutes a serious operational problem" [14] . As such, there is no singular list of reasons for which a hospital might declare that they experienced a serious operational problem, rather these are based on local standards and are highly context dependent, and thus can be likened to self-reported symptom elicitation from a patient. Moreover, the exact reason (in narrative form) underlying the serious operational problem is not collected by the regulator as far as we are aware.

Alongside extracting whether a trust declared a 'serious operational problems', occupancy of general and acute beds, as well as mechanical ventilator occupancy (as a marker of intensive care unit strain) were extracted. Occupancy was defined as the proportion of surge capacity occupied on each calendar day. The full cleaning and preparation of the SitRep data is described in detail elsewhere [4] , however, for this analysis an additional step was necessary as there are three dates where no data is available: 14 th May, 24 th May and 24 th November. To address this issue, a naïve imputation method was applied where the occupancy information for the preceding 24-hour period were used to forward-fill the dates in question.

Outcome. All-cause in-hospital mortality with follow-up until death, discharge, or transfer, where both latter conditions are considered absorbing states indicating survival (the data did not distinguish whether a discharge was part of a palliative care plan or not, and thus this is a limitation of the dataset). Individuals still on the ward on the date of final observation (i.e. 22 nd December), were excluded from the primary analysis, but included in sensitivity analyses. Otherwise, no date restriction was applied to the in-hospital follow-up ensure that patients with long stays were not automatically assumed to survive, which might bias our results.

Descriptive summaries were generated as follows: medians with interquartile ranges for continuous variables and counts and proportions for categorical data.

Bayesian model specification. Bayesian hierarchical models (with default, flat priors on categorical features and student's t priors on smooths and group level effects) were used to estimate, as odds ratios, the association between mortality and: 1) an indicator for whether a serious operational incident occurred on the date of a patient's admission, and; 2) the proportion of the days in a patient's stay that had a serious operational incident occurring within their relevant trust.

Adjustments were made for the following coefficients: Age (in years), calendar week of ICU Admission, Sex, indicators for the presence of the following chronic co-morbidities: Respiratory Disease(s) (including Asthma), Immunosuppressive Disease, Renal Disease, Hypertension, Heart Disease, Liver Disease, Neurological Disease and Diabetes. Week of ICU Admission was treated linearly following investigation with a spline that showed no evidence of non-linearity. Age was treated as a continuous variable and was re-parameterised using a cubic spline with 4 knots to investigate and subsequently represent non-linearity in the effect. Sex and the various comorbidity indicators are dichotomous. Chronic Respiratory Disease(s) represents the union of two initially separate covariates in the data: Respiratory Disease and Asthma, setting instances where both are either missing or no to be negative; otherwise positive. Alongside the aforementioned population level effects, the primary models included some group level effects: Ethnicity was included with intercept coefficient for each of 7 groups: White, Asian Subcontinent, Asian (Other), Black, Mixed, Other and Missing; Obesity was included with intercept coefficients for each of 3 groups: Obese, Non-Obese and Missing; and Trust was included with intercept coefficients for all 90 trusts.

The model parameter space was sampled using Hamiltonian Monte Carlo with 3 chains of 3,000 iterations each, using the Stan statistical programming language [21] . The target proposal acceptance probability was modified from the default value to 0.95 to improve convergence with the hierarchical shrinkage prior. All models discussed had fewer than 1 in 1000 divergent transitions and R-hat diagnostics of 1.00. The minimum bulk effective sample size was 2000 for group coefficients and 2700 for population coefficients, and 3000 for smooth ones. All modelling was carried out using the BRMS package [22] , in R [23] .

Sensitivity analysis. To support the primary analysis, multiple sensitivities were formulated and undertaken to ensure the results seen were robust to changes in model structure and various assumptions. Specifically, these sensitivities covered: filtering for different degrees of missingness at trust-level by removing all trusts from the modelling data with a total of 25% / 50% / 75% of all comorbidity information missing; adjustment for the time taken in days between a patient's admission to hospital and their escalation to ICU; adjustment for occupancy on the calendar date of admission, and; adjustment for patient-level deprivation index as a proxy for socioeconomic status-derived through taking the weighted average of all super-output areas that fall within outward code available for every individual. Finally, sensitivity analyses were carried to ensure that the lack of final outcomes for some patients did not bias the modelling results. The primary model was fitted only on those patients that had a definite final outcome and associated date within the specified study cohort date range. As such, we fit the same primary models but on different datasets: including those patients that were said to be still on the unit at the time of the extract (n = 509), assuming they all survived; and another including those patients with clear final outcomes, but no final outcome date provided (n = 9).

The study was approved by the Warwick Biosciences Research Ethics Committee (BSREC 120/19-20-V1.1) and sponsorship is being provided by University of Warwick (SOC.28/ [19] [20] . The raw data was collected by Public Health England (PHE) as part of their statutory responsibilities, which allows them to process patient confidential data without explicit patient consent, and using the additional statutory powers granted during the COVID-19 pandemic were empowered to share the data with specific academic groups for the purposes of research (i.e., the COPI notices). In this study we utilised a de-identified version of this dataset, with the assent of the BSREC, who recognised that the aforementioned statutory justifications were sufficient to proceed without informed consent from individual patients.

Patient and public involvement. No patients were involved in the design, interpretation of the results, or dissemination of this study. (Fig 1) , which resulted in 122,008 patient-days observed. 2,539 deaths were recorded (37.7%), equating to a mortality rate of 20.8 per 1000 patient-days. Baseline characteristics are summarised in Table 1 . 47/90 trusts (52.8%) reported at least one day of serious operational problems, with a total of 2,701 trust-days (of 21,716; 12.4%) associated with serious operational problems. Table 2 summarises the characteristics of trusts that experienced serious operational problems and those that didn't.

Fig 2 illustrates the distribution of SOPs at patient and trust levels alongside the COVID-19 epidemic curve. Notably, the administration-reported SOPs are not solely a reflection of the incidents that might be reported due to exceeding nationally defined occupancy thresholds (i.e. 85%), as 92.8% of serious operational problems (i.e. 2,506 of the 2,701 trust days) occurred on dates when reported occupancy was less than the national 85% threshold.

The unadjusted odds ratio (OR) for mortality based on whether a patient was admitted during a period of serious operational problems was 1.12 (95% posterior credible interval (PCI): 0.98 to 1.28, 90% PCI: 1.00 to 1.25). Following adjustment for patient-level factors (full model specification shown in Fig 3) , the OR was 1.34 (95% CI: 1.07 to 1.68, 90% CI: 1.11 to 1.62). The results of the sensitivity analyses, detailed in Table 3 , illustrate that the associations are not explained by data missingness, occupancy on date of admission, or modelling structure and assumptions. 

The unadjusted OR for risk of mortality for the proportion of a patient's admission duration that had serious operational problems (i.e. 100% stay compared to 0% of stay) was 1.14 (95% CI: 0.99 to 1.31, 90% CI: 1.01 to 1.28), whilst the fully adjusted OR for risk of mortality for the proportion of the admission duration with SOPs was 1.47 (95% CI: 1.10 to 1.96; 90% CI: 1.16 to 1.87, see Fig 4) . Again, this finding was robust in all sensitivity analysis (Table 3 ).

Our study shows the declaration of serious operational issues by hospitals appears to be associated with a substantially increased critical care mortality for patients with COVID-19. The size of association is comparable to those observed for high-risk patient-level factors such as the presence of major co-morbidities, for example diabetes [17] . Importantly, we demonstrate that declaration of serious operational issues does not simply reflect hospital occupancy levels, as over 93% of declarations were reported on days when hospitals did not exceed nationally agreed upon occupancy standards and the association was not attenuated after adjustment for ventilated bed occupancy.

Our study is the first to evaluate the potential influence of hospital trust-level serious operational problems on critical care outcomes for patients with COVID-19. The findings are concordant with the previous studies that have evaluated occupancy levels (one type of serious 

The association between serious operational problems and COVID-19 specific intensive care mortality risk operational problem) and shown that operating at extremes of critical care capacity is associated with worse COVID-19 outcomes [7] [8] [9] . The fact that a positive association was seen in our study when adjusting for occupancy (both on the date of admission and during admission) in sensitivity analysis provides clear evidence that considering patient occupancy levels alone is insufficient to provide a complete picture of operational strain in secondary care. Further work to disentangle the effects of specific serious operational problems, in particular staff absence levels, would be of considerable interest in datasets where such information is available.

where in fact the real peak of the first wave in the UK was around the 22 nd of April 2020. Moreover, the substantial variance seen in mortality rate throughout the summer months is due the 7-day rolling average across intermittently low numbers of daily admissions recorded in CHESS during this period. Interestingly, the distribution of serious operational problems (Purple) implies that despite the fewer trusts experiencing SOPs at the end of the study, more patients were affect (Blue), possibly due to larger size trusts experiencing SOPs during the winter months.

https://doi.org/10.1371/journal.pone.0255377.g002 

The association between serious operational problems and COVID-19 specific intensive care mortality risk

Strengths of this study include the use of a national-level dataset (CHESS) which provided near complete capture of COVID-19 intensive care admissions in England in 2020. CHESS is a mandatory collection for hospitals in England, meaning ascertainment of mortality outcomes should be well recorded. Moreover, the use of a robust Bayesian framework allowed us to more accurately model the uncertainties implicit in the analysis, and reflect these in the reported parameter values, which are often lost by the maximum likelihood (or least squares) estimated methods are often employed for similar analyses. A limitation of the CHESS dataset is the lacked of validated patient level clinical data, in particular comorbidity for which data completeness is variable at hospital trust level, and a lack of reliable information on patient acuity at admission. Our robust sensitivity analysis, showing consistent estimates of association when filtering for different degrees of missingness at trust-level, mitigates concern over potential bias from hospital trust level coding variation. Whilst previous analysis of UK national critical care during COVID-19 suggests differences in patient acuity at admission are not associated with mortality [18] , further analysis incorporating patient-level severity information such as the APACHE-II score alongside hospital level operational data would be of considerable interest. A final limitation is that operational problems in this study were reported at the trust level, not specifically relating to the functioning of the intensive care unit, and thus there is a risk of an ecological bias which must be acknowledged.

Although limited by the fact that our results should not be interpreted causally, this study highlights the potential for using generic serious operational problem flags to capture hospital operational issues where more granular data capture is not appropriate or feasible. This is likely to be an important tool both for responding to the current COVID-19 pandemic, but also more generally for service evaluation and performance monitoring long after. Moreover, the marked association between trust-level operational problems and individual-level outcomes in critical care setting highlights the need for additional research to identify the specific causal factors that might be driving the association. Novel modelling methods are more than capable of handling unstructured natural language data, and might be able to identify modifiable factors around which to develop mitigation strategies to improve future patient care-as 

The association between serious operational problems and COVID-19 specific intensive care mortality risk such, policy makers should consider prioritising the collection of free-text descriptions, and developing mechanisms to facilitate the sharing of this data with researchers alongside the generic flags upon which this study is based.

Using a national dataset, our findings demonstrate that serious operational problems occurring at hospital trust level are associated with a substantial increase in mortality in patients with COVID-19 admitted to intensive care units. These generic serious operational issue flags potentially offer insight on local-level operational strain beyond that provided by assessment of bed occupancy levels which did not explain the associations we observed. However, there are also a number of other factors that may have contributed to the deterioration in quality of care which would not be captured in such a flag, from human error to organisational culture, and which we were not able to correct for. In essence, further research to disentangle the causal pathways between operational and organisational factors and mortality outcome, and prospective evaluation of the SOP binary indicator to determine if it is a valid prognostic marker (including outside of the pandemic context) for deteriorating quality of care, are both necessary.

Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The lancet

Hospital bed capacity and usage across secondary healthcare providers in England during the first wave of the COVID-19 pandemic: a descriptive analysis

Report 13: Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries. Imperial College COVID-19 Response Team

Volume of activity and occupancy rate in intensive care units. Association with mortality. Intensive care medicine

The association between mechanical ventilator availability and mortality risk in intensive care patients with COVID-19: A national retrospective cohort study. medRxiv

The role of organizational characteristics on the outcome of COVID-19 patients admitted to the ICU in Belgium. The Lancet Regional Health-Europe

Association of intensive care unit patient load and demand with mortality rates in US Department of Veterans Affairs hospitals during the COVID-19 pandemic

Nurse staffing, medical staffing and mortality in intensive care: an observational study. International journal of nursing studies

NHS hospital staff illness or absence up to three times usual level

Improving survival of critical care patients with coronavirus disease 2019 in England: a national cohort study

Urgent and Emergency Care Daily Situation Reports 2020-21

MSitRep) Critical Care Bed Capacity and Urgent Operations Cancelled

Fixing Emergency Department Staffing, Systems & Support to deliver excellent patient centered care

COVID-19 Hospitalisation in England Surveillance System (CHESS) 2020

Type 2 diabetes and COVID-19-Related mortality in the critical care setting: a national cohort study in England

Between-centre differences for COVID-19 ICU mortality from early data in England

Children evacuated from London hospital after fox falls through ceiling

A probabilistic programming language

brms: An R package for Bayesian multilevel models using Stan

R: A language and environment for statistical computing. R Foundation for Statistical Computing

The authors thank NHS Improvement & NHS England for providing access to the SitRep data and Prof. Matt Keeling for establishing data access on behalf of Warwick University.

Conceptualization: Bilal A. Mateen.