key: cord-034373-7v7r44do
authors: Stevens, Jennifer P.; O’Donoghue, Ashley; Horng, Steven; Tandon, Manu; Tabb, Kevin
title: Healthcare’s earthquake: Lessons from complex adaptive systems to develop Covid-19-responsive measures and models
date: 2020-10-23
journal: NEJM Catal Innov Care Deliv
DOI: 10.1056/cat.20.0505
sha: 
doc_id: 34373
cord_uid: 7v7r44do

New metrics and forecasting models are key to understanding and anticipating the challenges of the Covid-19 pandemic.

Complex adaptive systems are distinct from complicated systems. A complicated system is like an engine, with multiple, elaborate components that, when taken apart, are reduced to small, indivisible parts. Complicated systems are deterministic; they can be anticipated and predicted. Many health care processes that are easily measured and improved, such as placing a central venous line, are complicated. These types of systems are straightforwardly monitored with data systems; tracking incremental changes over time allows us to distinguish between meaningful variation and random noise.

Complex systems, while they appear to have multiple patterns, are more akin to fractals or biologic processes, in that the closer you look, the more you see. Complex systems have numerous emerging and evolving connections with individual agents and are largelynondeterministic--that is, there can be many possible outcomes for a given set of circumstances. For example, an emergency room on a Friday night is filled with connections, visible and invisible hierarchies, and structures, with an unpredictable outcome for the evening. As described by Sargut and McGrath, one of the biggest challenges health care leaders face in dealing with complex systems is the "vantage point" problem,1 where the individual actors and leaders are unable to truly see the whole,2 especially in rare events like Covid-19.

Covid-19 is health care's "big one" of an earthquake, a rare event turning our complex system of health care on its head."

Covid-19 is health care's "big one" of an earthquake, a rare event turning our complex system of health care on its head. Many of the variables driving the epidemiology of the disease are only now emerging, several months into the pandemic. Carefully constructed dashboards created by institutions for patient flow and safety metrics have been largely up-ended as patients are cared for in different clinical environments. New ICUs were created overnight in operating rooms, postanesthesia care units, and medical-surgical floors. Existing process measures that served as reliable surrogates for outcome measures are no longer reliable as clinical workflows are re-engineered. New financial incentives, such as the payment parity of telehealth,3 , 4 have created new processes of care overnight. New drug and device shortages created care plans that evolved by necessity.

Where do all these factors leave health care leaders in their efforts to manage health care operations? We review three strategies for leadership during the current rare event: expanding the vantage point; creating interest and acceptability for evolving metrics; and building forecasting strategies that reflect the complex environment.

Expand the vantage point: Engage diverse thinkers in your dashboard design Standard health care leadership groups may include members of the finance team, members of the information systems group, and leadership from medical and nursing staff. Many hospitals adapted incident command structures, largely derived from emergency management designs, to create ad hoc leadership and reporting structures during Covid-19 surges.5 -7

While these systems primarily create a shared mental model for action, resource demands, and communications, they also bring to the fore additional voices and vantage points for managing health care delivery beyond the bounds of the immediate surge, including which metrics to follow and how to evaluate data. For example, hospital epidemiologists guided our leadership team in designing a dashboard to monitor for a second surge of Covid-19 cases. Our nursing leaders asked whether our health care staff of color were disproportionately sick from Covid-19.

As health care leaders, it is time to consider how we can continue to broaden our vantage point. For instance, consider the value of a patient representative to herald shifts in the community's pandemic response; or of a physician who cares for patients most likely to be affected by delayed care during surges, such as oncology; or of members of the clinical and operations staff from communities of color served by the health care network. These voices can supplement the usual voices in incident command structures and leadership structures to help health care leaders see more of the complex system and help them anticipate.

Metrics and variables in health care data structures allow us to see shifts and changes while we can still intervene. But if we are entirely dependent on specific metrics, we may find ourselves furiously watching one series of metrics while the system itself crumbles around us.

Instead, we propose creating openness to new metrics, by being watchful for undervalued variables, variables with shifting value, and variables with new values. Some examples:

Undervalued variable: Patient demographics. Patient demographics were a largely unreported variable in many health care dashboards, even as different public health agencies reported them. What did we miss because of this in our Boston-based health system? At Beth Israel Deaconess Medical Center, one of the two large academic health centers in our network, we had two separate surges that drove our patient population, identified in Figure 1 . The first surge was composed disproportionately of patients of color from densely populated areas who were admitted early in the surge (blue line). The second, more protracted surge was composed of a greater proportion of white patients (red line). The two separate surges meant our health care system faced more of a plateau " rather than a peak, as shown in the total population of patients with Covid-19 (green line). The protracted peak had implications for our clinical staff, our personal protective equipment needs, and the continued delay in much of our health care network's operations.

Shifting value: Percentage of Covid-19 patients in the ICU. Other metrics may change their meaning and usefulness mid-pandemic, even if we do not actively change their definition. One such example is the percentage of Covid-19 patients in our ICUs. Initially, this metric was critical during the surge of Covid-19 cases in our community, with many of our Covid-19-positive patients admitted with severe acute respiratory distress syndrome, pneumonia, and influenza-like illnesses. These patients needed immediate critical care services, making Covid-19-positive patients synonymous with symptomatic patients. Further, this meant that health care resources that our health care system needed, such as ICU beds and ventilators, were also synonymous with Covid-19-positive patients.

In June, many of our hospitals began performing urgent and elective procedures again. As part of providing elective care and creating a safe environment for all patients and staff, we started Covid-19 testing for all patients on admission. This increase in testing created a new denominator problem -suddenly, we had a large number of asymptomatic patients admitted with Covid-19, rather thanbecause of Covid-19, who no longer required the same ICU resources and other levels of care. The result was our Covid-19 patients suddenly look dramatically less sick with a much smaller percentage in the ICU (Figure 2 ).

Without understanding the shifting value of this metric, we might mistake our plummeting Covid-19-positive ICU census for new knowledge about how to clinically manage patients with the disease, or presume the disease had mutated to a less sick form. Instead, in a time of low community prevalence, we find a persistent number of Covid-19 positive patients but with less of an impact on critical care resources, which itself has raised the value of a new metric, the number of symptomatic patients with Covid-19.

New value: respiratory illness and influenza-like illness in the ED. Influenza-like illness, the most common presentation of active and symptomatic Covid-19 infection,8 was an early harbinger of the epidemic to come in Boston. Both throughout February and again following an international conference hosted by the Biogen in the final week of February now recognized to be a superspreader event,9 we noted a higher percentage of influenza-like illness presentations to our ED at the BIDMC than in past years during the same time, an early signal of what was to engulf our state in six weeks' time ( Figure 3) .

As we embark on the 2020-2021 influenza season while the Covid-19 pandemic continues, we may face a catastrophic combination of diseases that cause respiratory failure. We propose anticipating these surges with the inclusion of a metric in Covid-19 dashboards that is already collected and reported as part of influenza monitoring: influenza-like illness presentations. Whether these be Covid-19 infections or flu, we expect that incorporating other signals may serve to provide additional early warning signals about shifts in resource needs to keep patients safe.

Finally, as health care systems face increasing shifts in the pandemic, we propose identifying forecasting tools that provide opportunities to learn about the complex system of our health care environment and Covid-19 itself, rather than depending on unrealistic assumptions. Many models that were used to forecast Covid-19, particularly early in the epidemic, fell short of this requirement. Some models relied on a Susceptible-Infected-Recovered models,10 which depend on several assumptions, such as all members of the population are equally likely to get infected ("susceptible"), or all patients who survive can never be re-infected ("recovered"). Others, like the Institute for Health Metrics and Evaluation (IHME) initially fit their curves to prior data from Italy and China to predict results in other, very different communities and countries.11

Unfortunately, oversimplified models are more likely to create chaos than to provide direction for health systems. Consider the political consequences on Election Day of oversimplified election predictions; what would happen if we confidently predicted the results of the 2020 election only using voters' income and ignored all recent events? Many Covid-19 national models do exactly this -make out-of-date or oversimplified assumptions -which lead to meaningless output and the potential to create major clinical consequences for patients and health systems.

For example, models that fail to accommodate the shifting populations of different communities may overstate how many people are at risk of infection, causing a hospital system to direct resources to the wrong clinics and communities. Models that do not accommodate shifting public health guidance, school policies, and changing business requirements may understate the numbers, rendering health systems unprepared for second waves of infection. Models need to reflect the shifting health and policy landscape -to allow for the complexity of the pandemic itself -for any health care organization to meaningful make use of them.

Models need to reflect the shifting health and policy landscape -to allow for the complexity of the pandemic itself -for any health care organization to meaningful make use of them."

Our institution has proposed one modeling solution to this. The solution incorporates hospital decision making, what is known about the virus, and how the Massachusetts population is moving around and interacting with one another -in short, a model that reflects the complexity of Covid-19. 12 We recognize that adding a range of different real-world variables risks creating a model overfitted to our data, which is to say a model that can only describe what has happened in the past but can tell us nothing about what lies ahead or generalize to any other setting.

To guard against overfitting while reflecting a complex reality, we did the following. First, we built a multi-hospital (multi-task) model using hospital census data from each of our hospitals within " hospital network, rather than depending on any single institution or the combined network census. Second, we asked the model to estimate some of the unknown features of the virus rather than making assumptions about these values in our model, as the scientific landscape of SARS-COV-2 has shifted and the infection itself is new to all of us. For example, rather than use information about the virus such as the number of days patients are infectious, hospitalization rate, etc. from published literature from earlier hit areas such as China or Italy, we learned these variables directly from our observed data using machine learning methods. And third, we incorporated information about how people were moving around and how much they were interacting with other people, using publicly available cellphone data, thereby incorporating the shifting policy interventions in our state and shifting norms of behavior. This hybrid approach allows us to learn from data we are observing, but also generate a model that allows us to forecast what might happen if certain variables changed. The result is a forecasting model that leverages the principles of complexity to guide hospital leadership, providing weekly updates to a group of health care leaders about how and when a new surge of infections may arrive.12

Healthcare is facing one of its greatest challenges, in part because we have wrapped ourselves comfortably in familiar metrics and dashboards that weren't designed to handle the problems of complex system. Healthcare couldn't see the "big one" coming, a dramatic reminder of the risks of oversimplifying a complex problem. To move forward, we have to build models that reflect the true complexity we are facing, to engage new voices that let us understand the next challenge we will face, and to remain flexible and curious about our metrics. We are still squarely in the middle of this earthquake and we have many aftershocks ahead. 

Learning to live with complexity

Understanding health care delivery science

Implications for Telehealth in a Postpandemic Future: Regulatory and Privacy Issues

Telehealth and patient satisfaction: a systematic review and narrative analysis

Covid-19 Best Practice Information: Emergency Operations Centers. 2020

World Health Organization. A systematic review of public health emergency operations centres (EOC)

Use of Incident Command System for Disaster Preparedness: A Model for an Emergency Department COVID-19 Response

Clinical Characteristics of Hospitalized Covid-19 Patients

How a Premier U.S. Drug Company Became a Virus "Super Spreader

Locally Informed Simulation to Predict Hospital Capacity Needs During the COVID-19 Pandemic

Wrong but Useful -What Covid-19 Epidemiologic Models Can and Cannot Tell Us

How One Boston Hospital Built a Covid-19 Forecasting System. HBR.org, 2020