key: cord-0531484-mgd6nlwm
authors: Thompson, Robin N; Hollingsworth, T Deirdre; Isham, Valerie; Arribas-Bel, Daniel; Ashby, Ben; Britton, Tom; Challoner, Peter; Chappell, Lauren H K; Clapham, Hannah; Cunniffe, Nik J; Dawid, A Philip; Donnelly, Christl A; Eggo, Rosalind; Funk, Sebastian; Gilbert, Nigel; Gog, Julia R; Glendinning, Paul; Hart, William S; Heesterbeek, Hans; House, Thomas; Keeling, Matt; Kiss, Istvan Z; Kretzschmar, Mirjam; Lloyd, Alun L; McBryde, Emma S; McCaw, James M; Miller, Joel C; McKinley, Trevelyan J; Morris, Martina; ONeill, Philip D; Pearson, Carl A B; Parag, Kris V; Pellis, Lorenzo; Pulliam, Juliet R C; Ross, Joshua V; Tildesley, Michael J; Tomba, Gianpaolo Scalia; Silverman, Bernard W; Struchiner, Claudio J; Trapman, Pieter; Webb, Cerian R; Mollison, Denis; Restif, Olivier
title: Key Questions for Modelling COVID-19 Exit Strategies
date: 2020-06-21
journal: nan
DOI: nan
sha: 5cfa205010ffbd80c6e0c5915bf5e172269e3b8a
doc_id: 531484
cord_uid: mgd6nlwm

Combinations of intense non-pharmaceutical interventions (lockdowns) were introduced in countries worldwide to reduce SARS-CoV-2 transmission. Many governments have begun to implement lockdown exit strategies that allow restrictions to be relaxed while attempting to control the risk of a surge in cases. Mathematical modelling has played a central role in guiding interventions, but the challenge of designing optimal exit strategies in the face of ongoing transmission is unprecedented. Here, we report discussions from the Isaac Newton Institute Models for an exit strategy workshop (11-15 May 2020). A diverse community of modellers who are providing evidence to governments worldwide were asked to identify the main questions that, if answered, will allow for more accurate predictions of the effects of different exit strategies. Based on these questions, we propose a roadmap to facilitate the development of reliable models to guide exit strategies. The roadmap requires a global collaborative effort from the scientific community and policy-makers, and is made up of three parts: i. improve estimation of key epidemiological parameters; ii. understand sources of heterogeneity in populations; iii. focus on requirements for data collection, particularly in Low-to-Middle-Income countries. This will provide important information for planning exit strategies that balance socio-economic benefits with public health.

As of 15 June 2020, the coronavirus disease 2019 (COVID- 19) pandemic has been responsible for more than eight million reported cases worldwide, including over 436,000 deaths.

Mathematical modelling is playing an important role in guiding interventions to reduce the spread of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2). Although the impact of the virus has varied significantly across the world, and different countries have taken different approaches to counter the pandemic, many national governments have introduced packages of intense non-pharmaceutical interventions (NPIs), informally known as "lockdowns".

Although the negative socio-economic impacts (e.g., significant economic costs and potential long-term mental health effects) are yet to be assessed fully, there is strong evidence that lockdowns have led to substantial reductions in transmission [1] [2] [3] .

As case numbers have either stabilised or declined in many countries, attention has now turned to developing strategies that allow restrictions to be lifted [4, 5] in order to alleviate the economic, social and other health costs of the lockdown. However, in countries with active transmission still occurring, daily disease incidence could increase again quickly, while countries that have suppressed community transmission successfully face the risk of transmission reestablishing due to reintroduction. In the absence of a vaccine or sufficient herd immunity to reduce transmission substantially, COVID-19 exit strategies pose unprecedented challenges to policy-makers and the scientific community. Given our limited knowledge of this virus, and the fact that entire packages of interventions were introduced in quick succession in many countries as case numbers increased, it is challenging to estimate the effects of removing individual measures directly and modelling remains of paramount importance.

Here, we report discussions from the "Models for an exit strategy" workshop (11-15 May 2020) which took place online as part of the Isaac Newton Institute's "Infectious Dynamics of Pandemics" programme. The Isaac Newton Institute in Cambridge is the UK's national research institute for mathematics, and many distinguished scientists (including nine Nobel laureates and 27 Fields Medallists) have attended programmes there. We outline progress to date and open questions in modelling exit strategies that arose during discussions at the workshop. Most participants were working actively on COVID-19 at the time of the workshop, often with the aim of providing evidence to governments, public health authorities and the general public to support the pandemic response. After four months of intense model development and data analysis, the workshop gave participants a chance to take stock and openly share their views of the main challenges they are facing. A range of countries were represented, providing a unique forum to discuss the different dynamics and policies around the world. Although the main focus was on epidemiological models, the interplay with other disciplines formed an integral part of the discussion. The purpose of this article is twofold: to highlight key knowledge gaps underlying the uncertainty in current predictions and projections, and to provide a roadmap for modellers and other scientists wishing to make useful contributions to the development of solutions. All evidence is summarised here in a policy-neutral manner.

The questions in this article have been grouped as follows. First, we discuss outstanding questions for modelling exit strategies that are related to key epidemiological quantities, such as the reproduction number and herd immunity fraction. We then explore different sources of heterogeneity underlying SARS-CoV-2 transmission and control, and consider whether or not differences in transmission between hosts, populations and across the world should be included in models. Finally, we discuss current challenges relating to data requirements, focussing on the data that are needed to resolve current uncertainties and how uncertainty in modelling outputs can be communicated to policy-makers and the wider public. In each case, we outline the most relevant issues, summarise expert knowledge and propose specific steps towards the development of evidence-based strategies. This leads to the development of a roadmap for future research (Fig 1) made up of three key steps: i) improve estimation of epidemiological parameters using outbreak data from different countries; ii) understand heterogeneities within and between populations that affect virus transmission and interventions, and; iii) focus on data needs, particularly data collection and methods for planning exit strategies in Low-to-Middle-Income countries (LMICs) where data are often lacking. If this roadmap can be followed, it will be possible to predict the effects of different possible exit strategies with more precision. This is of clear benefit to global health. 

The time-dependent reproduction number, !(#) or ! ! , has emerged as the main quantity used to assess the transmissibility of SARS-CoV-2 in real time [1, 6, 7] . Within a given population with active virus transmission, the value of !(#) represents the expected number of secondary cases generated by someone infected at time #. If this quantity is and remains below one, then an ongoing outbreak will eventually fade out. Although easy to understand intuitively, inferring R(t)

from case reports requires the use of mathematical models. As factors such as contact rates between infectious and susceptible individuals change during an outbreak in response to public health advice or movement restrictions, the value of !(#) has been found to respond rapidly. For example, across the UK, countrywide and regional estimates of !(#) dropped from approximately 2.5-4 in mid-March [1, 8] to below one within days of lockdown being introduced [8, 9] . At the time of writing, one of the criteria in the UK and elsewhere for relaxing the lockdown is for the reproduction number to decrease to "manageable levels" [10] . Monitoring !(#), as well as case numbers, as individual components of the lockdown are relaxed is critical for understanding whether or not the outbreak remains under control [11] .

Several mathematical and statistical methods for estimating temporal changes in the reproduction number have been proposed in the last 20 years. Two popular methods include the Wallinga-Teunis method [12] and the Cori method [13, 14] , which are designed to estimate two subtly different versions of the reproduction number (the case reproduction number and the instantaneous reproduction number, respectively [15] ). These methods use case notification data along with an estimate of the serial interval distribution (the times between successive cases in a transmission chain) to infer the value of !(#). Other approaches exist (e.g. based on compartmental epidemiological models [16] ) and can be used alongside different data (e.g. time series of deaths [1, 8] or phylogenetic data [17, 18] ).

Despite this extensive theoretical framework, practical challenges remain when dealing with real-time reporting. In particular, reproduction number estimates often rely on case notification data and are therefore subject to reporting delays between case onset and being recorded.

Observed data at time t therefore do not include up-to-date knowledge of current numbers of infections, and "nowcasting" using past data is required [6, 8, 19] . After the fact, temporal changes in reproduction number estimates can be constructed by first "back-calculating" data describing when new cases arose from case notification data. The Richardson-Lucy method, which has been applied to infer incidence curves from time series of deaths [20] , as well as other methods that account for reporting delays [21] , provide useful avenues to improve the practical estimation of !(#) given incomplete data. Furthermore, changes in testing practice (or capacity to conduct tests) leads to temporal changes in case numbers that cannot be distinguished easily from changes in transmission. This remains an important target for current research into COVID- 19 . A more immediate way to assess the reproduction number that does not require nowcasting is by observing people's transmission-relevant behaviour directly, e.g. through contact surveys or mobility behaviour [22] . These methods do, however, come with their own limitations: because these surveys do not usually collect data on infections, care must be taken in using them to understand and predict ongoing changes in transmission.

Other outstanding challenges in assessing changes in !(#) include the need to understand that methods cannot be accurate when case numbers are low, and the requirement to account for temporal changes in the serial interval or generation time distribution of the disease [23] . Indeed, in periods when there are few cases, there is little information with which to assess virus transmissibility, and so the methods described above for tracking transmission-relevant behaviour directly are particularly important. When case numbers are low, the "transmission potential" might be more important than realised transmission [24] . The effect of population heterogeneity on the reproduction number also requires further investigation, as current estimates of !(#) tend to be calculated for whole populations (e.g. countries or regions). Understanding the characteristics of individuals and constituent groups contributing to this value is important to target interventions efficiently with limited resources [25, 26] . For this, data on infections within and between different subpopulations (e.g. infections in care homes and in the wider population) are needed.

Finally, it is well documented that presymptomatic individuals (and, to a lesser extent, asymptomatic infected individuals -i.e. those who never develop symptoms) can transmit SARS-CoV-2 [27, 28] . For that reason, negative serial intervals may occur when an infected host displays COVID-19 symptoms before the person who infected them. Although methods for estimating !(#) with negative serial intervals exist [29, 30] , the inclusion of presymptomatic or asymptomatic transmission in publicly available software for estimating !(#) should be a priority. Increasing the accuracy of estimates of !(#) in the ways described here, as well as supplementing these estimates by other assessments of transmissibility (e.g. estimates of growth rates of case numbers [31] ), is of clear importance. When lockdowns are relaxed, this will permit a fast determination of whether or not interventions that are removed are leading to a surge in cases.

Herd immunity refers to the accumulation of sufficient immunity in a population through infection and/or vaccination to prevent further substantial outbreaks. It is a major factor in determining exit strategies, but data are still very limited. Dynamically, the threshold at which herd immunity is achieved is the point at which !(#) (see above) falls below one (for an otherwise uncontrolled epidemic), resulting in a negative epidemic growth rate. However, reaching the herd immunity threshold does not mean that the epidemic is over or that there is no risk of further infections. Great care must therefore be taken in communicating this concept to the public, to ensure continued adherence to preventative measures such as social distancing.

Crucially, whether immunity is gained naturally through infection or through random or targeted vaccination affects the herd immunity threshold, and the threshold depends critically on the immunological characteristics of the pathogen. Since SARS-CoV-2 is a new virus, its immunological characteristics -notably the duration and extent to which prior infection confers protection against future infection, and how these vary across the population -are currently unknown [32] . Lockdown measures have heavily impacted contact structure and hence the accumulation of immunity in the population, and are likely to have led to significant heterogeneity in acquired immunity (e.g. by age, location, workplace). Knowing the extent and distribution of immunity in the population will help guide optimal exit strategies that have only a limited risk of a resurgence in infections.

As interventions are lifted, whether or not !(#) remains below one depends on the current level of immunity in the population as well as the specific exit strategy followed. A simple illustration is to treat the current reproduction number, !(#), as a deflation of the original (basic)

reproduction number (! " ):

in which )(#) is the immunity level in the community at time t and *(#) is the overall reduction factor from the preventative measures. If )(#) > 1 − # $ ! , the time-dependent reproduction number remains below one even when all interventions are lifted: herd immunity is achieved. However, recent results [33, 34] show that, for heterogeneous populations, herd immunity occurs at a lower immunity level than 1 − # $ ! . The threshold 1 − # $ ! assumes random vaccination, with immunity distributed uniformly in the community. When immunity is obtained from disease exposure, the more socially active individuals in the population are over-represented in cases from the early stages of the epidemic. As a result, virus transmission preferentially infects those with higher numbers of contacts, thereby acting like a well-targeted vaccine. This effectively reduces the herd immunity threshold. However, the extent to which heterogeneity in behaviour lowers the herd immunity threshold for COVID-19 is currently unknown.

We highlight three key challenges for determining the herd immunity threshold for COVID-19, and hence for understanding the impact of implementing or lifting lockdowns in different populations. First, most of the quantities for calculating the herd immunity threshold remain largely unknown and require careful investigation. For example, determining the immunity level in a community is far from trivial for a number of reasons: antibody tests may have variable quality in terms of specificity and sensitivity; it is currently unclear if mild or asymptomatic infected individuals become immune and/or if they carry enough antibodies to give a positive test result; how long immunity lasts is currently unknown, and more. Second, estimation of ! " , despite receiving significant attention at the start of the pandemic, still needs to be refined within and between countries as issues with early case reports come to light and are addressed. Third, as discussed later, there is strong evidence that SARS-CoV-2 does not spread uniformly in communities [35] . An improved understanding of the main transmission routes, and which communities are most influential, will help determine how much lower disease induced herd immunity lies as compared to the classical immunity threshold 1 −

To summarise, it is of paramount importance to obtain more accurate estimates of the current immunity levels in different countries and regions, and to understand more clearly how population heterogeneity affects virus transmission and the accumulation of immunity.

Information on how many people are currently infected and have been infected in the past are key inputs to formulate exit strategies, monitor the progression of epidemics and identify social and demographic sources of heterogeneities. Seroprevalence surveys provide a simple and direct way to estimate the fraction of the population that has been exposed to the virus but has not been detected by regular surveillance mechanisms [36] .

Contacts between pathogens and hosts that elicit an immune response can be revealed by the [40] , and in certain circumstances (for example for measles) used to infer population immunity to a pathogen [41] . From a statistical perspective, cross-sectional serological surveys hold the structure of current status data since left and right censoring are in place, so we only know if a person has been infected or not (with some uncertainty) but the precise timing of infection remains unknown.

The available testing methodology is imperfect, since tests are not always perfectly accurate, and this can impact the accuracy of predictions made using compartmental models that are fitted to data on numbers of individuals testing positive. Care must be taken over the sensitivity and specificity of the test with regards to the outcome that is being assessed: whether it is the 

When ! " is greater than one, an emerging outbreak will either grow to infect a substantial proportion of the population or become extinct before it is able to [43, 44] . If instead ! " is less than one, the outbreak will almost certainly become extinct before a substantial proportion of the population is infected. If new susceptible individuals are introduced into the population (for example due to births), there is a non-zero probability that the disease will persist after its first wave and become endemic [45] . Such theoretical analyses can be extended to populations with household and network structure [46, 47] and scenarios in which ! " is very close to one [48] .

Data from multiple different diseases are consistent with model predictions in showing that extinction can be a slow process, often involving a long 'tail' of cases with significant random fluctuations away from the long-term trend ( Figure S1 ). Smallpox is the only previously endemic human disease to have been eradicated, and extinction took many decades of vaccination.

Prevalence and incidence of polio and measles have been reduced substantially through vaccination but both diseases persist. The 2001 Foot and Mouth Disease outbreak in the UK and the 2003 SARS pandemic were new epidemics that were driven extinct without vaccination before they became endemic, but both exhibited long tails before eradication was achieved. The 2014-16 Ebola outbreak in West Africa was eliminated (with vaccination at the end of the outbreak [49] ), but eradication took some time with flare ups occurring in different countries [50, 51] .

Past experience therefore raises the possibility that SARS-CoV-2 may not be driven to complete extinction in the foreseeable future, even if a vaccine is developed and vaccination campaigns can be implemented. As exemplified by the outbreak of Ebola in the Democratic Republic of the Congo that has still not been declared over [52] , there is an additional challenge of assessing whether the virus really is extinct rather than persisting in individuals that do not report disease [50] . There is a distinct possibility that SARS-CoV-2 could become endemic, either persisting in populations with limited access to healthcare or circulating in seasonal outbreaks as the virus evolves. Appropriate communication of these scenarios to the public and policy-makers -particularly the possibility that SARS-CoV-2 may never be eradicated -is of obvious importance.

A common challenge faced by modellers working in outbreak situations is the tension between making models more complex (and possibly therefore seeming more realistic and convincing to stakeholders) and maintaining simplicity (for scientific parsimony when data are sparse and for expediency when predictions are required at very short notice by policy-makers) [53] . How to strike the correct balance is not a settled question, especially given the growing amount of available data on human demography and behaviour. Indeed, outputs of multiple models with different levels of complexity can provide useful and complementary information. Many sources of heterogeneity between individuals (and between populations) exist, including the strong skew of severe COVID-19 outcomes towards the elderly and individuals from specific groups. Here, we focus on two sources of heterogeneity in human populations that must be considered when modelling exit strategies: local/geographical contact structure and health vulnerabilities. There has been considerable success in modelling local contact structure, both in terms of spatial heterogeneity (distinguishing local and long-distance contacts) and in local mixing structures such as households and workplaces. However, challenges include estimating the value of !(#)

and how it will change under interventions. In spatial models with only a small number of near-neighbour contacts, the number of new infections grows slowly, so that each generation of infected is only slightly greater than the previous one. As a result, in those models, !(#) cannot significantly exceed its threshold value of one [54] . In contrast, models accounting for transmission within closely interacting groups explicitly contain a mechanism that has a multiplier effect on the value of !(#) [46] . Another challenge is in modelling the spatial aspect of human populations: the spatial distribution of individuals is important, but long-distance contacts make populations more connected than depicted by simple percolation-type spatial models [54] . Clustering and pair approximation models can capture some aspects of spatial clustering [55] , but display exponential rather than linear growth in case numbers [56] .

While modelling frameworks exist to include almost any kind of spatial stratification, ensuring that model outputs are meaningful for exit strategy planning relies on appropriate calibration with data. This brings in challenges of merging multiple data types with different stratification levels. For example, case notification data may be aggregated at a broad regional level within a country, while mobility data from past surveys might be available at finer scales within regions.

Another challenge is to determine the appropriate scale at which to introduce or lift interventions. Although government policies are usually directed at whole populations within relevant administrative units (country-wide or smaller), more effective interventions and exit strategies may exist that target specific parts of the population [57] . Here, modelling can be helpful to account for operational costs and imperfect implementation that will offset expected epidemiological gains. Where host vulnerability is an issue, is it necessary to account for considerations other than reported risk factors, as these may be proxies for underlying causes? Another consideration is that, once communicated to the public, the results of modelling could create behavioural feedback that might help or hinder exit strategies; some sensitivity analyses would be useful. As with the questions around spatial heterogeneity, modelling variations in host vulnerability could improve proposed exit strategies, and modelling might be used to explore how these are targeted and communicated [5] . Finally, heterogeneities in space and vulnerabilities may interact;

modelling these may reveal surprises that can be explored further.

In combination with contact tracing, NPIs reduce the opportunity for transmission by breaking up contact networks (closing workplaces, schools, preventing large gatherings), reducing the chance of transmission where links cannot be broken (e.g. wearing masks, sneeze barriers) and

identifying infected individuals (temperature checks, testing). Network models [60, 61] aim to split pathogen transmission into opportunity (number of contacts) and transmission probability, using social network data that can be measured directly (through devices such as mobility tracking and contact diaries) and indirectly (through traffic flow and co-occurrence studies). This brings new issues: for example, are observed networks missing key routes of transmission, such as indirect contact via contaminated surfaces, or adding noise in the form of contacts that are low risk [62] ? How we measure and interpret contact networks also depends on the geographical and social scale of interest (e.g. wider community spread or closed populations such as prisons and care homes; or sub-populations such as workplaces and schools) and the timescale over which we want to use the network to understand or predict infection chains.

In reality, individuals belong to households, children attend schools and adults mix in workplaces as well as in social contexts. This has led to the development of household models [46, [63] [64] [65] [66] , multilayer networks [67] , bipartite networks [68, 69] and networks that are geographically-and socially-embedded to reflect location and travel habits [70] . These Contact tracing, followed by isolation or treatment of infected contacts, is a well-established method of disease control. The structure of the contact network is important in determining whether or not contact tracing will be successful. For example, contact tracing in clustered networks is known to be more effective than in equivalent non-clustered networks [71, 72] , since a potentially infected contact can be traced from multiple different sources. Knowledge of the contact network enhances understanding of the correlation structure that emerges as a result of the epidemic. The first wave of an epidemic will typically infect many of the highly connected nodes and will slowly move to less connected parts of the network leaving behind islands of removed and susceptible individuals. This can lead to a correlated structure of susceptible and recovered nodes that may make the networks less vulnerable to later epidemic waves [73] , and has implications for herd immunity (see above).

In heterogeneous populations, relatively few very well-connected people can have a large impact on the spread of infectious disease and be major hubs for transmission. Such individuals are often referred to as super-spreaders [74, 75] and some theoretical approaches to controlling epidemics are based on targeting them [76] . However, particularly for respiratory diseases, whether it is a case of individuals being super-spreaders or whether any infected individual has the potential to generate a super-spreading events is debated [26, 77, 78] .

As control policies are gradually lifted, the disrupted contact network will start to form again.

Understanding how proxies for social networks (which can be measured in near-real time using mobility data, electronic sensors or trackers) relate to transmission requires careful consideration.

Using observed contacts to predict virus spread might be successful if these quantities are heavily correlated, but one aim of NPIs should be at least a partial decoupling of the two, so that society can reopen but spread remains controlled. The extent to which this is possible is unclear and is likely to vary between regions. Currently, a key empirical and theoretical challenge is to understand how households are connected and how this is affected by school opening (see [79, 80] , now adopted in several countries, provide the opportunity not only to improve our understanding of critical epidemiological characteristics (such as incubation periods, generation intervals and the roles of asymptomatic and presymptomatic transmission) but also to make many of the comparisons just outlined.

An early intervention implemented in many countries was school closure, which is frequently used during influenza pandemics [81, 82] . Further, playgrounds were closed and strict social distancing has kept children separated. However, compared with influenza, the role of children in SARS-CoV-2 transmission is unclear. Early signs from Wuhan (China), echoed elsewhere, showed many fewer cases in under 20s than might be expected. There are three aspects of the role of children in transmission: i) susceptibility to infection; ii) infectiousness once infected, and; iii) propensity to develop disease if infected [83] [84] [85] . Evidence for variation in age-specific susceptibility to infection and infectiousness is mixed, with infectiousness the more difficult to quantify. However, evidence is emerging of lower susceptibility to infection in children compared to adults, although the mechanism underlying this is unknown and it may not be generalisable to all settings. Once infected, children appear to have a milder course of infection than adults, and it has been suggested that children have a higher probability of a fully subclinical course of infection.

Reopening schools is of clear importance both in ensuring equal access to education and enabling caregivers to return to work. However, the risk of transmission within schools and the potential impact on the rate of community transmission needs to be understood in order for policy-makers to balance the potential benefits and harms. As schools begin to reopen there are key questions that models can help with, and major knowledge gaps that prevent clear answers.

The most pressing question at a regional and national level is the extent to which school restarting will affect population-level transmission, characterised by !(#) (see above). Clearer Return to school affects the number of physical contacts that teachers and support staff have.

Schools will not be the same environments as prior to lockdown, since physical distancing measures will be in place. These include smaller classes and changes in layout, plus increased hygiene measures to decrease transmission. This is critical not only to decrease transmission from children to teachers (and vice versa) and between children, but also to decrease the risk of transmission between teachers. Some teachers may be unlikely to return because of the presence of underlying conditions and a need to "shield", and if there is transmission within schools, there may be absenteeism following infection. Models must therefore consider the different transmission effects of pre-and post-lockdown school environments. Post-lockdown, with strong social distancing in place in the wider community, reopening schools could link subcommunities of the population together, and models can be used to estimate the wider effects on population transmission as well as those within schools themselves. These estimates are likely to play a central role in decisions surrounding when and how to reopen schools in different countries.

As the pandemic progresses, so does the need for different modelling approaches that account for population structure and heterogeneity (see above). While these effects can be approximated in standard compartmental epidemiological models [3] , such models can become highly complex and cumbersome to specify and solve as more sources of heterogeneity are introduced. An alternative modelling paradigm is agent-based modelling. Agent-based models (ABM) allow complex systems such as societies to be represented, using virtual agents programmed to have behavioural and individual characteristics (age, sex, ethnicity, income, employment status, etc.) as well as the capacity to interact with other agents [87] . In addition, ABM can include the effects of societal-level factors such as the influence of social media, regulations and laws, and community norms. In more sophisticated ABM, agents can also anticipate and react to future While ABM arise from a different tradition, they can incorporate the insights of compartmental models; for example, agents must transition through disease states (or compartments) such that the mean infection rates correspond to the rates that quantify flows in compartmental models.

However, building an ABM that represents a population on a national scale is a huge challenge and one that cannot be accomplished in a timescale useful for the current pandemic. ABM often include many parameters, leading to challenges of model parameterisation and a requirement for careful uncertainty quantification and sensitivity analyses to different inputs. On the other hand, useful ABM do not have to be all-encompassing. There are already several models accessible to the public that illustrate the effects of various policies such as social distancing on small simulated populations, and such models can also be very helpful as "thought experiments" to identify the potential effects of candidate policies such as school re-opening, the consequences of non-compliance with government edicts, the impact of restrictions on long-distance travel, and other features.

There are two areas where action should be taken, both of which are long-term and intended to assist with dealing with the almost inevitable next pandemic rather than this one. First, more data about people's ordinary behaviour are required: what individuals do each day (through time-use diaries), whom they meet (possibly through mobile phone data, assuming consent can be obtained), and how they understand and act on government regulation, social media influences, broadcast information and so on [88] . Second, we should start building a large, modular ABM that represents heterogeneities in populations and that is properly calibrated as a social 'digital twin' of our own society, with which we can carry out virtual policy experiments. Had these developments occurred before, they would be useful in the current situation. As a result, if these are done now, they will aid the planning of exit strategies in future.

In most countries, criteria for ending COVID-19 lockdowns rely on tracking trends in numbers of confirmed cases and deaths, and assessments of transmission rates (see above). However,

there are a number of other challenges in determining when interventions should be relaxed in LMICs.

In some countries, data about the COVID-19 epidemic and about the general population and context can be unreliable or lacking. Because of limited healthcare access and utilisation, there can be fewer opportunities for diagnosis and subsequent confirmation of cases in LMICs compared to other settings, unless there are active programs [89] . Distrust can make monitoring programs difficult, and complicate control activities like test-trace-isolate campaigns [90, 91] .

Other options for monitoring -such as assessing excess disease from general reporting of acute respiratory infections or influenza-like-illness -require historical baselines that may not exist [92, 93] . In general, while many countries will have a well-served fraction of the population, dense peri-urban and informal settlements are typically outside that population and may rapidly become a primary concern for transmission [94] . Since confirmed case numbers in these populations are unlikely to provide an accurate representation of the underlying epidemic, reliance on alternative measures such as clinically diagnosed cases may be necessary to understand the epidemic trajectory.

In addition to the challenges in understanding the pandemic in these settings, metrics on health system capacity (including resources such as beds and ventilators), as needed to set targets for control, are often poorly documented [95] . Furthermore, the economic hardships and competing health priorities in low-resource settings change the priorities associated with lifting restrictions -hunger due to loss of jobs and changes in access to routine health care (e.g. HIV services and childhood vaccinations) as a result of lockdown have the potential to cost many lives in themselves, both in the short-and long-term [96, 97] . To assess the costs and benefits of lifting restrictions appropriately, additional data on these conditions may be required. In many settings, these data are unavailable.

These challenges suggest three key elements of modelling efforts for moving forward in datalimited settings: i) identification of policy responses that are robust to missing information; ii)

value-of-information analyses to prioritise additional data collection and curation efforts, and; iii) development of methods that use metadata to interpret epidemiological patterns.

In scenarios in which additional data are cost-prohibitive to collect, models may be able to provide a clearer picture by incorporating more metadata, such as testing and reporting rates through time, sample backlogs, and indirect measures of COVID-19 such as numbers of diagnosed potential cases. By identifying the data that could be useful, modelling could encourage countries to make data that are already collected readily available. For example, in high incidence settings, burial/death certificate data may be available, and these data can provide information on key demographics that influence the infection fatality rate. These can in turn reveal potential COVID-19 deaths classified as other causes and hence missing from COVID-19 attributed death notifications.

In general, supporting LMICs calls for creativity in the data that might be incorporated in models and in the response activities that are undertaken. This will have to account carefully for aspects that may be reasonably be simplified in high income settings, particularly the downsides of lockdowns. Some LMICs have managed the COVID-19 pandemic successfully so far (e.g.

Vietnam, as well as Trinidad and Tobago [98] ). However, additional support in LMICs is required: data limited settings represent uniquely high stakes.

Understanding the effects of the different components of lockdown is important to understand how -and in which order -interventions should progressively be released. The impact of previous measures must be understood both to inform policy in real-time and to ensure that lessons can be learned from the current pandemic.

Models vary from those that include few parameters but can offer powerful and robust insight into the potential impact of different strategies, to highly complex simulations aiming to capture all nuances affecting transmission (see above). Complex simulations are often sensitive to uncertainties in the many assumed parameters and model structure. However, all models ultimately require information to make their predictions relevant to the ongoing pandemic. Data from PCR tests for presence of active virus and serological tests for antibodies, together with data on COVID-19 related deaths, are freely available via a number of internet sites (e.g. [99] ). However, metadata associated with testing protocols (e.g. reason for testing, type of test, breakdowns by age and underlying health conditions) and the definition of COVID-19 related death, which are needed to quantify sources of potential bias and correctly parameterise models, are often unavailable. New sources of data also exist, ranging from tracking data from mobile phones [100] to social media surveys [101] and details of interactions with public health providers [102] . Although potentially valuable, these data sources bring with them biases that are, currently at least, imperfectly understood. These types of data are also often subject to data protection and/or costly fees, meaning that they are not readily available to many modelling groups. Mixing patterns by age were reasonably well-characterised before the current pandemic [103, 104] (particularly for adults of different ages) and have been used extensively in existing models. However, there are gaps in these data and uncertainty in the impacts that different interventions have had on mixing. Predictive models for policy tend to make broad assumptions on the effects of elements of social distancing [105] , although results of studies are beginning to emerge that attempt to estimate effects in a more data-driven way [106] . The future success of modelling efforts to understand when controls should be relaxed or tightened depends critically on whether, and how accurately as well as how quickly, the effects of different elements of lockdown can be parameterised.

Given the many differences in lockdown implementation between countries, cross-country comparisons offer an opportunity to estimate the effects on transmission of each component of lockdown measures [1] . We recommend that a repository, open to all, is developed with timelines of interventions in different countries as well as expert anecdotal knowledge describing the actual degree of take-up by populations. The University of Oxford Coronavirus Government Response Tracker [98] is a useful starting point, which could be expanded and integrated with links to surveillance data, behavioural studies and surveys of population behaviour for each country. There is also a need to coordinate the design of epidemiological studies, involving longitudinal data collection and case-control studies, to enable models to track and compare the progress of the epidemic and the impacts of control policies internationally. This would allow more refined conclusions than those that follow from population data alone. Countries with substantial epidemiological modelling expertise should support epidemiologists elsewhere with standardised protocols for collecting data and using models to inform policy. Of course, there is also a need for sharing of models to be used "in the field" in different settings. Collectively, these efforts will ensure that models are parameterised as realistically as possible for the particular settings in which they are to be used. In turn, as interventions are relaxed, this will allow us to detect the earliest possible reliable signature of a resurgence in cases should it occur, leading to an unambiguous characterisation of when it is necessary for some interventions to be reintroduced.

SARS-CoV-2 transmission models have played a crucial role in shaping policies in different countries, and their predictions (and the scenarios that they have been used to explore) have been a regular feature of media coverage of the pandemic [105, 107] . Understandably, both policymakers and journalists generally prefer single "best guess" figures from models, rather than a range of plausible values. However, the ranges of outputs that modellers provide include important information about the quality of the information being presented and guard against over-interpretation of model results. Not presenting information about uncertainty can convey a false confidence in predictions, and it is critical that modellers present uncertainty in a way that is understandable and useful for policy-makers and the public [53] .

There are numerous and often inextricable ways in which uncertainty enters the modelling process. As discussed above, any model includes assumptions that inevitably vary according to judgements regarding which features should be included in the model [1, 94] and which datasets are used to inform the model [108] . Within any model, ranges of parameter values can be considered to allow for uncertainty about clinical characteristics of COVID-19 (e.g. its infectious period and case fatality ratio) [109] . Alternative initial conditions (e.g. numbers and locations of imported cases seeding national outbreaks, or levels of population susceptibility) can be considered. In modelling exit strategies, when surges in cases starting from small numbers may occur and where predictions will depend on characterising the parameters as accurately as possible, stochastic models may be of particular importance. Not all the uncertainty arising from such stochasticity will be reduced by collecting more data. It is inherent to the process. narrowly defined requirement, such as short-term predictions of cases and deaths, more tractable tools for comparing the outputs from different models in real-time would be valuable. One possible approach is to assess and compare the models' past performance at making predictions [110, 111] . The use of ensemble estimates, most commonly applied for forecasting disease trajectories, is a way to synthesise multiple models' predictions into a single estimate [112] . The assessment and comparison of past performance can then be used to weight models in the ensemble. Such approaches typically lead to improved point and variance estimates.

To deal with parameter uncertainty, a common approach is to perform sensitivity analyses in which model parameters are repeatedly sampled from a range of plausible values, and the resulting model predictions compared; both classical and Bayesian statistical approaches can be employed [113] [114] [115] . Methods of uncertainty quantification provide a framework in which uncertainties with regard to model structure, values of epidemiological parameters, and data can be considered together. In practice, there is usually only a limited number of possible policies that can be implemented. An important question is often whether or not the optimal policy can be identified given the uncertainties we have described, and decision analyses can be helpful for this purpose [116, 117] .

In summary, communication of uncertainty to policy-makers and the general public remains a challenging area. Different levels of sophistication may be required for different audiences.

There are many subtleties: for instance, almost any epidemic model can provide an acceptable fit to data in the early phase of an outbreak, since models almost invariably predict exponential growth. This can induce an artificial belief that the model must be based on sensible underlying assumptions, and the true uncertainty about such assumptions has vanished. Clear presentation of data is vital, for example including denominators as well as numerators in numbers of deaths, or relating the number of cases to the number of individuals who have been tested. Improved communication of model and parameter uncertainty is important as models are used to predict the effects of different exit strategies.

In this article, we have highlighted a number of ongoing challenges in modelling the COVID-19

pandemic, and uncertainties faced by most countries devising lockdown exit strategies. It is important, however, to put these issues into context: a few months ago, at the start of 2020, the virus was unknown, and its pandemic potential only became apparent at the end of January. The speed with which the scientific and public health communities came together to tackle this challenge and the openness in sharing data, methods and analyses are unprecedented. At very short notice, epidemic modellers were able to mobilise a substantial workforce -mostly on a voluntary basis -and state-of-the-art computational models. Far from the rough-and-ready tools sometimes depicted in the media, the modelling effort deployed since January is a collective and multi-pronged effort benefitting from years of experience of outbreak modelling, often combined with long-term engagement with public health agencies and policy-makers.

Drawing on this collective expertise, the virtual meeting convened in mid-May by the Isaac Newton Institute generated a clear overview of the steps needed to improve and validate the scientific advice to guide lockdown exit strategies. Importantly, the roadmap outlined in this paper is meant to be feasible within the lifetime of the pandemic. Unlike other scientific fields, epidemiology does not have the luxury of waiting for all data to become available before fully validated models can be developed. As discussed here, the solution lies in using diverse and flexible modelling frameworks, which can be iteratively revised and improved as more data become available. Equally important is the ability to assess the data critically and bring together evidence from multiple fields: numbers of cases and deaths reported by regional or national authorities only represent a single source of data, and expert knowledge is even required to interpret these data correctly.

In this spirit, our first recommendation is to improve estimates of key epidemiological parameters. This requires close collaboration between epidemic modellers and the individuals and organisations that collect epidemic data, so that the caveats and assumptions on each side are clearly presented and understood. That was a key message from the first section of this study, in which the relevance of theoretical concepts and model parameters in the real world was demonstrated: far from ignoring the complexity of the pandemic, models draw from multiple sources of expertise to make sense of imperfect observations. By acknowledging the simplifying assumptions of models, we can assess their relative impact and validate or replace them as new evidence becomes available.

Our second recommendation is to seek to understand important sources of heterogeneity that appear to be driving the pandemic and its response to interventions. Agent-based models are one possibility for modelling complex dynamics, but standard epidemic models can also be extended to include age groups or any other relevant strata in the population as well as spatial structure.

Network models provide computationally efficient frameworks to capture different types of epidemiological and social interactions. Importantly, multiple modelling frameworks provide avenues for collaboration with other fields, especially social sciences.

Our third and final recommendation regards the need to focus on data requirements, particularly in resource limited settings such as LMICs. Understanding the data required for accurate predictions in different countries requires close communication between modellers and governments, public health authorities and the general public. While this pandemic casts a light on social inequalities between and within countries, modellers have a crucial role to play in sharing knowledge and expertise with those who need it most. In LMICs, cost-effective guidance can be provided by models validated with global data. In the last few months, countries that We are now moving into a stage of the COVID-19 pandemic in which data collection and novel research to inform the modelling issues discussed here are both possible and essential for global health. These are international challenges that require an international collaborative response from diverse scientific communities, which we hope that this article will stimulate. This is of critical importance for global health, not only to tackle this pandemic but also to improve the response to future outbreaks of emerging infectious diseases.

Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11

The effect of control strategies to reduce social mixing on outcomes of the COVID-19 epidemic in Wuhan, China: a modelling study

Epidemiological models are important tools for guiding COVID-19 interventions

How and when to end the COVID-19 lockdown: an optimisation approach. medRxiv

Segmentation and shielding of the most vulnerable members of the population as elements of an exit strategy from COVID-19 lockdown

Temporal variation in transmission during the COVID-19 outbreak. C Repos

Impact assessment of nonpharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study

MRC Biostatistics Unit COVID-19 Working Group. Nowcasting and Forecasting Report

Government Office for Science. Government publishes latest R number

Our plan to rebuild: The UK government's COVID-19 recovery strategy

Estimating in real time the efficacy of measures to control emerging communicable diseases

Different epidemic curves for severe acute respiratory syndrome reveal similar impacts of control measures

A new framework and software to estimate timevarying reproduction numbers during epidemics

Improved inference of time-varying reproduction numbers during infectious disease outbreaks

Estimating individual and household reproduction numbers in an emerging epidemic

Assessing the impact of nonpharmaceutical interventions on SARS-CoV-2 transmission in Switzerland. medRxiv

Birth-death skyline plot reveals temporal changes of epidemic spread in HIV and hepatitis C virus (HCV)

Modeling the growth and decline of pathogen effective population size provides insight into epidemic dynamics and drivers of antimicrobial resistance

Nowcasting pandemic influenza A/H1N1 2009 hospitalizations in the Netherlands

Reconstructing influenza incidence by deconvolution of daily mortality time series

Adjusting COVID-19 deaths to account for reporting delay

Using mobility to estimate the transmission intensity of COVID-19 in Italy: A subnational analysis with future scenarios

A note on generation times in epidemic models

Estimating temporal variation in transmission of COVID-19 and adherence to social distancing measures in Australia

Estimating reproduction numbers for adults and children from case data

Superspreading and the effect of individual variation on disease emergence

Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science (80-)

Time from symptom onset to hospitalisation of coronavirus disease 2019 (COVID-19) cases: Implications for the proportion of transmissions from infectors with few symptoms

Estimating the generation interval for COVID-19 based on symptom onset data

Estimating the time interval between transmission generations when negative values occur in the serial interval data: using COVID-19 as an example

Challenges in control of Covid-19: short doubling time and long delay to effect of interventions

Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period. Science (80-)

The disease-induced herd immunity level for Covid-19 is substantially lower than the classical herd immunity level

Individual variation in susceptibility or exposure to SARS-CoV-2 lowers the herd immunity threshold. medRxiv

UK Government Office for National Statistics. Coronavirus (COVID-19) Infection Survey pilot: England

Use of serological surveys to generate key insights into the changing global landscape of infectious disease

Quantifying antibody kinetics and RNA shedding during early-phase SARS-CoV-2 infection. medRxiv

What policy makers need to know about COVID-19 protective immunity

Interpreting diagnostic tests for SARS-CoV-2

Seventy-five years of estimating the force of infection from current status data

Benefits and challenges in using seroprevalence data to inform models for measles and rubella elimination

Viral kinetics and antibody responses in patients with COVID-19. medRxiv

When does a minor outbreak become a major epidemic? Linking the risk from invading pathogens to practical definitions of a major epidemic

Stochastic epidemic models and their statistical analysis

Intervention to maximise the probability of epidemic fade-out

Epidemics with two levels of mixing

The use of chain-binomials with a variable chance of infection for the analysis of intrahousehold epidemics

Extinction times in the subcritical stochastic SIS logistic epidemic

Efficacy and effectiveness of an rVSV-vectored vaccine expressing Ebola surface glycoprotein: interim results from the Guinea ring vaccination cluster-randomised trial

Rigorous surveillance is necessary for high confidence in end-of-outbreak declarations for Ebola and other infectious diseases

Sexual transmission and the probability of an end of the Ebola virus disease epidemic

Ebola Virus Disease: Democratic Republic of the Congo External Situation Report 88

Influencing public health policy with data-informed mathematical models of infectious diseases: Recent developments and new challenges

Five challenges for spatial epidemic models

The effects of local spatial structure on epidemiological invasions

Pair approximations for spatial structures? Oberwolfach Rep

Management of invading pathogens should be informed by epidemiology rather than administrative boundaries

UK Government Office for National Statistics. Coronavirus-related deaths by ethnic group

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

Mathematics of epidemics on networks: from exact to approximate models

Epidemic processes in complex networks

A novel field-based approach to validate the use of network models for disease spread between dairy herds

Analysis of a stochastic SIR epidemic on a random network incorporating household structure

Reproduction numbers for epidemic models with households and other social structures II: Comparisons and implications for vaccination

Reproductive numbers, epidemic spread and control in a community of households

Reproduction numbers for epidemic models with households and other social structures. I. Definition and calculation of R0

Modeling the impact of social distancing testing contact tracing and household quarantine on second-wave scenarios of the COVID-19 epidemic

Epidemics on random intersection graphs

Epidemics on random graphs with tunable clustering

Social networks with strong spatial embedding generate nonstandard epidemic dynamics driven by higher-order clustering. bioRxiv

Disease contact tracing in random and clustered networks

Time is of the essence: impact of delays on effectiveness of contact tracing for COVID-19. medRxiv

Network frailty and the geometry of herd immunity

Spread of epidemic disease on networks

Random Graph Dynamics

Graphs with specified degree distributions, simple epidemics, and local vaccination strategies

Social encounter networks: Collective properties and disease transmission

Social encounter networks: Characterizing Great Britain

World Health Organization. The First Few X (FFX) Cases and contact investigation protocol for 2019-novel coronavirus (2019-nCoV) infection

Characterising pandemic severity and transmissibility from data collected during first few hundred studies

Closure of schools during an influenza pandemic

Estimating the impact of school closure on influenza transmission from sentinel data

Agedependent effects in the transmission and control of COVID-19 epidemics. medRxiv

School closure and management practices during coronavirus outbreaks including COVID-19: a rapid systematic review

Impact assessment of nonpharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study

Stepping out of lockdown should start with school re-openings while maintaining distancing measures. Insights from mixing matrices and mathematical models. medRxiv

Agent-based models

Computational models that matter during a global pandemic outbreak: A call to action

Health security capacities in the context of COVID-19 outbreak: an analysis of International Health Regulations annual report data from 182 countries

Historical parallels, Ebola virus disease and cholera: Understanding community distrust and social violence with epidemics

The ongoing Ebola epidemic in the Democratic Republic of Congo

A review of the surveillance systems of influenza in selected countries in the tropical region

The INDEPTH network: Filling vital gaps in global epidemiology

Local response in health emergencies: key considerations for addressing the COVID-19 pandemic in informal urban settlements

Introducing The Lancet Global Health Commission on High-Quality Health Systems in the SDG Era

The impact of COVID-19 control measures on social contacts and transmission in Kenyan informal settlements. medRxiv

An appeal for practical social justice in the COVID-19 global response in low-income and middle-income countries

University of Oxford Blavatnik School of Government. Coronavirus government response tracker

An interactive web-based dashboard to track COVID-19 in real time

Mobile phone data for informing public health actions across the COVID-19 pandemic life cycle

Early epidemiological analysis of the coronavirus disease 2019 outbreak based on crowdsourced data: a population-level observational study

Analysis of temporal trends in potential COVID-19 cases reported through NHS Pathways England. medRxiv

Projecting social contact matrices in 152 countries using contact surveys and demographic data

Contagion! The BBC Four Pandemic -The model behind the documentary

Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand

Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions

Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia

Four key challenges in infectious disease modelling using data from multiple sources

How will country-based mitigation measures influence the course of the COVID-19 epidemic?

Prequential data analysis. Current Issues in Statistical Inference: Essays in Honor of D Basu

Optimising renewal models for real-time epidemic prediction and estimation. bioRxiv

Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the

Control fast or control smart: When should invading pathogens be controlled?

Accurate quantification of uncertainty in epidemic parameter estimates and predictions using stochastic compartmental models

Fitting dynamic models to epidemic outbreaks with quantified uncertainty: A primer for parameter uncertainty, identifiability, and forecasts

Infectious disease pandemic planning and response: Incorporating decision analysis

Improving the evidence base for decision making during a pandemic: The example of 2009 influenza A/H1N1

The authors would like to thank the Isaac Newton Institute for Mathematical Sciences, Cambridge (www.newton.ac.uk), for support during the virtual "Infectious Dynamics of Pandemics" programme where work on this paper was begun. 

The authors declare that no ethical concerns exist.

Data sharing is not applicable to this manuscript as no new data were created or analysed in this study.

The authors declare that no competing interests exist.

RNT, TDH, VI, HH, DM and OR organised the workshop and designed the study. All authors attended the workshop, contributed to discussions and wrote sections of the manuscript. RNT compiled the manuscript. All authors edited the manuscript and approved the final version for publication. Figure S1 .