key: cord-0832639-hi8nqrc5
authors: Teerawattananon, Yot; Sarin, KC; Chi, Y-Ling; Dabak, Saudamini; Kazibwe, Joseph; Clapham, Hannah; Lopez, Claudia; Leung, Gabriel; Sharifi, Hamid; Habtemariam, Mahlet; Blecher, Mark; Nishtar, Sania; Sarkar, Swarup; Wilson, David; Chalkidou, Kalipso; Gorgens, Marelize; Hutubessy, Raymond; Wibulpolprasert, Suwit
title: Recalibrating the notion of modelling for policymaking during pandemics
date: 2022-03-02
journal: Epidemics
DOI: 10.1016/j.epidem.2022.100552
sha: 0aa8403fae904f28db9a77d55aba3a8205ac33a2
doc_id: 832639
cord_uid: hi8nqrc5

COVID-19 disease models have aided policymakers in low-and middle-income countries (LMICs) with many critical decisions. Many challenges remain surrounding their use, from inappropriate model selection and adoption, inadequate and untimely reporting of evidence, to the lack of iterative stakeholder engagement in policy formulation and deliberation. These issues can contribute to the misuse of models and hinder effective policy implementation. Without guidance on how to address such challenges, the true potential of such models may not be realised. The COVID-19 Multi-Model Comparison Collaboration (CMCC) was formed to address this gap. CMCC is a global collaboration between decision-makers from LMICs, modellers and researchers, and development partners. To understand the limitations of existing COVID-19 disease models (primarily from high income countries) and how they could be adequately support decision-making in LMICs, a desk review of modelling experience during the COVID-19 and past disease outbreaks, two online surveys, and regular online consultations were held among the collaborators. Three key recommendations from CMCC include: (Caiado BCaC, 2020) A ‘fitness-for-purpose’ flowchart, a tool that concurrently walks policymakers (or their advisors) and modellers through a model selection and development process. The flowchart is organised around the following: policy aims, modelling feasibility, model implementation, model reporting commitment. Holmdahl and Buckee (2020) ‘Reporting standards trajectory’, which includes three gradually increasing standard of reports, ‘minimum’, ‘acceptable’, and ‘ideal’, and seeks collaboration from funders, modellers, and decision-makers to enhance the quality of reports over time and accountability of researchers. Malla et al., (2018) A framework for “collaborative modelling for effective policy implementation and evaluation” which extends the definition of stakeholders to funders, ground-level implementers, public, and other researchers, and outlines how each can contribute to modelling. We advocate for standardisation of modelling process and adoption of country-owned model through iterative stakeholder participation and discuss how they can enhance trust, accountability, and public ownership to decisions.

There is a growing reliance by low-and middle-income countries (LMICs) to adapt readily available COVID-19 disease models developed in high-income countries (HICs). Lack of technical capacity to develop models locally and the demand for urgent decision-making may be contributing to this trend. While the need for local models is undebated (1), the impact of such adapted models may be impeded due to challenges surrounding their use, from inappropriate model selection, inadequate and untimely reporting of evidence, and lack of iterative stakeholder engagement in policy formulation and implementation (2) . Models built in HICs for HICs, by their sheer size, resources available, and number of people involved in their development, have had more opportunities to be validated, iterated, and contextualised. However, the same cannot be said for LMICs where effective knowledge transfer in public health are hindered by complexities in creating and accessing evidence, contextualising knowledge translations strategies, among others factors (3) . Hence, the full set of locally relevant constraints may not be adequately reflected in these global models that are adapted for LMICs. Together, these elements can contribute to the misuse of models and hinder effective policy implementation. In the absence of guidance to identify and address such challenges, the true potential of such models may not be realised. In response, we convened a Policy Group comprising decision-makers from LMICs who were engaged in responding to the pandemic in their respective countries under the COVID-19 Multi-Model Comparison Collaboration (CMCC). Members of this group represented the Ministry of Public Health, Thailand, Africa Center for Disease Control and Prevention (CDC), Mayor"s Office of the City of Bogota, Prime Minister"s Office of Pakistan, Indian Medical Research Council (ICMR), Government of India, National Treasury of South Africa, COVID-19 Task Force of Hong Kong, and Government of Iran. The CMCC is a collaboration between policymakers, disease modelling experts, and development partners, a global initiative led by a group of organisations representing different constituencies namely, the Bill and Melinda Gates Foundation (BMGF), the International Decision Support Initiative (iDSI), the World Bank Group (WBG), and the World Health Organisation (WHO). As part of this engagement, we organised consultations, conducted two online surveys, and a desk review of modelling experience during the COVID-19 and past disease outbreaks. The online surveys were conducted among decisionmakers from LMICs, COVID-19 disease modellers and researchers, and international funders. The purpose of these surveys was to understand the limitations of existing COVID-19 disease models (primarily from HICs) and how they could be adequately adapted to LMICs. This engagement led to the development of our policy report, 'Guidance on Use of Modelling for Policy Responses to , which intends to support decision-makers, their advisors, and the research community to build greater collaboration for effective policy implementation and evaluation. Our technical report, 'Model Fitness-for-Purpose Assessment Report' (6) , provides a comprehensive guide on model comparison, outlining their purpose, strengths and J o u r n a l P r e -p r o o f 4 weaknesses, data needs, etc., and how they can be tailored to the local context. These reports are complimentary, therefore, best read in conjunction. In this paper, we summarise key findings and recommendations from the report and discuss their implications on research and policy implementation. Throughout this paper, we advocate for the standardisation of modelling process and the adoption of country-owned models through stakeholder participation and discuss how they can enhance strength of evidence, trust, accountability, and public ownership to decisions. While our recommendations are tangible in nature, they should be viewed as frameworks with general principles to follow while modelling for decision making. We urge countries to contextualise our recommendations, depending on the resources available, policy problem, and stakeholders involved and their roles in the country, and draft individualised action plans as necessary. The strength of our approach lies in the global stakeholder engagement undertaken to understand the issues related to model use and resulting decision tools that were reviewed and endorsed as "necessary". Applied to this outbreak, these tools can ensure that decision-makers and their advisors are equipped to adequately use such models and ensure they are fit-for-purpose for local decision-needs.

Many types of COVID-19 disease models exist with varying structure, assumptions, strengths, limitations, and abilities to address a given decision problem (6) . It is worth noting that these models are not designed to answer all COVID-19 related questions decision-makers may have (7) . For instance, the issue of vaccine hesitancy cannot be understood using disease models. This requires estimating the magnitude (number of people) of vaccine hesitancy in the country and understanding their profile (their reasons for being hesitant). Such information may be retrieved via nation-wide surveys and not through disease models. Other examples include measuring realworld effectiveness of COVID-19 vaccines or efficacy of mask wearing in preventing infection which require other forms of research technique (for example, randomised controlled or observational studies using regression analysis), and therefore, not addressable by disease models. However, these are crucial pieces of information used by disease models to estimate parameters such as the and predict the outcomes of the pandemic such as number of cases, required hospital beds, etc. Disease models are therefore not a panacea for all policy questions, and it is important to distinguish questions not addressable by them and communicate to decision-makers who may need to resort to other research methods or groups for answers. Indepth discussion on the limitations of such models have been discussed elsewhere (2, 6) . Furthermore, one must recognise all models can be prone to inaccurate predictions, for example, consistent overestimation of COVID-19 deaths or resources required in hospitals during the early stages of the pandemic (2). However, models remain the best tool available to us to project future outcomes and draft appropriate response. A large part of model inaccuracies may be attributed to poor quality and limited data or model structure not capturing the extent of disease dynamics and local setting, especially when dealing with a novel virus. Hence, predictions must be accompanied by uncertainties and their implications on whether reasonable decisions can still be J o u r n a l P r e -p r o o f 5 made. This can be used by decision-makers when communicating the level of confidence they have on a certain policy based on model findings. For those reasons, blindly adopting a model without understanding its purpose and limitations can be costly in lives, time, and resources wasted, and must be avoided. Hence, there is a need for guidance that concurrently walks decision-makers and modellers in the approach for fine tuning the model selection and development process. Catering to this need, we designed a fitness-for purpose flowchart, depicted in Figure 1 . The flowchart is a model selection and development tool which is organised around the following: policy aims, modelling feasibility, model implementation, model reporting and commitment. The aim is to guide policymakers through a sequence of questions that should be raised while considering the use of models without discussing in detail about technical issues, for instance, each data needed to populate the model and how to compare, assess, calibrate, or validate models. Comprehensive guidance on these issues is available in Tables 2-6 of our technical report (6) . The flowchart helps assess the suitability of models to answer a given policy question, availability of appropriate models, and feasibility in implementing such models to a local setting. The flowchart ends with questions around reporting standards (discussed in the subsequent section). The use of this flowchart is a collaborative process that offers benefits beyond just the selection of the right model. Direct communication with policymakers or their advisors means the modellers can identify whether their modelling efforts are serving the decision needs of the hour. If not, modelling efforts can be repurposed. Decision-makers can further contribute to this process by bringing insights on policymaking and country context (political or wider context), as well as collaborations that can support data collection, model calibration, and validation where appropriate. Modellers can help shape decision-makers" perspectives on the decision-problem and viable interventions by pointing to relevant literature or international experience. This twoway communication can help identify gaps in the current modelling effort for e.g., lack of local data, in-house technical capacity to produce, appraise, and consume evidence, etc. which the funders, decision-makers, and modellers can collectively find means to invest in and improve on. This can help mitigate issues resulting from inaccurate predictions from models or finding other methods to address policy questions as highlighted earlier. We discuss this further in our collaborating modelling framework.

Not all factors highlighted in the flowchart can always be fully met, therefore, modellers and decision-makers need to balance the trade-offs and decide where to compromise. The extent to which a model is fit-for-purpose will ultimately be a matter of judgement resulting from an ongoing dialogue between policymakers and modellers, informed by the answers to these questions. Hence, we encourage an iterative model development and selection process to ensure evolving disease dynamics, newly available evidence, and evolving decision problems are adequately captured and addressed by models. This flowchart has been applied during the pandemic and was adopted in Singapore and Thailand while modelling the impact and conducting economic evaluation of several COVID-19 vaccines which subsequently informed the national vaccination policy (8) . Upholding scientific standards and accountability "Crises are no excuse for lowering scientific standards", say Ethicists from Carnegie Mellon and McGill Universitie (9) . Since the outbreak, the speed at which evidence is being generated is unparalleled and maintaining the quality of evidence and accountability of evidence producers and users have been problematic (10, 11) . Decisions based on inappropriate, inadequate, and untimely evidence can have lasting health and economic repercussions (12) . The absence of internationally accepted guidelines to address such issues warrants a framework for upholding scientific standards and accountability of stakeholders during unprecedented times. J o u r n a l P r e -p r o o f 7 To this aim, we propose a "Reporting Standards Trajectory" (RST) comprising three gradually increasing standard of reports, "minimum", "acceptable", and "ideal", that seeks collaboration from funders, modellers, and decision-makers to enhance the quality of reporting model findings over time. As shown in Figure 2 , each stage of reporting are bound by a set of criteria that are guided by key principles that govern a high quality of evidence from models (13): Translation of policy question to research question, selection of appropriate model, contextualisation of the model, model validation, incorporating uncertainties, and declaration of conflict of interest. Furthermore, modellers are asked to report a preferred mode of contact, which provides decisionmakers a means to communicate with modellers for clarification or future iteration, make their models (codes, data, etc.) publicly accessible, get their work peer-reviewed for public scrutiny and gain validation from wider scientific community. This is particularly relevant given the recent evidence of poor quality and low publication rate (ranging from 6-21%) in peer-reviewed journals among pre-printed manuscripts related to . This suggests that failures in peer review may also constitute a public health risk if the results of non-reviewed studies were used to inform policy implementation. While the goal is always to meet the ideal standard, we acknowledge that decision-makers may be constrained by time during crises and may find it difficult to prescribe the proposed RST to modellers. However, we think the value of adopting such a tool to be greater than the additional workload it may create, by providing a mechanism to monitor the type and quality of evidence they receive, putting them in a commanding position to assess the level of confidence with which they can make decisions. Nonetheless, the RST provides flexibility to decision-makers and funders in countries with saturated local capacity (to provide and consume evidence) and rapidly evolving policy needs, by starting with the minimum standard of reporting at their initial engagement with modellers. Urgent decision needs, for e.g., closure of international borders may be aided with, at least, the minimum standard of reporting. For non-immediate decisions, for e.g., reopening of schools, an acceptable if not ideal reporting standard should be sought. We see the RST as a practical tool to deal with the constraints on all stakeholders during pandemics while re-emphasising the use of the highest possible quality of evidence.

J o u r n a l P r e -p r o o f 8 The RST echoes the significance and benefits of a collaborative approach to modelling for policy. First, it allows the funders, and decision-makers to screen the modellers who have the calibre to deliver the required standard of reporting by demanding their commitment to gradually improve the standard to meet the ideal standard. This way, decision-makers can be assured that the evidence used in their decisions will be appropriate, adequate, and timely, which all sit at the heart of evidence-informed decision-making. Second, it can improve transparency of the process and increase accountability of all stakeholders involved. Funders and decision-makers can seek justification from modellers when they fail to improve or meet criteria in the RST, and the general public can demand that decision-makers and funders only engage with modellers who have committed to such RST and seek justification when they do not or consistently use substandard quality of evidence in their decisions. This way modellers are obliged to report their finding even when they contradict results of other modellers or recommend against a favourable action of a decision-maker. While the RST cannot be a binding constraint, they can certainly be used as tool to increase public ownership to such models, maintain accountability, and legitimise decisions. However, adopting the RST is driven by policy urgency and local capacity of a country. These factors are subjective and differ by context. Hence, a wider stakeholder engagement is advised (explored below) to have a consensus and make a valued judgement on what is feasible and acceptable. This applies to both, choosing the appropriate reporting standard at the initial stage and the timelines to transition to a higher quality of reporting. 

We capitalise on an existing framework on the collaborative process between modellers and stakeholders (15) , to address challenges including transparency, inclusive decision-making, and accountability that hinder successful policy implementation (16) . This has been done in two ways: First, the addition of the fitness-for-purpose flowchart and the RST provide a practical guide and tangible avenues for decision-makers and funders to participate in modelling efforts. Second, by extending the definition of stakeholders to funders, researchers from other fields, ground-level implementers, and general public, depicted in Figure 3 , we define their roles and contributions in this process and allow each to be held accountable for their actions or the lack thereof. This process is iterative and should be applied to continuously monitor, evaluate, and revise decisions. Funders can play a crucial role in this process by bringing in modellers with a reputation for maintaining professional standards to generate evidence on pressing decision problems. By seeking commitments from the decision-makers and modellers to adopt the fitness-for-purpose flowchart and the RST, funders can ensure adequate levels of collaboration required for an effective modelling to inform policy and reap the benefits discussed above. The uncertainty surrounding the virus and its interaction with society means continuous learning remains key to understanding and tackling the outbreak (7, 17) . Hence, this collaborative effort should extend to researchers from other fields, implementers, and general public who can provide society-based information such as nuanced local culture, values, behaviour, environment, and how different (7). In LMICs, religious gatherings, intergenerational living, large informal sector with daily wage earners, and migrants living in congested areas, may prohibit the ability to maintain social distancing (18) . Modelling outcomes and policies devoid of such context, for example, stringent lockdowns in densely populated and poor communities may result in unintended consequences like excess mortality (19) , starvation, and poverty (20), or create new hot spots for the virus (21) . Furthermore, in light of yet another coronavirus disease and emerging climatic hazards, adopting the interdisciplinary One Health approach becomes crucial. By bringing experts from various fields, it offers understanding of the complex interrelationship between animals, humans, and the environment, and offers preventative interventions that can reduce the threat of outbreak such as bio-surveillance of live animal markets, identifying sources of pathogens, improved biosecurity, public education on zoonotic diseases, etc (22) . A detailed guide from WHO on addressing zoonotic diseases using the one health approach is discussed in this book (23) . Early engagement with implementers (at different levels of government, agencies, etc.) can harmonise efforts, help coordinate a swift response that pandemics demand, and relay ground level information where logistical challenges can appear while implementing a response (24) . Hence, modelling process and results need to be integrated into the wider pandemic considerations and not viewed as a standalone activity. As depicted in figure 3 , this can be done by bringing in other researchers who can monitor and evaluate the value of recommendations made by such models as well as find new areas for research. For example, conducting the costeffectiveness of a particular intervention (for e.g., imposing a lockdown) (25) by considering wider societal costs such as excess mortality, increased hunger poverty, impact on routine immunisation, increased mental health issues, national gross domestic product (GDP), etc. against the reduction in infection and deaths due to COIVD-19. A collaborative modelling effort can provide insights into such trade-offs which remains crucial for a sustainable response on our road to recovery from COVID-19. Broadening the definition of stakeholders during pandemics can inform the value, feasibility, acceptability, and allow necessary fine-tuning of any policy intervention recommended by models (26, 27) . Such an inclusive and transparent process can bring a sense of public ownership to decisions and instil trust, increasing cooperation and adherence to policy measures from the public, which remains vital in fighting the COVID-19 pandemic. Leveraging decisions with such a process can help avoid public scrutiny and keep distant selected voices and vested interests (28) . It is important to note that such engagement is demanding on the time and capacity of researchers and decision-makers. For those reasons, it important to establish this process early on to ensure appropriate planning and resourcing for this modality of engagement. This framework for stakeholder engagement has application beyond disease modelling and was adopted while developing Thailand"s triage protocol to prioritise critical care resources during the first outbreak of COVID-19 which can offer actionable details of this framework (29) .

Coronvirus: why we need local models to successfully exit lockdown. . The Conversation

Wrong but Useful -What Covid-19 Epidemiologic Models Can and Cannot Tell Us

Knowledge translation for public health in low-and middleincome countries: a critical interpretive synthesis

Forecasting for COVID-19 has failed. International journal of forecasting

COVID-19 Multi-Model Comparison Collaboration (CMCC) Policy Group. Guidance on Use of Modelling for Policy Responses to COVID-19

COVID-19 Multi-Model Comparison Collaboration (CMCC) technical group report. Model fitness-for-purpose assessment report

New coronavirus outbreak: Framing questions for pandemic prevention

Avoiding Trouble Ahead: Lessons Learned and Suggestions for Economic Evaluations

Crises are no excuse for lowering scientific standards, say ethicists

Scientists criticise UK government's 'following the science' claim. The Guardian 2020. 11. Retraction Watch

The public health response to COVID-19: balancing precaution and unintended consequences

The International Decision Support Initiative Reference Case for Economic Evaluation: An Aid to Thought. Value in health : the journal of the International Society for

Publication rate and citation counts for preprints released during the COVID-19 pandemic: the good, the bad and the ugly

Modelling for policy: The five principles of the Neglected Tropical Diseases Modelling Consortium. PLoS neglected tropical diseases

Governance of the Covid-19 response: a call for more inclusive and transparent decision-making

COVID-19 and the policy sciences: initial reactions and perspectives

Does One Size Fit All? Realistic Alternatives for COVID-19

Low-Income Countries Center for Global Development. 2020(Commentary and Analysis)

Estimated impact of the COVID-19 pandemic on cancer services and excess 1-year mortality in people with cancer and multimorbidity: near real-time data on cancer care, cancer deaths and a population-based cohort study

Covid-19 Singapore: A 'pandemic of inequality' exposed

From SARS to COVID-19: A previously unknown SARS-related coronavirus (SARS-CoV-2) of pandemic potential infecting humans -Call for a One Health approach

World Health Organization. Taking a multisectoral one health approach: a tripartite guide to addressing zoonotic diseases in countries: Food & Agriculture Org

How primary healthcare in Iceland swiftly changed its strategy in response to the COVID-19 pandemic

Tackling covid-19: are the costs worth the benefits?

Shoe leather epidemiology in the age of COVID: lessons from Cuba

Are highperforming health systems resilient against the COVID-19 epidemic? The Lancet

Difficult trade-offs in response to COVID-19: the case for open and inclusive decision making

Developing a Thai national critical care allocation guideline during the COVID-19 pandemic: a rapid review and stakeholder consultation. Health research policy and systems

Trust in Public Health Is Essential Amid the COVID-19

Journal of hospital medicine

Trust and the Coronavirus Pandemic: What are the Consequences of and for Trust? An Early Review of the Literature

The CMCC was established by a group of partners coming together: Bill and Melinda Gates Foundation (BMGF), Data 4 SDGs partnership, Department for International Development (DFID), UK, the International Decision Support Initiative (iDSI), Norwegian Agency for Development Cooperation (NORAD), Ministry of Higher Education, Science, Research and Innovation (MHESI), Royal Thai Government, the World Bank (WB), the World Health Organization (WHO), the US Centers for Disease Control and Prevention (US CDC), and United States Agency for International Development (USAID). The findings

the online meetings and surveys to share their experience in using evidence for models. YT, SKC, and YL wrote the first draft of the whole paper and revised the drafts. All other authors reviewed and contributed to the revision of the final draft

Thailand, and supports evidence-informed priority-setting and decision-making for healthcare. HITAP is funded by national and international public funding agencies. HITAP"s international work is supported by the International Decision Support Initiative (iDSI), with the aim of providing technical assistance on health intervention and technology assessment to governments in low-income and middleincome countries. iDSI is funded by the Bill & Melinda Gates Foundation (OPP1202541), the UK"s Department for International Development, the Rockefeller Foundation. HITAP is also supported by the Access and Delivery Partnership

Authors" statement YT, SKC, YL, SD, JK, and HC together convened the engagement (online meetings) with  policymakers, conducted the online surveys, and searched the literature. SW, CL, GL, HS, MH,  MB, SN, SS, DW, KC, MG , and RH participated in the online meetings and surveys to share their experience in using evidence for models. YT, SKC, and YL wrote the first draft of the whole paper and revised the drafts. All other authors reviewed and contributed to the revision of the final draft.

☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Highlights 1. COVID-19 models built for high-income settings need to be adapted appropriately when used for low-and middle-income settings due to contextual difference 2. Upholding scientific standards equally important to providing evidence for policymaking during pandemics 3. Wider stakeholder engagement with iterative process is required for effective policy implementation 4. COVID-19 Multi-Model Comparison Collaboration (CMCC) offers recommendations that can ensure models are fit-for-purpose, timely reporting of good quality of evidence, transparency and accountability of all stakeholders involved