key: cord-0879645-n78tk8dj
authors: Stratil, J. M.; Arnold, L.
title: WHO-INTEGRATE COVID framework Version 1.0: Criteria and considerations to guide evidence-informed decision-making on non-pharmacological interventions targeting COVID-19
date: 2020-07-07
journal: nan
DOI: 10.1101/2020.07.03.20145755
sha: 07a603937513804b47c386a42f1df7813dd6367d
doc_id: 879645
cord_uid: n78tk8dj

Background: Decision-making on matters of public health requires the balancing of numerous, often conflicting factors. One approach to ensure relevancy and comprehensiveness of the criteria underpinning the decision is a broad societal discourse and participatory decision-making process. However, this often was not feasible within the time constraints imposed on by the SARS-CoV-2 pandemic. While not able or intended to replace stakeholder participation, evidence-to-decision frameworks can serve as a tool to approach relevancy and comprehensiveness of the criteria considered, even if not all voices of affected stakeholders could be heard in the process. Objective: The objective of this research project was to develop a decision-making framework adapted to the challenges decision-makers face when deliberating on national and sub-national level on non-pharmacological interventions (NPIs) measures to address the global SARS-CoV-2 pandemic. Methods: We used the WHO-INTEGRATE framework Version 1.0 as a starting point. In phase I, we adapted the framework through brainstorming exercises and application to exemplary case studies. In phase II, we used the best-fit framework synthesis technique with the output of phase I serving as a priori framework. We conducted a content analysis of comprehensive strategy documents intended to guide the policy makers on the phasing out of the lockdown measures in Germany. Based on factors and criteria identified in this process, we adapted previous versions into the WHO-INTEGRATE COVID framework (WICID framework) Version 1.0. Results: Twelve comprehensive strategy documents were included in the content analysis. The revised WICID framework consists of eleven criteria, which are expanded on through 49 aspects contained within them, and the metacriterion quality of evidence. The criteria cover implications for the health of individuals and populations due to and beyond COVID-19, infringement on liberties and fundamental rights, acceptability considerations, societal, environmental, and economic implications, as well as resource and feasibility considerations. Discussion: In a third phase, the framework will be expanded through a comprehensive document analysis focusing on key-stakeholder groups across the society. The WICID framework V1.0 can be a tool to support comprehensive evidence-informed decision-making processes.

The response to the global SARS-COV-2 pandemic highlighted the challenges and complexities of evidence-informed decision-making inherent to matters of public health [1] [2] [3] . These include, among other decision-making under time constraints, under uncertainty due to limited evidence, balancing numerous conflicting factors, and the challenge of ensuring fair decision-making processes under such circumstances.

Due to exponential growth in the number of infections, the issue of timing is of crucial importance in a pandemic. A delay of decisions on the implementation of interventions (e.g. social-distancing regulations) by days and even hours can have grave consequences. For example, a modelling study concluded that had the same non-pharmacological control measures been implemented in the USA one week earlier, around 36,000 deaths (55.0%) could have been averted by the beginning of May 4 . Therefore, to receive evidence-informed guidance timely, many countries, regions, and governmental institutions set up expert committees to support guidance and recommendations to public health and health policy decision makers 5 . One challenge these expert groups face is the challenge of limited availability and often poor-quality scientific evidence with often questionable transferability and applicability to the context of decision-making 6 7 . However, even if strong scientific evidence (e.g. on the effectiveness of an intervention) was availablewhich in most cases, it is notthis evidence in itself is insufficient to make sound recommendations, as evidence-informed decision-making is a deeply value-laden process [8] [9] [10] . The second challenge therefore is that decision-makers must balance numerous and often conflicting normative and technical aspects for a decision-making process [11] [12] [13] , posing the questions to the expert committees and decision-makers: which criteria should be considered and how should these be weighed against each other? For example, when recommending or implementing school closures, decision-makers need to balanceamong othersthe risk associated with COVID-19 infections for children, parents and teachers, the implications for the course of the pandemic overall, the economic implications of absent parents bound by care responsibilities, educational outcomes for children, the implications on wellbeing and mental health for all those involved, and legal ramifications 14 .

The extent to which recommendations or final decisions (e.g. on not to lift the school closures at a given point of time) are considered acceptable and legitimate depends on procedural considerations (on how the decision was made). Various approaches on achieving legitimacy and acceptability despite the inherent challenges of evidence-based decision-making have been proposed. These approaches emphasize e.g. the importance of transparency throughout the process, the inclusion of the perspectives of relevant stakeholders, an appropriate composition of the decision-making panel, and the option of appeals and revision 8 15-20 . According to the Accountability for Reasonableness (A4R) framework by Daniels and Sabin 19 , a key aspect is the condition of 'relevance': the decision or recommendation must rest on evidence, reasons, and principles that all fair-minded parties can agree on, to be relevant to deciding how to meet the diverse needs of affected stakeholders 19 . Such approaches can increase the acceptability and perceived legitimacy of a decision 21-23 even ifgiven varying and sometimes contradictory interestsno consensus regarding the right selection and weighting of criteria can be achieved 21 .

Involving (representatives of) all relevant stakeholder groups in the process to identifying reasons and principles of relevance for a given decision-making process is considered ideal 8 15 24 25 . However, this ideal is often difficult if not impossible to meet under the time constraints imposed by the rapid progression of the SARS-CoV-2 pandemic. While not intended to nor able to replace stakeholder participation, Evidence-to-Decision (EtD) frameworks are one approach to support decision-making processes in this balancing act 26 27 . EtD frameworks, which tend to comprise sets of criteria as well as procedural guidance, are intended to ensure that all relevant criteria are considered, the best available evidence is assessed, and the underlying rationale is made explicit and transparent 28 . When developed and applied well, these EtD frameworks can help identify and integrate the criteria of relevance for a given decision-making process, even if not all voices of relevant stakeholders could be heard.

One of these frameworks, is the WHO-INTEGRATE framework version 1.0 29 . It was developed as part of a larger research project commissioned by the World Health Organization (WHO) to inform the development of guideline recommendations on complex public health and health system interventions 30 . It was developed with a strong conceptual and normative foundation 29 primarily based on an evaluation of WHO norms and values derived from key WHO documents (e.g. the WHO constitution) and widely used public health ethics frameworks [31] [32] [33] [34] [35] [36] [37] [38] . To ensure the relevance and comprehensiveness of the framework, this normative approach was combined with a comprehensive literature review of decision criteria used in real world decision-making 39 , an assessment of complexity features 29 as well as key-informant interviews with WHO guideline developers and focus-group discussions with public health and health policy decision-makers from low-, middle-and high income countries on four continents 40 . As with most EtDs, the WHO-INTEGRATE framework is a generic framework. To apply the framework to a decision-making process on a specific intervention and context, adapting the framework is necessary.

The objective of this research project is to adapt the WHO-Integrate framework to be applicable to the requirements of developing recommendation and decision-making on a national and sub-national level on non-pharmacological interventions (NPIs) intended to address the global SARS-CoV-2 pandemic; taking a plurality of viewpoints of affected stakeholders into account. The adapted WHO-INTEGRATE COVID framework (WICID framework) is intended to reflect the challenges arising decision-making on matters of public health in relation to COVID-19, but to be sufficiently generic to be applicable to a wide range of NPIs, contexts, and decision topics.

While procedural criteria, norms, and principles (how a decision is made) beyond the principle of relevance are crucial for achieving fair processes with outcomes considered acceptable and legitime, this research project focuses on the substantive decision-making criteria (what criteria should the decision be based on?).

To develop a EtD framework adapted to the requirements of public health decision-making in the context of COVID-19, we used the WHO-INTEGRATE framework as a starting point and employed an approach analogous to the "best fit" framework synthesis 41 . This adaption of the WHO-INTEGRATE Framework is conducted in three phases: In phase I, we adapted the framework through brainstorming exercises and applying it to case studies, in order to develop the a priori framework. Phase II, intended to advance this framework based on a content analysis of purposive sample comprehensive strategy papers on phasing out lockdown measures in Germany, which were coded against the a priori framework. The documents were intended to inform political decision-makers on strategic decisions surrounding the phasing out of NPIs implemented in Germany in March 2020. This group of documents was chosen as a starting point, as we assumed that these documents are intended to provide a broad, differential, and multi-perspective consideration of potential benefits and harms resulting from various NPIs as well as other considerations (e.g. ethical, legal, or feasibility), rather than exclusively or primarily focusing on health implications. Phase III, which is yet to be completed, we will advance the WICID framework version 1.0 developed in phase II, by integrating the perspectives of a diverse set of affected stakeholders across the society, in an approach analogous to the one employed in phase II. Phase III allows for an assessment of the comprehensiveness of the framework criteria and the integration of factors insufficiently covered in the expert-based strategy documents.

In phase I, we adapted the WHO-INTEGRATE framework through (i) discussion within the research team, (ii) an assessment of real-world decision-making criteria derived from an comprehensive overview-ofreviews 39 , and (iii) a brainstorming exercise guided by the application of the WHO INTEGRATE Framework version 1.0 on two case studies (reopening high schools and reopening small businesses such as book shops).

This preliminary version of the adapted framework was used as a priori framework and imported into the software MAXQDA20 (verbi, Berlin) to serve as a coding frame to be used in phase II. In this process, we added two additional codes: "evidence" to, cover considerations regarding uncertainty and the evidence to be used to inform on these considerations, and the code "TBD" (to be discussed), which was to be applied to passages where one coder was not certain about how to code the passage against the coding frame.

We used the technique of "best fit" framework synthesis [41] [42] [43] , which offers a mean to build on an existing framework, conceived for potentially different but relevant circumstances. "Best fit" framework synthesis begins by creating a framework of a priori themes (in this case: the WHO-INTEGRATE framework adapted in phase I) and coding data from included documents (in this case: the comprehensive strategy documents) against that thematic or conceptual framework. A new framework composed of a priori and new themes is created by performing a thematic analysis on any data that cannot be accommodated within the a priori framework. As the a priori framework, we used the adapted version of the WHO-INTEGRATE framework created in phase I, and translated this into the coding frame we used coded the strategy documents against (provided in the supplementary table TA2) 41 .

We decided to focus phase II on comprehensive strategy documents by expert commissions or expert groups which were intended to inform policy and public health decision-makers in Germany. We did so, as we assumed, they would be more likely to provide broad comprehensive, multi-perspective recommendations (in contrast to e.g. scientific publications or statements by individual groups, which do not claim nor intend to reflect multiple relevant perspectives in coming to a conclusion).

We defined these as documents (a) intended to provide a comprehensive strategy or strategic suggestions for phasing out the lockdown measures, (b) not exclusively or primarily focused on mitigating the health related-consequences of the SARS-CoV-2 pandemic but also on other societal, economic, or health outcomes, (c) addressing the interplay of various NPIs (e.g. not exclusively focusing on testing), and (d) focusing on multiple considerations to be considered in this process (e.g. not exclusively focused on health impact considerations). Position papers of stakeholder groups reacting or positioning themselves to a document, measure, or event without providing comprehensive strategy guidance were excluded but will be considered in phase III.

While the main discourse on implementing lockdown measures in Germany was focused on suppressing the spread of the outbreak and on averting a collapse of the health care system, the debate on the controlled phasing out of the implemented lockdown measures was more nuanced: focusing on the challenge to balance the implications of the measures e.g. on health, society, or the economy. Therefore, we concentrated on strategy documents that focused on the latter.

To focus on strategy documents with impact on real world decision-making, we focused on papers developed by expert groups or task forces implemented by federal or national governments as well as ministries of the German federal Government. As we assumed that some guidance documents are not publicly available, we expanded the sample to comprehensive strategy documents on the exit from the lockdown measures developed by non-government affiliated expert groups which were intended to guide public health and health policy decision making.

In order to ensure that the strategy documents addressed a broad range of societal and economic implications beyond the health sector, we focused on strategy documents whichdirectly or indirectlyaddressed one of the following NPIs: (a) closure and reopening of schools, (b) closure and reopening of businesses, and (c) "shelter-in-place" regulations. We excluded strategy papers not or only marginally concerned with these three NPIs. Strategy papers primarily concerned with hygiene measures or the testing capacity were also excluded. Table TA1 in the annex displays the inclusion and exclusion criteria for phase II. All identified borderline cases were retained to be analyzed in phase III.

The search was conducted through multiple approaches channels:

• Two researchers (JMS, LA) independently searched in the search engine Google TM with various versions of keyword combinations of the terms and synonyms of "strategy" or "expert commission" and "COVID-19" in German. • Two researchers (JMS, LA) independently searched the websites of major newspaper outlets in Germany (including: Die Zeit, Frankfurter Allgemeine Zeitung, Die Welt, Deutschlandfunk, Der Spiegel) outlets using the website's search engine with similar keywords • One researcher (JMS) searched the websites of the 16 German federal states, the national government and selected national government ministries, focusing on the section of press releases. • We contacted a sample of experts involved in public health decision making or expert groups to provide us with strategy document; either directly or through the platform of the interdisciplinary Kompetenznetz Public Health COVID-19 (www.public-health-covid19.de). • We posed freedom of information requests posed to the federal states' governments, the national government, as well as selected national government ministries to provide us with strategy documents developed by expert groups, if available.

As the documents we intended to identify are grey literature documents and most likely will be written in German, we did not conduct a literature search in scientific data bases for these types of documents.

In the processes, the researchers applied the level 1 codes (referring to the criteria in the framework) and level 2 codes (referring to the aspects in the framework) of the coding frame to passages in the document making references to criteria, considerations, or values covered within the codes. When the content of such a passage was perceived as not adequately covered by the coding frame, new level 2 or level 1 codes were created. When the researchers identified passages containing references to criteria, considerations, or values of relevance which were assumed to be covered by a specific level 1 or level 2 code, but which seemed to expand on this or provide details or nuances (e.g. a passage on the implications of an measure risking the insolvency of small enterprises within the code economic implications), the researchers took note of these passages for later review. Unclear passages were assigned the code TBD code for later review.

After coding all strategy documents, both authors critically reflected on content saturation and dimensions of the framework insufficiently covered within the strategy documents. As most criteria of the preliminary framework were adequately covered and there was a considerable overlap of consideration across documents, we assumed content saturation to be reached.

Next, one researcher (JMS) conducted a thematic analysis of the passages assigned to the newly created codes as well as those passages noted down for expanding on or providing nuance within existing codes. A draft of an adapted phase II framework was created by reflecting on whether there was a need to adapt the a priori framework to cover the content in the coded passages. This included whether: (a) criteria (reflecting level 1 codes) should be created, (b) new or preexisting criteria should be merged or separated, (c) new aspects (reflecting level 2 codes) should be added to the framework, (d) new or preexisting aspects should be merged or separated, (e) new or preexisting aspects should be moved to another position within the framework, (f) the wording of the criteria or aspects needs to be adapted.

This draft of the adapted framework, including the passages in the newly created codes as well as those passages noted down for expanding on or providing nuance to existing codes, were critically reviewed by a second researcher (LA). Afterwards, the results of steps (a-f) were critically discussed within the research team (JMS, LA) to solve conflicts and revise the adapted phase II framework accordingly.

Afterwards, two researchers jointly went through all passages noted down for later review (code TBD) and assessed through discussion, whether any criterion, consideration, or value within these passages was not covered within the framework. Where necessary, criteria or aspects were added, revised, reworded, moved to another position within the framework.

In a final step, two researchers (JMS, LA) went through each of the coded passages to critically reflect on whether the criteria, considerations, or values contained within these passages was adequately covered within the newly adapted phase II framework.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint

In phase I, we used the WHO-INTEGRATE EtD Framework Version 1.0 as a starting point. Following the process outlined in the methods section, we developed a preliminary, adapted framework, which consisted of 8 criteria with 36 aspects and the metacriterion quality of evidence. The criteria focused on the implication of the measure(s) on the course of the pandemic and CVOID-19-related health implications from (I) a populations and system perspective and (I) for individuals, (III) health implications other than COVID-19 (e.g. wellbeing, health behavior, access to standard medical care), (IV) acceptability of the measures and implication for to individual rights and liberties, (V) equity, equality, and fair distribution of benefits and burdens, (VI) societal, economic, and environmental considerations, (VII) resource and feasibility considerations, and (VIII) implications for and interactions with other components of the health system.

The framework is provided in the supplementary table TA4. A preliminary version of this phase I framework as well as the exemplary case studies are part of a strategy document developed for the federal ministry of interior and the federal chancellery 45 .

We identified four strategy papers developed by expert groups developed for federal states, the national government, or ministries of the national government. Two of the strategy documents were developed by the corona expert commission of the federal state government of North Rhine-Westphalia, which were publicly available 46 . Two strategy documents were developed by an informal expert group for the German Federal Ministry of the Interior, one of which was publicly available through a press report the other was provided to us through the Kompetenznetz Public Health COVID-19 45 47 .

We identified additional eight comprehensive strategy documents, which wereto the best of our knowledgenot directly commissioned by federal state governments, the national government, or ministries of the national government. Four of which were published by the National Academy of Sciences Leopoldina 48-50 , one by a political party in the federal city-state of Hamburg 51 , one by a research institute aligned with political foundation Hans-Böckler-Stiftung 52 , one by the Boston Consulting Group 53 , and one developed by researchers from diverse institutions under the coordination of two researchers with affiliation at the university Wuerzburg and the IFO institut 54 .

The adapted WHO-INTEGRATE COVID-framework consists of 11 substantive decision-making criteria, containing 49 decision-making aspects, and the meta-criterion quality of evidence, to be applicable across all criteria and aspects (outer circle, Figure F1 ). Depending on the measure at hand, the criteria and aspects are intended to be applied on and reflected for different population groups (center-most circle, figure F1 ). Depending on the measure and type of decision-making process, the decision-makers are intended to deliberate on the criteria and aspects taking one or multiple different perspectives (inner circle, figure F1 ). Analogous to the WHO-INTEGRATE framework, it aims to accommodate different features of complexity: depending on impact the measure is assumed to have on the system it is implemented in, not only direct implications but also indirect effects, not only local but also regional, national and even global implications, and not only immediate but also, short, medium and long term implications should be considered ( figure  F2 ). 

The eleven criteria in the WHO-INTEGRATE COVID framework consist of: three criteria focus on the balance of health benefits and harms: (I-III), two criteria focus on the accordance with human rights principles and socio-cultural acceptability (IV-V), one criterion focuses on equity, equality, and nondiscrimination (VI), two criteria focus on the societal implications (VII-VIII), and three criteria focus on feasibility and health systems considerations (IX-XI). The criterion I. implications for the course of the pandemic and its impact on health covers implications of the measures for the course of the SARS-CoV-2 pandemic, including the number of new infections, the resulting health-related consequences (e.g. COVID-19 related mortality), the implications for the capacity of the health care system to treat COVID-19 patients, as exceeding the capacity this capacity is associated with a pronounced increase in mortality. It furthermore covers the direct SARS-CoV-2 related health risk of individuals affected by the measures (probability of infection and consequences).

II. implications for quality of life, social well-being, and mental health aims to capture the how the measures affect overall wellbeing and quality of life, which includes the degree to which the capability to shape everyday life according to one's own wishes and needs is affected (e.g. through restricting daily habits) or the experience of self-efficacy and of sense of coherence. It furthermore covers the social wellbeing of individuals (e.g. the experience of loneliness) and the social cohesion of communities (e.g. cohesion of families or non-family communities). The criterion furthermore captures the implications for the mental health of individuals and populations (e.g. depression, anxiety disorders), including risk-factors -such as the experience of stress or fear, ability to practice coping mechanisms, receive support and, and the adverse consequences resulting from poor mental health (e.g. suicides).

III. implications for the physical health, health behavior, health risks, and health care beyond COVID-19 focuses on behavioral, environmental, and interpersonal risk factors for health (including accidents and violence) other than those directly related to COVID-19 (e.g. physical activity) and their consequences for health. It furthermore covers the implications of the measures for availability, accessibility, acceptability, and quality of medical and care services and institutions (e.g. willingness to seek emergency care in the case of myocardial infarctions).

The criterion IV. Proportionality and accordance with individual autonomy and fundamental rights covers whether the measure is in accordance with and how itdirectly or indirectlyaffects autonomy, selfdetermination, individual liberties, and fundamental individuals' rights (e.g. privacy and data protection implications of a contact tracing-app). It furthermore covers the intrusiveness of the intervention (with e.g. providing information being a measure with a low intrusiveness; and restricting choice one with a high intrusiveness 36 ). The aspect of proportionality addresses whether intrusiveness and infringement of individual rights and liberties are proportionate to the expected benefit (or expected avoidance of harm).

V. acceptance of and willingness to implement measures focuses to the degree of acceptability of the measure and their consequences to the general population and different affected population groups. This includes the willingness to implement, adhere to, or enforce the measure (e.g. whether reopening schools without any protective measures is acceptable to teachers with a high-risk profile). While is acceptability is an end in itself, this criterion is strongly linked to feasibility and assumed effectiveness.

The criterion VI. equity, equality and the fair distribution of benefits and burdens covers the implications of the measure for vulnerable population groups, whether and how it affects stakeholder groups differently and thereby the risk for increasing or reducing inequalities (e.g. men benefiting less from a measure in comparison with women, exacerbating health inequities), considerations of equity (treating people differently according their need to allow them the same capability of achieving an outcome), considerations of equal treatment (e.g. not treating people differently without sufficient justification), and implications for the risk of individuals or population groups to be stigmatized or be discriminated against.

VII. societal and environmental implications and considerations aims to capture the implications for civil society, social life, and culture from an individual-level and system-level perspective, and the implications for the functioning and cohesion of the society. It furthermore covers the implications of the measures for social determinants of health, including household income, social participation, and education. A further aspect covered is the implications for the ecosystems resulting from the measures.

Analogous to criterion VII, the criterion VIII. economic implications and consideration address the implications for economic actors, their activities, as well as the economy as a whole. Including the implications of the measures for the work force (e.g. closing of schools forcing parents to stay at home to take care of their children).

Within the criterion IX. resource requirements and resource implications, the framework covers the requirements of the measures for different resources in the context of the availability of these resources as well as how the measure affect the availability and quality of these resources (e.g. how many face masks . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . would be required to provide every teacher with a high-risk profile with one, are these masks available, and would this lead to a shortage of masks for e.g. health service providers).

X. feasibility and implementation considerations covers the practical, technical, and political feasibility of implementing the measure as well as the legal conformity. Other aspects address feasibility related characteristics of the measures, such as the flexibility in extension, adjustment, or withdrawal of the measures and the possibility of evaluating the implications of the measure and reacting adequately to new information (e.g. ability to test those affected by the measure and conduct contact-tracing if needed).

The final substantive criterion, XI. interaction with and implications for the health system, addresses how the intervention will interact (synergistically or adversely) with other measures to control and contain the SARS-CoV-2 pandemic and other not-directly COVID-19 related components of the health system (within a broad understanding of health system in accordance with the WHO 55 )

The meta-criterion quality of evidence is intended to be applied across all criteria and aspects, e.g. taking the quality of the evidence on health impacts alongside with its strength and direction into account. The metacriterion quality of evidence, reflects the confidence that the available evidence is appropriate and adequate to support the decision or recommendation. Evidence is interpreted in a broad sense, beyond an understanding focusing on quantitative evidence of effectiveness derived from systematic reviews. Different forms of evidence can be used and be the most appropriate type of evidence to inform on the criteria (e.g. an appropriate form to assess the accordance with selected fundamental rights can be a legal assessment). Decision-making under uncertaintyas often the case in a pandemiccan require a decision based on stakeholder experience and judgement, when stronger evidence is unavailable. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint

Depending on the measure and decision-making context, criteria and aspects within the framework should be considered for different population groups to adequately assume relevant implications. The stakeholder groups to be considered will depend on the type of measure (e.g. closing schools vs. closing nursing homes to the public). Building on the WHO-INTEGRATE framework and the strategy documents, we suggest to consider the implications for (a) the general population, (b) those intended to benefit from the intervention (e.g. young school children in the case of school reopening), (c) those intended to implement the measures (e.g. teachers), populations with a (d) high risk and (e) low risk in regard to COVID-19 (e.g. young and healthy parents; senior citizens with preexisting conditions living in the same household), and (f) other affected stakeholder groups (e.g. employers), with an emphasis on vulnerable and marginalized populations (e.g. school children with a family with a low socio-economic status). Depending on focus, further disaggregation of affected populations can be considered here (e.g. disaggregating the implications of the measure for different genders).

Depending on the measure and decision-making process, decision-makers need to reflect on the different criteria from different perspectives to inform their deliberations (center-most ring in figure F1 ). For example, the health implications of a measure on the SARS-CoV-2 related health risk (e.g. reopening of schools) can be approached from a population perspective (e.g. incidence rate of infections and associated mortality rates), a systems perspective (e.g. the implications for the capacity of the health system), or the individual perspective (e.g. the risk for individual teachers working in these schools).

The implications of the NPIs can be far reaching. For example, school closures cause children to stay at home, resulting in the possible need of parents to reduce working hours to engage in childcare. Without childcare facilities for essential workers, this can lead to shortages in staff, for example in the health care sector, limiting its ability to provide medical services and care. In line with the WHO-INTEGRATE framework, this framework approaches decision-making from a complexity perspective 2 : the measures are regarded as an "events in a system" 56 , with the (intended and unintended) implications of the measure resulting from the interaction of the measures' components with each other and components of the larger system. Within this perspective, disentangling the effects caused by the measure itself from the interplay of context and measure can be challenging if not impossible to do 57 ; posing challenges for transferability and generalizability of evidence. Analogous to the ripple effects caused by dropping an object in a pool of water, the effects initiated by the introduction of the measure to the system can be followed outwards incrementally. The more profound the impact of the measure to the system, the further way the effects of the measure throughout the system can be observed. For example, the shutdown of a few companies in a region for a short time can lead to locally felt adverse economic consequences. However, a marginally more impactful event of closing the same companies for a marginally longer time and thereby exceeding an economic threshold can lead to the insolvency of these companies, causingdepending on the companiesa disruption of globalized production chains with economic consequences that can have regional, national, and even global effects. Therefore, depending on how profound the impact of the measure is assumed, decision makers need to consider whether the measure is likely to lead not only to immediate and local, but also regional, national, or global consequences over the short, medium, and long term. When reflecting on the measures, decisionmakers should not only focus on the direct effects along the intended causal pathway, but also anticipate implications caused across several degrees of indirectness (figure F2).

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint Figure F2 : Complex system perspective on the implications caused by a measure being introduced into a system

Within this research project, we adapted the WHO-INTEGRATE framework to decision-making processes on non-pharmaceutical interventions intended to suppress or mitigate the effects of the SARS-CoV 2 pandemic. We used brain storming exercises and content analysis of comprehensive strategy papers aimed at informing political decision making in Germany on the phasing out the implemented lockdown measures. The resulting WHO-INTEGRATE COVID-framework version 1.0 consists of 11 substantive decision-making criteria, containing 49 decision-making aspects, and the meta-criterion quality of evidence. Depending on the needs of the decision-making processes, these are intended to be applied on different affected stakeholder groups and to be used from different perspectives. In line with the underlying complexity perspective, rather than only focusing on direct, immediate, and local effects of the measure, the ripple effects caused by the introduction of the measure to a given system should be followed to adequately capture the implications a measure might have.

The current version of the WHO-INTEGRATE COVID framework is intended as a version 1.0, intended to be expanded in a third phase. The limited diversity of expert groups established to inform policy makers on the handling of the SARS-CoV-2 pandemic, including their disciplinary background, has faced criticism 5 . While intended to provide comprehensive guidance taking multiple relevant considerations into account, our approach of adapting the WHO-INTEGRATE COVID framework based on strategy documents comes with the risk of criteria considered as relevant by stakeholder groups not represented in experts groups to be overlooked. The third phase of the research project aims to address this issue by including the perspectives of various stakeholder groups across the society and expanding the WHO-INTEGRATE COVID framework version 1.0 with considerations previously not adequately covered.

This will be done in an approach similar to phase II: conducting a content analysis of key-documents representing the opinions and perspectives of stakeholder representatives across the society and using the results to validate andwhere neededexpand the framework version 1.0. Using a sample of NPIs with broad societal implication as a starting point (closure and reopening of schools, closure and reopening of businesses, and "shelter-in-place" regulations), we will include opinion pieces, position papers, or press statements aimed at informing political decision-making during the ongoing debate of phasing out the . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . lockdown measures. A first set of stakeholder group clusters (e.g. social and welfare organizations) will be selected based on an initial brainstorming phase and expanded in an iterative process based on the findings of the content analysis following a snowballing approach. While it will be not feasible within the scope of this project to cover all relevant organizations and institutions within a given stakeholder group clusters, we will analyze documents put forward by a heterogeneous sample which will be expanded in an iterative process depending on whether content saturation is assumed to be reached within a given cluster. While this approach is not able to capture the voices of all affected stakeholders nor to identify all considerations regarded of relevance by these stakeholder groups, this approach allows for a broad representation of societal values in decision-making across the society and can be dynamically expanded should the need arise.

There are several approaches to using evidence-to-decision frameworks. While some are tailored to specific decision-making processes (e.g. on vaccination policies 58 ) and provide a fixed set of "ready-to-use" decision-making criteria 25 , others are more generic and require some form of adaption. The WHO-INTEGRATE COVID framework was adapted to inform decision-making on COVID-19 focused NPIs and the challenges arising from this. However, it was intended to be adequately generic to be applicable across a broad range of NPIs and decision-making contexts. While the WHO-INTEGRATE COVID framework with its eleven plus one criteria can be used as a "ready-to-use" evidence-to-decision framework, we believe the framework to be most useful as a guide to systematically reflecting on the measure in the given context and adapting it based on the specific needs of the decision-making process, using the WHO-INTEGRATE COVID framework as guidance and frame in the process.

Within the latter approach, the decision-makers are intended to develop a comprehensive logic model 59 60 or systems map of the measure as well as the context they are implemented in. This is done in order to identify how different components of the measure and of the system might interact with each other and thereby to identify relevant pathways through which intended and unintended consequences could arise. After creating this initial logic model, the WHO-INTEGRATE COVID framework can be used to expand on dimensions not adequately covered, e.g. by exploring the causal pathways from different perspectives, assessing the implications for different affected population groups, or using the criteria and aspects to assess its comprehensiveness. Using the criteria and aspects of the framework as a starting point and building on the expanded logic model, those involved in the decision-making process are now intended to develop and select criteria they assume to be of relevance for deliberating on the measure and adapting them to the context at hand. Using the example of school reopening, this coulda among othersinclude the risk of new outbreaks, health implications for teachers, for students, for parents, and other family members with an elevated risk profile (e.g. the students grandparents), the implications for the wellbeing of these groups, educational implications, etc. Given the complexity of the decisions at hand, it is likely not all criteria considered to be relevant can be covered in depth in the deliberation process. However, the rational for selecting a subset should be provided. Next, efforts should be made to acquire appropriate sources of evidence to inform on the criteria. Taking the direction a criterion points the decision to (e.g. implications for reducing educational inequalities pointing towards reopening the schools), along side the quality of evidence for this effect and the weight assigned to the criterion into account, the group of decision-makers are now intended to engage in the deliberation on which measures to take and how these should be adapted.

Public health ethics frameworks are intended to guide public health and health policy decision-makers on principles and values to consider. Various public health ethics frameworks have been published 38 61-63 , some . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . of which aimed to support decision-making on matters of public health more generally 32 33 36 64 65 and others developed for decision-making on public health emergencies and pandemics [66] [67] [68] [69] [70] . Building on these foundations, institutions such as the German ethics council 71 or the ethics working group of the German Network Public Health COVID-19 72 73 have put forward documents outlining relevant values and principles for decision-making in the current public health crisis. These include the duty to provide care, health, nondiscrimination, security, equity, individual liberty, privacy, proportionality, protection of the public from harm, reciprocity, and solidarity, among others 72 .

Rather than being an alternative to them, the WHO-INTEGRATE COVID framework was developed to be in line with these documents. The WHO-INTEGRATE framework, on which the adapted framework was built, was developed to have a firm foundation in WHO norms and values and informed by key public health ethics frameworks 29 . The intention of this adapted framework was to translate these principles and values into criteria better applicable for real-world decision making processes (e.g. by translating the general moral considerations of producing benefits and avoiding, preventing, and removing harms 32 intoamong others the criteria I,II,III,VII, and VIII on the different positive and negatives social, economic, or health-related effect a NPI might for individuals and populations). Furthermore, our framework aims to place criteria derived from these values and principles alongside with other considerations of relevance for real-world decisionmaking often not covered in depth in public health ethics frameworks, such as considerations of feasibility or the wider implications for the (health) system. While some values and principles have a direct representation in the framework criteria (e.g. individual liberty, privacy, or proportionality) others are introduced on the level of the perspectives or the populations the criteria should be applied to (e.g. reciprocity being reflected in the consideration of those intended to implement the interventions).

Furthermore, the framework is not intended as an alternative to these guidance documents, as whether a decision or recommendation is in line with these values or principles not only depends on which criteria are selected, considered, and weighted against each other in the deliberation process.

While the WHO-INTEGRATE COVID framework can be one tool to support decision-makers in selecting decision-making criteria of relevance, this in itself is insufficient to achieve fair decision-making processes with results considered acceptable and legitimate 19 20 . It is important that not only the final decision, but also the underlying rationaleincluding the criteria and evidenceis made transparent. Other important values and principles underlying fair decision-making processes include accountability, inclusiveness, openness and transparency, reasonableness, and responsiveness 72 73 . While a comprehensive approach of stakeholder engagement is likely not possible with a pandemic, efforts to including the voices of affected stakeholders, for example through rapid reaction statements from stakeholder representation organizations, can not only improve acceptability and legitimacy but also lead to better outcomes. Especially, the needs of affected marginalized and vulnerable population groups without strong political capital (e.g. people affected by homelessness) are at risk of being overlooked. Furthermore, special attention needs to be placed to the composition of the stakeholder groups 5 .

The WHO-INTEGRATE COVID framework was developed by building on the WHO-INTEGRATE framework 29 which was developed in principles-based approach to ensure a solid, comprehensive normative foundation and on previous research such as the result of an overview of systematic reviews on real-world public health and health system decision-making criteria 39 , and expanding this foundational framework through a broad set of comprehensive strategy documents informing real-world decision-making processes in Germany.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . However, the WHO-INTEGRATE COVID framework is only concerned with substantive criteria and does not comprise procedural criteria. We recognize that that a decision-making process guided by procedural norms, values, and principles is essential to come to decisions which are considered acceptable and legitimate by those (negatively) affected. While beyond the scope of the current publication, we plan to address this in an upcoming publication.

The WHO-INTEGRATE framework was developed to be applicable on national and sub-national levels throughout the world. However, to adapt the framework we focused on strategy documents intended to inform the German government. While be believe the WHO-INTEGRATE COVID frameworks to be useful and applicable to other regions within and outside Europe, the need for adapting the framework to the respective decision-making contexts should be considered.

The framework in its current version 1.0 was develop based primarily on comprehensive strategy documents developed by expert groups. The composition and the intention of the expert groups is likely to have had an effect on the criteria, consideration, values, and principles covered within them. While we believe possible shortcomings and blind spots are in part compensated by the use of the WHO-INTEGRATE framework as a basis for the adaption, there is a risk of relevant considerations and criteria missing from the current framework. We will address this issue in phase III of this research project, in which we aim to integrate views and perspectives from key stakeholder groups across the society.

Despite multiple approaches to identify the comprehensive strategy documents, which was done through independent searches by two authors and further informed by external input, we acknowledge the possibility of having missed on individual documents. Due to the rapid cycle of publishing on COVID-19-related issues, we see a risk of having missed strategy documents published from mid-June 2020 onwards. Furthermore, likely other relevant strategy documents exist, but were not disclosed by the governments or leaked to the public and therefore are not captured in our analysis. We will add an update of the phase II searches for comprehensive strategy documents as first task within phase III of the project.

Another limitation is the distinction between comprehensive strategy documents and position papers by affected stakeholder groups, which was not always clear cut. However, we believe this only to be a minor limitation, as all borderline documents were retained and will be included in the phase III of the project.

The WHO-INTEGRATE COVID framework represents a comprehensive COVID-19 focused EtD framework intended to guide policy and public-health decision-makers on NPIs. It is rooted in WHO norms and values, criteria and considerations used to inform real-world decision-making, and a complexity perspective. It offers a set of 11+1 decision-making criteria which are expanded on through 49 aspects assigned to them. While adapted to COVID-19 related challenges, it is intended to be generic in a way to be applicable across a broad range of decisions-making processes, contexts, and on a diverse set of measures. 

Consequences of the measures for individual companies, sectors of the economy and other economic actors and their capability for economic activities (e.g. through restrictions on the economic activity itself, the interruption of production chains, the availability of personnel, or sales opportunities at home and abroad) including the reversibility of the consequences (e.g. avoidance of insolvencies)

Implications of the measure for the national and regional economy, its resilience, the risk of recession and the possibility of recovery after the crisis (including planning security, willingness to invest, confidence in the future or the risk of hostile takeovers)

implications of the measures for innovation and economic development opportunities . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint 

• The document addresses measures related to the handling of the SARS-CoV-2 pandemic in Germany • The document is intended to inform real-world policy and public health decision-makers in Germany • The document is intended to provide a comprehensive strategy or strategic suggestions for phasing out the lockdown measures • The document addresses societal, economic, or health considerations beyond health considerations associated with SARS-CoV-2 (e.g. number of infections) • The document focuses addresses multiple considerations to be considered in this process (e.g. not exclusively focused on health impact considerations).

• The document addresses considerations for multiple NPI or considerations of relevance for multiple NPIs on a more abstract level • Position papers of stakeholder groups reacting or positioning themselves to a document, measure, or event without providing comprehensive strategy guidance are considered in phase III. • To document was developed by an expert group or task forces implemented by or commissioned to write the document by the federal or national governments as well as ministries of the German federal Government OR by non-government affiliated expert groups intended to guide public health and health policy decision making on the federal or national level.

• The document is not primarily targeted to inform policy or public health decision makers • The document does not provide a comprehensive strategy to phase out the lockdown, but is rather a reaction or positioning paper to an event, decision, or measure or provides general statements on aspects to consider (e.g. statement not to forget migrant workers) • The document is only or primarily concerned with direct health consequences of COVID-19 and/or the mitigation of these effects (e.g. the right number of tests to be performed), without discussing broader considerations (e.g. regarding societal implications, economic implications, feasibility and resource considerations) • The document is exclusively focused on the effectiveness of a measure (e.g. impact of school closures on COVID-19 transmission), without taking broader considerations (e.g. societal, economic, feasibility, or ethical considerations) into account. • The document is providing general information on COVID-19, the pandemic, or NPIs; not intended to guide decision-making (e.g. what is allowed in a federal state) • The document is a scientific study, guidance documents and guidelines to inform patients or health care providers on treatment and therapeutic approaches of COVID-19 (e.g. guidelines on the safety of ibuprofen) • The document is focused on health care system and medical care planning without relation to the selected • The document addressesdirectly or indirectlyaddressed one of the following NPIs: (a) closure and reopening of schools, (b) closure and reopening of businesses, and (c) "shelter-in-place" regulations.

NPIs (e.g. on triage-procedures, necessity of intubation therapy, relation of intensive vs. palliative treatment, number of ICU-beds necessary) 9.4 a priori framework adapted in phase I and coding frame • the implications of the measures on exposure to living environments and environments that affect health (e.g. domestic violence, unhygienic conditions)

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted July 7, 2020. . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint resources -infrstructure and other

• availability of, demand for and implications on availability of infrastructure other resources and (e.g. premises)

resources -protective equipment and hygiene products

• availability of, need for and impact on availability of protective equipment and hygiene products resources -human resources

• availability of, demand for and impact on availability of human resources (e.g. professionals, teaching staff) interaction with health system interaction with other measures or components of the health care system* interaction with COVID-19 related measures

• positive (synergistic) or negative interaction with other COVID-19 containment measure(s) interaction with health system (non covid)

• positive (synergistic) or negative interaction with other components of the health care system* *(broad understanding of health care system, beyond medical care) capacity of health care system (COVID-19)

• impact of the measure(s) on availability, accessibility, acceptability and quality of emergency and mainstream health care (beyond COVID-19)

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted July 7, 2020. . https://doi.org/10.1101/2020.07.03.20145755 doi: medRxiv preprint

Intelligent Policy Making for a Complex World: Pragmatism, Evidence and Learning

Implications of a complexity perspective for systematic reviews and guideline development in health decision making

The need for a complex systems model of evidence for public health

Differential Effects of Intervention Timing on COVID-19 Spread in the United States

Governance of the Covid-19 response: a call for more inclusive and transparent decisionmaking

A fiasco in the making? As the coronavirus pandemic takes hold, we are making decisions without reliable data

Coronavirus disease 2019: The harms of exaggerated information and non-evidence-based measures

Priority Setting for Universal Health Coverage: We Need Evidence-Informed Deliberative Processes

Balancing efficiency, equity and feasibility of HIV treatment in South Africadevelopment of programmatic guidance

Beyond cost-effectiveness, morbidity and mortality: a comprehensive evaluation of priority setting for HIV programming in Uganda

From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking

Setting priorities for health interventions in developing countries: A review of empirical studies

Transforming our world: the 2030 agenda for sustainable development. A/RES/70/1

School closure in response to epidemic outbreaks: Systems-based logic model of downstream impacts

Value Assessment Frameworks for HTA Agencies: The Organization of Evidence-Informed Deliberative Processes

Accountability for reasonableness

Expanded HTA: Enhancing Fairness and Legitimacy

Priority setting for new technologies in medicine: qualitative case study

The Ethics Of Accountability In Managed Care Reform

Beyond accountability for reasonableness

Accountability for reasonableness

Procedural Justice, Participation, and the Fair Process Effect in Groups and Organizations

What is wrong with evidence based policy, and how can it be improved?

WHO Handbook for Guideline Development

Priority setting in health care: learning from international experience. Health policy

The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective

Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions

Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review

Public health ethics: mapping the terrain. The Journal of law, medicine & ethics : a journal of the American Society of Law

Putting public health ethics into practice: a systematic framework

Beyond evidence--to ethics: a decision-making framework for health promotion, public health and health improvement

Ethical issues in epidemiologic research and public health practice

Public health: ethical issues. London: Nuffield Council on

Political and institutional influences on the use of evidence in public health policy. A systematic review

An overview of ethical frameworks in public health: can they be supportive in the evaluation of programs to prevent overweight?

Development of the WHO-INTEGRATE evidence-to-decision framework: An overview of systematic reviews of decision criteria for health decision-making

The WHO-INTEGRATE framework as a tool for evidence-informed, deliberative decisionmaking processes. A qualitative study exploring the views of those developing and using WHO guidelines

Best fit" framework synthesis: refining the method

A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents

Using framework-based synthesis for conducting reviews of qualitative studies

Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups

Kreativität -Verantwortung -Transformation. Grundlegende Voraussetzungen für die progressive Lockerung des COVID--19 Lockdown

Expertenrat Corona der Landesregierung

COVID-19-Eindämmung: Übergang von Verlangsamung zu Viruskontrollphase. Politische Handlungsfähigkeit erlangen Erwartungshorizonte für Gesundheit

Erste Ad-hoc-Stellungnahme: Coronavirus-Pandemie in Deutschland: Herausforderungen und Interventionsmöglichkeiten (21. März 2020)

Zweite Ad-hoc-Stellungnahme: Coronavirus-Pandemie in Deutschland: Coronavirus-Pandemie -Gesundheitsrelevante Maßnahmen

Dritte Ad-hoc-Stellungnahme: Coronavirus-Pandemie in Deutschland: Herausforderungen und Interventionsmöglichkeiten (21. März 2020)

Autorenpapier unter Federführung von Katharina Fegebank. Corona-Pandemie -Voraussetzungen für eine Exit-Strategie in Hamburg. 09

Schneller Ausstieg oder bedachte Lockerung? -Zur Zukunft von Kontaktbeschränkungen in der Covid-19-Krise. IMK Policy Brief Nr

BCG Perspektive zum Restart-Management in der Corona-Krise Munich

Die Bekämpfung der Coronavirus-Pandemie tragfähig gestalten

Everybody's business. Strengthening health systems to improve health outcomes : WHO's framework for action

Theorising interventions as events in systems

Evidence-making interventions in health: A conceptual framing

Decision support in vaccination policies

Towards a taxonomy of logic models in systematic reviews and health technology assessments: A priori, staged, and iterative approaches

Logic models help make sense of complexity in systematic reviews and health technology assessments

The evolution of public health ethics frameworks: systematic review of moral values and norms in public health policy

Theoretical models and operational frameworks in public health ethics

Public health ethics theory: review and path to convergence. The Journal of law, medicine & ethics : a journal of the American Society of Law

Looking Ahead: Addressing Ethical Challenges in Public Health Practice. global health law, ethics, and policy

An ethics framework for public health

Pandemic influenza preparedness: an ethical framework to guide decision-making

Ethics in a time of contagion: a relational perspective. The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres

Influenza pandemic preparedness: legal and ethical dimensions

Ethical issues in pandemic planning

Ethical discussion on criteria for policy makers in public health authorities for preventative measures against a pandemic caused by a novel influenza A virus

Solidarity and responsibility in the Corona-Crisis. Ad Hoc recommendation

Public Health Ethics and Covid-19 -The ethical dimensions of public health decisionmaking during a pandemic German Network Public Health COVID

Ethical principles for the Coronavirus Global Response: Ethical guidance for the development of fast and equitable access to safe, quality, effective and affordable diagnostics, therapeutics and vaccines against COVID-19