key: cord-0004483-09gp99xs authors: Stratil, J. M.; Baltussen, R.; Scheel, I.; Nacken, A.; Rehfuess, E. A. title: Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making date: 2020-02-11 journal: Cost Eff Resour Alloc DOI: 10.1186/s12962-020-0203-6 sha: 4be9882705e5d0021b60347ea0ac3e2ae6d91b40 doc_id: 4483 cord_uid: 09gp99xs BACKGROUND: Decision-making in public health and health policy is complex and requires careful deliberation of many and sometimes conflicting normative and technical criteria. Several approaches and tools, such as multi-criteria decision analysis, health technology assessments and evidence-to-decision (EtD) frameworks, have been proposed to guide decision-makers in selecting the criteria most relevant and appropriate for a transparent decision-making process. This study forms part of the development of the WHO-INTEGRATE EtD framework, a framework rooted in global health norms and values as reflected in key documents of the World Health Organization and the United Nations system. The objective of this study was to provide a comprehensive overview of criteria used in or proposed for real-world decision-making processes, including guideline development, health technology assessment, resource allocation and others. METHODS: We conducted an overview of systematic reviews through a combination of systematic literature searches and extensive reference searches. Systematic reviews reporting criteria used for real-world health decision-making by governmental or non-governmental organization on a supranational, national, or programme level were included and their quality assessed through a bespoke critical appraisal tool. The criteria reported in the reviews were extracted, de-duplicated and sorted into first-level (i.e. criteria), second-level (i.e. sub-criteria) and third-level (i.e. decision aspects) categories. First-level categories were developed a priori using a normative approach; second- and third-level categories were developed inductively. RESULTS: We included 36 systematic reviews providing criteria, of which one met all and another eleven met at least five of the items of our critical appraisal tool. The criteria were subsumed into 8 criteria, 45 sub-criteria and 200 decision aspects. The first-level of the category system comprised the following seven substantive criteria: “Health-related balance of benefits and harms”; “Human and individual rights”; “Acceptability considerations”; “Societal considerations”; “Considerations of equity, equality and fairness”; “Cost and financial considerations”; and “Feasibility and health system considerations”. In addition, we identified an eight criterion “Evidence”. CONCLUSION: This overview of systematic reviews provides a comprehensive overview of criteria used or suggested for real-world health decision-making. It also discusses key challenges in the selection of the most appropriate criteria and in seeking to implement a fair decision-making process. Page 2 of 15 Stratil et al. Cost Eff Resour Alloc (2020) 18:8 Background Decision-making in public health and health policy is complex [1] [2] [3] . Processes that consider evidence and other considerations in a structured manner require a careful deliberation of many and often conflicting normative and technical criteria [4] [5] [6] [7] [8] [9] [10] . The choice of which of these criteria should be employed in the form of criteria has a profound impact on the outcome of the decision-making process. In many decision-making processes directly addressing health, criteria have mostly been concerned with effectiveness and cost [10] [11] [12] [13] . This is at odds with the complexity of realworld decision making, were normative and feasibility considerations may act as key drivers of decisions (e.g. infringement of population health interventions on individual rights or interactions of interventions with other components of a health system) [14, 15] . The values and perceptions of different stakeholders with respect to normative and technical considerations often vary greatly both within and across societies. As there are various reasonable and defendable perceptions of which values and principles should guide the decision-making process and as there is no consensus on the right or best criteria, reasonable disagreement about the right decision or action is likely in pluralist societies [16] . Of course, many health-relevant decisions in public health policy and practice are made without adhering to structured decision-making processes based on predefined sets of criteria, populating those with evidence and weighting the results. But in various areas of public health policy and practice, such structured processes are relied on in the evaluation or comparison of alternative interventions or modes of actions. This inter alia includes the allocation of resources [9] , the setting of research priorities [8, 17] , decision-making about public health interventions [18] , the assessment of health technologies for funding or reimbursement [19] [20] [21] , or investment or disinvestment considerations [22] . Selecting the most appropriate and relevant criteria is a challenging but critical task in all of these structured decision-making processes. The criteria used across different types of decisions have been addressed in multiple reviews. Several reviews have explored the criteria used when applying multi-criteria decision analysis (MCDA) [13, [23] [24] [25] [26] [27] [28] , an"umbrella term to describe a collection of formal approaches which seek to take explicit account of multiple criteria in helping individuals or groups explore decisions that matter" [29] . Other reviews have explored the criteria employed in the context of health technology assessments (HTA), which intend to examine social, economic, organizational and ethical considerations in relation to health technologies in a comprehensive manner [30] ; these covered both the criteria to inform decisions about health technologies by national or sub-national HTA institutions [19, [31] [32] [33] , and the criteria used for selecting the technologies or interventions a HTA is to be conducted on [8, 34] . In general, reviews have addressed criteria used for making decisions on funding or implementing health interventions or technologies [9, 10, 22, [35] [36] [37] [38] , prioritizing research topics [39] [40] [41] or coverage decisions [7, 12, 42, 43] . Reviews include criteria used on various levels of decision making (national, regional, or local), in different contexts (e.g. high-vs. low-income countries), and proposed by various stakeholder groups (e.g. decision makers, beneficiaries/patients). Against this background, we conducted this study as part of a larger research project to develop a new evidence-to-decision (EtD) framework. The WHO-INTE-GRATE EtD framework was developed to be firmly rooted in WHO norms and values and reflective of the changing global health landscape, and to encompass a comprehensive set of criteria suitable for decision making on clinical practice, public health, and health system interventions [15] . Within the development process of the framework, we conducted this overview of systematic reviews de-novo with the objective to provide a comprehensive overview of criteria used or intended to be used in real-world health decisions. More details on the role of this review in the development process of the WHO-INTEGRATE EtD framework is provided in our publication Rehfuess/Stratil et al. [15] . Our search strategy combined the terms "decision-making", "decision maker*", "decision analysis", "multi-criteria decision analysis", "priority setting", "resource allocation", "policy-making" and "policy-maker*" and their synonyms with the terms "criterion" and "criteria" as well as the terms "review*", "literature search", "mapping", "meta analysis" and their synonyms. Searches were conducted in PubMed and focused on the occurrence of these search terms in title and abstract. As the term "criteria" is used in many adjacent fields (e.g. referring to treatment or diagnostic decisions), we complemented these systematic searches with hand searches of the references of all included studies. Title and abstract screening was conducted using the software Rayyan [44] . Title and abstract screening as well as full-text screening was independently undertaken by two authors (JMS and AN), based on the inclusion and exclusion criteria shown in Table 1 . We included studies which had conducted systematic searches of the literature and had comprehensively reported real-world criteria used in health decision-making. Studies focused on clinical decision-making (i.e. concerned with the decisions of individual patients) as well as studies focussing on the concepts or measurements of individual criteria (e.g. cost-effectiveness) were excluded. Where discrepancies could not be resolved by the two screening authors, a third author (ER) was consulted. Screening of the records identified through the updated literature searches was conducted by the author (JMS) and a research assistant (ST). The original literature searches were undertaken in September 2016 and updated in July 2018. Information extracted from the included reviews were (i) study objective, (ii) type of health decision, (iii) the types of studies included, (iv) the strategy used to identify primary studies or documents, (v) information on how the criteria were compiled in the primary studies, (vi) the topic of the health decision in the primary studies (e.g. public health interventions, pharmaceuticals), (vii) the regional context of the primary studies (e.g. high-or lowincome countries), (viii) the decision-making level (e.g. national, regional, local) and, importantly, (ix) the criteria themselves. We critically appraised included studies. As no adequate, validated critical appraisal tool was available, we adapted items of the CASP systematic review checklist [45] and AMSTAR 2 [46] to our research question (Additional file 1). Our critical appraisal tool focuses on (i) the formulation of a clear research question regarding the decision-making process to be explored, (ii) a comprehensive search strategy, (iii) the adequate selection of eligible studies, (iv) the comprehensive extraction of criteria, (v) the critical appraisal of primary studies, (vi) the adequate description of the identified criteria (vi) the consideration of potential conflicts of interest, and (vii) the use of pre-established methods. The critical appraisal was conducted by one author (JS) and cross-checked by a research assistant (ST). Given the intended primary use of the WHO-INTE-GRATE framework in the development of WHO guidelines, the analysis focuses on substantive criteria (i.e. "What are the considerations or criteria a decision should be based on?"; e.g. cost, health benefit, available resources) rather than procedural criteria (i.e. "How should the process through which a decision is made be organized?"; e.g. transparency, participation of key stakeholders, opportunity for revising decisions). This information was extracted onto an Excel spreadsheet by one author (JMS) and spot checked by a research assistant (ST). Wherever possible, criteria were extracted as stated in the primary studies. Where the reviews only reported synthesised criteria without a direct link to the primary studies, we extracted these synthesised criteria (e.g. "disease burden" and "burden of illness" as reported in primary studies summarized in a single "burden of disease" criterion in the included review). Categories, referring to the classification system developed or used in the reviews, were not extracted (e.g. "cost" and "cost-effectiveness" as reported in primary studies summarized under the criterion "financial considerations" in the included review). The criteria were then re-organised by one author (JMS) by combining (i) identical criteria (e.g. "burden of disease" and "burden of disease") and (ii) criteria described through similar terms with the same meaning (e.g. "burden of disease", "burden of illness" or "disease burden"). The criteria were then synthesised in a mixed inductive and deductive approach: The publication is a review based on systematic searches of the literature The publication is concerned with the criteria considered as part of a health decision between two or more options or the weighting of multiple options, made by a governmental or non-governmental organization on a supranational, national, or programme level The health decision is made from a population perspective regarding the general population, patients, healthcare personnel, health decision-makers or other similar stakeholders in public health and healthcare The publication reports on a comprehensive set of criteria identified through the literature searches (at least 3 criteria) The publication is written in English, German, Spanish or Italian The publication is not a literature review The publication did not utilize a systematic search strategy The publication is focused on selected criteria (e.g. cost-effectiveness) rather than sets of criteria guiding a decision The publication relates to a health decision made outside of an organizational context (e.g. General Practitioner's office) and/or from an individual perspective (e.g. treatment choices for an individual patient in clinical practice) The publication primarily addresses issues of how to measure, weight or calculate a criterion (e.g. cost-effectiveness, quality of life) For the deductive approach, we used an intermediate step in the development of the WHO-INTEGRATE framework [15] , the seven so-called preliminary criteria "Health-related balance of benefits and harms", "Human and individual rights", "Acceptability considerations", "Societal considerations", "Considerations of equity, equality and fairness", "Cost and financial considerations", "Feasibility and health system considerations", as well as "Evidence considerations" (Note that these categories were revised in the final WHO-INTEGRATE Framework [15] ). "Evidence considerations" was singled out to align with the role of evidence as a meta-criterion in the WHO-INTEGRATE Framework: rather than taking evidence as one of several substantive decision-making criteria into account, the framework argues for reflecting on the quality of evidence of each criterion and considering these aspects alongside. We used these-what we refer to as-criteria as level one of the category system. During the synthesis, we remained open-minded about revisions of the category system to be able to capture new considerations relevant for decision making in an appropriate manner. For the inductive approach, we started from the criteria as reported in primary studies and reviews and grouped similar criteria into groups of-what we refer to as-sub-criteria (level two of the category system) and decision aspects (level three of the category system). Criteria relating to decision-making principles, procedural criteria and research priority setting were extracted and categorized separately. In cases where the exact meaning of a criterion was unclear, the primary publication was consulted wherever possible. Were uncertainty remained, these cases were discussed with a research assistant (ST) or with other members of the research team (ER; RB). After an initial sorting of criteria identified through the included reviews into the three-level category system, this was discussed and refined through discussions between JMS, ST, RB and ER; one author (JMS) subsequently conducted a second round of sorting of the extracted criteria to ensure that all criteria, sub-criteria and decision aspects would be placed correctly within the category system. The literature search yielded 4448 unique records, of which 106 were assessed for eligibility based on their full text. A further 88 records were identified through hand searching (see Additional file 2 for PRISMA diagram). We included 36 reviews in this overview of systematic reviews [4-8, 10-13, 19-23, 27, 28, 31-37, 39, 41-43, 47-54] . All of these were published after 2006, with 15 reviews published in 2018 or 2017 and only 5 reviews published before 2010. 16 reviews provided criteria used for or intended to guide various priority setting exercises [5, 6, 8, 11, 12, 31, 32, 34-36, 38, 41, 42, 49, 51, 55] , with one review focused on research priority setting (in the field of child health and nutrition) [39] . Six reviews were framed in the context of multi-criteria decision analysis [6, 13, 23, 27, 28, 48] . Three reviews explored criteria used to guide investment or disinvestment decisions [22, 27, 37] . Two reviews assessed criteria to guide the selection of topics for HTA [8, 34] , and eight reviews captured criteria used in HTA [8, 11, 19, 20, 31, 32, 48] . Four reviews focused specifically on the evaluation of or decisions on vaccines [4, 21, 47, 55] . 19 reviews exclusively included studies or documents from high-income countries while five had an explicit focus on decision-making processes in low-and middle-income countries [10, 35, 52, 55] . The number of criteria extracted from each publication ranged from 31 [4] to 360 [6] . The criteria, sub-criteria and decision aspects based on the -criteria extracted from the reviews are provided in Table 2 . An additional category containing synthesized criteria extracted from the included reviews is provided in Additional file 3. The first level of the category system encompasses seven substantive criteria, i.e. "Healthrelated balance of benefits and harms", "Human and individual rights", "Acceptability considerations", "Societal considerations", "Considerations of equity, equality and fairness", "Cost and financial considerations", and "Feasibility and health system considerations". In addition to these substantive criteria, we also identified an eight criterion "evidence" ( Table 3) . As criteria may be used in different decision-making processes and different decision-making contexts, not all criteria may apply. One important distinction, for example, is whether the problem to be addressed (e.g. a specific disease) has already been decided on or not. If so, the decision is about selecting one out of several options to address the problem, and considerations regarding the priority of the problem itself (e.g. burden or severity of disease or disability) are no longer relevant. As noted in the methods section, we sorted criteria into a category system based on content. This way of organizing the criteria could be modified by adding additional dimensions. For example, one could also adopt a temporal perspective where criteria may relate to the point in time before an intervention is decided on or implemented the process of implementing the intervention or the short-term or longer-term outcome of the intervention. As an illustration, equity considerations can be framed as relating to the starting point (e.g. priority of a given health issue due to high health inequity), as an criterion of relevance to the implementation process (e.g. distribution of adverse events across all those affected by the intervention) or as an outcome (e.g. reduced health inequity several years Uptake of intervention [15, 20, 34, 35] Magnitude of benefit/effect/impact [2, 4, 11, 14, 18, 31, 33, 36] Additional or indirect effects [2, 6, 33, 34] Type and composition of effect/ benefit/impact [2] Impact on mortality, survival, longevity and life expectancy [1, 2, 4, 11, 16, 19, 21, 24-26, 28, 34-36] Last chance therapies [23, 24] Impact on morbidity and disability [1, 2, 16, 35] Potential changes in health consequences [24, 25] Impact on (health-related) quality of life [2, 8, 11, 12, 14, 19, 20, 22, 25, 26, 28, 29, 31, 33, 35, 36] Impact on patient-reported outcomes [2, 12, 16, 21, 26] Valuation of health outcomes by patients and desirability of the effects [2] Preventive benefits/effects or preventive approaches [1, 2, 4, 5, 16, 21, 25, 26, 31] Character of benefit or effect Onset of effect and time until benefit [2, 11, 13] Duration, sustainability and lasting effect [2, 11, 13, 15, 27, 31] Individual and population level of benefit Clinical benefits/effectiveness/impact [2, 3, 5, 11, 14, 17-22, 24, 25, 28, 31-33, 36] Individual level benefit, effectiveness or impact [1, 2, 4, 18, 21, 22, 25, 26, 36] Marginal benefits (for every patient) [20, 26] Population level benefit, effectiveness or impact [1-5, 9, 12, 16, 18, 19, 24-27, 31, 34, 35] Threshold effectiveness on populations (herd immunity) [15, 34] Balance of benefits and harms Balance of (health) benefits and harms [5, 14, 18, 19, 23, 24, 34] General considerations surrounding harm/risk General safety, risk and tolerability of intervention [2, 4, 5, 7-9, 11-17, 19-26, 29, 31, 33-36] Magnitude and likelihood of adverse events [26, [33] [34] [35] Valuation of health outcomes by patients and desirability of the effects regarding harms [34] Short and long term risk and safety profile [26] Over diagnosis and over treatment [16, 26] Stigmatization [2, 26] Risk of failure of intervention [15, 34, 35] Burden of treatment [2, 26, 33] Risk of inappropriate use [8, 16, 25, 26] Impact on disease patterns and reduced long-term effectiveness [15, 34] Other or additional adverse events [2] Health-related need and priority Health-related needs: in general [1, 2, 11, 14, 22, 23, 24] Burden and impact of disease: in general [ Size of affected population and number of potential beneficiaries [1, 2, 4, 5, 8, 10, 11, 13-17, 20-26, 28, 31-34, 36] Maximum potential for disease burden reduction [10] Severity of disease/condition: in general [1, 2, 4, 5, 9-12, 14, 16-19, 21-26, 28, 29, 31, 33-36] Severity of disease/condition: long term outcomes [34, 35] Severity of disease/condition: life threatening disease/condition and prognosis without treatment [1, 5, 11, 16, 21, 28, 29, 34] Severity of disease/condition: late stage or end-of life status of disease/condition [5, 21, 24, 25, 28, 29, 36] Outbreaks and epidemic potential [15, 34] Urgency and emergencies [1, 2, 13, 25] Human and individual rights Human rights considerations [2, 11, 12, 20, 36] Autonomy and informed consent [2, 11, 35] Privacy and confidentiality [26, 35] Intrusiveness of intervention [22] Acceptability considerations Perceived priority of the problem Public perception of disease burden, disease risk or severity [15, 34, 35] Acceptability Acceptability in general [2, 4, 6, 14, 15, 24, 26, 27, [34] [35] [36] Acceptability of cost and financial outcomes [2, 14, 25, 26, 34] Acceptability by beneficiaries Acceptability by beneficiaries: in general [2, 6, 8, 11, 15, 16, 26, 35] Comfort, convenience and user experience [2, 11, 12, 14, 15, 20, 24-26, 31, 33, 35] Acceptability by those providing intervention Acceptability by those providing intervention [15, 16, 34, 35] Social and cultural acceptability Social and cultural acceptability [2, 4, 8, 9, 11, 15, 18 Demands, interest and pressures by industry [2] Pressures, demand and interest of beneficiaries and patient representatives [2, 17, 20, 22, 25, 32, 33] Pressures, demand and interest of those providing intervention [2, 17, 20, 25, 26, 32, 33] Media attention and coverage [1, 32] Societal considerations Societal needs and priority Social burden of disease (individual/population) [8, 18, 21, 29, 34, 35] Social needs [24] Economic burden of disease on society (in general) [32] Social and societal impact Social impact or benefits [2, 3, 5, 6, 14, 21, 23-26, 31, 33, 35] Impact on non-health outcomes (in general) [15, 18, 26, 34] Impact on poverty [1, 2, 4, 14, 25, 26, 36] Relevance to social development of the country [10] Value of hope [11] Raise profile of condition [14] Impact on economy Impact on economy [2, 25, 26] Impact on productivity and population in productive age [2, 12, 16, 25] Relevance to economic development of the country Innovativeness (potential to encourage innovation) [2, 9, 10, 14, 21, [23] [24] [25] 31] Environmental impact Environmental and/or ecological impact of intervention (in general) [14, 16, 21, [29] [30] [31] 33] Impact on future generations Impact on future generations [19] [1-4, 9-11, 14, 15, 18, 19, 22-24, 26, 27, 29, 33-36] Fairness considerations: in general [2, 3, 6, 11, 16, 26, 36] Impact on (health) (in-)equity/(in-)equality [2, 6, 10, 14, 26, 31, 34, 35] Distribution of benefits and harms [10, 26] Accessibility Accessibility in general [2-4, 6, 9, 14-17, 19, 21, 24, 29, 31, 33-35] Equity in accessibility [1, 2, 4, 6, 7, 11, 25, 26, 35] Physical and spacial accessibility [2, 4, 6, 16, 26] Timeliness of access (time spent waiting for treatment) [2, 3, 22, 36] Informational accessibility [11] Financial accessibility of intervention/Affordability [4, 9, 10, 12, 14, 19, 25, 26, [34] [35] [36] Affordability: risk of catastrophic health expenditure [11, 16, 25] Affordability: cost and Financial impact on beneficiaries [11, 13, 14, 16, 22, 25, 26, 31, 34, 35] Availability Availability/lack of suitable alternatives [1, 2, 5, 11, 12, 14, 17, 19-25, 28, 32, 34-36] Limitations of alternative interventions [2, 11, 14, 15, 25, 26, 34, 35] Rare diseases/orphan disease [12, 14, 16, 21, 24, 28] Unmet needs [11, 12, 16, 21, 24, 29] Responsibility Ability to reduce own health risk and conditions arising from patient behavior [4, 5, 11, 16, 24, 25] Non-discrimination Non-discrimination [11, 23, 24] Consideration regarding specific populations Consideration of high-risk populations [2, 11, 14, 26] Consideration of vulnerable populations [1, 2, 4, 11, 21, 25, 26] Consideration of Socio-economic status [4, 16, 25, 26, 36] Consideration of sex, gender, and/or sexual orientation [2, 6, 16, 25, 26, 34] Consideration of race/ethnicity [25, 36] Consideration of care giver responsibilities [29] Consideration of age-groups [1, 2, 4, 11, 14, 16, 26, 28] Consideration of place of living [11, 16, 25] Consideration of identity and ideology [25] Other group related considerations [11] Cost and financial considerations Cost over time [20, 21, 25, 32, 34] Long term cost/budget impact [25] Overall cost/budget impact [1, 2, 11, 22, 26, 36] Direct cost [11, 25, 32, 34, 35] Indirect/additional/hidden cost [11, [34] [35] [36] Marginal cost [11, 25] Opportunity cost [2, 11, 21, 31] Appropriateness of intervention for specific context [2, 16, 30] Legislative and regulatory considerations Adherence to legal requirements, constrains and implications [2, 8, 10, 15, 16, 18, 25, 26, [31] [32] [33] [34] [35] Adherence to other directives, standards and requirements [2, 6, 14, 22, 30, 34] Political considerations Political acceptability, interests and pressures [1-4, 6, 10, 11, 14, 18, 21, 25, 26, 33-36] Donor and global interests and pressures [2, 3] Political impact [2, 25, 33, 35] Alignment with priorities [1-4, 6, 13-15, 21, 23, 24, 26, 29, 30, 34, 35] Mission, mandate and goals of health system [2, 13, 14, 21, 25] after introducing the intervention). Further additional dimensions could be a focus on individuals, populations or systems (e.g. clinical health benefits for the individual, reduction of the disease burden of a population, or impact on the performance of a health system following an intervention). In the organization of the criteria, we kept such additional organizational dimensions in mind. The most frequently reported criteria were healthrelated impact of interventions, cost, cost-effectiveness and political interests or priorities; these were covered in all of the included reviews. Rarely used criteria were concerned with the environmental or societal impacts of interventions, and (non-financial) resource availability/needs. The granularity (level of detail with respect to sub-criteria/aspects) varied widely depending on the criterion: the criteria related to cost or financial considerations included general (e.g. "resource use" or "cost") as well as very specific usages (e.g. distinct ways to quantify cost-effectiveness). In contrast, criteria related to the societal or environmental impacts of interventions, as well as considerations regarding equity or equality were usually reported in very generic terms. Evidence in general or evidence regarding specific criteria was mentioned in most included reviews, most often using generic terms, such as "evidence". In some cases, the criterion evidence referred to specific measures, primarily "evidence of effectiveness/efficacy" and sometimes "evidence on cost". In other reviews, this included criteria regarding the relevance of the available evidence for a given context (e.g. "relevance of evidence" or "generalizability of evidence") and criteria regarding the quality of evidence (e.g. "certainty of evidence", "credibility of evidence" or "validity of evidence") (see Table 3 and Additional file 4). We also identified several considerations of specific relevance to research priority setting, covering considerations regarding the answerability of the research question, research ethics or avoidance of duplication of research. As those were not the primary focus of this Strategic considerations Strategic planning and considerations [1, 2, 13, 19, 25, 31] Existing cooperation [2, 10, 14, 22] Decisions and practice of other institutions or stakeholders [2, 11, 17, 24, 29, 34] Alignment with recommendations, guidelines and standards [1-3, 7, 11, 14, 16, 21, 24, 26, 29, 34] Historical context and past decisions [1-3, 11, 13, 14, 17, 19-22, 25] Availability of incentives [1, 2] Impact on future decisions [6, 14] Keeping promises and commitments [2, 13] Characteristic of intervention (technical) Complexity of intervention [2, 4, 6, 16, 21] Scalability of intervention [9, 27] Ability to evaluate intervention [2, [33] [34] [35] Reversibility of intervention [2] Flexibility of implementation [2] Uniqueness [21] Frequency of use and expected level of usage/activity [1, 2, 6, 8, 14, 25, 26, 32] Dependence on maintenance [26, 30] Sustainability of intervention utilization (e.g. Supply of parts, vaccines) [2, 14, 15, 27, 29, 34] Position of intervention in care pathway [5, 12, 20, 24] Additional uses of intervention [5, 21, [23] [24] [25] publication, they are not further discussed here but listed in Additional file 4. Furthermore, the included publications reported several decision-making principles (i.e. guiding concepts from which different criteria derive). The distinction between decision-making principles and substantive criteria is not always clear cut. For example, human rights can be regarded as an underlying principle from which other criteria derive (as used in the human rights-based framework by Bustreo et al. [56] ), as well as a specific criterion (assessing whether the intervention is in accordance with human rights). From the publications included in our overview of systematic reviews, the following criteria were extracted: Beneficence, non-maleficence, fairness, diversity, fair innings, proportional shortfall, concern for the worse off, justice, formal justice, social justice, distributive justice, principles of human rights, principle of human dignity; marginal utility principle, principle of need and solidarity, collectivism, cohesion, mutuality, rule of rescue and Rawls' difference principle (see Additional file 4). Table 2 provides an overview of the criteria, sub-criteria and decision aspects for the seven substantive criteria. Criteria relating to evidence are reported in Table 3 . The results of the critical appraisal are provided in Additional file 5. Only one publication, Niessen et al. [11] , met all eight items; 11 out of 36 publications met five or more items. Most publications did not conduct a critical appraisal of included studies, did not report independent extraction of criteria by two reviewers, and did not state explicitly, that the review had been undertaken based on a protocol or otherwise pre-established methods (Additional file 5). Drawing on 36 included reviews, we identified a set of 200 unique decision aspects. These were sorted into 7 substantive criteria und 45 sub-criteria as well as a separate criterion on evidence. The substantive criteria cover health-related balance of benefits and harms; human and individual rights; acceptability considerations; societal considerations; considerations of equity, equality and fairness; cost and financial considerations; and feasibility and health system considerations. We found that some criteria, sub-criteria and decision aspects are well developed in the literature, such as those referring to the health implications of an interventions or to the costs of an intervention. In contrast, several others lacked a clear conceptualisation, notably those relating to societal implications or equity and equality considerations. The wide range of decision aspects were used to refine the criteria and sub-criteria in the WHO-INTEGRATE EtD framework, as well as to inform the development of definitions and guiding questions provided as part of the framework. In addition to their use in the WHO-INTEGRATE framework, we postulate that the list of criteria, subcriteria and decision aspects can be helpful to decisionmakers in their own right: To the best of our knowledge, this is the most comprehensive and up-to-date list of real-world criteria available for health decision-making. It could therefore provide a valuable tool for informing decision-makers wishing to select those criteria relevant for a given type of decision and decision-making context. This comprehensive list is likely to be most relevant to decisions in public health or healthcare. Due to the focus of the present study, the applicability for research priority setting or the evaluation of diagnostic or testing devices is likely to be more limited, as we may not have covered all relevant publications. Most of the reviews included in our overview of systematic reviews did not meet all or even a majority of the items of our critical appraisal tool. This finding does, however, need to be interpreted in view of the following considerations. First, a validated critical appraisal tool appropriate for the topic does not exist-neither at the level of systematic reviews nor at the level of primary studies. Only three of the included reviews undertook some form of critical appraisal: Whaitaka et al. [36] and Burchett et al. [4] used an adapted CASP Qualitative Checklist and Niessen et al. [11] used custom qualityof-research assessment scales. Second, the low score of Table 3 Overview of evidence considerations Evidence Considerations evidence in general [1, 2, 11-13, 15, 17, 21, 23, 31, 33, 36] strength of evidence in general [2, 11, 12, 21, 25, 36] Certainty of evidence (in general) [2, 5, 6, 11, 16, 21-24, 26, 28, 33, 36] Quality of evidence [2, 4-6, 11, 14-22, 25, 26, 28, 29, 33, 34, 36] Completeness of evidence [2, 6, 12, 14, 25, 26] Validity of evidence [2, 6, 25, 26, 29, 36] Credibility of evidence [29, 34, 36] directness of evidence [2, 11, 28] Consistency of evidence [2, 6, 14, 25, 26, 29, 33] Precision of evidence effect [2] Relevance of evidence [2, 6, 11, [25] [26] [27] [28] [29] Applicability and generalizability of evidence [2, 11, 22] Type and quality of evidence sources [2, 11, 17, 21, 34, 35] Experience based evidence [26, 34, 35] Evidence requirements [2, 5, 25] Page 11 of 15 Stratil et al. Cost Eff Resour Alloc (2020) 18:8 some of the included reviews is likely due to poor reporting rather than poor conduct (e.g. regarding pre-established methods, or data extraction in duplicate).Third, the value and relevance of criteria for a given decision-making process does not necessarily depend on the quality of the review they were derived from. For example, even if the criterion "environmental impact" was merely mentioned in a single systematic review of low quality, this would not invalidate its relevance for a decision-making process focused on interventions with pronounced environmental (adverse) effect (e.g. large-scale usage of DDT in malaria prevention). With our overview of systematic reviews, we build on several previously published reviews, notably, the review by Guindo and colleagues, which represented the most extensive general overview of criteria until now [6] . Rather than focusing on specific decision-making processes (e.g. priority setting in low-and middle-income countries), we sought to cover the full range and heterogeneity of criteria and their use across various health fields. We followed an approach focusing on descriptive ("what criteria are used?"), rather than prescriptive ("what criteria should be used?") approach. Several overviews of more prescriptive frameworks have been published in the field of public-health ethics in recent years [58] [59] [60] . A similar undertaking-providing decisionmakers with a basis to select appropriate criteria-was conducted by Vermeulen and Krabbe, who provided an overview of the most widely recognized arguments and principles used in decision-making [18] . Their more prescriptive publication, which explores decision arguments and principles, and our more descriptive publication complement each other. In contrast to some of the other reviews of criteria for decision making [6, 10] , we abstained from quantifying how often criteria were cited for several reasons: First, the focus of this publication was to provide an overview of criteria that can be used for decision-making, rather than to provide an overview of which criteria are (widely) used in different decision making settings, as was the purpose in other publications [5, 6, 35] . Second, the quantification of how often or rarely a criterion is used does not necessarily imply its relevance for a given decision-making process: we believe that relevance should be informed by normative considerations. Third, there is a pronounced heterogeneity in the included studies: this begs the question, whether a criterion used in decisionmaking in a local hospital should count as much as the criteria used in the health technology assessment process of a national or supra-national organization. Finally, the quantification of the use of criteria is complicated: not only were many studies cited in several included reviews [61] , but some of the reviews referred to other reviews as their data sources [6, 38] . Our focus of the literature search on a single data base (PubMed) and the reliance on a selection of terms such as "criteria/criterion" might have missed relevant studies conducted on this issue. These decisions were made due to significant time and resource constraints relating to the development of the WHO-INTEGRATE framework over a relatively short period of time. We countered this potential limitation by thoroughly searching the references of all included studies, which yielded some additional publications. Furthermore, during the extraction of the criteria from included reviews we noted that we seemed to have reached saturation, as from the mid-way point, additional extracted studies yielded no or minimal additional criteria. Expanding the search to additional databases, especially those in the fields of political sciences and health economics, with a more inclusive search strategy may yield valuable additional insights from a broader range of disciplines. A significant strength of our publication is that-to the best of our knowledge-it is the most extensive overview of criteria used in or proposed for health decision making. We included studies from several different health fields, conducted on various levels of decision-making and topics and in heterogeneous contexts around the world. We classified this comprehensive and diverse set of criteria according to a theory-based categorization system comprising three levels, i.e. criteria, subcriteria and decision aspects. In doing so, as a team we critically reflected on extracted criteria and their underlying rationale, seeking to be as consistent as possible in how we sorted criteria reported in included reviews into higher-order categories. The very large number of criteria and sub-criteria identified in this publication highlights the complexity of health decision-making It can serve as a resource when considering which criteria to include in sound multi-criteria approaches (i.e. adhering to principles of completeness, lack of redundancy, mutual independence, operationalizability and clustering) and how to use these. At the centre of any decision-making process will be the challenge of who selects which criteria and how they should be weighted or ranked against each other. As various stakeholders with diverging but reasonable motives are likely to disagree on which criteria are the right ones, the focus often shifts from selecting the right criteria to making decisions using a good or fair process [16, 62] . Numerous procedural conditions which characterize such a fair process have been proposed, including in the Accountability for Reasonableness framework [16] , among others [63] [64] [65] [66] . A fair and transparent process and especially an adequate representation and participation of all relevant stakeholder groups is essential for achieving legitimacy [62, 63] . One approach to overcoming reasonable disagreement about criteria for decision making is to reflect on the underlying normative principles and to make them explicit, e.g. by exploring the roots of a conflict which may lie in (potentially) conflicting normative arguments, e.g. if improving the life of a large number of people has to be weighed against the interest of those suffering from rare diseases with no alternative treatment. While we extracted such principles in our overview of reviews, others have focused explicitly on these [18, 58, 60] and several frameworks to guide the discussions and selection process have been proposed in the public health ethics literature [59, 60, 64, 65, [67] [68] [69] [70] [71] [72] [73] [74] . Furthermore, underlying motives and drivers of stakeholders should be taken into account when reflecting on proposed criteria, as these can manifest themselves as trojan horses cloaked in ethical rhetoric [75] . Some calls for strengthening the consideration of criteria beyond evidence of effectiveness or incremental cost-effectiveness ratios are motivated by vested interest in a specific outcome [27, 76] . Such conflicts of interest should not necessarily lead to dismiss the arguments made, but it should lead to a critical reflection regarding the relevance and appropriateness of the proposed criteria for a given decision-making process and the power relations in the discourse [75, 77] . The criteria by themselves are often highly interconnected and at times conflicting. An example is the criterion "age", which can serve as a "surrogate" criterion for other normative and (harder to measure) considerations. For example, a focus on interventions targeting younger people may be motivated by their potential for achieving a longer life span (greater health impact) or to reduce productivity losses (positive impact on the economy). At the same time, "age" can be considered with respect to non-discrimination or equity: explicitly reflecting on age in order not to prioritize one group of people based on age as a characteristic (ageism). An explicit reflection on and discussion of such conflicts within criteria is important. Furthermore, the criteria identified in the included reviews are partly overlapping (e.g. cost, effectiveness, and cost-effectiveness). Depending on the decision-making process and the tools used (e.g. MCDA), accounting for overlaps and redundancies may be of relevance. This can, for example, be achieved through selecting non-overlapping criteria or through increasing the granularity of the criteria. In particular the MCDA-literature has developed methods and guidance on how to identify and handle overlapping criteria [78] . The example of "age" as a criterion that can have conflicting interpretations highlights the need to set up a mechanism for handling conflicts within criteria and balancing interests in place. The same holds true for conflicts between criteria (e.g. positive impact on population health, negative impact on the natural environment), which occur on a regular basis in decisionmaking processes. Populating criteria with evidence presents a third important challenge [15] . Evidence collection and synthesis approaches are well developed for some criteria (e.g. health impacts) although some challenges remain. For a few criteria, approaches are virtually non-existent in the literature on health decision-making (e.g. environmental implications) while for others there is a lack of clarity regarding the best methods to be employed (e.g. societal or environmental impact assessments) [15] . It is highly likely that suitable methods exist outside of the health decision-making or broader healthcare and public health literature and learning from other disciplines may offer solutions to this challenge. Guideline development, HTA and other decisionmaking processes aiming to integrate evidence and criteria for decision making in a structured manner usually operate under significant time and resource constraints. To avoid treating criteria beyond effectiveness and cost-effectiveness as an "after thought", evidence will need to be collected or analysis on these other criteria. This will require the development of rapid and pragmatic approaches to keep such decisionmaking processes feasible. The comprehensive list of criteria from and for real-world health decision-making presented here was an essential building block in the development of the WHO-INTE-GRATE framework. We postulate that it can also be a useful stand-alone tool to inform health decision-making processes not employing an EtD framework. To make the best possible use of this list, solutions to the challenges of Intelligent policy making for a complex world: pragmatism, evidence and learning Briefing Note-Policy dialogue: what it is and how it can contribute to evidenceinformed decision-making. Geneva: World Health Organization Goodbye to the simple solutions: the second phase of priority setting in health care National decision-making on adopting new vaccines: a systematic review Real-world' health care priority setting using explicit decision criteria: a systematic review of the literature From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking Stated and revealed preferences for funding new high-cost cancer drugs: a critical review of the evidence from patients, the public and payers Priority setting for health technology assessments: a systematic review of current practical approaches Societal values in the allocation of healthcare resources: is it all about the health gain? Setting priorities for health interventions in developing countries: a review of empirical studies Methods research report (Prepared by the Johns Hopkins University Evidence-based Practice Center under contract No. 290-2007-10061-I) Decision-making in priority setting for medicines-A review of empirical studies on behalf of the National Leading-Edge Cluster Medical Technologies 'Medical Valley EMN' . Balancing costs and benefits at different stages of medical innovation: a systematic review of multi-criteria decision analysis (MCDA) Integrating social justice concerns into economic evaluation for healthcare and public health: a systematic review The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective Accountability for reasonableness Priority setting for health technology assessment at CADTH Value judgment of health interventions from different perspectives: arguments and criteria Health technology funding decision-making processes around the world: the same, yet different Role of centralized review processes for making reimbursement decisions on new health technologies in Europe Decision support in vaccination policies Case studies that illustrate disinvestment and resource allocation decisionmaking processes in health care: a systematic review Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decisionmaking FIFARMA Amplifying each patient's voice: a systematic review of multi-criteria decision analyses involving patients. Appl Health Econ Health Policy Making good decisions in healthcare with multi-criteria decision analysis: the use, current research and future development of MCDA. Appl Health Econ Health Policy Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging good practices task force Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature Using multi-criteria decision analysis to appraise orphan drugs: a systematic review Multiple criteria decision analysis-an integrated approach Health technology assessment and health policy today: a multifaceted view of their unstable crossroads Revealed and stated preferences of decision makers for priority setting in health technology assessment: a systematic review The health systems' priority setting criteria for selecting health technologies: a systematic review of the current evidence Health technology prioritization: which criteria for prioritizing new technologies and what are their relative weights? Health Policy How to choose health technologies to be assessed by HTA? A review of criteria for priority setting Using economic evidence to set healthcare priorities in low-income and lower-middle-income countries: a systematic review of methodological frameworks Evaluating healthcare priority setting at the meso level: a thematic review of empirical literature Hospital managers' need for information on health technology investments Priority setting of health interventions. Review of criteria, approaches and role of assessment agencies Setting health research priorities using the CHNRI method: VII. A review of the first 50 applications of the CHNRI method A review of selected research priority setting processes at national level in low and middle income countries: towards fair and legitimate priority setting How are health research priorities set in low and middle income countries? A systematic review of published reports Scoping literature review on the basic health benefit package and its determinant criteria A systematic review of coverage decision-making on health technologies-evidence from the real world. Health Policy Rayyan-a web and mobile app for systematic reviews AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both Comparison of NITAG policies and working processes in selected developed countries Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries Setting healthcare priorities: a description and evaluation of the budgeting and planning process in county hospitals in Kenya A review of NICE methods and processes across health technology assessment programmes: why the differences and what is the impact? Appl Health Econ Health Policy Pharmaceutical priority setting and the use of health economic evaluations: a systematic literature review. Value Health Methods for medical device and equipment procurement and prioritization within low-and middle-income countries: findings of a systematic literature review Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario Conceptual frameworks and key dimensions to support coverage decisions for vaccines Accounting for equity considerations in cost-effectiveness analysis: a systematic review of rotavirus vaccine in low-and middle-income countries 18:8 • fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations maximum visibility for your research: over Ready to submit your research ? Choose BMC and benefit from World Health Organization. Women's and children's health: evidence of impact of human rights Setting healthcare priorities in hospitals: a review of empirical studies The evolution of public health ethics frameworks: systematic review of moral values and norms in public health policy Public health ethics theory: review and path to convergence An overview of ethical frameworks in public health: can they be supportive in the evaluation of programs to prevent overweight? Priority setting of health interventions: the need for multi-criteria decision analysis A strategy to improve priority setting in health care institutions Beyond accountability for reasonableness Public health ethics: mapping the terrain Beyond evidence-to ethics: a decision-making framework for health promotion, public health and health improvement The idea of justice An ethics framework for public health Putting Public health ethics into practice: a systematic framework Theoretical models and operational frameworks in public health ethics Pandemic influenza preparedness: an ethical framework to guide decision-making Principles for the justification of public health intervention Looking ahead: addressing ethical challenges in public health practice Preparedness: medical ethics versus public health ethics The unethical use of ethical rhetoric: the case of flibanserin and pharmacologisation of female sexual desire Multiple criteria decision analysis for health care decision making-emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force Hypoactive sexual desire disorder: inventing a disease to sell low libido The use of MCDA in HTA: great potential, but more effort needed Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations We would like to acknowledge the contribution of the research assistants Anja Nacken (AN) and Sarah Thumbeck (ST) who supported us in the data analysis. We furthermore want to acknowledge of all those involved in the development of the WHO-INTEGRATE EtD Framework V1.0, in particular Anayda Portela and Susan L Norris, from the World Health Organization, who supported us on the conceptualization of our approach. Page 13 of 15 Stratil et al. Cost Eff Resour Alloc (2020) 18:8 selecting criteria, of resolving conflicts between criteria or their interpretation, and of identifying and appraising evidence towards these criteria will need to be found. The WHO-INTEGRATE framework seeks to address some of these challenges, by providing a set of criteria selected based on a strong normative basis and by offering a methodological toolbox, which suggests both comprehensive and pragmatic approaches to populating criteria with evidence [15] . Supplementary information accompanies this paper at https ://doi. org/10.1186/s1296 2-020-0203-6.Additional file 1. Critical appraisal tool. Funding was provided by the World Health Organization's Department of Maternal, Newborn, Child and Adolescent Health through a grant received from the United States Agency for International Development (USAID). We also gratefully acknowledge that JMS's position was funded by the Bavarian Health and Food Safety Authority and that IBS's input was funded by the Norwegian Agency for Development Cooperation (Norad). All data generated or analyzed during this study are included in this published article and its Additional files. Not applicable. Not applicable. This research was conducted within a research project initiated by the World Health Organization in which the WHO-INTEGRATE EtD Framework V1.0 was first developed and published by the authors. The authors declare that they have no competing interests. The funding organizations, in particular the USAID, the Bavarian Health and Food Safety Authority and Norad had no influence on the research process or content of this manuscript.Author details 1 Institute for Medical Information Processing, Biometry and Epidemiol-