key: cord-0824738-yekw4jk4
authors: Kan, Kaying; Jörg, Frederike; Lokkerbol, Joran; Mihalopoulos, Cathrine; Buskens, Erik; Schoevers, Robert A.; Feenstra, Talitha L.
title: More than cost‐effectiveness? Applying a second‐stage filter to improve policy decision making
date: 2021-06-01
journal: Health Expect
DOI: 10.1111/hex.13277
sha: d92fe250e47f0b002a333e2160811eda1dc2a96a
doc_id: 824738
cord_uid: yekw4jk4

BACKGROUND: Apart from cost‐effectiveness, considerations like equity and acceptability may affect health‐care priority setting. Preferably, priority setting combines evidence evaluation with an appraisal procedure, to elicit and weigh these considerations. OBJECTIVE: To demonstrate a structured approach for eliciting and evaluating a broad range of assessment criteria, including key stakeholders’ values, aiming to support decision makers in priority setting. METHODS: For a set of cost‐effective substitute interventions for depression care, the appraisal criteria were adopted from the Australian Assessing Cost‐Effectiveness initiative. All substitute interventions were assessed in an appraisal, using focus group discussions and semi‐structured interviews conducted among key stakeholders. RESULTS: Appraisal of the substitute cost‐effective interventions yielded an overview of considerations and an overall recommendation for decision makers. Two out of the thirteen pairs were deemed acceptable and realistic, that is investment in therapist‐guided and Internet‐based cognitive behavioural therapy instead of cognitive behavioural therapy in mild depression, and investment in combination therapy rather than individual psychotherapy in severe depression. In the remaining substitution pairs, substantive issues affected acceptability. The key issues identified were as follows: workforce capacity, lack of stakeholder support and the need for change in clinicians’ attitude. CONCLUSIONS: Systematic identification of stakeholders’ considerations allows decision makers to prioritize among cost‐effective policy options. Moreover, this approach entails an explicit and transparent priority‐setting procedure and provides insights into the intended and unintended consequences of using a certain health technology. PATIENT CONTRIBUTION: Patients were involved in the conduct of the study for instance, by sharing their values regarding considerations relevant for priority setting.

Countries have limited public resources to invest in health care.

Technological innovations and resource constraints continuously challenge health-care priority setting. 1 Indeed, maintaining comprehensive, high-quality, sustainable and affordable health-care packages entails difficult choices. Accordingly, numerous frameworks have been proposed to guide health-care decision making. [2] [3] [4] [5] [6] Most developed countries currently apply formal health technology assessments (HTAs), notably for pharmaceuticals, to substantiate reimbursement decisions. Generally, HTAs follow distinct phases: problem definition during a scoping phase, an evaluation according to different assessment criteria during an assessment phase and an appraisal of all available information by an independent multidisciplinary committee to provide policy recommendations during an appraisal phase. 7, 8 This organizational structure reflects growing recognition that the evaluation of evidence and public engagement techniques should be incorporated into priority-setting approaches. 9 HTA can be strengthened by a systematic approach to include robust evidence about patients' perspectives and by ensuring effective engagement of patients in the entire HTA/appraisal process to create a fair deliberative process. 10 Several forms of public or patient engagement (e.g. telephone surveys, questionnaires or public meetings) can occur at various levels of the HTA process. 11 For example in the Netherlands, the assessment criteria are set and followed by stakeholder consultation rounds, where medical experts, patients, professional associations or other relevant health-care stakeholders may be consulted during the appraisal. In the UK, the National Institute for Health and Care Excellence (NICE) can consult a citizen council to elicit public perspectives on overarching moral and ethical issues that NICE should consider when providing guidance (e.g. societal values to be considered in decisions about trade-offs between equity and efficiency). 12

The degree of public engagement (deliberative or nondeliberative participation) differs by country, 13 and the extent to which citizens' or patients' inputs influence the final decision remains unclear. 11, 14 Involving experts such as clinicians and health providers may help to provide important insights into various domains and the context in which technologies are used. However, to improve patient outcomes, and to take into consideration the needs of the group that is affected most by decisions regarding health technologies, requires the involvement of patients.

Therefore, a deliberative decision-making process, in which experts and clinicians as well as patients are systematically involved, may be the way forward. 11, 13 Apart from the regulatory framework, guiding principles or criteria applied within this framework are critical. A World Health Organization survey found that the majority of member states reported using a formal HTA process to inform coverage decisions. 14 The main criteria applied in HTA were safety, clinical effectiveness and economic and budgetary considerations. Acceptability to patients and health-care organizations, equity and ethical issues, and feasibility considerations rarely receive systematic attention. 14 Moreover, the findings of HTA-performing organization(s) are considered advisory rather than mandatory for policy decisions. A comparison of criteria applied within international HTA frameworks reveals that some criteria are perceived to be important across systems, but there is no consensus on a universal set of core criteria to inform priority setting. 15 The importance of incorporating a broad range of criteria other than effectiveness and cost-effectiveness in the decision-making process is twofold. First, it is rare that all consequences and costs can be included within technical cost-effectiveness calculations (e.g. informal caregiver impacts are often not considered or valued).

Furthermore, there are 'due-process' considerations in decision making that do not feature explicitly in estimations of incremental cost-effectiveness ratios. For example, parenting interventions designed to prevent anxiety disorders in children appear to provide good value for money. 16 However, several key stakeholders have highlighted issues pertaining to this intervention, including community concerns associated with the stigmatization of positively workforce capacity, lack of stakeholder support and the need for change in clinicians' attitude.

Conclusions: Systematic identification of stakeholders' considerations allows decision makers to prioritize among cost-effective policy options. Moreover, this approach entails an explicit and transparent priority-setting procedure and provides insights into the intended and unintended consequences of using a certain health technology.

Patient contribution: Patients were involved in the conduct of the study for instance, by sharing their values regarding considerations relevant for priority setting.

cost-effectiveness analysis, decision making, health technology assessment, major depressive disorder, patient participation, priority setting screened preschool children and parents' reluctance to participate in such interventions. 16 To date, while many studies recommend incorporating additional criteria and perspectives in health-care priority setting, few studies have actually demonstrated how such an approach could be implemented. Importantly, methods for weighing the opinions of different stakeholders, incorporating these opinions and deciding what criteria to use remain unclear. 17 Thus, the aim of this study was to demonstrate a structured methodology for eliciting and weighing additional criteria jointly with the results of costeffectiveness analyses by systematically involving diverse stakeholders in a different way than having a deliberative commission.

We used the principles guiding the 'second-stage filter' approach, derived from the Assessing Cost-Effectiveness (ACE) prioritysetting approach, 18 and built onto a set of cost-effectiveness analyses regarding depression treatment, undertaken by Lokkerbol et al 19 in the Netherlands. We began our qualitative analysis of important criteria, other than cost-effectiveness, with an assessment of the core criteria commonly used in existing ACE studies. 20, 21 Subsequently, we investigated other considerations relevant to the local Dutch mental health-care context for inclusion in the 'second-stage filter analysis'. We have adopted a systematic approach using focus group discussions (FGDs) and semi-structured interviews to elicit information from key stakeholders: health-care professionals, patients, and health provider and health insurer representatives.

We used a case study in depression care to illustrate the application of a structured appraisal methodology using preselected criteria and eliciting additional criteria. Qualitative research techniques were used with key stakeholders to elicit and evaluate criteria other than cost-effectiveness for a list of potential cost-effective substitute interventions.

Lokkerbol et al developed a health economic substitution algorithm to identify pairs of treatment interventions from among those currently in use that could result in a more cost-effective health-care system for patients with major depressive disorders. 19 Intervention pairs were identified via health-care substitution, by (partly) investing in a more cost-effective intervention and simultaneously disinvesting in a less cost-effective intervention. The cost savings from disinvestment would cover the investment in more cost-effective interventions. Comparable economic evaluation approaches that jointly consider investment and disinvestment decisions are called the 'step in the right direction approach', and is also used in programme budgeting and marginal analysis. 22, 23 We applied the economic analysis described by Lokkerbol et al, 19 using regional estimates for the incidence and prevalence of depression in Friesland, a province in the Northern Netherlands. Estimates were based on NEMESIS-2, 24 and we used updated intervention costs in accordance with the latest Standard of Care for depressive disorders. 25 

The ACE priority-setting approach was developed in Australia and has been used extensively to support health-care policy in areas not covered by formal HTA. This structured approach is aimed at reducing methodological inconsistencies across economic evaluations.

It explicitly considers both formal cost-effectiveness analyses for reducing methodological confounding and 'due-process' decisionmaking considerations, largely obtained via a Steering Committee of stakeholders. Legitimacy is achieved through explicit discussions of other criteria important to decision making-commonly referred to as the 'second-stage filter' criteria.

Previous ACE studies have identified the following key criteria essential for decision making: equity and ethical issues, acceptability to key stakeholders, strength of evidence, feasibility considerations, and other important beneficial or harmful effects not captured in the technical analysis. 18 Importantly, these criteria can change according to the requirements of each decision-making context. We used these second-stage filter criteria in our study as a starting point to elicit information from key stakeholders for the list of substitute interventions derived from the model-based cost-effectiveness analysis. 19 

FGDs and semi-structured interviews were conducted to elicit information on criteria from key stakeholders. While focusing on the above-mentioned criteria, the interview guide also allowed for the inclusion of other relevant issues (for details, see Supplementary Material A). We used a phenomenological approach to obtain stakeholders' views and values on important priority-setting considerations in depression care. This approach elicits individuals' experiences of a certain phenomenon. 26 It simultaneously attempts to set aside preconceived assumptions about experiences of a particular situation.

Key stakeholder participants were patients with depression or a history of depression, health-care professionals who treat depression, the director of the largest regional specialist mental health-care organization and a medical advisor of the health insurer with the largest market share (approximately 61%) in the region.

FGDs with patients stimulated exchanges of knowledge and experiences and their perceptions regarding potential substitute interventions. We conducted semi-structured interviews with the remaining stakeholders. Because of time constraints and logistical issues, five of the ten health-care professionals preferred to fill in a questionnaire covering the same topics.

Patients and health-care professionals were selected through purposive sampling. Patients were eligible to participate in this study if they currently or previously suffered a depressive disorder and underwent several treatments; were willing to pro- The largest specialist mental health-care organization and health insurer were selected through convenience sampling. The health insurer perspective was represented by the medical advisor in mental health. The director of the mental health-care organization represented the provider perspective. All participants were contacted by telephone or email for an eligibility check. Three patients dropped out of the study due to personal circumstances (a broken toe, a funeral and a hospital admission, respectively) and one interview with a clinician was unusable due to faulty audio equipment.

Data collection took place between December 2017 and August 2018. All interviews were audio recorded and transcribed verbatim. Field notes were taken during the FGDs and semi-structured interviews.

Data were analysed using ATLAS.ti version 8.2, a qualitative analysis software. One researcher coded the data (K.K.), and two researchers (F.J. and T.F.) participated in peer-debriefing to improve the credibility and validity of the results. Thematic content analysis was performed on the data using the one-sheet-of-paper method. 27 Patients and health-care professionals provided feedback on the findings to verify the accuracy of the interpretations of the transcripts. Themes were derived from the data but were mostly driven by the interview guide topics.

We constructed a summary table based on our analysis of the interview data. Subsequently, conclusions on issues identified for each criterion, displayed by coloured cells, were converted into appropriate recommendations on each substituted intervention pair for decision makers. Three researchers independently judged the importance of the identified issues. The researchers discussed their differences, eventually reaching consensus. Quotes by different stakeholders were used to illustrate the findings. Table 1 presents the intervention substitution pairs using the health economic substitution algorithm. Substitution of an intervention by an alternative can potentially improve cost-effectiveness within the mental health-care system. A detailed description of non-standard interventions is given in Supplementary Material C. Table 1 ). Next, we present the salient findings, including judgements, for each of the substitution pairs. Table 2 presents a summary of the findings for substitution pairs 1 and 2: investment in therapist-guided Internet-based cognitive behavioural treatment (iCBT) and disinvestment in individual cognitive behavioural therapy (CBT) for (1) mild and (2) moderate depression. Issues identified for each criterion are described in more detail below, and the findings are illustrated with respondents' quotations (Table 3) . Similar tables for all remaining substitution pairs (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) are provided in Supplementary Material D.

Most of the health-care professionals who were unfamiliar with iCBT believed that it could be meaningful for patients who are not severely depressed. They indicated that it should be at least as effective as regular face-to-face CBT. Health-care professionals already familiar with iCBT reported seeing good outcomes and did not have any doubts about the effectiveness in clinical practice (Table 3, quote 1). One health-care professional remarked that some of her colleagues were more reluctant to apply iCBT because they lacked information regarding its effectiveness in clinical practice. The representatives of the mental health-care organization and health insurer felt that iCBT was worth investing in, and both stakeholders endorsed provision of this intervention and evaluations of its effects on patients with mild depression. By contrast, several stakeholders were reluctant to offer therapist-guided iCBT to patients with moderate depression because they anticipated limited treatment adherence within this group.

All the stakeholders indicated that the substitute intervention would be justified for the majority of patients with mild depression. They all agreed that a minority of patients with mild depression might not prefer this type of intervention for reasons such as (computer) illiteracy or low socio-economic status ( Table 3 , quote 2). Most stakeholders considered iCBT to be more appropriate for younger patients, as most of them are familiar with digital devices. One patient did not recommend it for adolescents, especially those who needed social connections. Conversely, others believed that iCBT provides a solution for young people who feel ashamed about undergoing psychological treatment. It could reduce no-show in patients who require therapy but whose time is occupied by work/school. Moreover, it could overcome barriers for a certain group of patients, as some patients find it difficult to tell their stories face-to-face.

Some patients and health-care professionals argued that iCBT would not be equivalent to existing therapies for moderately depressed patients because some patients find structure in daily routine, discipline, stability and independent functioning challenging. In such cases, expectations of motivating oneself to participate in an Internet-based treatment may not be realistic.

Other important equity and equality considerations included provision of treatment in patients' own environments, which saves travel time and costs. The treatment also elicits patients' insights and the possibility of re-reading the information in patients' own time and pace. Finally, it may increase the productivity of health-care professionals, reducing waiting lists.

None of the stakeholders expected major implementation issues for patients with either mild or moderate depression. Some health-care professionals and the director of the mental health-care provider expected that iCBT would require more efficient staff deployment (Table 3, 

Stakeholders prioritized investments in iCBT over individual CBT for mild depression. One clinician stated that task-oriented interventions like iCBT could even be more effective than CBT for some patients, as treatments are structured and entail regular online TA B L E 2 Priority-setting considerations based on different stakeholders' perspectives on investment in therapist-guided, Internetbased cognitive behavioural treatment (iCBT) and disinvestment in individual cognitive behavioural therapy (CBT) for mild and moderate depression Level of evidence -Clinicians doubt treatment effects in routine practice; not all clinicians have experience in providing Internet-based treatments. Clinicians with experience in providing Internet-based treatments have no doubts -Evidence of treatment effectiveness is required for mild and moderate depression and for primary care services versus specialized care services Equity and equality considerations Potential causes of increased inequalities/inequities are: -An ageing population, (computer) illiteracy, low socio-economic status of patients, patients with sight and hearing deficiencies, psychiatric comorbidities, and intellectual disabilities -Treatment effects are more obvious in younger patients. However, for adolescents, in particular, connecting with others could be important in a digital world -Patients are not open to this form of therapy -Treatment is more suitable for patients with mild depression, but its suitability for patients with moderate depression is questionable because of their decreased treatment adherence and difficulties putting things into perspective Potential causes of decreased inequalities/inequities are: -Patients' insights -Individual clinicians' increased caseloads and reductions in waiting lists -Less travel time and costs -Insights of patients who experience barriers to therapy sessions and/or patients who have difficulty expressing themselves -Treatment occurs in patients' own environments, and they have opportunities to re-read the information at leisure

Positive: -In organizations that already provide this form of treatment, clinicians are qualified, and no issues are anticipated -Implementation is less intensive for clinicians; whereas a CBT session requires an hour, a therapist-guided Internet-based cognitive behavioural treatment requires 30 minutes Neutral: -More efficient deployment of staff is required -The recommendation is to start with a pilot initiative to gain some experience Issues: -Training of qualified personnel is required regarding to e-health guidance and a different form of contact -The application should be interactive, user-friendly and catchy -Mental health-care procurement is dependent on the resources of care organizations -Training is required when intervention is provided as a component of primary care services

Positive: -Acceptable in its blended form in combination with face-to-face therapy -Could be more effective: working method is more task-oriented -Good experiences -Structured moments of contact via the Internet, during which clinicians can provide supportive and stimulating feedback Neutral: -Patients' preferences remain important -Suitable for moderate depression along a continuum (if not too severe); CBT treatment is appropriate for more severe cases Issues: -The extent of clinicians' openness and willingness to adapt to this method of interacting with patients -Clinicians' handling of patients' self-responsibility -Suitability and quality of the e-health material (adaptation for different target populations, content tailored for specific patients, and based on CBT principles) and ease of use for both clinicians and patients -Challenges of convincing patients and clinicians to accept the innovative intervention -Requires patients' trust in clinicians (as a precondition) -Requires patients to work independently during treatment, demonstrating self-discipline and responsibility -Not suitable or questionable suitability for patients with moderate depression. Suggestions: begin with medication and then continue with therapist-guided Internet-based cognitive behavioural treatment. Requires patient stability and structure in daily routine. Not suitable for patients with suicidal thoughts. More intensive initial treatment, more frequent sessions, or in conjunction with medication -CBT remains important for treating moderate depression -Clinicians' unfamiliarity with this type of treatment -Possible resistance from the older generation of clinicians; doubts about treatment effectiveness 

Both patients and health-care professionals felt that patients' trust in therapy and the therapeutic relationship were crucial for good treatment outcomes. For patients without a social network or structured routine, travelling to treatment centres could have benefits, as it promotes some scheduled activities. However, providing more iCBT could lead to a reduced need for antidepressant prescriptions and CBT treatment, thereby saving costs and shortening waiting lists. Finally, patients and health-care professionals felt that iCBT could increase patient involvement in therapy, leading to patients regaining control over themselves more rapidly, with expected consequences for recovery times (Table 3, quote 7) .

In sum, for treating mild depression, investment in therapistguided iCBT and disinvestment in individual CBT seems to be a promising potential substitute intervention pair with no major issues anticipated. However, for treating moderate depression some concerns remain. Practice assistant), training of staff is required to increase feasibility of implementation. The main acceptability issues stemmed from the clinicians' work culture, including reluctance to pass on patients to their colleagues, who might be better equipped to provide necessary interventions, or to transfer responsibilities to their patients.

Finally, independent of a specific substitute intervention, personalized treatment and shared decision making remain essential, and were prioritized over and above economic considerations by all of the stakeholders. Further, stakeholders noted that patients' social networks or contexts could significantly affect treatment outcomes.

Economic evaluations do not usually cover this aspect. From the perspectives of the mental health-care provider and health insurer, the long-term cost-effectiveness of interventions and prevention of future recurrences were deemed important.

The aim of this study was to demonstrate how the structured ap- 

Recent trends like value-based health care, but also the longstanding tradition of HTA stress that cost-effectiveness is not the only relevant aspect for priority-setting decisions. Current policy processes address these broader perspectives mostly in the appraisal phase, using deliberative panels. The current study piloted an alternative approach that adds to deliberative panels by offering a structured approach resulting in an overview of stakeholders' views and that has the potential to increase the extent to which their inputs influence final decisions. 

Improving the health-care system through interventions that meet the efficiency criterion may have limitations because they do not include additional considerations relevant to priority setting. Access to diverse stakeholders' views and values regarding these criteria, via a systematic embedded qualitative approach, enables decision makers to make better-informed decisions and appropriate judgments when setting health-care priorities. It also informs decision makers on issues that could impede successful adoption of the intervention, allowing them to tackle these issues. Furthermore, addressing such criteria in an open and explicit manner increases the transparency of the priority-setting process.

The authors would like to thank the interviewees of the study.

All authors have no conflict of interest to declare.

Kan, Jörg, Feenstra, Lokkerbol and Mihalopoulos conceptualized and designed the study. Kan, Lokkerbol, Feenstra and Jörg analysed the data. All authors interpreted the data. Kan, Jörg, Feenstra and Mihalopoulos drafted the manuscript. All authors revised the manuscript and finally approved the version to be published.

We are legally and ethically not allowed to publicly post our data set. Participants provided informed consent for the data collection and the use of audio recorded anonymized data for this particular study only.

Kaying Kan https://orcid.org/0000-0001-5337-3001

Priority setting in health care: Lessons from the experiences of eight countries

Overcoming barriers to priority setting using interdisciplinary methods

Decisionmaking frameworks and considerations for informing coverage decisions for healthcare interventions: A critical interpretive synthesis

Health technology assessment: A comprehensive framework for evidence-based recommendations in Ontario

To each according to need: A communitybased approach to allocating health care resources

Priority setting processes for healthcare

National Institute for Health and Care Excellence

Beoordeling stand van de wetenschap en praktijk

Key principles for the improved conduct of health technology assessments for resource allocation decisions

Patients' perspectives in health technology assessment: A route to robust evidence and fair deliberation

s5kjg 27kof_j9vvi j5epm j1ey0/ vi3k9 g5171 yv/f=/blg14 649.pdf 12. National Institute for Health and Care Excellence

Survey on the involvement of consumers in health technology assessment programs

Gobal Survey on Health Technology Assessment by National Authorities

Priority setting in health care: Learning from international experience

The population costeffectiveness of a parenting intervention designed to prevent anxiety disorders in children

Patient involvement in HTA: What added value? Pharmaceuticals

Priority setting in health: Origins, description and application of the australian assessing cost-effectiveness initiative

Rationalization of innovation: The role of health-economic evaluation in improving the efficiency of mental health care (dissertation)

The economic analysis of prevention in mental health programs

Assessing cost-effectiveness of obesity prevention policies in australia 2018 (ACE-obesity policy)

Cost effectiveness/utility analyses. do current decision rules lead us to where we want to be?

Program, priorities and budgets

First-incidence of DSM-IV mood, anxiety and substance use disorders and its determinants: Results from the Netherlands mental health survey and incidence study-2

Husserl's phenomenology

Making sense of qualitative data analysis: An introduction with illustrations from DIPEx (personal experiences of health and illness)

Adherence to internet-based and face-to-face cognitive behavioural therapy for depression: A meta-analysis

Comparing telehealth-based and clinic-based group cognitive behavioral therapy for adults with depression and anxiety: A pilot study

Standardized web-based cognitive behavioural therapy of mild to moderate depression: A randomized controlled trial with a long-term follow-up

Efficacy and effectiveness of online cognitive behavioral treatment: A decade of interapy research

COVID-19 from a psychiatry perspective: Meeting the challenges

Perspectives on the COVID-19 pandemic and individuals with serious mental illness

Real-world' health care priority setting using explicit decision criteria: A systematic review of the literature

Stakeholder views on criteria and processes for priority setting in norway: A qualitative study

From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking

Pakketadvies in de praktijk: wikken en wegen voor een rechtvaardig pakket. Diemen, the Netherlands: National Health Care Institute

Why patients should be involved in health technology assessment

Determining social values for resource allocation decision-making in cancer care: A canadian experiment

More than cost-effectiveness? Applying a second-stage filter to improve policy decision making