key: cord-0029548-jqijsyg5 authors: Almajed, Suaad; Alotaibi, Nora; Zulfiqar, Sana; Dhuhaibawi, Zahraa; O’Rourke, Niall; Gaule, Richard; Byrne, Caoimhe; Barry, Aaron M.; Keeley, Dylan; O’Mahony, James F. title: Cost-effectiveness evidence on approved cancer drugs in Ireland: the limits of data availability and implications for public accountability date: 2021-08-30 journal: Eur J Health Econ DOI: 10.1007/s10198-021-01365-2 sha: fd4256edf14b362b9e3b1297fd31d92cbbc0ce49 doc_id: 29548 cord_uid: jqijsyg5 BACKGROUND: We surveyed evidence published by Ireland’s National Centre for Pharmacoeconomics (NCPE) on the cost-effectiveness of cancer drugs approved for funding within the Irish public healthcare system. The purpose is threefold: to assess the completeness and clarity of publicly available cost-effectiveness data of such therapies; to provide summary estimates of that data; to consider the implications of constraints on data availability for accountability regarding healthcare resource allocation. METHODS: The National Cancer Control Programme lists 91 drug-indication pairs approved between June 2012 and July 2020. Records were retrieved from the NCPE website for each drug-indication pair, including, where available, health technology assessment (HTA) summary reports. We assessed what cost-effectiveness data regarding approved interventions is available, aggregated it and considered the consequences of reporting constraints. RESULTS: Among the 91 drug-indication pairs 61 were reimbursed following full HTA, 22 after a rapid review process and 8 have no corresponding NCPE record. Of the 61 where an HTA report was available, 41 presented costs and quality-adjusted life-year (QALY) estimates of the interventions compared. Cost estimates and corresponding incremental cost-effectiveness ratios (ICERs) are based on prices on application for reimbursement. Reimbursed prices are not published. Aggregating over the drug-indication pairs for which data is available, we find a mean incremental health gain of 0.85 QALY and an aggregate ICER of €100,295/QALY, which exceeds Ireland’s cost-effectiveness threshold of €45,000/QALY. CONCLUSION: Reimbursement applications by pharmaceutical manufacturers for cancer drugs typically exceed Ireland’s cost-effectiveness threshold, often by a considerable margin. On aggregate, the additional total net cost of new drugs relative to current treatments needs to be more than halved for the prices sought on application to be justified for reimbursement. Commercial confidentiality regarding prices and cost-effectiveness upon reimbursement compromises accountability regarding the fair and efficient allocation of scarce healthcare resources. This study considers Ireland's health technology assessment (HTA) framework for the assessment of cancer drugs and surveys the completeness of the cost-effectiveness evidence made public. It uses this assessment of the constraints on publicly available data to motivate a discussion of the implications for accountability regarding the allocation of scarce healthcare resources. Cancer drugs are a particular class of intervention that present persistent challenges to achieving value for money for many healthcare systems [1] . Many countries conduct HTAs to determine if drugs provide sufficient value for money to merit reimbursement. HTAs combine systematic reviews, trial data and modelling to estimate the clinical and cost-effectiveness of candidate interventions. Policy makers consider these estimates alongside other considerations when forming their recommendations, such as budget impact and ethical concerns regarding access to care. Ireland's HTA framework for pharmaceuticals has been described and examined previously [2] [3] [4] [5] [6] [7] [8] . Ireland's taxfunded public health system is managed by the Health Service Executive (HSE). The provision of pharmaceuticals by the HSE is subject to the 2013 Health Act, Schedule 3, Part 3 of which obliges the HSE to consider the costeffectiveness and budget impact of candidate interventions alongside seven other points of consideration [9] . This is manifest in the requirement of manufacturers to submit a pharmacoeconomic evaluation (PE) to the HSE's Corporate Pharmaceutical Unit when seeking approval for the reimbursement of new therapies. The health economic decision criteria regarding new drugs in Ireland are set as part of an ongoing series of agreements between the pharmaceutical industry, the HSE and government, the most recent of which dates from 2016 [10] . An appendix to the agreement details what level of decision maker within the HSE can authorise approval of a new drug. This differs by levels of budget impact and cost-effectiveness, with the latter expressed at two thresholds of €20,000/quality-adjusted life-year (QALY) and €45,000/QALY. While the agreement does not articulate how these decision thresholds relate to the task of balancing the cost of new interventions with their opportunity cost, the €20,000/QALY and €45,000/QALY limits are widely interpreted as Ireland's prevailing cost-effectiveness thresholds [7, [11] [12] [13] . Clearly only the upper threshold will ultimately be relevant if decisions can be escalated to a higher level within the HSE. The €45,000/QALY threshold is not binding in that drugs exceeding it are not necessarily rejected, but can be put forward for further consideration, which may include additional confidential price negotiations. The National Centre for Pharmacoeconomics (NCPE) is the independent expert review body commissioned by the HSE to evaluate HTA submissions [14] . The NCPE conducts two tiers of analyses. A rapid review (RR) is a preliminary analysis of an information summary on the candidate technology presented by manufacturers containing clinical evidence and economic considerations including the treatment cost and anticipated budget impact [15] [16] [17] . There are several possible recommendations from the NCPE following a RR: that the intervention be forwarded for further consideration without conducting an HTA; a full HTA should be conducted due to questions regarding costs or cost-effectiveness; an HTA should not be conducted and the intervention should not be considered for reimbursement at the submitted price; finally, that a full HTA is not recommended until further data on either or both efficacy and safety is provided [18] . The criteria for a full HTA are described on the NCPE's website [14] . They broadly correspond to an anticipated large budget impact, questionable clinical efficacy or potentially poor value for money. On completing a full HTA the NCPE issues the Final Appraisal Report documenting their findings and a reimbursement recommendation to the HSE Drugs Group and publishes a summary of that report (henceforth referred to as an HTA summary) on the NCPE website [4] . The Final Appraisal Report is not made public. The NCPE can make four different post-HTA reimbursement recommendations: that the drug be considered for reimbursement at the assessed price; a conditional recommendation that the drug be considered for reimbursement if the price can be reduced in subsequent negotiations; a conditional recommendation that the drug not be considered for reimbursement unless the price can be reduced in subsequent negotiations; finally, simply to recommend against consideration for reimbursement [18] . Importantly, the second and third conditional recommendations explicitly refer to the possibility for postsubmission price negotiations to achieve a better price. In all cases the NCPE also state that their recommendations should be considered within the criteria for reimbursement stated in the 2013 Health Act. The National Cancer Control Programme (NCCP) is a directorate within the HSE responsible for population cancer control in Ireland [19] . Part of this function is to manage and deliver cancer care in collaboration with care providers. In the case of cancer drugs, the NCPE appraisal report will also be submitted to the NCCP's Technology Review Committee. This committee considers the NCPE's PE assessment and can issue one of three recommendations: rejection; adoption; or adoption subject to a price reduction [20] . This recommendation is issued to the NCCP's Director, who then brings it to the HSE Drugs Group for consideration. If a cancer drug is approved by the HSE Drugs Group, it is then added to the list of approved cancer drugs maintained by the NCCP [21] . Meeting minutes from both the Technology Review Committee and the Drugs Group are published [22, 23] . The NCPE HTA summaries publish costs and ICERs on the basis of the list prices on application. If there are post-HTA price negotiations prior to approval, then these costs, ICERs and list prices will not be representative of the agreed prices on adoption. Although the meeting minutes of both the Technology Review Committee and the Drugs Group are published, the pricing details of approved drugs are redacted. This means there is no publicly available source for the agreed prices, associated costs and ICERs of cancer drugs on reimbursement. Approved cancer drugs are funded under three schemes, the first of which is the community drug scheme of the Primary Care Reimbursement Services (PCRS), which funds self-administered drugs for community-resident patients. The second is the Oncology Drug Management System (ODMS) for high-cost drugs administered in hospitals. The final source is individual hospital budgets that sponsor drugs administered in hospitals. Previous studies have assessed aspects of Ireland's HTA appraisal process, including the choice of appraisal pathways, appraisal times and analyses of particular classes of therapies [3, 5, 7, 24] . The objective of this study is to survey the available evidence on the cost-effectiveness of publicly funded cancer drugs in Ireland and to assess the clarity, consistency and completeness of that data. A secondary objective is to use the available data to derive aggregate cost-effectiveness estimates for approved cancer therapies. The third objective is to use this appraisal of available costeffectiveness data to inform a discussion on the implications of commercial confidentiality regarding reimbursed prices for accountability regarding the allocation of scarce healthcare funds. No previous study has provided such an analysis to our knowledge. We compiled a data set by combining publicly available sources on approved cancer drugs in Ireland. The primary source was the list of all approved treatments maintained by the NCCP [21] . As each drug can have multiple clinical indications we describe each separate drug and indication combination as a drug-indication pair. The NCCP's list of approved drugs names and dates all drug-indication pairs approved since May 2012, details under which funding scheme reimbursement was made and provides links to NCCP regimen summary documents. The list was assessed in February 2020 and the search updated in July 2020. We then consulted the NCPE website to find information for the same drug-indication pairs that the NCCP list as approved [25] . The website details what applications have been made for which drugs and for what indications, the dates of each initial application and reports if the drug-indication pair has been subject to RR alone or has undergone full HTA. The website also provides HTA summaries for those pairs subject to full assessment. Recently the NCPE has offered both plain English and technical HTA summary documents. Where both are available we refer to the technical summary. Nine reviewers recovered data for the drug-indication pairs (RG, CB, AB, DK, NOR, SA, SZ, NA, ZD). Each reviewer was responsible for assessing a portion of the NCCP's list and cross checking the data extraction of another reviewer. Each drug-indication pair was categorised according to mechanism of action (MOA) and cancer types according to the International Classification of Diseases (ICD). Cancers were classified using the ICD 10 codes reported in the associated NCCP regimen summaries. Similarly, the drugs' MOA were classified using the World Health Organisation (WHO) Anatomical Therapeutic Chemical (ATC) codes from the regimen summaries and cross-referenced to the ATC listings maintained by the WHO [26] . Some of the cancer and drug type categories were merged where numbers were small. The NCCP regimen summaries link to European Medicines Agency (EMA) product characteristics descriptions for each therapy, which were used to identify the market authorisation holder for each drug. The orphan status for each drug-indication pair was determined using the European Commission's register of medicinal products and cross-referenced against the Orphanet database [27, 28] . For those drug-indication pairs in which a full HTA was conducted, the reviewers examined the NCPE HTA summaries. In some cases a single drug approval record on the NCCP website corresponds to multiple indications. We considered each pair separately unless the relevant NCPE HTA summary aggregated the cost-effectiveness of the indications together. In these cases we considered the multiple indications to correspond with a single drug-indication pair. Similarly, in cases in which additional subgroup analyses were presented alongside the primary patient group, we only consider the primary patient group. For those cases that are disaggregated between indications the NCPE HTA summaries typically report incremental cost-effectiveness ratios (ICERs) for the different indications separately but the report budget impact for the indications combined. Accordingly, our results are presented in the same way. We extracted and reported information from the HTA summaries regarding the comparisons made between treatments, ICERs, budget impact and the NCPE recommendation. Note that the ICERs and budget impact from the HTA summaries relate to list prices, not final reimbursed prices. We retrieved the identity of the applicant firm from the summaries, which is not always synonymous with the market authorisation holder reported by the EMA. We recorded the date the HTA summary was published, the summary length and if the NCPE website reported post-assessment price negotiations were conducted for the drug-indication pairs. We also appraised the time taken to reimbursement as the difference in time in months between the first mention of the drug-indication pair on the NCPE website and the date of listing by the NCCP as an approved drug. Note this total reimbursement time not only includes the time taken for the NCPE to appraise the intervention, but also includes any additional time taken to receive clarifications or amendments from manufacturers to submissions and for any price negotiations subsequent to the NCPE's appraisal. Where health effects and ICERs were reported for both QALYs and life years gained, we recorded the outcomes for QALYs. Where summaries reported both the manufacturer-estimated outcomes and outcomes based on what the NCPE stated was the most plausible set of assumptions, we recorded the latter. Where the NCPE summaries reported ICERs based on both deterministic and probabilistic analysis, we recorded the latter. In some summaries, the basecase ICERs were explicitly reported, while in others, several ICERs were given for the intervention relative to various comparators. To identify a single ratio, we recorded the ICER based on a comparison to the current standard of care. Some summaries did not report what the current standard of care is. In these cases, we recorded the highest of the reported ratios as this corresponds with the ICER on the efficient frontier. We recorded on what basis the budget impact was recorded within the HTA summaries. This included gross and net budget impact over either 1 or 5 years. In cases in which a budget impact is reported as a range, we recorded the midpoint of that range. We assessed the costs of the interventions reported in the HTA summaries. Standard CEA practice is to base reimbursement decisions on the incremental differences between discounted treatment costs of alternative interventions. Under standard methods such costs are the total treatment costs net of any resulting changes in related care costs such as hospitalisation or treatment reoccurrence. Irish HTA guidelines recommend that analyses are conducted with a lifetime time horizon and are assessed using the health payer perspective [13] . While the NCPE HTA summaries did not typically explicitly state what the cost estimates relate to, we assumed the reported costs accord with standard CEA practice in Ireland. We recorded the incremental costs and QALYs of the intervention of interest versus the relevant base case comparator where reported. In some instances, costs and QALYs were reported for some comparisons but not others. If the costs and QALYs for what appears the primary comparison of interest could be inferred from the reported figures from other comparisons, we used these. In other cases, the incremental costs and QALYs corresponding with the NCPE's preferred parameter set were not reported. In these cases, we recorded the incremental costs and QALYs reported for the most relevant scenario for which outcomes were reported. In each drug-indication pair, the recorded ICER, costs and QALYs all correspond to the same incremental comparison. We conducted a descriptive analysis of the compiled data to demonstrate the completeness of the publicly available evidence and to provide an overview of the relationships between the variables recorded. We compiled the unweighted arithmetic mean of the costs, QALYs, budget impact and ICERs. We also computed a weighted arithmetic mean of the incremental costs and QALYs weighted by the reported 5-year gross budget impact and calculated a weighted aggregate ICER from this. We use the analysis of the published data and aggregate cost-effectiveness estimates presented in the following results section to inform a discussion regarding the implications of data availability for considerations of accountability regarding healthcare resource allocation. Appendix Tables 4, 5, 6, 7, 8 provide detailed records of each drug-indication pair assessed. Table 4 records the drug name, approved indication, approval date and the internet addresses of both the NCCP regimen listing and record on the NCPE website. Table 5 details the drugs' proprietary names, market authorisation holders, ICD10 codes and mechanism of action. It also records if the indication includes metastatic disease and the drugs' current orphan status. In two instances it is known that the applicant firm is not synonymous with the market authorisation holder. The applicants for dabrafenib and ponatinib are GlaxoSmith-Kline and ARIAD Pharmaceuticals, respectively. Table 6 details aspects of the appraisal process, including at what stage in the NCPE's appraisal was the drug recommended for consideration for reimbursement and under what funding pathway it was reimbursed, the total time taken to from application to reimbursement, the length of the HTA summary (if applicable) and whether the NCPE website reports if reimbursement was made following post-assessment price negotiations or not. Table 7 details the information extracted from HTA summaries. This includes the basecase ICER, incremental costs and QALYs, 5-year gross budget impact and records if costs and QALYs were reported for all the treatment strategies mentioned within the summary. Table 8 reports the number of approved drug-indication pairs by market authorisation holder and the associated total 5-year gross budget impact where reported within the NCPE HTA summaries. The NCCP list of approved cancer therapies contained 77 drugs, drug combinations or distinct drug dosages as approved between May 2012 and July 2020. Following the separation and merging of indications described in the methods, there are 91 drug-indication pairs in total. While most drugs only have one approved indication, the four most commonly approved drugs (including combination therapies) are nivolumab, idelalisib, ponatinib and pembrolizumab with 7, 5, 4 and 4 approved indications, respectively. The NCPE website reports reimbursement was made following postappraisal price negotiations in 65 of the 91 drug-indication pairs. NCPE HTA summaries are only published for drug-indication pairs subject to full HTA review. There were 61 pairs with published HTA summaries available. The summaries range from 3 to 8 pages long, have a mean length of approximately 6 pages and have increased in length modestly over the period assessed. We found 8 pairs for which there is no clear record of a cost-effectiveness appraisal on the NCPE website, either RR or full HTA. There was no mention of siltuximab (pair 28) on the NCPE website at all at the time of analysis. In the remaining 7 cases (pairs 18, 19, 22, 37, 44, 47 and 48) , there were records of the drugs, but not for the indications in question. While there were 61 pairs with HTA summaries, not all reported health economic outcomes. ICERs or an outcome of dominance were reported in all 61, but relevant incremental costs and QALYs were only reported in 41 summaries. There were 48 summaries that reported the 5-year gross budget impact, yielding a total of €1.2Bn. As gross budget impact does not account for possible substitutions of older therapies by new drugs in the same indication, the €1.2Bn total will exceed the net budget impact at the list prices on application. There were a further 8 studies that reported a 1-year budget impact. We assessed the sample of pairs for which HTA summaries are available to assess how the completeness of data changed over time. In particular, we assessed what proportion of the sample included any two or more of an estimate of incremental costs, incremental QALYs or a 5-year budget impact (either gross or net). Dividing the sample in two chronologically according to the date of completion of the NCPE HTA review, 52% of the sample included any two or more data items for the first half of the sample, rising to 77% for the second half of the sample. While only a crude measure, this indicates that the completeness of reporting of cost-effectiveness data within the NCPE HTA summaries has increased over time. Figure 1 shows a correlation matrix of a selection of data from this analysis compiled with a series of histograms on the diagonal. The bars in the histogram and the points in the scatter plots shaded light grey correspond to approvals following a full HTA, those in dark grey correspond to approvals without full HTA. The sample size in each plot varies as the number of available data points varies (given as n in each plot). The first histogram shows the distribution of approvals over time. The first approval listed by the NCCP is in May 2012 and the most recent in July 2020. While there is no distinct trend in approvals, clearly more have been approved annually from between 2016 and 2019 than earlier in the sample. Although the number of approvals without requiring full HTA has been few, the number increased in 2018 and 2019. Table 1 summarises at what stage of the HTA appraisal process had a drug reached prior to approval and under what funding pathway was the drug reimbursed. The table also details the time taken to reimburse in months. Of the 91 pairs, 3 were recommended for consideration at RR and subsequently approved. A further 19 were subsequently approved following a RR, but without progressing to full HTA. In these cases, a recommendation for consideration for reimbursement was not made at RR, but it appears subsequent negotiation avoided the need for a full HTA, even if recommended by the NCPE. Similarly, there were 3 and 58 recommendations at or after the full HTA stage, respectively. The three pairs recommended for consideration at HTA were aflibercept, dabrafenib and pembrolizumab as first line monotherapy for melanoma (pairs 14, 21, 34, respectively) . This indicates that only a small minority of applications are considered sufficiently cost-effective to recommend consideration for reimbursement at the full HTA stage and most required further negotiation before they could progress to subsequent approval. The overall mean time taken from application to reimbursement is 18 months. The mean reimbursement time appears shorter under RR than full HTA. Similarly, the reimbursement time for those pairs approved following post-HTA negotiations is longer than those at the other appraisal stages. The second histogram in Fig. 1 shows that most approvals are made in less than 24 months, but there are some outliers at over 4 years. The distribution of RR approvals is shown in dark grey, most of which are within 12 months. As mentioned in the methods, the total time to reimbursement assessed here is not synonymous with the time taken to appraise a given intervention, a detailed description and examination of which in an Irish context is given by Connolly et al. [3] . Somewhat over half of the approved drug-indication pairs are reimbursed under the PCRS. Only one pair (14-aflibercept) was listed for funding by individual hospital budgets. The remaining approvals were under the ODMS. There is no apparent difference in the reimbursement time between PCRS and ODMS funded drugs. Table 2 summarises the approved pairs according to their therapeutic class and mechanism of action and contains aggregate estimates of budget impact, incremental costs and QALYs and ICERs from the HTA summaries. Lymphomas, leukaemias and lung cancer are the top three disease categories by number of approvals. They count for half of all approvals between them. Of the total approved indications, 48 included metastatic disease. Regarding the therapeutic categories, protein kinase inhibitors and monoclonal antibodies are the two notably large groups, accounting for two thirds of all approvals. At the time of investigation, 11 pairs held positive orphan status. The number of approvals in the reported disease and therapeutic categories is small in many cases. Accordingly, it is important not to over-interpret the estimates disaggregated by category. The three largest disease categories in terms of gross budget impact were skin, lung and breast cancer, 1 3 which together account for almost two thirds of the total. Naturally, the total budget impact depends in part on the number of approvals, so we also present the mean budget impact per approval within each category. While the disparities in mean budget impact are smaller, the same three categories still carry the three largest budget impacts. These mean figures still reflect, in part, the anticipated patient population size within each category. The mean incremental cost provides a per-patient estimate of the discounted net incremental cost of care and, therefore, does not reflect the size of the indications. On this basis, skin, lung and breast cancer do not appear notably different from the others. Similar conclusions apply to the therapeutic categories, in that the very large total budget impact of monoclonal antibodies and protein kinase inhibitors appear largely to reflect the number of approvals as the mean budget impact and incremental costs much more aligned with the overall means. While the mean budget impact of metastatic disease is effectively the same as the overall mean, orphan treatments appear to have a lower mean budget impact. The third histogram shown in Fig. 1 shows the heavily right skewed distribution of 5-year gross budget impact, with clear mode between €5-10 M. Almost 60% of approvals have an estimated 5-year gross budget impact less than €20 M, while approximately 20% have a budget impact greater than €40 M. Table 3 reports selected findings from the analysis including aggregated outcomes. The range of incremental health gains reported is 0.10 to 2.94 QALYs, with an unweighted mean of 0.80. An unweighted mean of the reported ICERs is €141,598/QALY. The estimated weighted mean intervention cost, QALY gains and resulting aggregate ICER when weighting according to the gross 5-year budget impact are €85,164, 0.85 and €100,295/QALY, respectively. Note that the unweighted and weighted ICERs differ in part because not all HTA summaries report budget impact or costs and QALYs, so the weighted values are based on fewer studies (the unweighted mean ICER from pairs used to determine the weighted aggregate ICER is 133,843/QALY, which is less than the overall unweighted mean). The distribution of incremental health effects in Fig. 1 shows most are below 0.75 QALYs, but only very few exceed 1.5 QALYs. The distribution of ICERs shown in Fig. 1 shows few below the €45,000/QALY threshold, a cluster between 2 and 3 times Overall, the correlation plots indicate few notable relationships. There is a negative association between the QALYs gained and ICERs, but it is not pronounced. There may be some indication that the variance in appraisal duration, budget impact and ICERs are increasing over time (Table 3 ). We need to explicitly acknowledge this study's main limitation at the outset of the discussion, namely, that we only have access to the drug costs, budget impact and ICERs corresponding to the list prices on application for reimbursement, not the actual agreed prices paid following negotiation. The agreed prices will be lower than those on application in most cases, although by what margin we cannot know. Furthermore, the effective prices will also be lower due to rebates and risk sharing schemes. The following discussion describes what is useful about the currently reported cost-effectiveness estimates. We then outline some of the inconsistencies and other reporting gaps we observed in the NCPE summaries. Finally, we consider the implications of constraints on complete reporting for accountability regarding resource allocation. Although the list prices and ICERs published in the HTA summaries are not informative of actual resource allocation decisions, NCPE summary publications do still provide useful information. First, publishing evidence of ICERs breaching the threshold reveals the list prices sought by manufacturers will lead to aggregate net harm to the health system. We found that 92% of the drug indication pairs for which information was available were not cost-effective at the prices on application. This provides justification for the sometimes long negotiating periods required to agree prices. Moreover, the published ICERs give an indication of how much costs need to be moderated relative to those at list prices. The weighted aggregate ICER from our analysis of just over €100,000/QALY is more than twice the current threshold. We see that the additional total net costs of new cancer drugs at list prices need to be reduced, on aggregate, by more than half if reimbursement is not to represent an unfair and ethically questionable use of resources (assuming that the QALYs gained treating cancer are of equal value as those foregone by other patients). Indeed, that interpretation rests on the assumption the current €45,000/QALY threshold is a fair representation of the opportunity cost of other care in Ireland. Since it is arguably a substantial underestimate [29, 30] , the additional total net costs would need to be very substantially less than half than those at list prices for new cancer drugs to avoid being net damaging to the Irish health system. Another important benefit offered by the HTA summaries is that they provide evidence on other aspects of the HTA appraisal, such as the treatment comparisons considered, the size of anticipated health gains and some indication of the strength of evidence presented. Public discourse on novel drugs often features descriptions of "game changers", "breakthrough" treatments or other such terms [16, 17] . While of course some drugs may achieve complete remission in some patients, we believe it important the public appreciate that new cancer drugs approved in Ireland over nearly a decade will provide a mean of 0.85 QALYs; less than 1 year equivalent of good health. A sober assessment of the health gains offered by new treatments may inform a more balanced consideration of the choices between new drugs and the great many other interventions that the Irish health service struggles to provide timely access to [30] . We noticed some inconsistent reporting within the NCPE HTA summaries. Budget impact was not reported on a consistent basis, as it was variously reported in gross or net terms and the timeframe varied between summaries. Similarly, the anticipated patient population size was not always reported. The incremental costs and QALYs were not reported in all summaries and were also not always reported for all strategies mentioned in the summaries, thus precluding comparisons between strategies other than those reported by the NCPE. Overall, however, while not formally assessed against any objective criteria, we consider the consistency of reporting to be high across the NCPE HTA summaries. Moreover, as noted in the results, we find the completeness of reporting of costs, QALYs and budget impact within the HTA summaries to have increased over time. One specific issue we noticed within the HTA summaries was the reporting of multiple ICERs for a given strategy based on comparisons to multiple interventions. A narrow textbook interpretation of CEA would suggest that this is incorrect and that any given intervention should only have one ICER on the efficient frontier. Moreover, the reporting of multiple ICERs including those based on comparisons to dominated strategies could cause confusion to decision makers regarding an intervention's cost-effectiveness. In practice, there may often not be a single relevant comparator. The current standard of care may vary between patients and depend on clinical judgement. Accordingly, the reporting of multiple ICERs is not necessarily reason for concern. While the choice of comparator may be context-dependent it still deserves scrutiny. We found only five drug-indication pairs that were either dominant or had ICERs within the threshold (pairs 34 and 40-43). In the example of pair 34 pembrolizumab was dominant relative to the comparator of ipilimumab which itself was previously found to be costineffective. Accordingly, pembrolizumab would not have been found to be cost-effective had the previous approval not exceeded the cost-effectiveness threshold. Accepting comparisons against cost-ineffective comparators prompts concerns of a spiral of increasing cost-ineffectiveness. If decision makers permit cost-ineffective choices to accumulate on top of each other, this will result in ever less efficient treatment within each indication. Accordingly, there may need to be more critical examination of the treatment status quo. The RR process is a pragmatic way to triage the analytical workload of assessing new interventions and avoiding a full HTA process. A drawback of the current RR process is that no summary information is published on interventions appraised under this pathway. It seems reasonable that at least some of the information considered at RR could be published. A small proportion of drug-indication pairs have no corresponding NCPE record. While they are few and the indications may be narrow, it is nevertheless disconcerting that at least some approvals appear to have effectively bypassed the controls applied to other interventions. Our conclusion regarding the availability of data is that the NCPE HTA summaries provide a largely consistent, if not fully complete record of the costs and QALYs of the interventions on application. While there are evidence gaps arising from the lack of any reporting for interventions approved following RR and some concerns about ICER estimates based on comparisons to cost-ineffective technologies, the largest concern remains the unavailability of agreed prices and ICERs on approval. Withholding agreed prices because of commercial confidentiality is an understandable consequence of price negotiation between the State and manufacturers. This confidentiality is valuable to manufacturers for their negotiations in other markets [31] . Offering confidentiality to manufacturers may help decision makers secure savings for the State and is common practice in many countries [32, 33] . It is also suggested that price confidentiality might offer broader benefits in facilitating what is known as price discrimination, thereby permitting access to more countries on a pricing basis that reflects variation in ability to pay than would be achieved if common prices were paid by all nations [34] . A cost of such price confidentiality is compromised accountability regarding public health expenditure. Without actual agreed prices, there is no meaningful assessment that external parties can make of healthcare resources allocation, effectively protecting decision makers from scrutiny. This is concerning, as many interests including clinicians, patient advocacy groups, politicians and manufacturers will seek to call attention to the benefit of new treatments, while insufficient attention may be given to the countervailing costs to other patients. The lack of meaningful accountability is particularly concerning given the mixed performance of Ireland's healthcare system when compared internationally. Aspects of the Irish health system perform well, evidenced by rapid improvements in life expectancy and better than average performance in amenable mortality estimates relative to other EU nations [35] . Ireland also performs well on composite measures of health outcomes in general and on cancer mortality in particular [36] . Despite this, the Irish health system performs poorly in terms of access as it both lacks universal coverage and exhibits long waiting lists for elective care and has been ranked lowest out of 35 European nations with respect to waiting times [35, 36] . Indeed, the fact that the Irish health system provides demonstrably poor access to basic and cost-effective services is indicative that resources are not being used most efficiently [29] . A potential solution to this accountability problem would be the publication of some form of aggregated cost-effectiveness metrics based on reimbursed prices. The aggregated costs and QALYs presented in this analysis are an example of such reporting. The estimated incremental costs and QALYs aggregated over all interventions newly adopted each year could be published annually. Alternatively, it would be possible to publish the number of interventions reimbursed within intervals of multiples of the threshold as previously suggested by O'Mahony and Coughlan [29] and exemplified by the final histogram in Fig. 1 of this analysis. Either proposal would offer a degree of accountability regarding the allocation of health spending without revealing the agreed price paid for any given drug. The above proposals would provide evidence of the effectiveness of current appraisal and price negotiation systems. It may be the case that current processes are highly effective at achieving value for money. The reporting proposal would give due credit without publishing confidential prices. Some may contend the onus should be on pharmaceutical manufacturers to be more transparent regarding the prices they charge. Indeed, the World Health Organisation has issued a resolution urging greater price transparency [37] . Despite this, we perceive the responsibility for compiling and publishing such analyses most naturally falls on public regulators rather than private commercial entities: in this case, that part of the decision-making process that is privy to the prices on reimbursement, namely, the HSE Drugs Group. While this manuscript has considered confidentiality regarding prices of cancer therapies, the same concerns and potential solutions apply to all drug spending. We have already noted the primary limitation of this study, but there are others to acknowledge. We only assessed approved cancer drugs. Necessarily the analysis excludes drug-indication pairs that were assessed by the NCPE but ultimately rejected. Similarly, our analysis can only assess officially approved drugs and cannot appraise off-label use. Furthermore, the health economic outcomes extracted from the 61 pairs subject to HTA are not likely to be representative of the other interventions approved after RR, which will likely have both lower budget impact and ICERs on aggregate. In cases in which two budget impact estimates were published, we formed a point estimate by taking the mid-point between the published values. Such an estimate does not account for any possible skewness in costs. We have not conducted a statistical analysis of the outcomes recorded within our review, partly because the sample sizes within the disease and therapeutic categories are too small in most cases, but also because we consider the value of such an analysis questionable without access to the actual agreed prices. Our analysis reports the number of drug-indication combinations for which the NCPE record reimbursement was made following post-appraisal price negotiations. Given that the NCPE is not a party to such negotiations we do not know if the NCPE's reporting of such negotiations is exhaustive or not. Finally, a concluding note regarding data availability on agreed prices in Ireland. Like other research [24, 38] , this analysis could only use list prices on application. Notably, a recent study investigating the potential of value of information analysis regarding cancer drugs in Ireland did have the benefit of access to confidential agreed prices [11] . Accordingly, there is precedent for the use of confidential pricing information in Ireland for research purposes. To date there has been no single analysis of the cost-effectiveness of cancer drugs in Ireland. Our analysis shows that each new approval for which data is published yields approximately 0.85 QALYs on average. Appreciating the size of this health gain may provide the public with a more realistic expectation of new cancer therapies. The aggregate ICER on application of approximately €100,000/ QALY indicates that the additional costs of new drugs relative to existing therapies would need to be more than halved for reimbursement at list prices to be a fair and ethically justifiable use of scarce health resources. While the current publication of HTA summaries by the NCPE provide much useful data, additional reporting is required to accommodate commercial confidentiality while delivering meaningful accountability regarding decision maker choices regarding drug spending in Ireland. The most suitable body to provide such reporting is the HSE Drugs Group. Author contributions JOM conceived the research question. RG, CB, AB, and DK were responsible for the initial study design and data collection and manuscript drafting. NA, SA, SZ, SD, and NOR refined the data collection and revised the manuscript. Final oversight of the manuscript production was conducted by JOM. Availability of data and materials All of the data used in this study are already in the public domain. Code availability Not applicable. Conflict of interest There are no conflict of interest to report. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Delivering affordable cancer care in highincome countries Recent developments in pricing and reimbursement of medicines in Ireland Pharmacoeconomic evaluation in Ireland: review of process The pharmacoeconomic evaluation process in Ireland To HTA or not to HTA: identifying the factors influencing the rapid review outcome in Ireland Pharmacoeconomic evaluation in Ireland Analysis of health technology assessments of orphan drugs in Ireland from Rapid reviews with health-technology assessments in reimbursement systems-an examination of Ireland as a case study Health (pricing and supply of medical goods) Act Framework Agreement on the Supply and Pricing of Medicines Examining the feasibility and utility of estimating partial expected value of perfect information (via a nonparametric approach) as part of the reimbursement decision-making process in Ireland: application to drugs for cancer Coronavirus disease 2019: Considerations for health technology assessment from the National Centre for Pharmacoeconomics Review Group Guidelines for the Economic Evaluation of Health Technologies in Ireland 2020 Submission Process Review Template: National Centre for Pharmacoeconomics New cancer drug hailed as 'game changer' by experts Brave Vicky starts on new cancer wonder Information about National Cancer Control Programme Cancer Drugs Approved for Reimbursement HSE. Drugs Group Minutes Identifying and revealing the importance of decision-making criteria for health technology assessment: a retrospective analysis of reimbursement recommendations in Ireland NCPE Community Register of orphan medicinal products: European Commission The Irish cost-effectiveness threshold: does it support rational rationing or might it lead to unintended harm to Ireland's health system? Surveying the cost-effectiveness of the 20 procedures with the largest public health services waiting lists in Ireland: implications for Ireland's cost-effectiveness threshold Review of studies reporting actual prices for medicines Actual costs of cancer drugs in 15 European countries Can price transparency contribute to more affordable patient access to medicines? European Union pharmaceutical markets: a case for differential pricing? Ireland: Country Health Profile 2017, State of Health in the EU Health Consumer Powerhouse Improving the transparency of markets for medicines, vaccines, and other health products: World Health Assembly A retrospective analysis of budget impact models submitted to the National Centre for Pharmacoeconomics in Ireland Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations See Tables 4, 5 , 6, 7, 8.