key: cord-0037691-4lx5ft5u authors: Martin, Janet; Lal, Avtar; Moodie, Jessica; Zhu, Fang; Cheng, Davy title: Hospital-Based HTA and Know4Go at MEDICI in London, Ontario, Canada date: 2016-05-02 journal: Hospital-Based Health Technology Assessment DOI: 10.1007/978-3-319-39205-9_12 sha: 90f58ffd8bef6ba31e301b1a53f71ae6a1c25ef9 doc_id: 37691 cord_uid: 4lx5ft5u The majority of healthcare in Canada is universally provided and publicly funded through the provincial government. decisions about whether to adopt them. This chapter focuses on HB-HTA in the teaching hospitals across the city of London, Ontario [1] [2] [3] [4] [5] [6] [7] . There are two hospitals in London, Ontario, Canada that provide service to the city and surrounding referral regions. The London Health Sciences Centre (LHSC) is one of the largest acute-care teaching hospitals in Canada providing adult and pediatric services. St. Joseph's Health Care London (SJHC) is also a large teaching hospital in London, with a focus on ambulatory care, chronic care, rehabilitation, and mental health services for adults and children. There are more than 15,000 physicians, residents, and staff providing care for more than 1.5 million patient visits annually. The combined annual budget for LHSC and SJHC is approximately $1.7 billion (Canadian dollars). HB-HTA in the London hospitals evolved over the past 15 years including programs under various names, which have recently been consolidated within the Centre for Medical Evidence, Decision Integrity & Clinical Impact (MEDICI) as a partnership between the hospitals and academia. It is useful to describe the progressive stages of HB-HTA in London, to understand the context for the scope and breadth of the program and approaches to assessment. Appendix 12.1 outlines some of the technologies and drugs evaluated over the course of the HB-HTA program in London, Ontario, and further information is available on our website [1] . • Our HB-HTA program began its earliest roots during a hospital-funded project in 1999 entitled the Evidence-Based Prescribing Initiative within the London Health Sciences Centre [7] . The objective of the initiative was to improve translation of evidence related to drug therapies (whether "new" or "established") into • The majority of healthcare in Canada is universally provided and publicly funded through the provincial government. • Hospitals receive their funding from the provincial government, usually through an annual budget based on historical allocations and/or activitybased funding. • Most decisions about drugs, medical devices, and medical/surgical procedures are made by the individual hospitals, according to local demands and budget limitations. hospital policy and practice through a process of collaborative systematic review with meta-analysis of the evidence, alongside deliberative discussion about the relevance for our local setting. The evidence-based analyses and deliberative discussions were presented by teams of clinicians together with the EBPI project leader to the appropriate policy committees (usually the drug and therapeutics committee and other relevant clinical, quality, and finance committees) to inform decisions about which drugs should be taken up versus which should be abandoned from practice. • Initially, new and expensive drugs were the primary focus. But, eventually, drug classes were also reviewed, for the purpose of simultaneous investment and disinvestments within drug classes. In addition, the focus was not only on assessment but also on translation into policy and practice, with evaluation of the impact post-implementation. At any time, three to nine assessment and implementation projects were in progress simultaneously, with one full-time program leader supplemented by a number of clinicians and administrators providing inkind time. This innovative approach to knowledge translation was focused on collaborative evidence discovery with the project leader together with hospital practitioners (physicians, nurses, dieticians, respiratory therapists, clinical ethicists, and pharmacists) and managers (budget holders and other hospital policy-makers) to identify, interpret, synthesize, and evidence for high-risk or high-cost drugs in order to improve relevance, buy-in, and ultimately decision translation into practice within LHSC. The success of this initial project resulted in an ongoing program of evidence-based evaluation for drug therapies at the London Health Sciences Centre, resourced primarily through hospital operational funding and supplemented by grant funding and in-kind time from practitioners and trainees. This initiative introduced a new standard for evidence-informed decision-making at our hospital, ushering in a culture of expectation for rigorous evidence reviews to undergird decisions, and was soon incorporated into the hospital policy-making process for any drug therapy being considered for adoption or disinvestment. A number of drugs assessed in early stages have since become the focus of reassessment, or have been useful to expand into full drug class reviews, with subsequent evidence-based guidelines for internal use. Some of the assessments culminated in collaboration with the Ministry of Health (MOH) and with the Ontario Hospital Association (OHA) in order to influence drug policy changes in hospitals across the province (drotrecogin alfa, rhAPC; proton pump inhibitors, PPI; intravenous immune globulin, IVIG; biologics for ulcerative colitis). This process also allowed for a few innovative drug price negotiation strategies based on best available evidence, moving us toward evidence-based drug procurement with risk-sharing agreement for selected drug purchase contracts. The EBPI was awarded with two national recognitions: the Innovative Practitioner Award and the Pharmacy Administration Award. In addition, this program was awarded the LHSC Medical Advisory Committee Award (1999) for local impact. Some of the methods and approaches developed during the EBPI continue within our hospital today as a common thread toward assessment of drugs and technologies [1] . Eventually, the need to apply evidence-informed decision-making to areas beyond drug therapies was recognized within the London hospitals. As a result, the High Impact Technology Evaluation Centre (HiTEC) was initiated in 2003, with hospital operational funding, grant funding, and in-kind time to perform assessments of drugs and other nondrug technologies as requested by senior hospital leadership, managers, or relevant clinician decision-makers. HiTEC operated as an on-demand request service, to facilitate evidence synthesis and economic evaluations of drugs, devices, and other technologies to inform hospital decisions and procurement processes. In addition, we undertook collaborative projects with Medbuy (a group purchasing provider) to inform negotiations with industry for proton pump inhibitors and erythropoietics. The success of these initiatives was awarded the LHSC/SJHC Medical Advisory Committee Award in 2006 in recognition for evidence-based planning and implementation (knowledge translation) [1] . Following the early successes of HiTEC, leaders from other areas of the hospital requested formal collaboration to enable more systematic assessment of anesthesia, surgery, and critical care. As a result, we inaugurated the Evidence-Based Perioperative Clinical Outcomes Research Group (EPiCOR) as an academic and hospital-based collaboration together with HiTEC through a combination of grants, local operating funds, and in-kind clinician support, from the departments of anesthesia and perioperative medicine, surgery, medicine, and pharmacy. The EPiCOR-HiTEC collaboration also expanded beyond local work at the London hospitals to include international efforts to develop HTAs for surgery, anesthesia, and critical care. In addition, EPiCOR-HiTEC collaborated with international surgical and medical societies to assess innovative hospital technologies and surgical techniques. Through this approach, HTAs, guidelines, and consensus statements were developed for local and international considerations related to adoption or disinvestment in off-pump coronary artery bypass surgery, stentless aortic valves, transmyocardial laser revascularization, surgical ablation of atrial fibrillation, upper gastrointestinal bleeding, percutaneous coronary intervention, minimally invasive mitral valve surgery, video-assisted thoracic surgery, endovascular vein harvest, antibiotic prophylaxis, thoracic endovascular aortic repair, transcatheter aortic valve implantation, and various drugs, technologies, and techniques for blood conservation . In 2012, we inaugurated the Centre for Medical Evidence, Decision Integrity & Clinical Impact (MEDICI), which consolidated and further expanded the mandate of ongoing programs and initiatives including HiTEC, EPiCOR, and Know4Go (Appendix 12.2) to foster HB-HTA initiatives locally and beyond while also enabling research, teaching, and service provision through broader collaboration for hospitalrelevant HTA. Since HB-HTA provides the "perfect microcosm" to test methods and gain firsthand knowledge of techniques for translating evidence into policy and practice, the mandate of MEDICI has expanded to include research, education, and methods for improving decision-making and knowledge translation. The following outlines the key mandates of MEDICI: 1. Practice and policy: To provide timely, contextualized evidence syntheses to enable real-world evidence-informed decision-making related to drugs, devices, procedures, and programs with a special focus on (a) hospitals at the local, regional, national, and international level and (b) global surgery, anesthesia, and perioperative care as an essential component of universal healthcare in the developing and developed world 2. Education: To provide educational and capacity-building opportunities in evidence-informed decision-making, health technology assessment, health economics, health policy, and knowledge translation locally, nationally, and internationally in the developed and developing world 3. Research: To conduct cutting-edge research to advance the front of health technology assessment, economic analysis, health policy analysis, decision-making science, and knowledge translation in the developing and developed world A brief outline of MEDICI is provided below. Further information and additional published and internal HTA reports are available elsewhere [1, . . Currently, the staff of MEDICI includes three part-time positions (director, medical director, health economist) and three full-time positions (one coordinator, one systematic reviewer and methodologist, one research assistant). The three part-time positions also hold other roles, such as teaching university courses and providing clinical services and administrative responsibilities within the university and hospital. Additionally, at any time a number of trainees and visiting researchers contribute to MEDICI activities, including postdoctoral fellows, global health fellows, clinical fellows, medical residents, visiting professors, graduate students (MSc of biostatistics and epidemiology, MSc of applied mathematics, master of library and information sciences), and undergraduate medical and health sciences students. Funding for MEDICI varies annually based on the magnitude and scope of the work requested by the funding partners. Typically, the funders include Schulich School of Medicine & Dentistry, London Health Sciences Centre, St. Joseph's Health Care, Lawson Health Research Institute, internal grants, external grants, and externally commissioned service contracts from other hospitals, clinical specialty societies, and other governmental or nongovernmental organizations. HB-HTA services at MEDICI include assessments of technologies, procedures, drugs, and programs through comprehensive systematic reviews or ultra-rapid systematic overviews [1, . In addition, when capacity allows, MEDICI supports clinician researchers to design appropriate research to address evidence gaps. Depending on available resources within MEDICI, requests for assessments are accepted through a number of channels, such as through the senior leadership team including the hospital CEOs and other senior administrative leadership, or through clinical leaders including departmental chairs, and directly by physicians or other practitioners. The hospitals have gone through a number of changes in CEOs and senior leadership over the years, resulting in changes in institutional management structures and decision-maker accountabilities. As a result, we have provided HB-HTA for a number of different committees and decision-making units within the hospital in order to remain flexible based on demand and tempered by our available human resources and funding flows. At this time, decision-making for health technology uptake and disinvestment is spread across committees and decision-making structures within the London hospitals, typically organized around clinical departmental structures according to budget accountabilities. At the time of writing, there is no centralized intake process or unified decision-making process for all technology requests for the London hospitals, and we see this as an opportunity for formal research. We are currently seeking grant funding to evaluate the impact of a centralized approach using the Know4Go and IDEAL frameworks [69], both locally and in collaboration with other hospitals in Canada and abroad. When requests focus on single technology assessment within the local hospitals, we typically use Know4Go to initially map the evidence and resource impacts based on a rapid review of published evidence and local data as a prioritization step to determine whether more in-depth analysis is worthwhile (Appendix 12.2). This rapid pre-assessment allows us to telescope the depth and breadth of the review based on the likely impact of the technology in question. If the pre-assessment suggests that the payback on comprehensive assessment efforts are likely to be commensurate with the potential magnitude of impact of the technology, and if no relevant up-to-date reviews pre-exist from other HTA agencies (including the Canadian Agency for Drugs and Technologies in Health, Health Quality Ontario, and Ontario Health Technology Assessment Committee), we perform "de novo" systematic reviews, meta-analyses, and meta-regressions as the first component of Know4Go. Subsequently, as needed, we determine the contextualized benefit index and local opportunity costs based potentially on local data analysis, economic modeling, sleeper analyses, and a survey of competing priorities. The comprehensiveness of the evidence, economic, sleeper analyses, and trade-off assessments depends on the question at hand, whereby high-stakes decisions receive more time and rigor than low-stakes decisions. In some cases, the evidence and/or economics is so compelling that the Know4Go and decision-making process can be truncated without performing extended analyses. The comprehensiveness and number of reviews conducted also depends on amount of human resources available within MEDICI. More recently, requests have increasingly focused on more complex "programs" of care (multiple embedded systematic reviews), with crosscutting issues of tech-nologies, techniques, and institutional issues (i.e., sleepers) embedded within the request (see Appendix 12.1 and Appendix 12.2). While program assessment or "portfolio-wide" assessments can be extremely informative (far beyond single technology assessments), they have also raised significant challenges for a small unit as ours, since program evaluations often represent large and complex assessments that required devoted full-time research resources for several months while reducing our capacity to turn over multiple individual assessments within the annual cycle. Research efforts are required to address this gap. In addition, we have supported evidence development and research sequencing for innovative early development and evaluation of devices or procedures using the Know4Go Framework (Appendix 12.2) and the IDEAL Framework [69]. Since our HTA process also involves identifying gaps in the evidence base, we have also conducted local randomized controlled trials when existing evidence was insufficient to inform the decision at hand (Appendix 12.1) [35, 41, 51] . However, the latter has been difficult to achieve consistently, due to the resources and timelines required. More commonly, we have conducted local database analyses, scenario modeling, or pragmatic "value of further information analyses" to better inform whether decisions should be (a) "yes" or "no" today or (b) "further research is required and is worth waiting for" or (Appendix 12.2). MEDICI has experienced increasing demand for external consultations and international collaborations with hospitals both in the developing world and in the developed world. Taking on international work and consultations has resulted in less capacity for local projects. This trade-off will be reconsidered over time as we consolidate our expanded service, education, policy, and capacity-building mandate, and as we shift resources to enable efficiencies from locally conducted HB-HTA toward our ultimate goal of a local-global collaboration to reduce duplication, and increase cost-effectiveness and timeliness of HB-HTA through collaborative efficiencies and a formal research program to provide a systematized approach to development and evaluation of HB-HTA methodologies. As an extension of our local work in technology assessment and knowledge translation, members of MEDICI have contributed to a number of provincial, national, and international initiatives, including Health Quality Ontario Quality-Based Procedures, the Choosing Wisely Campaign, Ontario Drug Benefit policies, the Drugs for Rare Diseases Policy Working Group, Ontario Blood Advisory Committee [70] [71] [72] , research on decision-making determinants [73] , policy advice and white paper on health technology assessment and management for Health Canada, the "Unleashing Innovation: Excellent Healthcare for Canada" conducted by the Advisory Panel on Healthcare Innovation, and the federal health minister's roundtable on healthcare innovations [1] . We have also been invited to expand a number of our local assessments to coproduce national or international surgical society guidelines and prioritysetting papers for a number of technologies and techniques (Appendix 12.1). Given that HB-HTA is particularly relevant to achieving the globally declared sustainable development goal of "universal healthcare provision for the majority of the global population by 2030," MEDICI is now collaborating with the World Health Organization (WHO) Emergency and Essential Surgical Care program to address issues related to global surgery, anesthesia, and critical care services [74] [75] [76] [77] . In 2014-2016, MEDICI collaborated with the WHO to address the risk and impact of Ebola virus disease on the provision of surgery services in West African countries. Additionally, we are working with WHO on opportunities to improve access to essential global surgery and anesthesia services while also reducing perioperative and anesthetic-related morbidity and mortality in the developing world through contextualized evidence assessments. Performing HB-HTA to scale with meaningful contextualization and local stakeholder engagement and empowerment will be essential to providing timely guidance on how to achieve this sustainable development goal and may in fact have greater impact on quantity and quality of life and greater return on investment than performing more marginal assessments for newer technologies in the local hospital settings of the developed world. In 2012-2014, we performed a return-on-investment (ROI) evaluation of the impact of HB-HTA using Know4Go in the London hospitals. Overall, the ROI was greater than 2-for-1 (i.e., $2 saved for every $1 invested in the HB-HTA program) [1] . In another before-after study of the impact of our approach to HB-HTA using Know4Go to address drug decisions within LHSC, we found that the implementation of Know4Go was associated with reduced drug cost growth in our hospital and reduced total drug costs per patient when comparing the 5 years prior versus 5 years post-implementation ( Fig. 12.1 ). This result was not too surprising, since we had focused especially on performing evidence-based assessments and sleeper assessments for targeted high-cost drugs in 2006 and beyond. During this period, we also developed an annual request for proposals through the drug and therapeutics committee to receive requests for assessments in a coordinated fashion and to elicit sleeper issues that might not be anticipated in prior assessments. We additionally implemented a 24/7 pager system, whereby special one-off requests for non-formulary drugs could be made, which allowed a core team from the Pharmacy & Therapeutics Committee to screen all such requests for approval or rejection on a case-by-case basis. This proved especially important for effective de-implementation of some high-cost drugs (i.e., aprotinin) and for preventing indiscriminate use of newer drugs. It also served as a "horizon scanning" device to foresee the need for upcoming assessments for drugs with increased demand through special request system. Nevertheless, while the results of our study suggest a possible association of our HB-HTA program on costs (i.e., "bending the cost curve"), these results should not be overinterpreted given the limitations of this study. This study was a retrospective before-after study, likely with many confounders. Association does not prove "causation" since many other changes were likely implemented in our hospital within the same time frame as we began developing and implementing Know4Go. In fact, our Know4Go approach to performing evidence-based assessment, costs, other implementation issues (i.e., social, legal, ethical, institutional factors), and trade-offs was a continued evolution and expansion of our earlier approach to evidence-based assessment introduced during the Evidence-Based Prescribing Initiative. As a result, our evolution since 1999 was one of progressively increasing the expectation of comprehensive evidence-based systematic reviews of the evidence. Additionally, it is important to note that Know4Go was not applied in its entirety, or with equal rigor, to all decisions. Due to our limited resources for HB-HTA, we developed a pragmatic approach to prioritizing requests using rapid assessment Know4Go and subsequently assessing subcomponents of Know4Go (at the least, evidence + economics; if pertinent, also assess sleepers and opportunity cost) to address higher-cost or higher-stake drugs more thoroughly than lower-stake and lower-cost drugs. Another concern is that the data from 2004 were incomplete, and we remain uncertain regarding the verity of the drug cost information for that year. Perhaps most importantly, the goal of HB-HTA is not primarily to impact costs. Therefore, a key limitation was that patient outcomes were not measured. These limitations are similar for many assessments of HB-HTA in the literature, and this highlights the need for more rigorous assessments in the future, such as through adequately powered controlled trials in order to establish increased confidence of the range of impacts of HB-HTA on a variety of outcomes such as clinical outcomes, institutional impacts, costs, and return-on-investment. The successes of our dynamic approach to HB-HTA include implementing a number of projects and processes that advanced the rigor of decision-making beyond status quo through assessment of evidence, economics, and other contextual factors, as well as quantification of opportunity cost based on a pragmatic approach that can be telescoped based on likely "return-on-additional-effort." These efforts also contributed to a culture of expectation of evidence-based decision-making, of assessing true value, and (increasingly) of assessing opportunity cost through a number of initiatives since 1999. Throughout the evolution of our HB-HTA services, there have been a number of important challenges. Importantly, our approach to growing a program based on an initial project, and through various versions of a mix of informal or formally recognized service for the hospital setting, has required significant effort, often as an added margin of hours through a "side of desk" approach, while also managing other job titles and clinical or hospital administrative responsibilities. The underlying challenge that is germane to this is the ever-present need to "prove" the value of the HB-HTA unit, often before gaining approval for continued annual operational funding. This constant need to "prove" our worth results in a dual challenge to produce HTA for the hospital proactively while also evaluating the impact of the HB-HTA program and procuring grants to provide funding to expand services and methodologies beyond the core-funded services. This constant need derives from the continual budget shortfall for hospitals in the publicly funded healthcare setting, where demand always exceeds available resources. This growing demand for technology assessments highlights the need for HB-HTA growth, and yet, HB-HTA operating funds must compete directly with direct patient care shortfalls. This is a tough competition to win, given the immediate gratification of offsetting direct patient care shortfalls relative to the more remote and longer-term sustained benefits of an HB-HTA service. Another challenge is the divergent tug-of-war between rigor of academic methods and timeliness of real-world decisions. The weeks, months, or years required by traditional approaches to HTA with systematic review, meta-analysis, economic evaluation, and post-implementation evaluation or other methods of local evidence generation does not align with the pace expected by decision-maker needs. This challenge is becoming more serious as the volume of evidence and data is growing exponentially, inducing greater efforts to complete evidence syntheses and HTAs. However, decision-makers and academic collaborators who devote in-kind time (with the hopes of publishing quickly) expect ultra-rapid or expedited systematic review timelines. This challenge is compounded by our approach to HB-HTA where multiple options and post-implementation outcomes need to be evaluated to ensure the predicted effects translated to reality in our hospital. This results in multiple layers of research across multiple topics, and insufficient time to publish all assessments in the peer-reviewed literature. We hope to address this challenge through grant-funded research on expedited methods using rapid crowd-sourced Know4Go. The counterposing challenge of trading-off quality and precision for timeliness will remain a ubiquitous challenge for HB-HTA until we find better ways of effectively automating our processes and finding other methodologic heuristics which do not jeopardize quality of decision-making and patient care. In our HB-HTA program, rapid reviews with draft trade-off table plots will be our hybrid compromise. However, we need to understand the risks of premature decision anchoring with rapid reviews, given the evolving understanding about risks of evidence reversals with immature evidence. When decisions are prematurely made based on early evidence, reversing those decisions may end up becoming more costly, particularly if the evidence reverses direction, and disinvestment with de-implementation is required. Since the future success of HB-HTA will rely on moving beyond our current traditional methods of HTA, we are working on the following areas of future development for our program through grants and service contracts: Collaboration, Nationally and Globally: In 2013, together with CADTH and the Ottawa Health Research Institute, MEDICI co-hosted a national HB-HTA symposium to explore the potential for building a network. We are submitting grant requests to fund this future endeavor to develop collaborative decision-making and integrated knowledge translation for the hospital setting around the globe [2] [3] . It is our goal to build an effective national and global network to support hospitals in decisions and KT related to health technology investment and disinvestment to enable efficient innovation and optimal healthcare, whether locally or internationally [2, 3, 78] . Iterative assessment, throughout the life cycle: Using our Know4Go Framework, we have been exploring ways to move beyond the paradigm of one-off single technology assessments, to progress to dynamic assessment of portfolios of opportunities. To better embrace the world of iterative and evolving assessments throughout the life cycle across a multitude of technologies, a number of methods will need to be further developed including pragmatic Bayesian analysis, pragmatic value-of-information analysis, dual assessment of evidence from clinical trials, along with real-world outcomes, among others. Automated efficiencies will be necessary, through technologies such as machine learning and cognitive computing, to ensure that the global evidence base can be identified, collected, synthesized, and made readily available for local contextualization, and so that evidence can be weighed against local considerations and continuous feeds of real-world local data. HB-HTA education and capacity building: Capacity building in HTA skills through training, workshops, and graduate courses (MSc/PhD) will need to be expanded, both in terms of numbers of trainees and also in terms of the scope of knowledge and skills developed. Such capacity-building initiatives need to be accessible both the "users" and "doers" of HTA. In summary, our collective mantra for the future HB-HTA research and development is: Share everything; repeat sparingly; adapt often; incentivize problem solving; reward decision-impact and knowledge translation. Early in the experience of HB-HTA within the London hospitals, it became clear that the traditional approach to HTA and decision-making was insufficient to meet the needs for local decision-makers for a multitude of reasons [1] [2] [3] [4] : • Evidence alone is essential, but insufficient for decision-making. • Economic evaluation is essential, but also insufficient for decision-making. • Additional domains of influence on decisions (i.e., the "sleepers" defined below) also need to be systematically evaluated and contextualized for decisionmakers. • Single technology assessments in isolation add little value to the decision-making process; when in reality, multiple technologies and interactions among them are likely to be important. • One-off assessments add little value to the decision-making process when technologies (and the evidence) evolve quickly across multiple versions and varied disease applications, with an inevitable learning curve and changing competing technologies, which require iterative assessments throughout the technology life cycle in order to be meaningful. • Decision-makers and internal advocates are easily distracted by "new" and purportedly "innovative" technologies when there is no explicit process for simultaneously revealing the best value for money among all opportunities and options (whether "new" or "old") for investment and disinvestment. We developed the Know4Go Framework to address these deficiencies. The Know4Go Framework addresses the contextualized evidence, economics, sleepers, and opportunity cost ( Fig. 12.2) . Specifically, Know4Go builds on the foundation of traditional HTA components including rigorous evidence synthesis and economic evaluation but also ensures that it goes beyond traditional HTA by systematically addressing decision-relevant issues not addressed by the evidence (i.e., the "sleepers") and by quantifying the opportunity cost of choosing one set of opportunities over another. The sleepers are those domains which may be equally important for guiding decisions and which may prematurely trump the decisions at hand and preempt fair consideration of the evidence and economic considerations if they are not adequately addressed and placed in their appropriate context vis-à-vis the evidence. Specifically, the "sleeper" domains include the social, legal, ethical, environmental/ institutional, political ramifications, along with entrepreneurial, research/innovation opportunities, and stickiness/reversibility factors (Fig. 12.3) . The sleepers are defined and assessed systematically and collaboratively with stakeholders at the beginning of the decision process to capture the initial emotive reactions to the perceived issues underlying the sleeper domains and again after the evidence and economics have been systematically reviewed in collaboration with the stakeholders to capture the more mature, well-informed contextualized perception of the importance of the "sleepers." This allows stakeholders to define and express their perceptions about the importance of each of the domains of potential sleepers underlying the decision, based on initial "gut" reaction and again after evidence-and economic-informed contextualized reaction. The difference in perceived importance of the sleepers for administrators relative to clinicians is collected by survey and the results presented to the stakeholders via radial plots to outline the amount of discrepancy in perception of the relative importance by the stakeholder groups. This systematic and visual approach to addressing the sleepers allows for stakeholders and producers of HTA to come to an agreement up front about what issues underlie the decision at hand and the likely perceived weight of importance of that issue on the ultimate decision to be made. In addition, as the HTA progresses through evidence assessment and economic evaluation, the sleeper domains can be repeatedly discussed and placed into a more informed context in light of the evidence and economic issues. Sometimes the perceived weight of importance of the sleeper domains differs significantly from the point of first "gut reaction" to the more informed point of decision-making after the evidence and economic consider- ations have been brought to bear. Furthermore, defining and evaluating perceptions of the sleepers up front, before the evidence review and economic evaluations have been conducted, also allows the scope of the evidence and economic evaluation to consider formal incorporation of sleeper concepts, when relevant. Using the Know4Go Framework, once the evidence has been synthesized through systematic review or meta-analysis, the evidence is contextualized to the local hospital perspective (or the health system perspective, depending on where the budget and health outcome accountabilities lie) by converting the evidence to a decision-relevant benefit index. This benefit index derives from number of patients who would likely benefit tangibly from this intervention (using metrics of your hospital's choice) and is based on contextualization of the global evidence base through local data-informed estimates of the number of eligible patients corrected by the absolute benefit and risk derived from the evidence. Furthermore, the local resource considerations and total budget impact for the institution (or the health system, depending on the budget accountabilities) are estimated using local institutional costing data. Each technology, technique, or drug under consideration is plotted as a ball on the Know4Go trade-off table in order to make transparent the likely benefit gained per resource expended from the institutional perspective ( Fig. 12.4) . Each ball represents an opportunity (drug, device, procedure, or program), and the size of the ball is telescoped based on the amount of uncertainty regarding the benefit index and local resource impacts. The colors of the balls are coded based on the relevant clinical programs and interrelatedness of the decisions at hand. Opportunities which fall below the "go, no-go" line which have not yet been taken up into practice are colored as white balls, which represent the opportunity cost (i.e., lost opportunity, not yet implemented) which should be prioritized first for uptake into practice. Additionally, we have added an additional feature within Know4Go to color the ball based on the maturity of the evidence as per the IDEAL Framework [5] . Plotting options on this trade-off table allows greater transparency for decisions to be made about whether a decision should be a "go" or a "no-go." Since we know that generally we are not willing to pay exceedingly more money for exceedingly small benefits, there is a limit which can be defined as the "go, no-go" line. Over time, this "go, no-go" line has defined itself in hospital settings using Know4Go, since the transparency of the Know4Go table has allowed us to regulate our decisions to generally accept the decisions, represented by balls falling under the line, and declining the requests for technologies and programs above the line. Furthermore, as we progressively plot technologies and programs that already exist within the hospital setting, the trade-off table has become a tool for explicitly identifying disinvestment opportunities (i.e., previous decisions for technologies can be plotted according to their benefit index and resource requirement on the trade-off table and will be above the line if they were low value for money, which reveals an opportunity for disinvestment). The Know4Go trade-off table also allows for a simultaneous approach to consideration of paired investment-disinvestment opportunities for budget-restricted hospitals considering new opportunities for which there is no available marginal budget. Identifying lower value-for-money technologies that appear above the "go, no-go" line provides a targeted list of technologies from which to disinvest in order to release resources for better investment. When used appropriately to consider the evidence, economics, and contextualized sleepers, the Know4Go trade-off table becomes a tool to ensure transparency and objectivity in improving value for money for all technologies, drugs, and programs adopted (and disinvested) in the hospital setting. In essence, this becomes an evidence-informed tool to fuel innovation that provides better value for money. Using Know4Go, we have also found that we can better prioritize requests for new technologies and other innovations in the hospital setting by using the tradeoff table as an initial prioritization framework. For example, in previous years when we held an annual cycle of requests for proposals for new technologies and drugs in our hospitals, the volume of requests superseded available human resources to assess each technology using a traditional HTA approach. We used the Know4Go trade-off table to perform prioritization of the submitted technologies using an ultra-rapid review process to anticipate the "ballpark" benefit index and budget impact to plot the "draft" balls. In this way, we could identify requests for technologies which we should not spend further time on, since they provided very low estimated value for money. This is first-draft Know4Go, used as a prioritization tool. After prioritization of multiple requests, those with highest likelihood of providing worthy value for money (i.e., under the "go, no-go" line) become the focus of detailed HTA, with full evidence assessment, economic evaluation, and sleeper assessment. Full assessment of a proposed technology, procedure, or program may also involve identifying other existing options within the hospital for disinvestment, in order to ensure resources can be released, and the opportunity cost can be minimized. Know4Go can be used to identify and prioritize a local research agenda. This is an area where HB-HTA units around the world could take a much more proactive approach. Since HB-HTA is in the business of performing evidence syntheses and economic evaluations, with local considerations of competing priorities and detailed consideration of local institutional needs, every HTA becomes an opportunity to highlight the gaps in the evidence base and the gaps in local knowledge. This tabulation of gaps becomes a list of potential "research opportunities," which also can be valued with a predicted benefit and cost (and plotted on the Know4Go trade-off table). This becomes ultimately an expedited "value of further information" analysis, also known as predicting the cost-effectiveness of undertaking research to answer the gaps in the evidence, to prioritize the local research agenda. It can also be embedded within the sleeper assessment (during consideration of the "r" domain for research/innovation) proactively within each opportunity assessment in order to determine whether decisions should be made (in light of the remaining uncertainty) or whether it would be cost-effective and "worthy" (given the time required and likelihood of success in reducing uncertainty to an extent that meaningfully advances decision-making) to consider devoting more resource to research in order to reduce the remaining uncertainty. Know4Go has been applied to a number of decisions in London and iteratively further developed from its earliest prototype version after learning from application to real-world decisions, in Canada and internationally. Its development and refinement continues, with feedback from those using it in different contexts. At this time, we are seeking grants to study a broader implementation of Know4Go for portfoliowide assessments of technologies and in hospitals locally and internationally. Waking the Sleepers: Know4Go Oxford Podcasts on iTunes: Know4Go Special Lecture at Centre for Evidence-Based Medicine EBHC Powerpoint and Teaching Sessions: Know4Go Lecture Available at www.IDEAL-Collaboration.net References 1. Centre for Medical Evidence Hospital and regional health technology assessment in Canada: current state and next steps Supporting the use of health technology assessments in decision makers The real cost of care: focus on value for money, rather than pricetags Role of the anesthesiologist in the wider governance of healthcare and health economics Evidence-based practice and health technology assessment: a call for anesthesiologists to engage in knowledge translation Evidence-Based Prescribing Initiative, Submitted to London Health Sciences Centre Senior Leadership Committee Consensus Conference and Statements of randomized controlled trials of off-pump versus conventional coronary artery bypass surgery Antibiotic prophylaxis for preventing surgical site infection in plastic surgery: an evidence based consensus conference statement from the American Association of Plastic Surgeons Drug, devices, technologies, and techniques for blood management in minimally invasive and conventional cardiothoracic surgery: a consensus statement from the International Society for Minimally Invasive Cardiothoracic Surgery (ISMICS) 2011. Innovations (Phila) Conflicts of interest ethics: silencing expertise in the development of international clinical practice guidelines Management of patients with nonvariceal upper gastrointestinal bleeding Minimally invasive versus conventional open mitral valve surgery: a meta-analysis and systematic review Minimally invasive versus open mitral valve surgery: a consensus statement of the international society of minimally invasive coronary surgery (ISMICS) 2010 Proton pump inhibitor treatment initiated prior to endoscopic diagnosis in upper gastrointestinal bleeding Surgical ablation for atrial fibrillation in cardiac surgery: a meta-analysis and systematic review Surgical ablation for atrial fibrillation in cardiac surgery: a consensus statement of the International Society of Minimally Invasive Cardiothoracic Surgery (ISMICS) Endoscopic vein-graft harvesting: balancing the risk and benefits Orogastric tubes do not improve transesophageal echocardiographic imaging during cardiac surgery: a randomized trial Endovascular aortic repair versus open surgical repair for descending thoracic aortic disease a systematic review and meta-analysis of comparative studies International consensus recommendations on the management of patients with nonvariceal upper gastrointestinal bleeding EACTS/ESCVS best practice guidelines for reporting treatment results in the thoracic aorta Stentless versus stented bioprosthetic aortic valves: a systematic review and meta-analysis of controlled trials Stentless versus stented bioprosthetic aortic valves: a consensus statement of the International Society of Minimally Invasive Cardiothoracic Surgery (ISMICS) Video-assisted thoracic surgery for lung cancer resection: a consensus statement of the International Society of Minimally Invasive Cardiothoracic Surgery (ISMICS) Video-assisted thoracic surgery in lung cancer resection: a meta-analysis and systematic review of controlled trials Transmyocardial laser revascularization: a consensus statement of the International Society of Minimally Invasive Cardiothoracic Surgery (ISMICS) Transmyocardial laser revascularization: a meta-analysis and systematic review of controlled trials Off-pump versus conventional coronary artery bypass grafting: a meta-analysis and consensus statement from the Endoscopic vascular harvest in coronary artery bypass grafting surgery: a meta-analysis of randomized trials and controlled trials Endoscopic vascular harvest in coronary artery bypass grafting surgery: a consensus statement of the International Society of Minimally Invasive Cardiothoracic Surgery (ISMICS) Evidence-Based perioperative drug therapy: getting evidence into policy and practice A cost analysis comparing surgical trays with redundant instruments to reduced trays Does preoperative rehabilitation for patients planning to undergo joint replacement surgery improve outcomes? A systematic review and meta-analysis of randomized controlled trials Randomized double-blind safety comparison of intravenous iron dextran versus iron sucrose in an adult non-hemodialysis outpatient population. A feasibility study Cost-effectiveness analysis of diabetic retinopathy screening with pharmacy-based tele-ophthalmology versus in-person eye examination Ketamine added to morphine patient-controlled analgesia for postoperative pain in adults: a systematic review and metaanalysis of randomized trials A systematic review of transthoracic and transesophageal echocardiography in non-cardiac surgery: implications for point of care ultrasound education in the operating room The increasing need for health-economic assessment in gastrointestinal and liver diseases Comparison of different regimens of proton pump inhibitors for acute peptic ulcer bleeding Effectiveness of disseminating consensus management recommendations for ulcer bleeding: a cluster randomized trial Tranexamic acid for routine use in off-pump coronary artery bypass surgery: evidence base "fait accompli" or more research needed? Cost-efficiency of laparoscopic versus open colon surgery in a tertiary care center Prognosis of diabetic patients undergoing coronary artery bypass surgery compared with nondiabetics: a systematic review and meta-analysis N-acetylcysteine in cardiac surgery: do the benefits outweigh the risks? A meta-analytic reappraisal Endoscopic versus open vein-graft harvesting Off-pump coronary artery bypass surgery and the kidney The efficacy of an intraoperative cell saver during cardiac surgery: a meta-analysis of randomized trials Intravenous proton pump inhibitors: an evidence-based review of their use in gastrointestinal disorders Is it safe to cover the left subclavian artery when placing an endovascular stent in the descending thoracic aorta? Oral calcitriol versus oral alfacalcidol for the treatment of secondary hyperparathyroidism in patients receiving hemodialysis: a randomized, crossover trial Meta-analysis of clinical efficacy and bleeding risk with intravenous glycoprotein IIb/IIIa antagonists for percutaneous coronary intervention Evidence-based Peri-operative Clinical Outcomes Research (EPiCOR) Group. Does off-pump or minimally invasive coronary artery bypass reduce mortality, morbidity, and resource utilization when compared with percutaneous coronary intervention? A meta-analysis of randomized trials What interventions should pharmacists employ to impact health practitioners' prescribing practices? Patient-controlled versus nurse-controlled analgesia after cardiac surgery--a meta-analysis The use of amiodarone for inhospital cardiac arrest at two tertiary care centres Evidence-Based Perioperative Clinical Outcomes Research (EPiCOR) Group. NSAID-analgesia, pain control and morbidity in cardiothoracic surgery Off pump coronary artery bypass graft surgery versus conventional coronary artery bypass graft surgery: a systematic review Raising the bar: a primer on evidence-based decision-making The safety of dosing dalteparin based on actual body weight for the treatment of acute venous thromboembolism in obese patients Is off-pump bypass grafting truly superior to conventional coronary artery bypass grafting? Evidence-Based Perioperative Clinical Outcomes Research Group (2005) Does off-pump coronary artery bypass reduce mortality, morbidity, and resource utilization when compared with conventional coronary artery bypass? A meta-analysis of randomized trials A Canadian clinical practice algorithm for the management of patients with nonvariceal upper gastrointestinal bleeding Cost-effectiveness issues in gastrointestinal practice Cost implications in minimally invasive surgery Waking the sleepers: know4Go Oxford podcasts on iTunes: know4Go special lecture at Centre for Evidence Based Medicine EBHC powerpoint and teaching sessions: know4Go lecture Application of a policy framework for the public funding of drugs for rare diseases Towards a functional definition of 'rare disease' for regulatory agencies and funding agencies An evaluation framework for funding drugs for rare diseases Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers Survey of the capacity for essential surgery and anaesthesia services in Papua New Guinea Ebola virus disease and Surgery in West Africa Cheng D; Evidence-based Peri-operative Clinical Outcomes Research (EPiCOR) Group. Perioperative and anaesthetic-related mortality in developed and developing countries: a systematic review and meta-analysis Anaesthetic-related mortality in sub-Saharan Africa Health technology assessment 2020: networks are essential Acknowledgements Special thanks is owed to those who contributed to early HB-HTA activities, and who helped to refine the application of Know4Go, including Kelly Zarnke, David Massel, Sonja Sawh, Richard Jones, and Neil Johnson. Funding is gratefully acknowledged from LHSC, SJHC, LHRI, and Schulich School of Medicine & Dentistry, and CIHR, CADTH, AFP Innovation Fund. Thanks too to our local and international students and fellows.