key: cord-0946069-810jc01t authors: Rong, Lisa Q.; Audisio, Katia; O'Shaughnessy, Sinead M. title: Guidelines and evidence-based recommendations in anaesthesia: where do we stand? date: 2022-03-19 journal: Br J Anaesth DOI: 10.1016/j.bja.2022.02.025 sha: 34c11f8ab765a07b620fd76c7d2006298a434b68 doc_id: 946069 cord_uid: 810jc01t Clinical practice guidelines are increasingly important to guide clinical care. However, they can vary widely in quality, and many recommendations are based on low-level evidence. The COVID-19 pandemic highlighted the need for new flexible formats for rigorously developed guidelines. Future guideline development should be standardised, graded, registered, and updated to ensure that they are ‘living’ works in progress. The number of clinical studies published has accelerated to the point where the amount of new medical information doubled every 72 days in 2020. 1 As a result, anaesthetists are increasingly reliant on summarised evidence in guidance documents, with the most important being clinical practice guidelines. Clinical practice guidelines are designed to improve and standardise patient care based on systematic reviews of evidence and expert consensus. Adherence to guidelines is increasing tied to reimbursement for clinical services and determination of quality of care. Unsurprisingly, practice guidelines are often widely distributed, downloaded, read, discussed, and cited. However, guidelines are imperfect and may lack integrity and quality, and contain low levels of evidence, recommendations discordant with evidence, and redundancy. Recent experience during the COVID-19 pandemic has also highlighted the need for guidelines to be adaptable yet rigorous. To improve this aspect, clinical practice guidelines should be graded and registered, and efforts should be made to combine resources when feasible. Clinicians should understand that guidelines are only as good as the evidence behind them and the people who make them. Surprisingly, standardised criteria for clinical practice guideline development do not currently exist, and methodology and approaches for development can vary widely. Issues regarding their variable quality have been reported for years and include insufficient levels of evidence for formal recommendations, lack of inclusion of key stakeholders, lack of independence leading to bias, and poor applicability. 2 Although this should have improved with the call to action by the WHO, the Institute of Medicine, and Guidelines International Network, recent studies have shown that clinical practice guidelines in anaesthesia are frequently limited by conflicts of interest, are based on low levels of evidence, and often the level of evidence and strength of corresponding recommendations are discordant. Recent discussions have highlighted other forms of guidelines as necessary to address the speed and volume of new information resulting from global health emergencies, such as the COVID-19 pandemic, that require rapid dissemination of urgent guidance, often despite lack of robust evidence. Table 1 describes current advantages and disadvantages of various guideline formats. Although the focus of this editorial is on academic-led guidelines, the issues raised may also be highly relevant to government and regulator-led guidelines, which are often less evidenced based and lack rigorous development processes. In the UK, the National Institute for Health and Care Excellence (NICE), a government organisation that publishes online recommendations for clinical practice, has built considerable influence. Its guidance is not based on standard guideline development and is published online only. In addition, there is limited opportunity to challenge or report concerns on controversial statements. Outside of peer-reviewed/academic guidelines, it is important to note that many governments or societypublished online guidelines exist with these additional limitations. What is wrong with guidelines? Editorial independence during guideline synthesis is a key quality metric. 3 In the context of increasing conflicts of interest in medicine, many organisations, including the American College of Physicians, US Preventive Services Taskforce, and UK NICE, require anyone involved in clinical practice guideline development to disclose all financial/intellectual conflicts of interest in the preceding 3 yr according to their magnitude: high (active relationships with direct financial stakes in the guidelines), moderate (intellectual interest that may profit from the guidelines), and low (inactive conflicts of interests peripherally associated with the guidelines). 4 In 2011, the National Academy of Medicine report Clinical Practice Guidelines We Can Trust recommended that all financial conflicts be disclosed, that only a small number of authors have conflicts, and that chairpersons should not have any significant financial and academic conflicts of interest. 5 Given their high citation rates, participation of journal editors in guideline preparation creates another potential conflict of interest. Unfortunately, studies in the USA, Japan, and Australia have found that more than 70% of clinical practice guideline authors have financial conflicts with large discrepancies between publicly reported and self-disclosed conflicts of interest, and that chairpersons often have significant potential financial conflicts of interest. 6 Finally, it is worth mentioning that the author list is not always reflective of relative contributions or influence, and valuable input from reviewers often goes unrecognised. These hidden contributors may also need to display transparency and disclose appropriate conflicts of interest. In addition, industry influence cannot be fully excluded based on statements of individual conflicts of interest. The influence of pharmaceuticals on guidelines produced by the American Pain Society preceding the opioid crisis is a case in point. Conflicts of interests are often grey areas. These statements are self-reported declarations, which remain undefined by journals and the wider scientific community. Some societies have attempted to define 'significant' interests in a landscape where it is increasingly difficult to be free from all conflicts. For example, clinicians or researchers especially knowledgeable on a topic may be more likely to be approached by companies for paid consultation. Usually only 'moderate or high' levels of conflicts are considered significant and undesirable in guideline development. However, it remains at the discretion of guideline committees and societies to appropriately screen and avoid contributory conflicts of interest. It is impossible to ensure that authors do not intend to benefit from guideline development that would create future conflicts of interest. The guideline in question remains unaffected, although this may prohibit involvement in future guideline involvement. One way to avoid this may be anonymous guidelines published solely in the name of a society. This may have unintended consequences, however, allowing authors to hide their prior conflicts of interest and cannot be recommended. We suggest instead that for the process to be transparent, a third-party regulator or administrator should examine all potential conflicts of interest and grade the level of conflict before finalising involvement in the guideline development group. Another key quality indicator of clinical practice guidelines is the strength of the underlying evidence. Assessing the level of evidence is more complex than might be expected. Commonly used tools for grading evidence include the Grading of Recommendations Assessment, Development and Evaluation (GRADE) 7 and the American College of Cardiology/American Heart Association (ACC/AHA) classification. 8 These instruments categorise evidence to Levels A, B, and C, with Levels A and B based on prospective studies and Level C representing more subjective sources, such as clinical experience, case reports, and expert opinion. Systematic reviews, meta-analyses, and RCTs are considered the highest level of evidence but can be of variable methodological quality. The number of systematic reviews and metaanalyses has increased nearly five times in the last decade compared with clinical trials, which have grown only of the 55% during the same time period. 9 If best practices are not followed, many reviews may be biased and flawed, with questionable and often conflicting conclusions. Only 3% of meta-analyses produced represent high-quality work. In addition, meta-analyses of small trials are subject to publication bias and may overestimate treatment effects that are later disproved by large trials. If incorporated into guidelines, these reviews can affect the validity of recommendations. Inclusion of methodological experts as a gatekeeper is critical to maintaining guideline integrity. Assessing RCT level of evidence can also be challenging. Using GRADE, studies can be upgraded or downgraded depending on the methodological rigour. 7 Small RCTs and single-centre RCTs may overestimate treatment effects because of unintentional investigator bias, and RCTs using composite outcomes to decrease sample size may have clinically less relevant outcomes 'drive' the composite outcome. Superiority trials with high crossover rates or protocol deviations may bias the trial towards the null hypothesis. In contrast, large observational studies, representing a more 'real-world' clinical setting and providing temporal trends in disease prevalence or interventions, may have significant value and may be upgraded in quality of evidence. Therefore, it takes experts in trial methodology, digging deep into the details of individual trials, to grade levels of evidence accurately. A recent systematic review evaluating the levels of evidence of North American and European perioperative care clinical practice guidelines in the last 10 yr found that a minority (16%) of recommendations were supported by Level A evidence, whereas 51% were supported by Level C evidence. 10 More Another problematic issue with clinical practice guidelines is the correct translation of existing evidence into recommendations. Different tools have been introduced to evaluate this aspect, with the Appraisal of Guidelines, Research and Evaluation (AGREE) and the GRADE instruments being the most commonly utilised. The GRADE system defines how evidence should be translated into treatment recommendations and provides an explicit evaluation of the comprehensive criteria for downgrading and upgrading quality of evidence ratings. 12 The validity of GRADE is supported by its endorsement by organisations, such as the WHO, the American College of Physicians, the American Thoracic Society, and the Cochrane Collaboration. Although the GRADE system has not been universally accepted, it has frequently been used to assess the quality of clinical practice guidelines in anaesthesia. In an analysis of 681 recommendations from 15 clinical critical care guidelines published from 2011 to 2017, Sims and colleagues 13 found that amongst 215 Class 1 recommendations, 69 (32%) were discordantly paired with evidence other than Level A or B. Despite GRADE recommendations based on low-level evidence, 22 of 69 recommendations (32%) were based on expert consensus, and 47 (68%) of the guidelines were of low or very low quality. 13 The AGREE instrument, updated to AGREE II in 2011, is arguably the gold standard for clinical practice guideline development, reporting, and evaluation. 14 It has six domains, with Domain 3, Rigour of Development, regarded as the single most influential domain and most reflective of guideline quality. Several studies have used the AGREE II instrument to evaluate clinical practice guideline quality in anaesthesia. A study using AGREE II to access the quality of 96 guidelines in anaesthesia published between 2013 and 2018 found that 74% of them had low overall quality scores, and only 26% (25 out of 96) were of high quality. 15 A recent review used the AGREE II instrument to analyse the quality of clinical practice guidelines from all anaesthesia subspecialties from 2016 to 2020 and reported an increased overall quality of guidelines, mainly driven by Domain 3. The absolute score of Domain 3, however, remained low with only 13.7% of clinical practice guidelines deemed to be of high quality. Furthermore, eight subsections of Domain 3 demonstrated a variability in scores with quality assessment of the underlying evidence base having the lowest scores. 16 Future validated Domain 3 centric scoring systems may improve the accuracy of quality assessment and make AGREE II an even more reliable tool for clinical practice guideline evaluation. A need for improved methodological approaches to anaesthesia guideline development is evident. Use of AGREE II together with the GRADE tool offers a potential solution. Mandating use of these tools together in guideline formation could lead to more-transparent high-quality documents. A recent review described high-quality guidelines as largely collaborative, with 78% having international involvement and 100% multi-institutional involvement. 16 International and institutional cooperation should be further emphasised in anaesthesia guideline development and also reduction in duplication of efforts. The ADAPTE, A Guideline Adaptation and Implementation Planning Resource (CAN-IMPLEMENT), and Grading of Recommendations Assessment, Development and Evaluation e Adaptation, Adoption, De Novo Development (GRADE-ADOLOPMENT) documents may play a role, having been designed for the purpose of efficiently developing and adapting clinical recommendations. 17 In addition, a standardised registry of clinical practice guidelines, similar to the International Prospective Register of Systematic Reviews (PROSPERO), should be implemented. Input from organisations, such as the WHO, Guidelines International Network, and Cochrane Collaboration, would be instrumental in supporting this process. Clinical practice guidelines undergo a standard process of peer review before publication. However, journals do not comment Table 2 Recommendations to improve clinical practice guideline development in anaesthesia. AGREE, Appraisal of Guidelines, Research and Evaluation; GRADE, Grading of Recommendations Assessment, Development and Evaluation. Use of both AGREE II and GRADE during anaesthesia guideline development and assessment Standardisation of AGREE II scoring with higher weighting assigned to Domain 3 International, multi-institutional cooperation in anaesthesia guideline formulation Mandated registration of clinical practice guidelines Publication of objective 'quality mark' for readers by journals Recognition of reviewers and their conflicts of interest, and disclosure of society funding of guidelines Third party to examine and screen significant conflicts of interest before guideline development Use of focused clinical practice guidelines, rapid statements, and living guidelines appropriately for rapidly developing crises, such as COVID-19, and in areas of anaesthesia that require frequent updating on the quality of the guidelines, and publication in a specific journal does not always imply quality, particularly if the editorial team is involved in their development. Publication of an objective marker of guideline performance by the journal, such as AGREE II, along with the guideline itself should be considered to provide readers with a recognisable 'quality mark' reflecting the underlying integrity of the guideline. This does not imply that guidelines lacking this quality designation are inherently flawed, but that their recommendations should be viewed with some caution and with the understanding that there are grey areas in what is considered 'best practice'. Similarly confusing for the reader are the various names and types of guidance documents (Table 1) . Journals should define the type of guidance document that is being presented because titles can be misleading. Focused clinical practice guidelines The current process of development and publication of clinical practice guidelines can take up to 2 yr. For COVID-19, the pandemic is just 2 yr old and hundreds of recommendations have been made. According to AGREE II, it is recommended that clinical practice guidelines are updated every 5 yr. For the COVID-19 pandemic, however, most guidelines have been based on low-level evidence and modified accordingly as higher-level evidence comes in, often reversing initial recommendations. In this context, updating clinical practice guidelines every 5 yr is inadequate. Conversely, for some recommendations, the evidence may not change significantly over time and updating them unnecessarily may waste resources. 18 In anaesthesia, the European Journal of Anaesthesiology (EJA) recently introduced a new format to streamline guideline updates, 'the focused clinical practice guidelines'. 19 Focused clinical practice guidelines adhere to the same methodological structure as traditional guidelines, but they cover only a very specific topic and follow an expedited systematic review process. Therefore, they require less time from inception to publication, have a more efficient updating process, and have improved readability. 20 Whilst focused guidelines were initially considered of inferior quality, studies have found that there were no substantial differences in methodology, research strategy, and quality between traditional and focused clinical practice guidelines. 21 However, it should be noted that these may be much less comprehensive than traditional guidelines, and the cost of efficiency may be duplication and confusion: many focused guidelines covering similar material may complicate the message to the clinician. In addition, there may be value to approaching guidelines from the broad 'forest' viewpoint than from the 'trees' perspective. More and more specific guidelines on smaller topics published in subspecialty journals are contributing to this trend, and it remains unclear what the optimal balance between efficient and comprehensive is and the best size and scope of a clinical practice guideline. It should be noted that consensus statements exist widely to provide guidance in situations where there is clinical equipoise yet there is not enough evidence on a topic for systematic review and subsequent guideline development. The professional community should regard these as suggestions for practice rather than as binding recommendations. The EJA provided its take on consensus documents as 'rapid statements', in response to the COVID-19 health crisis emergency. These statements are less rigorous than clinical practice guidelines, but they are adaptable to the urgent needs arising from health crises, such as COVID-19, by aiming to disseminate the best available evidence as quickly as possible. 19 Rapid statements have been published prolifically during the COVID-19 pandemic, but many are of inferior quality and duplicative. 22 Strict standards should still be used in these statements. There is a growing body of literature detailing the application of GRADE and guidance for conducting rapid systematic reviews in the context of a pandemic. 23 Using high standards in the development of clinical practice guidelines may in fact be even more crucial in times of crisis, and rigour and speed should both be emphasised. The WHO recently introduced the concept of 'living clinical practice guidelines'. 18 This approach combines continuous literature surveillance, rapid and constant updating of prioritised systematic reviews, and virtual consultations with expert panels to ensure that the latest evidence and updated recommendations can reach health workers worldwide as quickly as possible. For COVID-19, two clinical practice guidelines, one for treatment and one for prevention, have been developed adopting this strategy. 24, 25 This concept has yet to be used in anaesthesia clinical practice guidelines, and a pertinent question remains regarding how and when updates should be made. Details on how living clinical practice guidelines should be incorporated in anaesthesia and what clinical questions would best suit this format depend on the rate of new information and interest in specific topics. The rate and degree of change should be set by (i) the rate of new evidence available and whether this evidence should change practice, and (ii) balancing the risks and benefits of frequent changes confusing readers. For medico-legal purposes and the evolving nature of practice recommendations, clear date stamps will be necessary to understand when a practice was recommended during previous versions of the guideline. Evidence, recommendations, and clinical practice guidelines exist in an imperfect world. However, there are clear steps to improve guidelines (Table 2) . A more formal process must be performed to translate these suggestions into working recommendations; however, the hope is that these concepts will stimulate further debate. The integrity of guidelines must be taken seriously, and their development should be free from conflicts of interest. GRADE and AGREE II should be used to determine the quality of guidelines, assess evidence, and translate them accurately into recommendations. The ideal timing for updating guidelines is unknown. High-quality RCTs take years to complete, and clinical practice guidelines, rigorously performed, can also take several years to be finalised. Although it is surprising that so many recommendations are based on low-level evidence, this does not necessarily mean they are untrustworthy. Many clinical questions can be initially addressed with well-performed observational studies. Guidance for clinical practice is necessary despite the limitations of available evidence, but guidelines should be flexible to accommodate new information. When RCTs become available, their findings must be taken into account, perhaps even outside the timeline of traditional clinical practice guidelines. The COVID-19 pandemic has demonstrated the need for living guidelines that balance speed and rigour in the face of mass dissemination of information. These guidelines should still adhere to strict methodological standards that minimise duplication. Finally, the process of grading clinical practice guidelines should be standardised, and clinicians should be aware that all guidelines are 'living' works in progress that have limitations and may be updated as additional evidence accrues. Challenges and opportunities facing medical education The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies The Lancet. Managing conflicts of interests in clinical guidelines Financial conflicts of interest and authorship of clinical practice guidelinesdtrust is fragile Committee on standards for developing trustworthy clinical practice guidelines. Clinical practice guidelines we can Trust Undisclosed financial ties between guideline writers and pharmaceutical companies: a cross-sectional study across 10 disease categories GRADE: an emerging consensus on rating quality of evidence and strength of recommendations Scientific evidence underlying the ACC/AHA clinical practice guidelines The mass production of redundant, misleading, and conflicted systematic reviews and metaanalyses Levels of evidence supporting the North American and European perioperative care guidelines for anesthesiologists between 2010 and 2020: a systematic review Levels of evidence supporting American College of Cardiology/American Heart association and European society of Cardiology guidelines Use of the GRADE approach in systematic reviews and guidelines Above the GRADE: evaluation of guidelines in critical care medicine AGREE II: advancing guideline development, reporting and evaluation in health care An appraisal of published clinical guidelines in anesthesiology practice using the AGREE II instrument Evaluation of the quality of clinical practice guidelines in anaesthesia publications 2016e2021 using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT Keeping global recommendations up-to-date: a 'living guidelines' approach to maternal and perinatal health Adapt or perish: introducing focused guidelines Evidence summaries: the evolution of a rapid review approach Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment Scope, quality, and inclusivity of clinical guidelines produced early in the covid-19 pandemic: rapid review Developing trustworthy recommendations as part of an urgent response (1e2 weeks): a GRADE concept paper A living WHO guideline on drugs for covid-19 A living WHO guideline on drugs to prevent covid-19 Difficult Airway Society guidelines for awake tracheal intubation (ATI) in adults European guidelines on perioperative venous thromboembolism prophylaxis: Patients with preexisting coagulation disorders and after severe perioperative bleeding European Guidelines on perioperative venous thromboembolism prophylaxis: Executive summary Update to living WHO guideline on drugs for covid-19 Controversies in airway management of COVID-19 patients: updated information and international expert consensus recommendations Expert consensus statement on the perioperative management of adult patients undergoing head and neck surgery and free tissue reconstruction from the society for head and neck anesthesia Prep, stop, block': refreshing 'stop before you block' with new national guidance Overview | Venous thromboembolism in adults | Quality standards | NICE Accountability for all aspects of the work: all authors. LQR is a member of the associate editorial board of the British Journal of Anaesthesia.Funding National Institutes of Health; National Heart, Lung, and Blood Institute (K23 HL153836-01A1) to LQR. Conception/design: LQR. Drafting/revising of paper: all authors. Final approval: all authors.