key: cord-0877128-tdrfcyhk authors: Schünemann, Holger J.; Santesso, Nancy; Vist, Gunn E.; Cuello, Carlos; Lotfi, Tamara; Flottorp, Signe; Davoli, Marina; Mustafa, Reem; Meerpohl, Joerg J.; Alonso-Coello, Pablo; Akl, Elie A. title: Using GRADE in situations of emergencies and urgencies: Certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what date: 2020-06-06 journal: J Clin Epidemiol DOI: 10.1016/j.jclinepi.2020.05.030 sha: 471b6fc6e0b57f0af67dcc5077e428008863c8d9 doc_id: 877128 cord_uid: tdrfcyhk ●. GRADEing the certainty of the available evidence is more important than ever because of the unprecedented pressure for action and the large number of people affected by decisions; ●. The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time; ●. In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence; ●. Requirements for emergency, urgency, rapid and routine GRADE assessment may differ but should transition from one to another. • GRADEing the certainty of the available evidence is more important than ever because of the unprecedented pressure for action and the large number of people affected by decisions • The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time • In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence • Requirements for emergency, urgency, rapid and routine GRADE assessment may differ but should transition from one to another The public, policy makers, and science communities are subject to many false, uninformed, overly optimistic, premature or simply ridiculous health claims. The Coronavirus disease 2019 (COVID- 19) pandemic and its context is a paramount example for such claims: from media and politician's attention to biased interpretation of case series of patients on hydroxychloroquine to the injection of disinfectants and use of azithromycin [1] . Yet, for some interventions, such as personal protective equipment and social distancing, there is an accumulating body of evidence in favor of their benefits [2] . To monitor misleading and appropriate claims and separate biased from unbiased research, understanding and expressing the certainty in effects of these and other clinical, public health or health policy interventions is critical to weed out misleading or wrongful claims. It is more critical than in other situations because policy-makers are under unprecedented pressure to react to claims and make decisions with varying degrees of certainty in the evidence and timing of their responses. And it is especially critical in this era because of the public access to information and expectation of a timely response. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach provides a transparent and structured approach to making judgments about the certainty of the evidence, and offers a transparent process to making recommendations and decisions [3] [4] [5] [6] . GRADE is the product of an open and inclusive community of people that has collaborated for 20 years (www.gradeworkinggroup.org) and is currently used by over 100 organizations globally, including the World Health Organization. The GRADE Working Group has used a carefully designed, rigorous, transparent and inclusive process based on cumulative evidence about research methods, bias and decision-making. GRADE offers solutions to the dilemma of expressing certainty in a body of evidence, which would have been very low certainty in efficacy of hydroxychloroquine because of imprecision and the non-randomized designs used [7] , and at least moderate certainty in physical distancing based on other indirect evidence from viral respiratory diseases research [2] . GRADE also offers a structured and transparent framework for making decisions, including recommendations, that should prove particularly helpful during the ongoing COVID-19 pandemic. In this article, after having made the point why we think expressing certainty in evidence to support a decision is critical above, we describe why GRADE matters now, perhaps more than ever and no matter what the specific topic is in relation to COVID-19. We finally also offer suggestions for how it can be used appropriately to support decision-making at global, national and local level. Decision makers should, and many are (!), asking 'what is the science?' and 'how good is this test or this intervention?'. GRADE, although appropriately sophisticated in its full execution, can answer these questions and be relayed to decision-makers by breaking its components down into straightforward questions about: 1) the certainty of evidence, and 2) the criteria for making decisions or recommendations. While ideally applied to rate the certainty of a body of evidence in a well-conducted and up-to-date evidence synthesis relevant to the question at hand (in terms of setting, population, intervention, comparator, and outcomes) with corresponding summary tables, such as evidence profiles; GRADE's application requires at least that "the evidence that was assessed and the methods that were used to identify and appraise that evidence should be clearly described" [4, [8] [9] [10] . Those providing and using evidence should ask the questions in box 1 to understand if they can be certain about the effects of an intervention (including tests, public health strategies or other options being considered), regardless of the time available. Based on the answers to these questions, an expression of certainty in the body of evidence can be articulated (GRADE uses four levels of certainty: high, moderate, low and very low) [4] . -Are the results precise enough or likely due to chance? -Is this all the research that has been conducted on the PICO question of interest? -Is there anything, in particular very large effects of an intervention, dose response gradients or unfavourable scenarios still leading to convincing effects, that makes us more confident? The questions proposed by GRADE to guide decision makers, including those formulating recommendations (and the recipients of them), are equally simple (Box 2), and can be tailored to the type of decision (clinical recommendations, public health recommendations/decisions, or health system recommendations/decisions) [11, 12] . Box 2. GRADE questions when recommending for or against an intervention or strategy -Are the expected health benefits greater than the harms or vice versa (this integrates considerations about the priority and severity of the problem, intervention effects, the values people place on the outcomes as well as the certainty in the effects)? -What is the magnitude of the resource requirements (and associated cost) related to the intervention/strategy and is it cost-effective? -What is the impact of the intervention/strategy on equity, including societal implications and environmental impact? -Is the intervention/strategy acceptable to different stakeholders (this criterion includes ethical and other considerations)? -Is the intervention/strategy feasible (this criterion includes health system, social, legal, political and other considerations)? Emergency, urgent, rapid and routine use of GRADE to assess the certainty of evidence Early in the COVID-19 pandemic it became evident that information is required with different levels of urgency. We previously described these different levels of time-based responses using GRADE (Figure 1 ), and categorized the use of GRADE to assess the certainty in: 1) emergency responses or in an ultra-short time frame of hours; 2) urgent responses, allowing one to two weeks to respond; 3) rapid responses, in up to three months; and 4) routine responses, beyond three months [10] . An example of the first scenario, i.e., providing GRADEd evidence within hours, is when it became apparent that a viral outbreak was the likely cause of what is now known as COVID-19. Under those circumstances, addressing the value of use of masks as personal protective equipment (PPE) or distancing, within hours became an emergency question to be answered. [13, 14] . Similarly, one would have high certainty that a large distance from the source is effective, yet the certainty in how much distance to keep, e.g. 1 meter or 2 meters or more, would be rated down for that same indirectness. For the second scenario of using GRADE with urgency (one to two weeks), the COVID-19 pandemic has made it clear that systematic reviews can be conducted to respond to urgencies (defined as responding within two weeks) without "cutting corners", for which rapid reviews have been criticized [15, 16] . To illustrate this point, we present three standard urgent systematic reviews that we have conducted within 7 to 11 days each, two of them in parallel and one including many complex meta-analyses, including Bayesian approaches and a metaregression. The three reviews addressed five questions on the use of masks, eye protection, and physical distancing; use of non-invasive ventilation; and handling of bodies of deceased individuals [17] [18] [19] [20] . We produced five GRADE evidence profiles with a rating of the certainty after reviewing over 80,000 citations, and including over 70 studies. The review on the use of masks and physical distancing featured studies directly addressing COVID-19 and provides low to moderate certainty about large effects, and high certainty for any observed reduction in transmission by physical distancing although the exact effect is only of moderate certainty. Thus, for a situation of urgency and beyond, GRADE assessments based on systematic reviews are possible, albeit in the hands of a large experienced systematic review team. Contrary to what some believe, however, applying GRADE does not add significant time to the systematic review process. The time required to produce evidence profiles and add appropriate ratings of the certainty was approximately one hour. We believe the investment of time has paid off. In the earlier phases of the review we saved time through streamlining the evidence assessments, structuring the questions and making decisions about what evidence to search for to address relevant outcomes as suggested by GRADE [9] . For example, to address the risk of COVID-19 transmission, we realized that studies addressing this risk during aerosolizing procedures such as non-invasive ventilation required a search for a different type of evidence, including mechanical and laboratory studies. While we conducted meta-analyses for many of our outcomes, something that it is often not possible, GRADE can be applied to narrative summaries of the evidence, and still provide guidance with informative statements about the findings [10, 21] . For the third scenario of using GRADE within three months, we previously described the use of GRADE using a framework for developing rapid recommendations, i.e. in up to three months [22] . In 2007 an expert committee at the World Health Organization (WHO) developed 23 separate recommendations, and a number of research recommendations for or against the pharmacological treatment of avian influenza [23] . Avian influenza was a serious public health threat of a pandemic at that time that, however, did not emerge, leaving more time to act compared with COVID-19 [23] . In addition to reviewing human studies, it included a review of mechanistic and animal research evidence. For COVID-19 addressing the question about hydroxychloroquine, remdesivir, antibody testing and other pharmacological interventions, rapid reviews (and subsequent monitoring) allow accumulation of evidence emerging during an outbreak for treatment, interventions, or strategies that may or may not be providing net benefit [24] . The accumulating evidence about mask use and physical distancing demands transitioning from urgent answers to rapid and routine answers, where new evidence is integrated, perhaps in living reviews, to provide best evidence for longer-term decisions [25, 26] . Indeed, there will be answers to COVID-19 related questions that may need, perhaps after provisional urgent evidence assessments, routine monitoring using GRADE beyond three months. This may include addressing rehabilitation needs for patients suffering respiratory or neurological consequences or the impact of telemedicine on patient outcomes and health care utilization after COVID-19 infection. In a partnership with the Norwegian Institute of Public Health (NIPH) (https://www.fhi.no/en/qk/systematic-reviews-hta/map/) and authors' institutions, we are making risk of bias assessments of individual studies and ratings of the certainty of a body of evidence for questions related to COVID-19 increasingly available which can be used globally by the evidence synthesis community, including Cochrane rapid COVID-19 reviews [27] . One of the goals is avoidance of duplication. Using our rationale above, to optimally inform the public, it is inappropriate for politicians and organizations to not transparently convey the certainty narratively or provide ratings of it. Policies have to be made regardless of the type of evidence that exists, even if there is low or very low certainty in the exact effect of an intervention or strategy, such as for the urgent answer about masks for the public. Those using GRADE are then often asked if policy makers do not shy away from making recommendations based on low or very low certainty evidence, and if they will be hesitant to communicate uncertainty of the evidence to the public. An adequate response is that evidence and studies are never perfect, and that there is always uncertainty To develop guidance, tools exist [28, 29] , and if used appropriately in the context of GRADE, we can inform decision makers appropriately and convey our certainty in any recommendation that evolves, on an emergency basis or over longer periods of time. When a structured and transparent approach is used, it facilitates the sharing of information and an understanding of Highlights • GRADEing the certainty of the available evidence is more important than ever because of the unprecedented pressure for action and the large number of people affected by decisions • The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time • In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence • Requirements for emergency, urgency, rapid and routine GRADE assessment may differ but should transition from one to another Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an openlabel non-randomized clinical trial Physical distancing, face masks, and eye protection to prevent person-person COVID-19 transmission: A systematic review and metaanalysis. The Lancet Grading quality of evidence and strength of recommendations GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables GRADE: an emerging consensus on rating quality of evidence and strength of recommendations Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations Virological and clinical cure in COVID-19 patients treated with hydroxychloroquine: A systematic review and meta-analysis GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes Using GRADE to respond to health questions with different levels of urgency GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction The GRADE Evidence to Decision (EtD) framework for health system and public health decisions Physical interventions to interrupt or reduce the spread of respiratory viruses GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions Evidence-based clinical practice guidelines for prostate cancer: the need for a unified approach Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews Reviews: Rapid! Rapid! Rapid! ...and systematic. Systematic reviews and the COVID-19-SURGE. Physical distancing, face masks, and eye protection to prevent person-person COVID-19 transmission: A systematic review and meta-analysis Ventilation techniques and Risk of Transmission for coronavirus disease including Covid-19: A living systematic review of multiple streams of evidence Safe management of bodies of deceased persons with suspected or confirmed COVID-19: a rapid systematic review Rating the certainty in evidence in the absence of a single estimate of effect Transparent development of the WHO rapid advice guidelines WHO Rapid Advice Guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus Comparative effectiveness of pharmacological interventions for Covid-19: a living systematic review and network meta-analysis Non-invasive and invasive ventilation for patients with COVID-19 and the risk of virus transmission to health care workers from aerosol generating procedures: An urgent systematic review of multiple streams of evidence for the World Health Organization Cochrane COVID Rapid Reviews website and Question Bank Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations Development of rapid guidelines: 1. Systematic survey of current practices and methods GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT Adolopment -a new term added to the Clinical Epidemiology Lexicon Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients Uncertainties in baseline risk estimates and confidence in treatment effects GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness nfectious Diseases Society of America Guidelines on the Treatment and Management of Patients with COVID-19 Surviving Sepsis Campaign: Guidelines on the Management of Critically Ill Adults with Coronavirus Disease 2019 (COVID-19) SURGE C-. Ventilation techniques and Risk of Transmission for coronavirus disease including Covid-19: A living systematic review of multiple streams of evidence Safe management of bodies of deceased persons with suspected or confirmed COVID-19: a rapid systematic review This article does not necessarily represent official views of the GRADE working group but describes the application of GRADE.