key: cord-0922652-eht4htd1 authors: Tricco, Andrea C.; Garritty, Chantelle M.; Boulos, Leah; Lockwood, Craig; Wilson, Michael; McGowan, Jessie; McCaul, Michael; Hutton, Brian; Clement, Fiona; Mittmann, Nicole; Devane, Declan; Langlois, Etienne V.; Abou-Setta, Ahmed M.; Houghton, Catherine; Glenton, Claire; Kelly, Shannon E.; Welch, Vivian A.; LeBlanc, Annie; Wells, George A.; Pham, Ba’; Lewin, Simon; Straus, Sharon E. title: Rapid review methods more challenging during COVID-19: commentary with a focus on 8 knowledge synthesis steps date: 2020-06-29 journal: J Clin Epidemiol DOI: 10.1016/j.jclinepi.2020.06.029 sha: bc6cb4babe440b229d046b2bcec395139e382ba5 doc_id: 922652 cord_uid: eht4htd1 nan COVID-19 has driven the need for timely evidence to inform decision-making. Rapid evidence products can be particularly helpful for decision-makers (e.g., citizens, patients, health care providers, policy-makers) during COVID-19. The main types of rapid evidence products include inventories, rapid response briefs, and rapid reviews [1] . In this article, we focus on rapid reviews, which are ''a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner [2] .'' In a rapid review, several mechanisms are used to streamline the methods, such as narrowing the scope of the topic, parallelization of tasks (e.g., conducting screening and data abstraction simultaneously), using review short cuts (e.g., one team member screens citations from the literature search versus two), and using automation (e.g., using computer software to rank the literature search results in terms of relevance) [3] . Guidance for the conduct of rapid reviews for health policy and systems research has been developed in collaboration with the World Health Organization (WHO) [3, 4] . The WHO guide recommends that researchers thoughtfully tailor their methods to the needs of decision-makers. The guide notes that there are several ways that rapid reviews can be streamlined to accommodate decision-makers' needs related to both the scope of the review and timeliness across all steps of the review process. In this article, we outline several challenges based on our collective experience conducting COVID-19 rapid reviews predominantly for health care provider and policy decision-makers. The types of rapid reviews that the Guidance is available on the conduct of rapid reviews. However, the COVID-19 pandemic has created several unique challenges. Challenges to the conduct of rapid reviews include the urgency of the request from decision-maker organizations, identification of and access to sources of evidence for inclusion in the rapid reviews, extrapolation of results from indirect evidence, and dissemination of results widely. There is a need for coordination of efforts internationally to reduce the risk of duplication, and to effectively use global collective evidence synthesis resources. We outline several methodological challenges to the conduct of rapid reviews that have become apparent during the COVID-19 pandemic using an 8-step framework that follows the knowledge synthesis process. We offer several suggestions to help address the methodological challenges encountered during the conduct of rapid reviews on COVID-19, as well as future research. authors have conducted vary in scope including public health measures, clinical management, health-systems arrangements, and economic and social responses [5] , providing a wide range of experiences to draw on. We specifically describe issues we have encountered throughout the rapid review process (Fig. 1) . The collective lessons we have learned through our collective experiences are also explored, as well as suggestions for future research. The conceptualization and scope of rapid reviews has been challenging during COVID-19. It is essential for researchers to involve decision-makers in the rapid review process to set and refine the review question, eligibility criteria, and the outcomes of interest [6, 7] . The pandemic and its implications are rapidly evolving, making it difficult to clearly articulate and finalize the research question for examination. In some instances, decision-makers have asked researchers to proceed with COVID-related priority rapid reviews and given rapid review teams flexibility to identify and prioritize specific questions based on local/national/international needs [8, 9] . In addition, decisionmakers require rapid reviews to be completed faster than everdoften within five to 10 days [2] and some within hours [10] . Other important stakeholders to partner with in the rapid review include clinicians and patients who can be involved with the entire rapid review process, and in particular, the selection of outcomes for inclusion in the review. It is important that rapid reviews address the need and include the views of front-line clinicians who are dealing with COVID-19. However, many clinicians are overwhelmed with clinical work due to COVID-19 and are unable to participate in scoping a rapid review as they might otherwise do. This might render the rapid review findings less relevant to clinical decision-making. Furthermore, in an era that has championed patients as partners in research, there is an imperative to involve patients and citizens as key stakeholders [11] . However, it is challenging to meaningfully engage patient partners within such limited timelines. Another problem is that there is little coordination of the evidence needs of decision-makers between local, provincial/regional, country and global levels, making it difficult for researchers, as well as research funders, to mitigate duplication [12] and research waste [13] . It is nearly impossible for a researcher to know whether the rapid review they are planning to conduct or were asked to conduct has already been completed. Indeed, there were 1,028 registered COVID-related protocols for systematic reviews of human studies in PROSPERO from March 1st to May 26th, 2020. Some organizations, such as the Cochrane Collaboration, are consulting with international partners (e.g., WHO) to prioritize rapid review topics [2] . In addition to posting topics, it is essential to register rapid review protocols to decrease duplication and provide a publicly available record of the rapid review. Initiatives involving several knowledge synthesis organizations internationally are working together to reduce duplication of rapid reviews [5, 14] , as well as provide a list of resources that are updated continuously (e.g., COVID-END). Use of guidance and tools on rapid review conduct [3, 15] can help researchers methodologically tailor the methods for their rapid reviews thoughtfully. Experienced rapid review teams facilitate review conceptualization, as well as the entire process in a short period. For example, experienced groups will know how many team members are required to complete the rapid review within a specific timeframe. Furthermore, experienced teams can make thoughtful decisions about the scope of the project and whether it is important to include types of studies (e.g., randomized trials, systematic reviews, guidelines) or sources of evidence (e.g., websites, organizational policies and procedures, news sources, social media) to ensure the review is feasible and relevant. Such teams will also know when it is appropriate to limit the number of outcomes using COVID-19-specific core outcome sets [11] . Teams can make use of collaborative tools, such as online meeting platforms and email, to interact with decisionmakers to help conceptualize the rapid review. Some teams consult with experts to provide their insight on contextualizing the rapid review findings via quick telephone calls [16] at the end of the review. The Cochrane COVID-19 consumer rapid response group is supporting teams and consumers to collaborate in the time available for completing a rapid review [17] . The process of involving consumers' needs to be practical, articulated clearly, and in line with the rapid review timelines. Further research is required on how to meaningfully prioritize topics and outcomes included in rapid reviews, as well as engage a range of stakeholders in rapid reviews to ensure all perspectives are included. A rapid review should include a comprehensive literature search [3] of multiple bibliographic databases (e.g., MEDLINE, Embase) [18] . However, searching the gray literature (i.e., unpublished and difficult to locate material) has become increasingly important because emergent literature on COVID-19 is scattered across numerous sources, such as websites, public health guidelines, organizational policies and procedures, news sources, social media, clinical trials, COVID-19 repositories, and preprint servers (e.g., medRxiv). Searching these sources is complex because of a lack of indexing and poor functionality of the search interfaces. The data are geographically diverse and often written in languages other than English. As researchers are increasingly using alternative methods of sharing their research findings (e.g., ResearchGate, Linke-dIn, preprint servers), librarians and information specialists must in turn adapt their typical search methods to account for this shift and may need to rapidly develop new skills. COVID-19 repositories and research/resource guides with lists of traditional and gray literature sources (e.g., WHO COVID-19, LitCovid, COVID-END) can be used to ensure the rapid review includes relevant studies that may not be captured by electronic databases [19] . For COVID-19 rapid reviews, studies in all languages should be considered for inclusion, which requires access to quick and trusted translations; membership in international networks such as Cochrane could help facilitate this. Teams can consider contacting their librarian as soon that they know a rapid review is on the horizon so that they can plan the literature search to meet the quick timeline. As well, based on our collective experience, some teams have prioritized specificity rather than sensitivity to make the literature searches more manageable for COVID-19 rapid reviews. Updating the literature search the same week as the rapid review becomes publicly available is one approach that ensures rapid reviews are up to date. The development and validation of a COVID-specific gray literature search will facilitate greater efficiency of literature searches for rapid reviews. A study on the trade-offs between taking the time to search for and appraise the quality of studies identified through alternative sources of evidence (i.e., preprints) and the impact on decision-making would provide useful information for rapid review conduct. Teams may have to consider which way to streamline the screening and data abstraction methods for the rapid review conduct. It is unclear what the impact of using several different approaches such as semi-automated screening tools (e.g., EPPI Reviewer), crowdsourcing [20e25], or having only one person involved with the screening [26] may have on results of the rapid review [27, 28] . Similarly, it is unclear whether certain data extraction tools (with or without data mining features) are accurate and reliable. Related to appraisal of methodological limitations, the literature included in COVID-19 rapid reviews may be of lower methodological quality due to the rapid nature that the primary studies themselves have been conducted. No matter what approach, methods must be transparently reported and limitations need to be discussed. Some review groups are conducting data abstraction across multiple team members with live (synchronous) sharing of data or using crowdsourcing approaches. As well, the use of online software (e.g., DistillerSR, Covidence) and parallelization of tasks will make rapid reviews more efficient. Appraising the methodological limitations takes time yet can be incorporated into applying GRADE (or GRADE CERQual) of the evidence [29] . This provides an indication of how trustworthy the rapid review results are and might be particularly important for COVID-19 rapid reviews, which often rely on nonepeer-reviewed sources. However, more time might be required to appraise evidence from nonepeer-reviewed sources. Some teams are limiting methodological assessments to only studies that are included in the analysis (whether qualitative or quantitative) to make the review more feasible. More research is required on whether the time tradeoff outweighs the potential of introducing error for various technological solutions for screening, data abstraction, and assessment of methodological limitations, such as crowdsourcing automation. This is particularly important for reviews that include many different types of study designs (e.g., qualitative and quantitative evidence) and sources of evidence (e.g., gray literature). The impact of including nonepeer-reviewed sources (e.g., preprints) on the feasibility of the rapid review can be explored. Because COVID-19 is an emerging disease, researchers may include indirect evidence from other pandemics or respiratory illnesses. However, it is challenging to extrapolate these findings to COVID-19, impacting interpretation of results. Furthermore, the outcomes examined in the primary studies included in the rapid review may vary, contributing heterogeneity and making any statistical pooling (e.g., meta-analysis) inappropriate. In addition, researchers are working at an incredibly fast pace and this makes it more challenging to interpret results in a thoughtful manner. The rapidity also makes it challenging to include the interpretation of results from all decision-makers. The interpretation of results needs to carefully consider any streamlined methods that were used. Researchers should be specific and transparent about what might have been lost in the process and what needs to be addressed in the future, perhaps through a more comprehensive review, and when such a review should be performed. If a meta-analysis was not feasible, it is important to report effect sizes with confidence intervals. In qualitative synthesis, it may not be possible to conduct subgroup analyses but this can be addressed in future updates. Some teams provide decision-makers with summary of findings tables without a descriptive writeup of results to facilitate completion in a shorter period. Working closely with decision-makers to interpret the rapid review results will ensure that the end product is relevant and fit for purpose (e.g., consulting experts to provide evidence contextualization at the review completion). Evaluation of technologies that automatically chart and summarize the content of included studies (e.g., technologies used to create a dashboard of research) [30] . 2.5. Dissemination of COVID-19 rapid reviews 2.5.1. Challenges The up-to-the-minute nature of the COVID-19 pandemic means that the traditional academic publishing model cannot keep up with the wave of evidence being produced. And decision-makers cannot wait for the rapid review to be published. At the same time, findings need to be presented in a complete and unbiased way and in a format that is clear to facilitate uptake of results. Developing a thoughtful plan for dissemination is even more challenging when the timeline is reduced leaving little time for communication or additional tasks. To quickly disseminate rapid review findings, researchers can consider other mechanisms, such as the Open Science Framework [31] , Zenodo [32], or preprint servers. Use of short (e.g., 1-page) evidence summaries can [33, 34] facilitate uptake of results with key messages highlighted upfront for the end user. Considering targeted dissemination mediums, such as infograms, podcasts, YouTube, Link-edIn, Twitter, ResearchGate, and media releases, might be required for dissemination above and beyond publishing in a peer-reviewed journal. Linkages with teams of data mobilizers and academic detailers, as well as communication teams, can facilitate dissemination of results. Use of evidence-informed dissemination strategies should be considered to ensure wide uptake of results [35] . Evaluation of the impact of rapid review evidence in the decision-making process during COVID-19 through quantitative (e.g., surveys) and qualitative (e.g., interviews, focus groups) research including an array of decision-makers (e.g., citizens, public, health care providers, policy-makers). 2.6. Updating COVID-19 rapid reviews 2.6.1. Challenges Decision-makers are requesting rapid reviews that are updated on a continuous basis (i.e., living reviews [18] ), which is an emerging methodology. For example, it is unclear how often the literature should be searched and when the optimal timing of updates, or full reviews, should take place. Communication of updates regarding how the results and conclusion have changed is also challenging. As well, many teams do not have funding to conduct continuous updates of rapid reviews, creating a sustainability issue. Some teams are making use of automation in searching and screening to convert their rapid reviews into living rapid reviews. As well, some teams are working with their decision-makers to reconsider funding structures to allow living rapid reviews to be conducted on an ongoing basis during COVID-19. Organizations such as Cochrane and the Campbell Collaboration have processes in place for the regular updates of published reviews. This may be particularly helpful for authors of rapid reviews that may want to return at a later stage to continue their review. It is unclear whether some topics are more amenable to a living rapid review than others. In summary, we have outlined several methodological challenges that have arisen during the conduct of COVID-19 rapid reviews. Although rapid reviews are being conducted quickly, we must use thoughtful methods and transparent reporting of our products [3, 36] . It is hoped that the suggestions presented here are helpful for the conduct of COVID-19 rapid reviews and can be used in future times of disaster and emergency. EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews Cochrane's work on Rapid Reviews in response to COVID-19. Cochrane Rapid reviews to strengthen health policy and systems: a practical guide. Geneva: World Health Organization Using rapid reviews to strengthen health policy and systems and progress towards universal health coverage COVID-19 evidence network to support decisionmaking Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review Engaging knowledge users in a systematic review on the comparative effectiveness of geriatrician-led models of care is possible: a cross-sectional survey using the Patient Engagement Evaluation Tool Available at https:// www.mcmasterforum.org/find-evidence/products?ProductTypes5Rapid %20evidence%20profile Strategy for Patient-Oriented Research. Canadian Institutes of Health Research One more time': why replicating some syntheses of evidence relevant to COVID-19 makes sense Open synthesis: on the need for evidence synthesis to embrace Available at https:// covidrapidreviews.cochrane.org/resources COVID-19 consumer rapid response group Cochrane Handbook for Systematic Reviews of Interventions version 6 Guide to COVID-19 evidence sources Quality control for crowdsourcing citation screening: the importance of assessment number and qualification set size Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial Decontaminating N95 masks with Ultraviolet Germicidal Irradiation (UVGI) does not impair mask efficacy and safety: A Systematic Review. Open Science Framework (OSF) Microwave-and Heat-Based Decontamination of N95 Filtering Facepiece Respirators: A Systematic Review 2020 Decontamination Interventions for the Reuse of Surgical Mask Personal Protective Equipment: A Systematic Review 2020 Efficacy and Safety of Disinfectants for Decontamination of N95 and SN95 Filtering Facepiece Respirators: A Systematic Review 2020 A full systematic review was completed in 2 weeks using automation tools: a case study Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review Rapid reviews may produce different results to systematic reviews: a Meta-Epidemiological Study COVID-19 primer How to write a plain language summary of a Cochrane EPOC Qualitative Evidence Synthesis Launch of Cochrane's dissemination Checklist and guidance Kappa opioid receptor-mediated depression of activity evoked in convergent dorsal horn cells by thermal and non-thermal noxious stimulation The authors thank Shazia Siddiqui for her help with formatting the paper and compiling the ideas from the coauthors. Supplementary data to this article can be found online at https://doi.org/10.1016/j.jclinepi.2020.06.029.