key: cord-1014135-c55khwym
authors: Hunter, Jennifer; Arentz, Susan; Goldenberg, Joshua; Yang, Guoyan; Beardsley, Jennifer; Lee, Myeong Soo; Myers, Stephen P.
title: Choose your shortcuts wisely: COVID-19 rapid reviews of traditional, complementary and integrative medicine
date: 2020-07-29
journal: Integr Med Res
DOI: 10.1016/j.imr.2020.100484
sha: 6257e92e7ef074cc0790efd172aab2a85eafa053
doc_id: 1014135
cord_uid: c55khwym

The COVID-19 pandemic has led to an explosion of rapid reviews geared towards providing time-sensitive answers for clinical and policy decision-makers. Rapid reviews strike a balance between rigour and rapidity to minimise bias and optimise transparency within specified constraints. Substantive guidance on how to strike that balance and undertake rapid reviews is readily available. Restrictions can be applied to the scope (PICO questions), methods (search strategy, inclusion criteria such as language, screening, analysis or synthesis) and reporting of results in order to prioritise speed. We propose that specific research disciplines, such as traditional, complementary, and integrative medicine (TCIM), have a uniqueness that may lead to unacceptable outputs if minimum methodological standards are not applied to rapid reviews. We recommend the minimum requirements to optimise the credibility of rapid reviews of TCIM.

The purpose of this paper is to clarify the minimum standards required for rapid reviews (RRs) of traditional, complementary and integrative medicine (TCIM) in the context of the COVID-19

pandemic. The intention is to guide researchers in TCIM to ensure that new RRs are of a minimum acceptable quality to maximise their credibility and impact.

During a global health crisis, such as the COVID-19 pandemic, RRs are needed to provide up to date summaries of important evidence. The methods for RRs are used as an alternative to the 'gold standard' systematic review (SR) methods when there are legitimate time, resource or other logistical constraints. [2] [3] [4] Restrictions may be applied to any aspect of the methods or reporting of a RR. For example, the scope of the question might be limited to a narrower population or set of outcomes. Methods may be streamlined by searching fewer databases, excluding grey literature, using single rather than dual reviewers for some or all review steps, not conducting a risk of bias appraisal and/or only conducting a qualitative analysis with a narrative summary. 4 Whilst the use of less rigorous methods may be unavoidable, concern has also been raised about poor quality reporting in RRs that is completely avoidable. 4 

The need to optimise methodological rigour has prompted the much anticipated release of the 'Interim Guidance from the Cochrane Rapid Reviews Methods Group'. 1 The guidance builds on earlier guidance from the World Health Organisation (WHO) 5 and leading evidence reviewers. 2 It complements the section of the Cochrane Collaboration website: 'Rapid Reviews in response to COVID-19'. The site also includes a set of streamlined templates, fast-track editorial services, and their 'Special Collection' series on COVID-19 that are brief, best-evidence summaries mostly reporting indirect evidence from published Cochrane reviews. Other initiatives to support rapid reviewers include fast-tracking of PROSPERO registration for COVID-19 systematic, umbrella/overviews and rapid reviews, and free access to Covidence software for researchers who are pursuing work on COVID-19.

To date, most of the COVID-19 RRs are evidence appraisals of interventions, which can also include meta-analyses. 4 Like any review, however, other questions and data sources can be rapidly reviewed. For example, there is a RR of traditional herbal medicine pattern recognition for COVID-19 symptoms. 6 

In line with RR recommendations, numerous COVID-19 RR protocols have been published. As of 9

July 2020, 142 RR protocols, including 2 protocols for Traditional Chinese Medicine (TCM), 7, 8 were listed on the Cochrane Coronavirus (COVID-19) Resources website, and 102 RR protocols, including five TCIM protocols [9] [10] [11] [12] [13] were registered with PROSPERO. A further 72 SR protocols categorised by PROSPERO as TCM (one of which included other types of TCIM interventions) were also registered.

Whilst none were RRs, all bar one of the TCM SR protocols 14 limited the population to SARS-CoV-2 infections and at this early stage in the pandemic a rapid turn-around would be possible. Table 1 summarises the methodological constraints proposed in the five RR TCIM protocols registered with PROSPERO. [9] [10] [11] [12] [13] The table does not included the two TCM RRs protocols listed on the

"A form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner."

Cochrane Rapid Reviews Methods Group 1 J o u r n a l P r e -p r o o f Cochrane website, as neither were Cochrane reviews and a citation for the registered protocol was not provided. 7, 8 Author JH conducted the search and extracted the data that was then verified by SA. 

Extended to include all viral respiratory infections and cytokine storms.

Search strategy: Limited to 3 major English language databases and 9 other English language databases that include clinical trial registries and pre-print severs. NI bibliography search. Title/abstract screening appears to be single, followed by dual screening of full text.

Limited to RCTs investigating efficacy. Limited to human studies for cytokine storm.

Data extraction: Limited to single reviewer data extraction, with a second reviewer checking for accuracy. * A revised protocol has since been published providing more clarity on priority outcomes, along with the decision to limit the meta-analysis to critical outcomes and only report critical outcomes in the summary of findings The RR protocols varied in the amount of detail provided. Most did not explicitly state when RR methodological compromises were being made. Presumably, standard SR methods would be applied unless stated otherwise.

In line with accepted RR compromises, it was common for the protocols to limit the number of databases to be searched and to use single, rather than dual reviewers for various tasks. However, 

Whilst "velocity does not have to impact transparency and appropriate methods", 3 we are concerned that this may be occurring in the context of RR of TCIM in the current pandemic. For instance, by the 1 July 2020, the Oxford COVID-19 Evidence Service (Centre for Evidence-Based Medicine (CEBM),

Oxford University) had published 193 articles on its website that were described as rapid evidence reviews, data analysis and thought-provoking writing. 17 The terms "treatment", "management" and "effective" were each entered into the website's Quick search function, yielding 26 articles of which 10 include the term "rapid review" in their title, abstract or main text (Table 2) . Five (half) of these RRs included TCIM interventions in the research question and/or search terms. None of the RRs had been peer reviewed, so a second search was conducted on PubMed using the following: "rapid review"[All Fields] with the filters: English, MEDLINE, Nursing journals, Humans, from 1 January to 31

June 2020. Of the twenty articles retrieved, 15 were RRs, 10 were specific to COVID-19 of which six aimed to answer clinical management questions. As these RRs are readily available and demonstrate a range of quality issues, they will be used for this illustration.

[INSERT Notably, the CEBM recommended minimum requirements 2 are much more flexible than the recently released interim guidance provided by the Cochrane Collaboration. Despite applying this lower standard, we found that the reporting standards of many of the included RRs were below par. All five of the RRs that considered TCIM were rated as having the lowest reporting standards. In contrast, the two Cochrane RRs easily met all the minimum reporting standards. All bar one of the CEBM RRs included a standard disclaimer statement noting the articles had not been peer reviewed and that all sources cited should be checked. With the exception of one RR, 26 , all the other peer reviewed articles had the highest reporting scores.

Despite CEBM's recommendation, 2 none of their RRs had published a basic protocol. However, this was not uncommon as the only two RRs to publish a protocol were the Cochrane reviews. 18, 19 Mullins et al,. 21 was the only RR without a protocol to note this was a limitation and provide a reason for not publishing a protocol.

In line with the CEBM's recommendations, 2 all the RRs provided adequate background information justifying the rationale for the review and bar one CEBM RR, 27 they all stated their research question/s in the body of the text. Most (14) reported information about the search strategy; this included searching the minimum requirement of one major database plus one other source. Yet, eight of the CEBM RRs and one peer-reviewed RR did not report their search flow results.

Wide variations were also observed in the other minimum reporting requirements. All of which were avoidable and unnecessarily limited the transparency and quality of the RRs. For example, whilst it is acceptable for some or all of the screening, appraising and data extraction to be conducted by one reviewer rather than the gold-standard of two reviewers independently doing each task, details about the methods applied should still be reported and the limitations noted.

Other variations reflected pragmatic constraints that were applied to the methods or reporting and for the most part, aligned with the more flexible approach recommended by CEBM. 2 Examples included only four RRs that formally critiqued the risk of bias/quality of the included studies and cited the appraisal tools. Another five narrated an overall summary of limitations of the included evidence, with limited or no information specific to each study. The remaining seven, appeared to defer to a simple hierarchy of evidence (i.e. levels of evidence) in which the findings of a SR supersede primary studies that in turn supersede non-human studies, possibly irrespective of their quality. Only two RRs, one of which was a Cochrane review, conducted a meta-analysis. 18, 20 The remaining chose to only narrate their findings. Various types of tables were used to summarise and present the data. Jones et al. 25 used a novel approach in which the references at the end of the review included three lists with the references that informed the three key review findings. Only the two Cochrane RRs presented a summary of findings table and graded the certainty of the evidence. 18, 19 Given the widespread global use of TCIM, it is commendable that the CEBM has included TCIM topics. This is in sharp contrast with the Cochrane COVID Rapid Reviews website and Question Bank that has only listed pharmacological interventions as priority Clinical Management questions, including for prophylaxis, and appears to also ignore the potential role of TCIM in the clinical management of pandemic-related impacts on health. 34 However, it is very concerning that given CEBM's highly influential position, all of the RRs that considered TCIM had consistently poor reporting standards that limited reproducibility and transparency. Already, at least one of these RRs has been used to inform national guidelines, yet the methodological limitations of the RR were not noted. 29 This is concerning as it increases the risk of bias.

At the outset, it is important to note that whilst RRs are similar to traditional SRs, they are not a replacement. Both should be protocol driven, however, RRs tend to have poorer reporting quality, provide less detail with fewer considerations and can miss key studies. In RRs any methodological shortcomings will be amplified 35 Trade-offs in comprehensiveness and rigour increase the risk of bias with resultant type 1 (false positive) and type 2 (false negative) errors, and reduce the certainty/quality of the review findings. 35 This concept is critical to consider in RRs and particularly for TCIM, as the conclusions of the review are rarely based on the totality of the literature, 36 but instead a limited subset, which is therefore more prone to bias.

Whilst no research is methodologically flawless, at the core of modern science are the concepts of reproducibility and transparency. Reproducibility simply means, if another investigator follows a published set of methods, they will obtain similar results. The notion of transparency follows, in that, the description of the methods and reporting of results must be comprehensive enough to replicate the study. Transparency in research also relates to ensuring that the results reflect the methods and that anomalous data which does not fit the a priori hypothesis is not buried, it is explored and openly discussed. It is for these reasons that like SRs, 37 a published protocol for a RR is recommended along with minimum reporting standards. 2, 6, 38 Rapid reviewers must also be careful not to misrepresent nor distort their conclusions and recommendations due to the conscious or unconscious 'spin' of their findings. Boutron and Ravaud define spin as "specific intentional or unintentional reporting that fails to faithfully reflect the nature and range of findings and that could affect the impression the results produce in readers". 39 The concept is closely linked to a type of cognitive bias called confirmation bias that increases the likelihood of favouring personal beliefs and ideologies when interpreting, reporting and recalling information. 40 Evidence is accumulating that this type of bias is highly problematic across scientific practices 40 and given the polarised opinions around TCIM, 41, 42 it is an issue of particular relevance for RRs in the field. Along with impacting the rigour of a RR, confirmation bias may also result in the end-user of a RR prematurely disregarding or accepting the findings without further critique. Whilst it is impossible to compensate for the outright dismissal of TCIM by those with rigid ideologies, a high-quality RR can provide a counterbalance for highly critical readers by ensuring that the methodological scholarship is above reproach.

In the case of TCIM, the risk of type 2 errors is particularly concerning as it can amplify cognitive biases and erroneously steer interest, resources and knowledge away from a potentially effective intervention. The CEBM RR of N-acetylcysteine (NAC) ( Table 2) is a good example. 32 In the conclusions, the reviewers downgraded promising clinical evidence due to the in vitro findings that any observed therapeutic efficacy of NAC was likely to be strain dependent. The review missed two key non-human studies, that were quickly identified on PubMed using the first two search terms in the addendum: N-acetylcysteine AND coronavirus. Both studies demonstrated that NAC also has anti-viral activity on the coronavirus, porcine epidemic diarrhea virus. 43, 44 The methodological decision to downgrade the certainty of clinical evidence based on in vitro evidence appeared to be a post-hoc decision and warranted further explanation. Further, the potential antiviral and antioxidant actions of NAC may not be precisely demonstrated in petri dishes and may be more clearly demonstrated in studies of humans with viral illnesses, irrespective of viral species. A registered protocol articulating the methodological rationale of this RR, coupled with a more robust search strategy, and reporting of the results would have improved the reproducibility and transparency of the findings. As it stands, the impact of type 2 errors that are made by influential bodies such as the CEBM, include prematurely dismissing further investigation of NAC as a potential adjuvant treatment for SARS-CoV-2 infections.

In light of the above, in addition to the excellent guidance from the Cochrane Collaboration and others, 2, 5, 38 we suggest those conducting a RR of TCIM carefully consider the following to optimise the rapidity, rigour and credibility of their reviews.

Generally, all RRs make some methodological sacrifices to ensure timely publication. It is important to note that these methodological shortcomings will be amplified in the RR setting, and it is therefore critical that a review follows an exact method, which requires careful planning. A structured protocol, even a basic protocol as recommended by CEBM, 2 will ultimately save time, reduce errors, and minimise bias. It will also allow for transparency and reproducibility which are essential for scientific credibility. Accompanying the 'Interim Guidance from the Cochrane Rapid Reviews Methods Group' is a RR protocol template that includes prompts for commonly used methodological compromises. 1 Typically, RR are better suited to narrow research questions applied to limited populations, interventions, comparisons and outcomes (PICO). 4 Broader questions and complex TICM interventions are more difficult, due the additional challenges with constraining search results and the need for a lengthier, detailed analysis conducted by content experts. Due to the paucity of TCIM evidence on a topic, rather than publishing an 'empty review', reviewers may decide to expand the inclusion criteria to include indirect evidence (e.g. all acute viral respiratory infections instead of only SARS-CoV2 infections). 15 However, this may also result in too many studies to rapidly appraise.

Additionally, such an approach may not be worthwhile as the final certainty/quality of the evidence will be downrated nevertheless, as it cannot be assumed that the evidence applies to SARS-CoV-2.

Prior to finalising the study design a quick scope of the literature is therefore recommended. This will help inform the search strategy, the types of studies to be included and whether a meta-analysis is possible and/or warranted. If there are a large number of potentially eligible primary studies and high-quality SRs have already been published, it might be worth considering an Overview (i.e. umbrella review of SRs), 37 or a best evidence synthesis (BES). 1 The BES approach begins with a search for SRs followed by a search for primary studies published after the last SR. 1 Methodological challenges to be aware of in undertaking Overviews and BES include dealing with the issue of overlap when a primary study is included in multiple SRs, and attempting to assess the methods and quality of primary studies included in a SR when limited information is reported. 37 Most of the methodological recommendations and minimum standards for RRs 1, 2, 5 apply to TCIM, however, some may not be appropriate. Language restrictions might not be acceptable for TCIM interventions when the modality originated in and/or the research has primarily been undertaken in a country where English is not an official language. 45, 46 Databases other than the minimum recommended MEDLINE, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) may also need to be searched. 1 Whilst these three databases are likely to be sufficient for non-TCIM interventions, unless the TICM therapy has recently been reviewed in a Cochrane SR that was not limited by language, key studies may still be missed. 45, 46 Consider using platforms such as Ovid, EBSCO Host and the Pan American Health Organization (PAHO) Virtual Health Library (VHL) that can simultaneously search multiple databases relevant to TCIM.

Reviewers should be cautious about using rapid appraisal tools such as the Critical Appraisal Skills Program (CASP) checklist for risk of bias, 47 as these methods are less robust than standard tools.

Such a shortcut may negatively impact the credibility of the review findings. Instead, consider highly regarded tools such as the recently released RoB-2 that is used to individually appraise each outcome reported in a study, 16 and then apply other limitations, such as only appraising the outcomes that are listed a priori in the RR as critically important, or only appraising the primary outcome of the included study.

An experienced research team who have previously collaborated, will help expedite the review. 2 At least one TCIM content expert is recommended. Including experienced reviewers fluent in languages other than English enables a broader database search, and saves time and money translating articles when screening and extracting data. Engaging a research librarian with SR experience and a thorough understanding of the TCIM research databases is extremely valuable. 48 Optimising the sensitivity and specificity of the search terms will minimise the time spent screening articles, and reduce the risk of missing key studies.

In conclusion, RRs of TCIM must still strive to optimise rigour, reproducibility and transparency.

Otherwise, any positive findings may be discredited nonetheless, and negative findings may be accepted without further critique. Above all, avoid 'spinning' 39 the findings, objectively report the results and explicitly discuss the potential biases that arise from any RR limitations that have been employed. 49 In line with the guidelines from leading evidence review bodies that are articulated in detail elsewhere, 1, 2, 5 recommended minimum requirements for RRs of TICM therefore include: 1. A basic protocol that is submitted for publication or made publicly available prior to commencing the review, with any post-hoc changes declared when reporting the RR.

A description of the methods, including explicit statements outlining any RR constraints 3. Search results, ideally including a PRISMA flow diagram. 4 . A structured, critical appraisal of the included studies, ideally using standardised tools.

A meta-analysis, if possible, of at least the most important (critical) outcomes of interest. 6 . The limitations the RR methods should be clearly acknowledged and accompanied by less authoritative evidence statements and remarks.

7. In the case of brief reports and plain language summaries only, either an appendix/addendum and/or an additional, publicly available document that reports additional key information on the methods and results. If the above is not possible, with the exception of not publishing a protocol, it should not be called a rapid review (nor a restricted review), it is either a non-systematic literature review or an evidence informed commentary / review article.

Systematic reviews of the research literature minimize bias and increase precision of reported effect estimates. This can be invaluable to policy makers and clinicians. Unfortunately, they are resource intensive and in the context of a rapidly evolving global pandemic, may not be a time sensitive way to synthesize evidence. In response to this concern, rapid reviews have been presented as a way of balancing the rigor and rapidity needed when there is an urgent need for high quality and timely evidence synthesis. While guidance for rapid reviews has been provided by authoritative organizations, unique issues exist in the field TCIM that need to be considered, such as strong confirmation biases, a higher prevalence of foreign language evidence sources, and unique TCIM databases. Ignoring these issues may lead to an unintended imbalance favouring rapidity over rigor if minimum requirements for RRs of TICM are not followed. It is for these reasons we recommend reinforcing and augmenting current RR guidance to maintain an acceptable balance in any rapid reviews of TCIM. 

This work received no grant from any funding agency.

This research did not require an ethical approval as it does not involve any human or animal experiment.

This review was not undertaken as part of a contractual relationship with any donor or sponsor.

The data will be made available upon request.

Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group

Redefining rapid reviews: a flexible framework for restricted systematic reviews

Reviews: Rapid! Rapid! Rapid! …and systematic

A scoping review of rapid review methods

Rapid reviews to strengthen health policy and systems: a practical guide

Herbal medicine and pattern identification for treating COVID-19: a rapid review of guidelines

The efficacy and safety of Lianhua Qingwen capsule on Covid-Virus. ID 146. Date added

Effectiveness of treatments recommended by guidelines (Traditional Chinese Medicine, hormonal, antiviral) for β-coronaviruses-associated respiratory diseases: a systematic review and meta-analysis

Protocol for a rapid review of zinc for the prevention or treatment of COVID-19 and other coronavirusrelated respiratory tract infections in humans

Accessed 9

A rapid review of Echinacea for the prevention or treatment of COVID-19 and other respiratory tract infections in humans PROSPERO 2020 CRD42020186339

Accessed 9

Could nutrition modulate COVID-19 susceptibility and severity of disease? A rapid systematic review

Accessed 9

Elderberry for prevention and treatment of viral respiratory illnesses PROSPERO 2020 CRD42020186339

Accessed 9

Shufeng Jiedu capsule for Covid-19 and other respiratory viral infections: a rapid review on current evidence PROSPERO 2020 CRD42020194320

Accessed 9

Effect of panax ginseng on preventing acute respiratory tract infection: a protocol for systematic review and meta-analysis (CRD42020181317

Rapid review protocol: zinc for the prevention or treatment of COVID-19 and other coronavirus-related respiratory tract infections

RoB 2: a revised tool for assessing risk of bias in randomised trials

COVID-19 Evidence Service: Treatment for COVID-19

Video calls for reducing social isolation and loneliness in older people: a rapid review

Convalescent plasma or hyperimmune immunoglobulin for people with COVID-19: a rapid review

Occurrence, prevention, and management of the psychological effects of emerging virus outbreaks on healthcare workers: rapid review and meta-analysis

Coronavirus in pregnancy and delivery: rapid review

The psychological impact of quarantine and how to reduce it: rapid review of the evidence

Lopinavir/ritonavir: A rapid review of effectiveness in COVID-19

Does BCG vaccination protect against acute respiratory infections and COVID-19: A rapid review of current evidence

How can healthcare workers adapt non-pharmacological treatment-whilst maintaining safety-when treating people with COVID-19 and delirium?

Rapid review for the anti-coronavirus effect of remdesivir

Mast cell stabilisers, leukotriene antagonists and antihistamines: A rapid review of the evidence for their use in COVID-19

Rapid Review: Diabetic retinopathy screening during the COVID-19 pandemic

Vitamin D: A rapid review of the evidence for treatment or prevention in COVID-19

Practical tips for clinicians helping patients with COVID-related anxiety/distress

Does physical exercise prevent or treat acute respiratory infection

A rapid review of the evidence for effectiveness in treating COVID-19

Activities delivered at home by family carers to maintain cognitive function in people with dementia socially isolating during COVID-19: Evidence for Non -technology based activities / interventions

COVID-19 Priorities: Version 7

A retrospective comparison of systematic reviews with same-topic rapid reviews

Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

Cochrane handbook for systematic reviews of interventions

Cochrane's work on Rapid Reviews in response to COVID-19

Misrepresentation and distortion of research in biomedical literature

Opinion dynamics with confirmation bias

The polarized debate over complementary and alternative medicine

Use and acceptance of complementary and alternative medicine among the general population and medical personnel: a systematic review

Porcine epidemic diarrhea virus infections induce apoptosis in Vero cells via a reactive oxygen species (ROS)/p53, but not p38 MAPK and SAPK/JNK signalling pathways

N-Acetylcysteine supplementation alleviates intestinal injury in piglets infected by porcine epidemic diarrhea virus

Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary

Systematic Reviews and Meta-Analyses of Traditional Chinese Medicine Must Search Chinese Databases to Reduce Language Bias

>; Accessed

The librarian's roles in the systematic review process: a case study

Rapid reviews for rapid decision-making during the coronavirus disease (COVID-19) pandemic, Norway