key: cord-295375-nakxfhxk
authors: Yu, Yang; Shi, Qianling; Zheng, Peng; Gao, Lei; Li, Haiyuan; Tao, Pengxian; Gu, Baohong; Wang, Dengfeng; Chen, Hao
title: Assessment of the quality of systematic reviews on COVID‐19: A comparative study of previous coronavirus outbreaks
date: 2020-04-28
journal: J Med Virol
DOI: 10.1002/jmv.25901
sha: 
doc_id: 295375
cord_uid: nakxfhxk

Several systematic reviews (SRs) have been conducted on the COVID‐19 outbreak, which together with the SRs on previous coronavirus outbreaks, form important sources of evidence for clinical decision and policy making. Here, we investigated the methodological quality of SRs on COVID‐19, severe acute respiratory syndrome (SARS), and Middle East respiratory syndrome (MERS). Online searches were performed to obtain SRs on COVID‐19, SARS, and MERS. The methodological quality of the included SRs was assessed using the AMSTAR‐2 tool. Descriptive statistics were used to present the data. In total, of 49 SRs that were finally included in our study, 17, 16, and 16 SRs were specifically on COVID‐19, MERS, and SARS, respectively. The growth rate of SRs on COVID‐19 was the highest (4.54/month) presently. Of the included SRs, 6, 12, and 31 SRs were of moderate, low, and critically low quality, respectively. SRs on SARS showed the optimum quality among the SRs on the three diseases. Subgroup analyses showed that the SR topic (P < .001), the involvement of a methodologist (P < .001), and funding support (P = .046) were significantly associated with the methodological quality of the SR. According to the adherence scores, adherence to AMSTAR‐2 items sequentially decreased in SRs on SARS, MERS, and COVID‐19. The methodological quality of most SRs on coronavirus outbreaks is unsatisfactory, and those on COVID‐19 have higher risks of poor quality, despite the rapid actions taken to conduct SRs. The quality of SRs should be improved in the future. Readers must exercise caution in accepting and using the results of these SRs.

address a specific research question. Practitioners mainly rely on SRs for evidence to provide evidence-based recommendations. Before the COVID-19 pandemic, outbreaks of severe acute respiratory syndrome (SARS) and the Middle East respiratory syndrome (MERS) were observed in 2003 and 2012, respectively; substantial numbers of SRs have been conducted on these outbreaks. The causative agents of these three outbreaks belong to the same genus of coronavirus (CoV), and SARS-CoV-2 shares a 79.5% sequence identity with SARS-CoV. Thus, the available SRs on SARS and MERS are helpful in guiding COVID-19 management. However, the quality of these SRs was unclear. In addition, besides these existing SRs, we also found that a large number of newly-conducted SRs with or without meta-analyses have been published rapidly, shortly after the COVID-19 outbreak, and the number of such SRs keeps rising. As we know, at the preliminary stage of any public emergency, primary studies are usually lacking, and most studies are observational. Thus, it is not easy to conduct an SR. These recent SRs on COVID-19 are more likely to have the potential methodological flaws. Therefore, we hypothesized that there may be a difference in methodological quality between these SRs on COVID-19 and previous SRs on SARS and MERS.

Up to now, no studies appraising the methodological quality of the SRs on COVID-19 and previous MERS and SARS outbreaks are available. In this comparative study, we investigated the present status of conducting SRs on COVID-19, MERS, and SARS, appraised the methodological quality of these SRs using the a measurement tool to assess systematic reviews (AMSTAR 2), and performed a preliminary examination of the potential risk factors associated with the quality of SRs, with the aim of providing suggestions from the aspects of methodological quality for conducting and using SRs during the COVID-19 pandemic.

We searched for eligible SRs (and/or meta-analyses) in both English 

We included all SRs which discussed one of the following diseases: COVID-19, SARS, and MERS. SRs were identified based on the following criteria: (a) label of "systematic review" in the title, abstract, or full text; and (b) a literature search was performed. There were no restrictions on the SR topic. SRs with or without meta-analysis were both eligible. Only SRs published in academic journals were included.

Studies were excluded based on the following criteria: (a) letters, editorials, and expert opinions; case reports and literature reviews; and other narrative reviews; (b) abstract only or data unavailable; or (c) studies in non-English or non-Chinese languages. When a duplicate or updated publications were identified, only the most recent one was included.

Two reviewers independently screened the titles and abstracts of all articles identified in the initial search and then checked the full texts of potentially eligible studies. Any disagreement was resolved by discussion or consultation with a third reviewer. The same reviewers used a predesigned table to collect the following information on the included SRs: first author, publication year, the growth rate of SRs, region or country in which the study was performed, study topic 

The methodological quality of the included SRs was independently assessed by two well-trained reviewers using the AMSTAR-2 tool. 5, 6 Any disagreement between the two reviewers was resolved by discussion, and in case of persistent disagreement, a resolution was arrived at by consulting with a third person.

The AMSTAR-2 is a critical appraisal tool for SRs of randomized control trials and/or observational studies. 6 The AMSTAR-2 contains 16 items, of which seven are critical domains (Table 1) . A questionspecific point scale ("Yes, No;" "Yes, Partial yes, No;" or "Yes, No, No meta-analysis") was used to score each item. Based on the degree of weaknesses detected in critical and noncritical items, the AMSTAR-2 classifies the overall confidence in the results of the SRs into four levels: high, moderate, low, and critically low. We graded each included SR on the AMSTAR-2 official website and generated the overall confidence using their online calculator. In addition, to compare adherence to items among SRs, a total score was calculated for each SR according to the response to each item, that is, "Yes," "Partial yes," "No meta-analysis," and "No" were scored as "+1," "+0.5," "0," and "−1," respectively.

Descriptive statistics were used to present the characteristics and methodological quality of the included SRs. Categorical variables were expressed as frequencies and percentages. The response to each AMSTAR-2 item was recorded and tabulated for all included SRs. Overall adherence to the 16 items were analysed with a percent stacked bar chart. The adherence scores in each disease group were presented as mean ± standard deviation (SD). Subgroup analysis was performed to compare the differences in quality among subgroups with different study characteristics. Statistical significance was tested using the Kruskal-Wallis rank test and two-sample Mann-Whitney U test. P < .05 was considered to be statistically significant, and all P values were two-sided. Excel 2019 and SPSS V25

were used for all data management and analyses. were supported by at least one funding source ( were of critically low quality. Not a single SR was rated as having high

quality. The quality of all SRs on COVID-19 was rated as low (29%, n = 5) or critically low (71%, n = 12), similar to that of SRs on MERS (low, 25%, n = 4; critically low, 75%, n = 12) but inferior to that of SRs on SARS (moderate, 38%, n = 6; low, 19%, n = 3; critically low, 44%, n = 7).

The potential factors affecting the quality of SRs were investigated in all samples. Subgroup analyses of different variables showed that the topic of the SR, involvement of a methodologist, and funding support were significantly associated with the methodological quality of the SRs ( Details of the assessment of each item are described in the Table S1 .

SRs provide the highest level of evidence in evidence-based medicine, and they are an important source of information for clinical practitioners and policy makers. However, due to the relatively low requirements and costs of conducting SRs, the number of SRs published in various fields is quite high. The increase in quantity has also brought concerns about the quality of SRs. 56 In this study, we found that a large number of SRs were conducted de novo after the For unknown emerging diseases, the scarcity of large samples and high-quality original reports are major challenges in scientific research. In particular, in the early stages of disease outbreaks, evidence is mostly acquired from observational studies, 58 

SRs specific to COVID-19, SARS, and MERS are being heavily relied upon during the current COVID-19 pandemic. The methodological F I G U R E 2 Adherence to each item in AMSTAR-2. AMSTAR, a measurement tool to assess systematic reviews; MA, meta-analysis quality of most SRs is unsatisfactory, and those on COVID-19 have higher risks of poor quality, despite the rapid actions taken to conduct SRs. Teams that may want to conduct a SR should focus on the study design and focus on improving the quality of the SR. SR findings should be used more cautiously, and it is not advisable that users accept the results of a single SR without critical appraisal. 

COVID-19) Situation

Cochrane Handbook for Systematic Reviews of Interventions

Scientific Journal Rankings2018

A Guide to the Core Journals of China

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews

AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

Asymptomatic Middle East Respiratory Syndrome Coronavirus (MERS-CoV) infection: extent and implications for infection control: a systematic review

Update on therapeutic options for Middle East respiratory syndrome coronavirus (MERS-CoV)

Prevalence of comorbidities in the Middle East respiratory syndrome coronavirus (MERS-CoV): a systematic review and meta-analysis

Prevalence of diabetes in the 2009 influenza A (H1N1) and the Middle East respiratory syndrome coronavirus: a systematic review and meta-analysis

A systematic, thematic review of social and occupational factors associated with psychological outcomes in healthcare employees during an infectious disease outbreak

Effect of integrated traditional Chinese medicine and western medicine on the treatment of severe acute respiratory syndrome: a meta-analysis

What have we learned about Middle East respiratory syndrome coronavirus emergence in humans? A systematic literature review. Vector Borne Zoonotic Dis

A rapid scoping review of Middle East respiratory syndrome coronavirus in animal hosts

Analyzing on the clinical symptoms and T cell subsets of patients with severe acute respiratory syndrome

Meta analysis on T cell subsets of patients with severe acute respiratory syndrome

Meta-analysis of comparative studies of integrative traditional Chinese medicine with western medicine and western medicine alone for SARS

Clinical symptoms and peripheral white blood cell counts of patients with severe acute respiratory syndrome: meta analysis

Meta-analysis on risk factors of SARS patients

Epidemic models of contact tracing: systematic review of transmission studies of severe acute respiratory syndrome and Middle East Respiratory Syndrome

Seroprevalence of IgG antibodies to SARS-coronavirus in asymptomatic or subclinical population groups

A meta-analysis to evaluate the effectiveness of real-time PCR for diagnosing novel coronavirus infections

Chinese herbal medicine for severe acute respiratory syndrome: a systematic review and metaanalysis

Chinese herbs combined with Western medicine for severe acute respiratory syndrome (SARS)

Middle East respiratory syndrome coronavirus: current knowledge and future considerations

Clinical determinants of the severity of Middle East respiratory syndrome (MERS): a systematic review and meta-analysis

A systematic review of therapeutic agents for the treatment of the Middle East Respiratory Syndrome Coronavirus (MERS-CoV)

Therapeutic options for Middle East respiratory syndrome coronavirus (MERS-CoV)-possible lessons from a systematic review of SARS-CoV therapy

Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis

Femoral head necrosis after severe acute respiratory syndrome: etiology and treatment

MERS transmission and risk factors: a systematic review

Global status of Middle East respiratory syndrome coronavirus in dromedary camels: a systematic review

SARS: systematic review of treatment effects

A systematic review of assessing the effect of integrated traditional Chinese medicine with western medicine for severe acute respiratory syndrome

Effect of integrated traditional Chinese and Western medicine on SARS: a review of clinical evidence

Systematic assessment on the effect of integrated Chinese traditional medicine with western medicine in the treatment of severe acute respiratory syndrome (SARS)

Steroid therapy and the risk of osteonecrosis in SARS patients: a dose-response meta-analysis

Transmission of Middle East respiratory syndrome coronavirus infections among healthcare personnel in the Middle East: a systematic review

Evidence-based rapid review on possibility of treatment of 2019-nCoV with subcutaneous α-interferon

A systematic review on the efficacy and safety of chloroquine for the treatment of COVID-19

The possibility of using Lopinave/ Litonawe (LPV/r) as treatment for novel coronavirus (2019-nCov) pneumonia: a quick systematic review based on earlier coronavirus clinical studies

Prevalence and impact of cardiovascular metabolic diseases on COVID-19 in China

COVID-19 patients' clinical characteristics, discharge rate and fatality rate of meta-analysis

Sanchis-Gomar F. Cardiac troponin I in patients with coronavirus disease 2019 (COVID-19): evidence from a metaanalysis

Procalcitonin in patients with severe coronavirus disease 2019 (COVID-19): a meta-analysis

Thrombocytopenia is associated with severe coronavirus disease 2019 (COVID-19) infections: a metaanalysis

Potential rapid diagnostics, vaccine and therapeutics for 2019 novel coronavirus (2019-nCoV): a systematic review

Clinical, laboratory and imaging features of COVID-19: a systematic review and meta-analysis

Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients

Clinical characteristics of hospitalized patients with SARS-CoV-2 infection: a single arm metaanalysis

Prevalence of comorbidities in the novel Wuhan coronavirus (COVID-19) infection: a systematic review and meta-analysis

A systematic review of lopinavir therapy for SARS coronavirus and MERS coronavirus-A possible reference for coronavirus disease-19 treatment option

Potential interventions for novel coronavirus in China: a systematic review

Traditional Chinese medicine syndromes of the novel coronavious pneumonica: a systemic review and meta-analysis

The Possibility of Ribavirinas in the Treatment of the Coronavirus Disease 2019: a Systematic Review

The influence of the team in conducting a systematic review

Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry

European Centre for Disease Prevention and Control. Evidence-based methodologies for public health-how to assess the best available evidence when time is limited and there is lack of sound evidence. Stockholm: European Centre for Disease Prevention

Systematic reviews: rationale for systematic reviews

The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study

An international registry of systematic-review protocols

Assessment of the quality of systematic reviews on COVID-19: A comparative study of previous coronavirus outbreaks

The authors declare that there are no conflict of interests.

YY and HC conceived the idea for this study. YY designed the study. 

http://orcid.org/0000-0002-2409-3681