key: cord-0027877-immcvwbf
authors: Paul, Mical; Leeflang, Mariska M.
title: Reporting of systematic reviews and meta-analysis of observational studies
date: 2020-11-17
journal: Clin Microbiol Infect
DOI: 10.1016/j.cmi.2020.11.006
sha: b6992b8316e0c49a890f5fe147cb87a7eb618f2a
doc_id: 27877
cord_uid: immcvwbf

nan

Reporting of systematic reviews and meta-analysis of observational studies Although randomized controlled trials (RCTs) are generally seen as providing more certainty about the evidence of an effect, many interventions in infectious diseases are examined in observational studies. These include rare cases of treatment that cannot be examined in a RCT (e.g. appropriate vs. inappropriate empirical antibiotic treatment) [1, 2] , complex interventions that are difficult to examine in RCTs (e.g. infection control or antibiotic stewardship) [3, 4] , comparative effective research and quality improvement programmes (comparisons between commonly used, approved, antibiotics, monotherapy vs. combinations or treatment durations) for which resources for RCTs are lacking [5e10], or studies rapidly launched during outbreaks, such as the coronavirus disease 2019 (COVID-19) pandemic [11e13] . Summarizing these in systematic reviews is appealing; however, systematic reviews of observational studies suffer from the same shortcomings as primary observational studies when causal inferences need to be made. Consistent errors in observational studies may even be magnified by narrowing the confidence intervals through meta-analysis.

In this editorial note, we provide our recommendations and expectations for systematic reviews and meta-analyses of observational studies examining an exposure or intervention. Guidance on performance and reporting of such studies has been published, and we recommend following these [14, 15] . We explain the specific pitfalls of such studies and what we require from authors of systematic reviews of observational studies.

As for any research, systematic reviews of observational studies should start with a protocol. We expect to see the protocol for the systematic review, preferably published in advance in PROSPERO (International Prospective Register of Systematic Reviews) or other database. This protocol should start with the review question, containing Patients, Intervention/Exposure, Comparator, Outcomes, Study design (PICOS or PECOS). The PECOS format has been proposed to distinguish between the active intervention in a RCT and the observation of an 'exposure' in observational studies [14] . In interventional trials, the dose and duration of the treatments are fixed by protocol, and restrictions on patient inclusion on the basis of time from disease onset are defined. Similarly, in observational studies, there should be criteria to define exposure [16] , and systematic reviews can and probably should restrict inclusion criteria to observational studies examining the exposure of interest. For example, in a systematic review of hydroxychloroquine therapy for COVID-19, rather than defining for inclusion all studies examining hydroxychloroquine among patients with COVID-19, an informative review can restrict inclusion to studies where hydroxychloroquine therapy was initiated early after diagnosis, examining an appropriate dose, for a minimal duration of treatment. Too tight a definition (e.g. one specific hydroxychloroquine dose) might result in zero studies included, but too broad a definition will result in less meaningful estimates and heterogeneity. Alternatively, this can be investigated in sensitivity or subgroup analyses, and would address the time hydroxychloroquine was started relative to the diagnosis of COVID-19, acceptable dosing and the minimal duration that would be considered as treatment.

The study designs included should be addressed. Inclusion of RCTs should always be considered. The types of observational studies considered and restrictions on observational study design for inclusion in the review should be specified. Possible exclusion criteria based on observational study design may include small sample sizes, before-and-after studies or lack of adjusted analysis.

An important feature of the protocol for a systematic review of observational studies is the need to tailor the risk of bias assessment tools to the review. We recommend the Cochrane tool for risk of bias of nonrandomized studies of interventions, ROBINS-I [17] . ROBINS-I is a domain-based assessment tool with signaling questions for each domain. It is comparable to the Cochrane ROB-2 tool for RCTs, which may be useful when including both RCTs and non-RCTs in the systematic review. The Newcastle-Ottawa scale is an alternative tool [18] . Both require advance planning and adaptation to the review question to enable assessment of the 'confounding' and 'deviations from intended interventions' domains (https://www.riskofbias.info/welcome/home/currentversion-of-robins-i/robins-i-tool-2016). The important risk factors for the outcomes and variables presumed to affect both the presence of the exposure and the outcome need to be defined. Review authors should assess whether these predefined variables were examined when analysing the association between exposure and outcome. For example, a systematic review assessing appropriate vs. inappropriate empirical antibiotic therapy for sepsis requires the assessment of age, any sepsis severity score and the place of acquisition as potential confounders in primary studies [19] . For a study of hydroxychloroquine and COVID-19, confounding domains should probably include age, comorbidities and disease stage/ severity. In addition, a list of potential cointerventions that could be different between intervention groups and that could affect outcomes needs to be specified in advance in order to compare their use between the treatment groups and to score studies on bias related to the interventions. For example, corticosteroids and availability of intensive support would be important cointerventions in the observational assessment of antiviral therapy for COVID-19, especially when comparing data among different centres. Wellconducted risk-of-bias assessment of observational studies within a systematic review is highly informative and could actually be the focus of the systematic review, as was the case in a systematic review of studies comparing monotherapy to combination therapy for carbapenem-resistant Gram-negative infections, where the main objective was to highlight the bias in existing studies to enable an evidence-based statement that these studies should not direct practice [20] .

The main challenge in meta-analyses of observational studies is pooling the adjusted results. Pooling the crude unadjusted numbers is simple, but it is not more informative than the univariate analysis in the original observational studies. Primary observational studies typically use different methods for adjustment of the analysis, such as multivariable regression analyses, propensity scores that can be used in several ways or instrumental variable analyses. Results from these analyses should be used in systematic reviews of observational studies. However, a commonly encountered problem is that depending on inclusion criteria of the review, only some of the included studies may report the adjusted association for the exposure variable of interest. Included studies might not include an adjusted analysis at all; the authors might have performed an adjusted analysis without analysing the exposure of interest, or they might not include quantitative results for factors that were not retained or that were not statistically significant in the final regression model.

We encountered all these possibilities in a systematic review and meta-analysis on appropriate vs. inappropriate empirical antibiotic treatment and mortality in sepsis. Because the studies that did not find an association between inappropriate empirical antibiotic treatment and mortality in the univariate analysis did not include it in the adjusted risk model or did not quantitatively report the results of these adjusted analyses, these studies were highly skewed towards a lack of association [19] . In this case, restricting the analysis to studies reporting the adjusted odds ratio for inappropriate empirical antibiotic treatment and mortality would have strongly biased this analysis in favour of appropriate empirical antibiotic treatment. A standard solution to all these obstacles is not available. (A simple solution would be not to perform a systematic review of observational studies.) Several options and the limitations of each are presented in Table 1 . Furthermore, even for a focused question, the studies will use different variables and variable definitions for adjustment. To address this source of variability in the adjusted estimates of association, review authors may limit the meta-analysis to studies adjusting for confounders that were defined as being important in the tailored ROBINS-I. Additionally, sensitivity or subgroup analyses by optimal, nonoptimal or no adjustment can be performed [14] . Finally, adjusted results of the original studies might be reported as odds ratios, hazard ratios or risk ratios with different dispersion measures (e.g. 95% confidence intervals, standard errors). A meta-analysis should pool the same type of measure.

The optimal strategy will probably be to conduct sensitivity analyses using different assumptions. None of these sensitivity analyses may be without bias, but if the bias works in different directions, this may be informative. Of six systematic review including observational studies published recently in CMI [13,21e25] , only two addressed the adjusted results of primary observational studies (presenting these without a meta-analysis in a review associating vaccines with antibiotic use [22] and using multivariate odds ratios when available and univariate results otherwise in a meta-analysis of hydroxychloroquine used to treat COVID-19 [13] ). A review of fluoroquinolones vs. trimethoprim/sulfamethoxazole used to treat Stenotrophomonas maltophilia infections reported a lack of adjusted results and pooled univariate results [23] . We would like to see both the crude results and the adjusted effect estimate presented. A comparison between the nonadjusted and adjusted associations can attest to the degree of confounding. Similarly, in systematic reviews including both RCTs and observational studies, their meta-analysis should be explicitly separated. If you believe that a meta-analysis of crude, unadjusted results is valid, then you must convince readers that confounding is negligible in this particular review. The complicated nature of an optimal meta-analysis of adjusted associations from observational studies limits the scope of a systematic review of observational studies typically to a single outcome of interest.

Certainty of the overall evidence presented in the systematic review can be determined using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, as for RCTs [26] . Because risk of bias will rarely, if ever, be low with observational studies, the certainty of the evidence will often be automatically downgraded to moderate, low or very low certainty of the evidence. GRADE allows upgrading the certainty of the evidence of observational studies when all confounding factors (including unobserved, residual confounding) work in the opposite direction of the observed association. This requires knowledge of the residual confounding and was impossible in the case of the systematic review on inappropriate empirical antibiotic treatment. Authors of systematic reviews of observational studies are advised to follow the GRADE-PRO handbook when performing the grading [27] .

As for the Discussion and Conclusion sections, we will be strict on overoptimistic interpretation of the results of systematic reviews relying on observational studies. If the aim of a systematic review is to provide evidence about causal effects of interventions and exposures, then this should be made explicit. But at the same time, the conclusions will usually need to be cautious and include a warning about the observational nature of the presented evidence.

In summary, performing a systematic review of observational studies requires knowledge of the review field, advanced understanding of observational study design and analysis, good planning and an understanding that the evidence summarized will most probably not reach high quality. We present in Table 2 subheadings for the Methods section that we expect to see in systematic reviews of observational studies in CMI; we expect judicious analysis of data from observational studies.

Neither author reports a conflict of interest relevant to this editorial.

Supplementary data to this article can be found online at https://doi.org/10.1016/j.cmi.2020.11.006. Types of data to be extracted should be defined, including nonadjusted and adjusted outcomes. Data on specific confounders extracted and adjustment methods used in study are special to systematic reviews of observational studies.

Methods of data extraction and risk of bias assessment, including whether duplicate independent extraction was performed. Data extraction and risk of bias assessment of observational studies are more complicated than in RCTs; a duplicate process with a consensus strategy is highly recommended.

If performing a meta-analysis, univariate and adjusted analyses must be addressed, including methods of handing different adjustment methods and studies that did not report an adjusted analysis. Describe approach of pooling studies (fixed/random effects meta-analysis), heterogeneity assessment and planned subgroup and sensitivity analyses. Grading

Preferably, quality of evidence summarized in review should be graded formally.

RCT, randomized controlled trial. 

Neonatal outcomes associated with maternal recto-vaginal colonization with extended-spectrum b-lactamase producing Enterobacteriaceae in Nigeria: a prospective, cross-sectional study

The impact of carbapenemase-producing Enterobacteriaceae colonization on infection risk after liver transplantation: a prospective observational cohort study

The effect of an antimicrobial stewardship programme in two intensive care units of a teaching hospital: an interrupted time series analysis

Impact of single-room contact precautions on hospital-acquisition and transmission of multidrug-resistant Escherichia coli: a prospective multicentre cohort study in haematological and oncological wards

The association of adequate empirical treatment and time to recovery from bacteraemic urinary tract infections: a retrospective cohort study

Combination therapy with ciprofloxacin and third-generation cephalosporin versus thirdgeneration cephalosporin monotherapy in Escherichia coli meningitis in infants: a multicentre propensity score-matched observational study

Early cidofovir administration might be associated with a lower probability of respiratory failure in treating human adenovirus pneumonia: a retrospective cohort study

Williams 2nd RJ. Fidaxomicin versus oral vancomycin for severe Clostridium difficile infection: a retrospective cohort study

Comparison of the toxicity of two treatment schemes with benznidazole for chronic Chagas disease: a prospective cohort study in two Spanish referral centres

Association between treatment duration and mortality or relapse in adult patients with Staphylococcus aureus bacteraemia: a retrospective cohort study

Umifenovir treatment is not associated with improved outcomes in patients with coronavirus disease 2019: a retrospective study

Ibuprofen use and clinical outcomes in COVID-19 patients

Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis

COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration

Observational studies examining patient management in infectious diseases

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

The Newcastle-Ottawa scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses

Systematic review and meta-analysis of the efficacy of appropriate empiric antibiotic therapy for sepsis

Polymyxin monotherapy or in combination against carbapenem-resistant bacteria: systematic review and meta-analysis

Acute kidney injury following the concurrent administration of antipseudomonal b-lactams and vancomycin: a network meta-analysis

Impact of vaccination on antibiotic usage: a systematic review and meta-analysis

Fluoroquinolones versus trimethoprimesulfamethoxazole for the treatment of Stenotrophomonas maltophilia infections: a systematic review and meta-analysis

Amphotericin B for treatment of visceral leishmaniasis: systematic review and meta-analysis of prospective comparative clinical studies including dose-ranging studies

Faecal microbiota transplantation for eradicating carriage of multidrug-resistant organisms: a systematic review

GRADE guidelines, 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach