key: cord-0882103-ktv2xpd6
authors: Padrão, Eduardo M.H.; Rahhal, Hassan; Valente, Fernando S.; Besen, Bruno A.M.P.
title: Methodological issues in meta-analyses of observational studies: the need for attention to the details
date: 2022-03-02
journal: Br J Anaesth
DOI: 10.1016/j.bja.2022.01.033
sha: 9725e37e6a6ef0b52b37666ca9d62acb82d147c8
doc_id: 882103
cord_uid: ktv2xpd6

nan

EditordWe read with interest the meta-analysis of awake prone position for spontaneously breathing patients by Fazzini and colleagues. 1 Systematic review and metaanalysis has been an important tool to bring insight into the care of patients with COVID-19. In this context, we appreciate all efforts to deal with the pandemic and minimise its burden. However, the main goal of systematic reviews cannot be forgotten: to find effect sizes as precisely as possible without bias, accounting for the risk of bias in study design (i.e. excluding studies that may bias the results) and the use of appropriate analytical tools. This paper caught our attention for several reasons.

First, the primary outcome was change in the Pa o2 :FiO 2 ratio, which is a physiological outcome of questionable patient-centredness that should not be a guide to clinical management per se. Moreover, the authors used a paediatric linear equation to estimate Pa o2 :FiO 2 ratio mean differences from the SpO 2 :FiO 2 ratio. This issue has been studied and the current recommendation is to use non-linear imputation of Pa o2 :FiO 2 ratios from SpO 2 :FiO 2 ratios. 2 This very important issue makes study results questionable and probably invalidates their primary outcome assessment.

More importantly, the data extracted to pool odds ratios for the secondary outcome of tracheal intubation were not the adjusted results. For non-randomised studies of interventions (NRSI), the Cochrane Handbook recommends extracting adjusted data and subsequently pooling the results. 3 Miguel Hernan furthers this concept stating that, although sample size may not be the most important issue, adequately designed and analysed observational studies (i.e. studies addressing confounding and other methodological issues such as immortal time bias) are quite important for evidence synthesis. 4 For example, our study, 5 which represents one of the highest weights in the meta-analysis of this outcome among the NRSI, presented both unadjusted and adjusted results. Although in the unadjusted analysis the hazard ratio was 1.21 (95% confidence interval [CI], 0.78e1.88), in our adjusted analysis for relevant characteristics the hazard ratio was 0.90 (95% CI, 0.55e1.49), with modifying the direction of the point estimate as a result of confounding by indication accounted for at least partially.

Using raw data without statistical adjustment introduces bias in the meta-analysis as confounding will not be properly dealt with. The estimates will be inherently biased, thus making it questionable to meta-analyse the data of clinical trials along with observational studies. To tackle this issue and as an example, we have reanalysed the secondary outcome of tracheal intubation rate, including only studies that dealt with confounding, at least to some extent. 5e8 We extracted the adjusted hazard ratios with their respective CIs from nonrandomised studies and extracted the same data from the clinical trial. 9 We performed a random effects meta-analysis using the HartungeKnappeSidikeJonkman method to avoid overly optimistic results. Our results are presented in Fig. 1 . The first finding is that all studies had point estimates either favouring awake prone positioning or neutral. The corresponding heterogeneity was much lower (I 2 ¼21%) than that presented in Fazzini and colleagues' 1 Figure 3 (I 2 ¼75%). This is expected, because the large observed heterogeneity can be explained by the inclusion of unadjusted (i.e. confounded) analyses in the results.

The final finding of this reanalysis is that awake prone positioning is associated with a reduced tracheal intubation rate (hazard ratio 0.79, 95% CI, 0.63e0.98), without worrisome inconsistency and with some impreciseness as a result of the moderately large CI. These results suggest that awake prone positioning, when it comes to the outcome of intubation, should be at least suggested in clinical practice per the GRADE approach. 10 This recommendation cannot be strong (recommend statement) because of some impreciseness and because of indirectness (gathering data from observational studies in the absence of more randomised clinical trials). The exercise we did with intubation hazard can also be done with the metaanalysis presented for mortality (Fig. 4 of Fazzini and colleagues 1 ), in which unadjusted (and therefore biased) analyses were done, coming to biased and potentially wrong conclusions.

Finally, we observed that subgroup analyses shown in the supplement grouped patients proned for >4 h or for <4 h. Our study results are included in that specific analysis considering that all patients were proned for >4 h. However, according to our results, only 29 (58%) patients tolerated prone positioning for >4 h. 5 Therefore, a correction is necessary: of the 57 proned patients included in the subgroup analysis, 1 only 29 should have been included.

Although observational studies have been increasingly recognised as important to be included in systematic reviews and meta-analyses, especially where data from randomised trials are not of low risk of bias, this is not straightforward. Adequate selection of studies, thorough risk of bias assessment, adequate extraction of data, and finally proper analysis are of utmost importance to draw unbiased inferences as precisely as possible. With proper methodology, our conclusions are different from the authors' conclusion. Pending the publication of further trials (the COVI-PRONE trial), which will increase sample size and therefore provide more precise effect estimates, awake prone positioning could at least be suggested (weak recommendation) as a strategy to avoid tracheal intubation for adult patients with COVID-19-related respiratory failure who are not in imminent need of invasive mechanical ventilation.

The authors declare that they have no conflicts of interest. 

Prone positioning for non-intubated spontaneously breathing patients with acute hypoxaemic respiratory failure: a systematic review and meta-analysis

Nonlinear imputation of Pa O2 /FIO2 from SpO2/FIO2 among mechanically ventilated patients in the ICU: a prospective, observational study

Chapter 24: including non-randomized studies on intervention effects

Cochrane Handbook for systematic Reviews of interventions. Version 6.2. Cochrane

Causal analyses of existing databases: no power calculations required

Awake prone positioning in COVID-19 hypoxemic respiratory failure: exploratory findings in a single-center retrospective cohort study

Early awake proning in critical and severe COVID-19 patients undergoing noninvasive respiratory support: a retrospective multicenter cohort study

Impact of prone position in non-intubated spontaneously breathing patients admitted to the ICU for severe acute respiratory failure due to COVID-19

Awake prone positioning does not reduce the risk of intubation in COVID-19 treated with high-flow nasal oxygen therapy: a multicenter, adjusted cohort study

Awake prone positioning for COVID-19 acute hypoxaemic respiratory failure: a randomised, controlled, multinational, openlabel meta-trial

Use of the GRADE approach in systematic reviews and guidelines