key: cord-0691845-t07jwfvt
authors: Martinuka, Oksana; von Cube, Maja; Wolkewitz, Martin
title: Methodological evaluation of bias in observational COVID-19 studies on drug effectiveness
date: 2021-04-01
journal: Clin Microbiol Infect
DOI: 10.1016/j.cmi.2021.03.003
sha: 23cfd77438092441d8b4aa6e10a83070acd66f0b
doc_id: 691845
cord_uid: t07jwfvt

BACKGROUND AND OBJECTIVE: Observational studies may provide valuable evidence on real-world causal effects of drug effectiveness in patients with Coronavirus Disease 2019 (COVID-19). Since patients are usually observed from hospital admission to discharge and drug initiation starts during hospitalization, advanced statistical methods are needed to account for time-dependent drug exposure, confounding, and competing events. Our objective is to evaluate the observational studies on the three common methodological pitfalls in time-to-event analyses: immortal time bias, confounding bias, and competing risk bias. METHODS: We performed a systematic literature search on October 23, 2020, in the PubMed database to identify observational cohort studies that evaluated drug effectiveness in hospitalized patients with COVID-19. We included articles published in four journals: The British Medical Journal (The BMJ), the New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), and The Lancet as well as their sub-journals. RESULTS: Overall, out of 255 articles screened, eleven observational cohort studies on treatment effectiveness with drug exposure-outcome associations were evaluated. All studies were susceptible to one or more types of bias in the primary study analysis. Eight studies had a time-dependent treatment. However, the hazard ratios were not adjusted for immortal time in the primary analysis. Even though confounders presented at baseline have been addressed in nine studies, time-varying confounding caused by time-varying treatment exposure and clinical variables was less recognized. Only one out of eleven studies addressed competing event bias by extending follow-up beyond patient’s discharge. CONCLUSIONS: In the observational cohort studies on drug effectiveness for treatment of COVID-19 published in four high impact journals, the methodological biases were concerningly common. Appropriate statistical tools are essential to avoid misleading conclusion and receive a better understanding of potential treatment effects.

With a growing number of publications on potential therapeutic candidates for the 34 coronavirus disease 2019 (COVID-19) treatment, high-quality observational studies have added value 35

to the assessment of drug benefit in the real-world health care setting [1, 2] . However, the 36 observational study design has important limitations and poses several challenges in the data analysis 37 particularly regarding the time-dependent nature of the data [3] . Ignorance of methodological biases 38 in observational studies with time to event analysis may lead to distorted results and false conclusions 39 on the exposure-outcome associations [4, 5] . The aim of this article is to review the observational 40 studies on evaluation of drug effectiveness in COVID-19 patients with regard to the presence of three 41 methodological biases referred to as immortal time bias, confounding, and competing risk bias. 42

Furthermore, this work also aims to give recommendations on avoiding these biases. 43

In contrast to randomized clinical trials, in observational cohort studies a drug of interest is 44 often prescribed after initiation of a study, e.g., later during follow-up. Immortal time typically occurs 45 when there is a delay or waiting period between cohort entry and the time of the first prescription that 46 is falsely accounted for as drug-exposed time. Thus, exposed subjects must survive the initial time 47 period to receive treatment if they are not assigned to the unexposed cohort [5, 6] . Exclusion or 48 misclassification of observation time often leads to immortal time bias and consequently to artificial 49 over-or underestimation of drug effectiveness [6, 7] . 50

Control of both time-fixed and time-varying confounding is crucial due to the lack of 51 randomization in observational real-world data [8] . In contrast to time-fixed confounding bias, time-52 varying confounding is commonly encountered in longitudinal observational studies [9] [10] [11] . In 53 clinical epidemiology, treatment exposures are often time-varying and the values of potential 54 confounders may change during the observational period leading to time-varying confounding 55

[12,13]. A lack of control of confounding may lead to biased estimates of treatment effects and causal 56 misinterpretation [11, 14] . 57

Another issue that is often observed in observational studies with survival or time-to-event analysis is 58 the competing risk bias. By definition, a competing risk is an event that modifies the chance of 59 J o u r n a l P r e -p r o o f occurrence of the primary event of interest and can occur when a patient is at risk of more than one 60 type of event [15] . 61

Competing risk events are frequently observed in hospital epidemiology when the follow-up ends 62 with hospital discharge. In turn, hospital discharge is a competing risk for the hospital death that is 63 often the outcome of interest [7, 16] . In survival analyses, the survivor function and the hazard 64 function are the two most common methods for representation of survival data. In the presence of 65 competing risks, the naïve Kaplan-Meier (KM) estimator takes the competing risks as censored 66

observations. As a result the KM analysis overestimates cumulative risks and produces upward biased • To avoid competing risk bias, a competing risk analysis has to be performed to describe results on all cause-specific hazards and visualized using cumulative hazard functions. If there is no follow-up beyond hospital discharge, the discharge has to be handled as a 'competing event' in the statistical analysis.

• Immortal time bias, time-fixed confounding and competing risk bias can be addressed simultaneously by applying a cause-specific Cox regression for an event of interest and a competing event with the inclusion of treatment as a time-dependent covariate.

• If robust time-varying clinical data are available and applicable, time-varying confounding should be addressed using a marginal structural Cox model or other g-methods for causal inference.

• Data analysis should be performed in a stepwise manner, starting from simple straightforward methods to increased model complexity. Lancet. Also, their sub-journals were selected. These journals were chosen because they are the 78 leading and most cited medical journals with high impact factors. This review required original 79 retrospective observational cohort studies with primary data including time-to-event data analysis. 80

Thus, comments, correspondence, opinions, researcher letters, and audio interviews were excluded. Systematic Reviews and Meta-analyses (PRISMA) flow diagram (Fig. 1) . 92

The systematic literature review was performed in the PubMed database on October 23, 2020. 94

The review was conducted according to the PRISMA guidelines for reporting systematic review [19] . 95

The list of keywords and detailed search strategy is described in the appendix (Supplementary 96

Material Tables S1 and S2). 97

The assessment of biases was performed independently by all three authors. Each 99 observational study was reviewed in-depth to determine possible presence of immortal time bias, 100 confounding bias, and competing risk bias by screening the methods as well as the results section. The 101 design, methods, and statistical techniques were evaluated. Studies were considered to be susceptible 102 to the immortal time bias if the time period before treatment allocation was not addressed in the 103 analysis and thus, time-dependent drug exposure was statistically handled as a time-fixed exposure. 104

The susceptibility to both time-fixed and time-varying confounding biases were evaluated. The 105 occurrence of time-fixed confounding biases was determined if baseline covariates were not adjusted. Studies were considered as being susceptible to competing risk bias if follow-up of patients was 112 ensured only until discharge and individuals who were discharged alive were censored at the time of 113 discharge for quantifying the probability of experiencing the event of interest. To identify whether the 114 biases were addressed, the study methodology (i.e., study design and applied analytical methods) and 115 results were evaluated. In addition, the supplementary materials of the included articles were checked. 116

Out of 255 articles screened, eleven observational cohort studies on the drug effectiveness for 118 COVID-19 treatment in hospitalized patients were included; six articles from The Lancet and its sub-119 journals, three articles from the Journal of the American Medical Association and its sub-journals as 120 well as one from the British Medical Journal and one from the New England Journal of Medicine 121 (Table 1) [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] . These observational studies investigated the effectiveness of drugs such as 122 anakinra, azithromycin, chloroquine or hydroxychloroquine, methylprednisolone, and tocilizumab. 123 J o u r n a l P r e -p r o o f These drugs were administered alone or in combination with standard therapy. All of these studies 124

were susceptible to at least one of the three discussed types of bias (Fig. 2) . The results and examples 125 of the identified biases are given in the following sections. 126

Overall, eight studies were susceptible to immortal time bias [20] [21] [22] [23] [24] [25] 27, 29] ; in the three 128 remaining studies, the start of the follow-up and/or the start of treatment exposure administration were 129 not clearly defined [26, 28, 30] . In two of these studies treatment was artificially converted into a time-130 fixed exposure by using different time scales for the treated and untreated cohorts, i.e. for the 131 untreated group the start of follow-up was at time zero whereas for the exposed cohort the start of 

Out of the eleven studies susceptible to confounding, nine studies used methods to account for 

Several time-to-event primary outcomes were investigated in the studies, such as 169 development of acute respiratory distress syndrome, admission to ICU, administration of invasive 170 mechanical ventilation (IMV), in-hospital death or 30-day in-hospital mortality, survival without 171 transfer to ICU, and overall survival. These endpoints were studied as a single event, or as a 172 composite endpoint of several events (Table 3) . 173

All in all, ten out of the eleven studies seem to be susceptible to competing risk bias as a 174

follow-up ended with hospital discharge and discharged patients were censored in probability 175

analyses [20] [21] [22] [23] [24] [25] [26] [28] [29] [30] . In nine out of the ten studies, the classical KM method was applied to detailed tutorial on causal inference and g-methods [40] . 219

In the presence of competing risks, we discourage to use KM plots for effect visualization due 220 to the high risk of potentially misleading conclusions. Instead, cumulative cause-specific hazards can 221 be calculated and should be plotted for the events of interest and for the competing events [41] . This 222 method allows to account for competing events and to display treatment exposure differences [16] . The immortal time bias, confounding bias, and competing risk bias alone or in combination 247

were present in all of the eleven reviewed observational studies on treatment effectiveness evaluation 248 for hospitalized patients with COVID-19. These biases may have led to a severe over-or 249 underestimation of COVID-19 treatment effectiveness estimates. Thus, the drugs may appear to be 250 either more effective or conversely with a little-to-no effect. In fact, making valid causal inferences 251 from real-world observational data is a demanding task that requires high-quality data and adequate 252 statistical methods as well as clinical knowledge and statistical expertise. 253

Even though every study has its unique features which should be addressed in a tailor-made 254 analysis, the summary points (Box 1) and the following recommendations could be taken into account 255 to prevent or decrease the occurrence and the severity of the methodological biases. To overcome 

The authors declare no conflicts of interest. No funding was received for this study. 266 

How do observational studies expand the evidence base for therapy? 273

Observational studies and their utility for practice

Bias in pharmacoepidemiologic studies using 277 secondary health care databases: a scoping review

Observational studies of treatment effectiveness: worthwhile 280 or worthless?

Time-related biases in pharmacoepidemiology

Immortal time bias in observational studies of drug effects

Survival biases lead to flawed conclusions in observational 286 treatment studies of influenza patients

Controlling for Time-dependent 289 Confounding using Marginal Structural Models

Handling time varying 292 confounding in observational research

Marginal structural models in clinical research: when and how to use 294 them?

Confounding in longitudinal studies in addiction treatment 296 research

Methods for time-varying 299 exposure related problems in pharmacoepidemiology: An overview

Estimation of Average Treatment Effects Based on Propensity Scores. The 302

An easy mathematical proof showed 304 that time-dependent bias inevitably leads to biased effect estimation

Competing risks in survival data analysis

Neuraminidase Inhibitors and Hospital Mortality in British 309 Patients with H1N1 Influenza A: A Re-Analysis of Observational Data

Interpreting and 312 comparing risks in the presence of competing events

Ignoring competing 315 events in the analysis of survival data may lead to biased results: a nonmathematical illustration 316 of competing risk analysis

The PRISMA 319 statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare 320 interventions: explanation and elaboration

Tocilizumab among patients with 323 COVID-19 in the intensive care unit: a multicentre observational study. The Lancet 324

Interleukin-1 326 blockade with high-dose anakinra in patients with COVID-19, acute respiratory distress 327 syndrome, and hyperinflammation: a retrospective cohort study

Hydroxychloroquine in Hospitalized Patients with Covid-19

Tocilizumab in 333 patients with severe COVID-19: a retrospective cohort study

Early Treatment With Tocilizumab and Mortality Among Critically Ill Patients With COVID-19. 337 JAMA internal medicine 2020

Anakinra for severe 339 forms of COVID-19: a cohort study

Clinical impact 342 of COVID-19 on patients with cancer (CCC19): a cohort study

Clinical efficacy of 345 hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational 346 comparative study using routine care data

RETRACTED: Hydroxychloroquine or 349 chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry 350 analysis

Treatment With Hydroxychloroquine or Azithromycin With In-Hospital Mortality in Patients 353 With COVID-19 in

Risk Factors Associated With Acute 356 Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease

A Proportional Hazards Model for the Subdistribution of a Competing Risk

Pandemic Publishing: Medical journals drastically speed up their publication 363 process for Covid-19

Statistical Analysis 366 of Clinical COVID-19 Data: A Concise Overview of Lessons Learned, Common Errors and 367 How to Avoid Them

Methodological challenges of analysing COVID-19 data during the 369 pandemic

Quantitative risk assessment: is more complex always better? Simple is not 372 stupid and complex is not always more correct

Analysis of time-to-event for observational studies: Guidance to the use of intensity models

Competing risks need to be 378 considered in survival analysis models for cardiovascular outcomes

Competing risks 381 analyses: objectives and approaches

Multistate Modeling of COVID-19 Patients Using a Large Multicentric Prospective Cohort of 385

Critically Ill Patients

Causal inference

Introduction to the Analysis of Survival Data in the Presence of 388 Competing Risks

Application of 391 multistate models in hospital epidemiology: advances and challenges

Harmonizing heterogeneous 394 endpoints in COVID-19 trials without loss of information -an essential step to facilitate decision 395 making

Remdesivir Use in 397 Patients Requiring Mechanical Ventilation due to COVID-19

Immortal time bias in observational studies of time-to-event outcomes

A competing risks analysis should 402 report results on all cause-specific hazards and cumulative incidence functions

J o u r n a l P r e -p r o o f