key: cord-0977902-qsa8jqls
authors: Renoux, Christel; Azoulay, Laurent; Suissa, Samy
title: Biases in evaluating the safety and effectiveness of drugs for covid-19: designing real-world evidence studies
date: 2021-02-10
journal: Am J Epidemiol
DOI: 10.1093/aje/kwab028
sha: 18760f459ef00b2b41a33cd7363ff345fe6fffec
doc_id: 977902
cord_uid: qsa8jqls

The coronavirus disease 2019 (COVID-19) pandemic caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has led to an unprecedented effort to generate real-world evidence on the safety and effectiveness of various treatments. A growing number of observational studies evaluating the effects of certain drugs have been conducted, including several assessing whether hydroxychloroquine improves outcomes in infected individuals and whether renin-angiotensin-aldosterone system inhibitors have detrimental effects. We review and illustrate how immortal time bias and selection bias were present in several of these studies. Understanding these biases and how they can be avoided may prove important for future observational studies assessing the effectiveness and safety of potentially promising drugs during the COVID-19 pandemic.

The coronavirus disease 2019 (COVID-19) pandemic caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has led to an unprecedented effort to generate real-world evidence on the safety and effectiveness of various treatments. While the randomized controlled trial (RCT) is widely accepted as the design providing the most definitive results, the generalizability of its findings to the real-world setting can be challenging. Indeed, RCTs often use strict selection criteria and treatment adherence that may differ with the realworld setting. 1 Moreover, compared with observational studies, RCTs may take longer to implement, and therefore their findings may take longer to reach the scientific community, which is a particular concern in the context of a rapidly evolving pandemic. Thus, by leveraging the rapidly accumulating data on patients hospitalized with COVID-19, a growing number of observational studies evaluating the effects of certain drugs have been conducted and published at an impressive pace. This includes assessing the effectiveness of hydroxychloroquine and evaluating whether angiotensin-converting enzyme inhibitors (ACEIs) and angiotensin II In cohort studies assessing drug effectiveness and safety, immortal time is typically introduced when treatment status is defined based on a prescription issued or received at some point during follow-up. The time period between disease diagnosis or cohort entry and the first treatment prescription is necessarily immortal since the patient had to survive or be outcome-free (owing to the censoring of events in the analysis) to be classified as exposed. Immortal time bias is then introduced when this immortal period between diagnosis or cohort entry and the first treatment prescription is misclassified as exposed rather than correctly accounted for as unexposed.

Immortal time bias is also introduced when this immortal period is excluded, with cohort entry defined as the date of treatment initiation for exposed patients and defined as the date of disease diagnosis or hospitalization for non exposed patients. Immortal time bias is common in cohort studies of drug effects and systematically biases the results downward in favor of the treatment under study. Consequently, this bias can make harmful treatments appear neutral, and neutral treatments appear protective.

Most recent cohort studies assessing the effectiveness of hydroxychloroquine on mortality in patients hospitalized with COVID-19 determined exposure, in their primary analysis, based on treatment received at any time during follow-up or typically within 48 hours of hospitalization. However, patients were considered exposed as of the date of hospital admission, thereby introducing immortal time bias ( Figure 1A ). [4] [5] [6] [7] [8] [9] Indeed, the time period between cohort entry (date of hospitalization) and treatment initiation is immortal since the patients had to survive or be outcome-free to be classified as treated. This immortal period between the date of hospitalization and treatment initiation was misclassified as exposed rather than correctly accounted for as unexposed. Immortal time bias was also introduced when patients receiving hydroxychloroquine during hospitalization were followed from the date of starting treatment while patients not exposed were followed from the date of hospitalization. In this instance, the immortal period between the date of hospitalization and treatment initiation was excluded from the analysis. 10 The magnitude of the bias and its overall impact on the results depends on the duration of the immortal period, the duration of follow-up, the number of exposed in the cohort, and the event rate. Although we focused on studies assessing the effectiveness of hydroxychloroquine on mortality, immortal time bias was also introduced in studies evaluating other drugs. For instance, immortal time bias was present in a recent cohort study evaluating the association between ACEIs/ARBs and all-cause mortality in patients with hypertension hospitalized with COVID-19. 11 Indeed, patients who received ACEs/ARBs at any time during the hospitalization were considered exposed from the date of admission until the end of follow-up, regardless of the timing of treatment initiation.

Another critical design aspect is the selection of patients to be included in, and particularly excluded from, the cohort. As such, the exclusion of patients based on an event or

treatment occurring at some point during follow-up can lead to selection bias ( Figure 1B ). For example, in some cohort studies, exposure was based on receiving hydroxychloroquine within 48 hours of hospitalization; patients initiating treatment more than 48 hours after hospital admission were excluded in the primary analysis. 8, 10 This exclusion may introduce selection bias in addition to immortal time bias. Both immortal time and selection bias were also at play in a study where the cohort was restricted to patients with at least six days of follow-up and a minimum of three days of treatment with hydroxychloroquine, but cohort entry was defined as the date of hospital admission. 12 Also, this study did not have any comparator group. Similarly, excluding patients who did not experience the outcome or were not yet discharged by the end of the study period is incorrect. 4 A flowchart describing cohort selection with numbers of patients excluded and reasons for exclusion should therefore be provided to assess the potential for such bias. It should be noted that these methodological issues are introduced by the investigators at the design or analysis stage, and thus, are not inherent 'flaws' of cohort studies. Moreover, as these biases are information and selection biases, methods used to deal with confounding, such as propensity scores, would not correct these biases.

Several options can be used at the design or analytical stage to prevent these biases. One approach is to define exposure at the date of hospitalization (cohort entry) and thus consider as optimally use the information from all patients exposed to the drug of interest over time.

Moreover, the study population does not include patients who die early after hospitalization.

Finally, one caveat of this approach is that it may introduce some exposure misclassification. A second approach is to use a time-varying exposure definition at the analytic stage. The study cohort comprises all consecutive patients hospitalized with COVID-19 during a specific time period with cohort entry defined as the date of hospitalization. For each patient, each day of follow-up is classified as either exposed or not exposed to the drug of interest, allowing patients to move from a period of non-exposure to a period of exposure. Once treatment is initiated, patients can be considered exposed for the remainder of the follow-up regardless of treatment discontinuation (analogous to an intention to treat approach), censored when treatment is stopped, or their person-days of follow-up after treatment cessation classified as unexposed. A grace period can be added after treatment discontinuation where patients are still considered exposed to account for the residual biological effect of the drug under study. Time-dependent

Cox proportional hazards models are then used to estimate hazard ratios for the association between current use of the drug under study and the outcome of interest.

Finally, a design approach aimed at emulating a target trial in this setting can be the prevalent new-user cohort design. 13 proposed by others, such as creating sequential cohorts at predetermined time intervals. 14, 15 Aside from these design issues, the potential for confounding by indication is always a concern in drug effectiveness studies, particularly in the current pandemic where no treatments are available. Indeed, off-label treatments are typically given preferentially to moderate to severe

patients at the time of hospital admission or to those with a worsening condition during followup; in extreme situations, the treatment is given for compassionate use to highly severe patients.

Thus, depending on the clinical context, the confounding may be intractable so that available statistical methods will not be able to control for this bias. The rapidly changing treatment recommendations may create an additional challenge in adequately balancing the exposure groups. Finally, traceable and transparent data are prerequisites to the above considerations, as recently reminded to the scientific community.

In summary, real-world data are useful to complement evidence from RCTs and can even predict their results in some settings. 16 However, recently published cohort studies assessing the effectiveness and safety of drugs in patients hospitalized with COVID-19 illustrate the importance of carefully designing and analyzing such studies. While much attention is paid to confounding, fundamental methodological principles must also be applied to derive meaningful conclusions. Methods exist, such as the prevalent new-user design, that allow to avoid these biases and permit a proper control for confounding. 13 

Invited commentary: every good randomization deserves observation

Immortal time bias in pharmaco-epidemiology

Problem of immortal time bias in cohort studies: example using statins for preventing progression of diabetes

Treatment with hydroxychloroquine, azithromycin, and combination in patients hospitalized with COVID-19

Low-dose hydroxychloroquine therapy and mortality in hospitalised patients with COVID-19: a nationwide observational study of 8075 participants

Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19

Hydroxychloroquine and tocilizumab therapy in COVID-19 patients-An observational study

Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data

Association of Treatment With Hydroxychloroquine or Azithromycin With In-Hospital Mortality in Patients With COVID-19 in New York State

Compassionate use of hydroxychloroquine in clinical practice for patients with mild to severe Covid-19 in a French university hospital

Association of Inpatient Use of Angiotensin Converting Enzyme Inhibitors and Angiotensin II Receptor Blockers with Mortality Among Patients With Hypertension Hospitalized With COVID-19

Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: A pilot observational study. Travel medicine and infectious disease

Prevalent new-user cohort designs for comparative drug effect studies by time-conditional propensity scores

Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease

Propensity Score Methods for Analyzing Observational Data Like Randomized Experiments: Challenges and Solutions for Rare Outcomes and Exposures

Using Real-World Data to Predict Findings of an Ongoing Phase IV Cardiovascular Outcome Trial: Cardiovascular Safety of Linagliptin Versus Glimepiride