key: cord-0821069-p1q56a65 authors: Shepshelovich, Daniel; Yahav, Dafna; Ben Ami, Ronen; Goldvaser, Hadar; Tau, Noam title: Concordance between the results of randomized and non-randomized interventional clinical trials assessing the efficacy of drugs for COVID-19: a cross-sectional study date: 2021-06-02 journal: J Antimicrob Chemother DOI: 10.1093/jac/dkab163 sha: 2f70dd57ed6532bff2040bbf04b3043ae238484d doc_id: 821069 cord_uid: p1q56a65 OBJECTIVES: To assess whether results of observational studies of potential anti-COVID-19 drugs were reproduced in subsequent randomized controlled trials (RCTs). METHODS: This was a retrospective cross-sectional study, including studies published online between 1 January and 27 October 2020 that evaluated potential COVID-19 treatments and reported all-cause mortality. RESULTS: Of 133 comparisons included in 117 studies, most were non-randomized (104/133, 78%). Hydroxychloroquine was the most common drug type, combined with azithromycin (n = 27, 20%) or alone (n = 22, 16%), followed by IL-6 inhibitors (n = 36, 27%) and corticosteroids (n = 26, 20%). Seventy-one percent (74/104) of non-randomized studies reported adjusted survival results and only 8% (8/104) adjusted for immortal time bias. Only two RCTs (2/29, 7%) reported significant survival benefit, both reporting treatment with corticosteroids, while 32/104 (31%) non-randomized studies showed statistically significant survival benefit associated with the intervention arm. The results of the majority (28/32, 88%) of non-randomized studies reporting survival benefit were not replicated by large-scale RCTs. CONCLUSIONS: The results of most non-randomized studies reporting survival benefit of potential anti-COVID-19 drugs were not replicated by large RCTs. Regulators and healthcare professionals should exercise caution and resist the pressure to approve and prescribe drugs of unproven efficacy and potential toxicity to optimize patient care and maintain public trust in medical science. Randomized controlled trials (RCTs) are recognized as the gold standard for assessing the efficacy and safety of novel medical interventions. However, RCTs are expensive and time consuming, limiting their effectiveness for delivering immediate results for urgent medical issues. 1 Facing the exponentially rising tolls of COVID-19-related hospitalizations and death, the medical and scientific communities faced an urgent need to identify effective treatments. Many early reports of potentially effective anti-COVID drugs were observational and some showed survival advantages associated with investigational interventions. [2] [3] [4] Some of those reported findings were not reproduced by subsequent RCTs, reflected by rapid changes in treatment recommendations. 5, 6 We aimed to describe the concordance between the reported survival benefit of RCTs and non-randomized studies of potential COVID-19 drugs. We hypothesized that the results of most observational trials will not be replicated by subsequent RCTs. We searched PubMed for studies published online between 1 January and 27 October 2020 reporting survival results of medical interventions in COVID-19 patients. The search term combined 'COVID-19 OR SARS-CoV-2' with interventions considered as promising anti-COVID-19 agents at the time of the study's design: hydroxychloroquine±azithromycin, corticosteroids, remdesivir, convalescent plasma, IL-6 inhibitors, antiretroviral drugs and anti-influenza drugs. Search terms included both drug categories and specific drug names. Three authors (N.T., D.Y. and D.S.) independently reviewed the identified studies and included only those comparing interventional drugs with placebo or with standard-of-care therapy. Studies reporting outcomes of patient groups treated with several anti-COVID-19 interventions were excluded. In studies including several treatment arms compared with placebo, each arm was considered separately for our final analysis. For each study, the following data were collected: journal impact factor (IF), study design (RCT versus non-randomized studies), single centre versus multicentre, patients' COVID-19 disease severity according to the NIH classification (mild, moderate, severe and critical), 7 sample size, patient age and gender, primary study endpoint (overall survival versus other), survival follow-up duration and funding source (industry versus other/none). For non-randomized studies reporting adjusted survival analysis, the adjustment method (propensity score weighting, patient matching or regression) was noted. Adjustments for immortal time bias were also noted. For studies reporting crude mortality rates without adjustments, the relative risk of death was extracted or calculated. Data were reported descriptively for all studies as well as separately for RCTs and non-randomized studies. Associations between study characteristics and study design were assessed using Fisher's exact test for categorical variables and the Mann-Whitney test for continuous variables. Pre-planned sensitivity analyses of survival benefit according to study design included: exclusion of studies unlikely to show statistically significant survival benefit (i.e. studies with patients with mild COVID-19 and those including <50 patients), exclusion of multi-armed trials and exclusion of studies published in journals with an IF <2. All statistical analyses were conducted using IBM SPSS Statistics v.25.0 (IBM Corp., Armonk, NY, USA). The literature search yielded 3528 unique titles. Exclusion reasons are shown in Figure S1 (available as Supplementary data at JAC Online). A total of 117 studies comprising 133 interventional arms were included (Table 1) . Eight studies included two intervention arms: six assessed hydroxychloroquine and a combination of hydroxychloroquine and azithromycin, and two evaluated hydroxychloroquine and lopinavir/ritonavir. Four studies included three intervention arms (hydroxychloroquine, azithromycin and a combination of both). Hydroxychloroquine was the most common drug type, combined with azithromycin (n = 27, 20%) or alone (n = 22, 16%), followed by IL-6 inhibitors (n = 36, 27%) and corticosteroids (n = 26, 20%). Most studies (n = 104, 78%) were non-randomized. Drug-type distribution was comparable in randomized and non-randomized studies, except for IL-6 inhibitors, which were more commonly represented in non-randomized studies [ Of 104 non-randomized study arms included, 74 (71%) reported adjusted survival analysis between the intervention and control arms, and 30 studies (29%) reported only crude mortality rates. Forty-one studies (55%) used propensity adjustment analysis, 24 (32%) used regression and 9 (12%) used matched cases-controls. Studies adjusted for a median of 8 parameters (IQR = 6-14). Only eight non-randomized studies (8%) contained any type of adjustment for immortal time bias. Statistically significant survival results of the included studies according to drug type and study design are shown in Table 2 . Only two (2/29, 7%) RCTs reported significant survival benefit associated with the intervention arm, both testing corticosteroid efficacy. Thirty-two (32/104, 31%) non-randomized studies showed statistically significant survival benefit associated with the intervention arm, mostly (28/32, 88%) not replicated by large RCTs. Results remained unchanged in multiple pre-planned sensitivity analyses (Table S1 ). Healthcare professionals have published their experience with potential anti-COVID-19 drugs in real-world settings, without randomization of patients to treatment and control arms. Here, we compared the survival benefit reported by these studies to findings from large RCTs evaluating the same drugs. Most reports of survival benefit by non-randomized studies were not reproduced by subsequent large RCTs. The lack of concordance between the positive results of nonrandomized studies and those reported by large RCTs is concerning. First, some non-randomized studies have led to the widespread prescribing of ineffective drugs, with potential adverse effects. [8] [9] [10] Second, massive prescribing of some of these drugs led to market shortages, denying effective treatments from patients with medical conditions known to respond to these drugs. 11 Third, the hype and hope created by non-randomized studies, 12 sometimes supported by official agencies, 13, 14 with treatments later reported to be ineffective, 15, 16 might have created confusion and mistrust in the soundness of medical science, even more so when the excitement-disappointment cycle was repeated in a short time span. 17 There are several potential explanations for the large survival benefit reported by many non-randomized studies not replicated by subsequent RCTs. Physicians may have reserved expensive, experimental drugs, or those in limited quantity, for patients thought to have better survival chances, introducing selection bias into the study results. Observational studies are also prone to immortal time bias, as patients who die early will not receive the investigational drug, leading to a biased positive treatment effect measure. 18 Publication bias might cause the rapid publication in prestigious journals of studies reporting survival benefit of experimental COVID-19 therapies. The challenges of conducting large RCTs in settings overwhelmed by COVID-19 are formidable, including the recruitment of many medical centres committed to a single protocol, patient recruitment in the face of experimental uncertainty and a potentially life-threatening infection, drug delivery, blinding and seamless data collection. However, even in this state of emergency, conducting large RCTs is both feasible and rewarding. Large, multiarm RCTs, such as the RECOVERY and SOLIDARITY trials, have provided the highest-quality data regarding the efficacy, or lack thereof, of potential COVID-19 treatments. 15, 16 Until such high-quality data are available, observational data should be used with caution to guide patient care. Healthcare professionals should consider only high-quality observational studies in their decision-making; those using a quasi-randomized design (e.g. drugs available in one hospital but not in another serving similar patients), following large patient cohorts for long durations, adjusting for patient characteristics and potential biases, using patient-centred endpoints and reporting results according to the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines. Results reported by such high-quality observational studies can be very similar to results reported by RCTs. 19, 20 Large clinical datasets worldwide can be analysed in real-time using predetermined methods to guide clinicians and regulators in uncertain times, until higher-quality data are available. Comparable data collection, analysis of different datasets, data sharing and transparency can facilitate rapidly available, high-quality observational data in times of crisis. Ultimately, different data types from variable sources should be integrated to optimize clinical and regulatory decisionmaking. This study has several limitations. Inclusion of different anti-COVID-19 drug classes, or of endpoints other than overall survival, might have led to different results. Multivariable analysis of the association between study characteristics and concordance of the results of non-randomized studies and large RCTs was not undertaken due to an insufficient number of RCTs to properly fit a multivariable model. Lastly, the evaluation of some anti-COVID-19 drugs is evolving and some included non-randomized studies might yet be supported by future RCTs. In conclusion, the results of most non-randomized studies reporting survival benefit of potential anti-COVID-19 drugs were not replicated by large RCTs. Until high-quality evidence is available, regulators and healthcare professionals should exercise caution and resist the pressure to approve and prescribe drugs of unproven efficacy and potential toxicity to optimize patient care and maintain public trust in the quality of medical science. This study was carried out as part of our routine work. H.G. reports personal fees from Roche (honorarium), Pfizer (honorarium), Novartis (honorarium and consulting) and Oncotest (honorarium), all outside of the submitted work. All other authors: none to declare. Innovation in the pharmaceutical industry: new estimates of R&D costs Compassionate use of remdesivir for patients with severe Covid-19 Treatment of 5 critically ill patients with COVID-19 with convalescent plasma Observational study of hydroxychloroquine in hospitalized patients with Covid-19 Coronavirus (COVID-19) Update: FDA Revokes Emergency Use Authorization for Chloroquine and Hydroxychloroquine Recommends Against the Use of Remdesivir in COVID-19 Prescription fill patterns for commonly used drugs during the COVID-19 pandemic in the United States The good, the bad, and the hoax: when publication instantaneously impacts treatment strategies for COVID-19 Chloroquine and hydroxychloroquine for the treatment of COVID-19: a systematic review and meta-analysis Hydroxychloroquine shortages among patients with systemic lupus erythematosus during the COVID-19 pandemic: experience of the Systemic Lupus International Collaborating Clinics Internet searches for unproven COVID-19 therapies in the United States Letter to Dr Rick Bright. Re: Request for Emergency Use Authorization for Use of Chloroquine Phosphate or Hydroxychloroquine Sulfate Supplied From the Strategic National Stockpile for Treatment FDA News Release. Coronavirus (COVID-19) Update: FDA Issues Emergency Use Authorization for Potential Repurposed antiviral drugs for Covid-19 -interim WHO Solidarity Trial results Randomised Evaluation of COVID-19 Therapy Model uncertainty, political contestation, and public trust in science: evidence from the COVID-19 pandemic The misleading "pooled effect estimate" of crude data from observational studies at critical risk of bias: the case of tocilizumab in COVID-19 Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials Use of health care databases to support supplemental indications of approved medications All authors participated in the research and preparation of the manuscript. Figure S1 and Table S1 are available as Supplementary data at JAC Online.