key: cord-0919810-uw7g3y71 authors: Chow, R.; Elsayed, S.; Lock, M. title: How robust are the results of one of the first positive trials exploring hydroxychloroquine for treatment of COVID-19? date: 2020-05-11 journal: nan DOI: 10.1101/2020.05.06.20093237 sha: cfaa14a325f94937d24de8efd4dc8c9b76368c30 doc_id: 919810 cord_uid: uw7g3y71 An outbreak of a novel human coronavirus infection emerged in Wuhan, China in December 2019. Two months later, the World Health Organization (WHO) announced SARS-CoV-2 as the name for the new virus and COVID-19 for the associated illness. On March 12, 2020, the WHO officially declared COVID-19 as a pandemic. The scientific community has raced to find effective therapeutic agents against the virus. Gautret et al 2020 is among one of the first purportedly positive trials of hydroxychloroquine for the treatment of COVID-19. However, it is imperative that a thorough analysis and understanding of trial data be undertaken prior to making claims about safety and efficacy. Our group assessed the statistical robustness of the trial using the Fragility Index (FI). The FI provides a numerical quantification of a clinical trial's conclusions. The index is based on iterative statistical calculations to determine the minimum number of events within a trial that would theoretically need to change from positive to negative in order for the trial's endpoint to convert from significant to non-significant; the higher the index, the more statistically robust the study results. For the Gautret et al trial, one endpoint had an FI of 1, two had indices of 2, and another had an index of 4. The primary endpoint of viral clearance on day 6 had an FI of 4. This indicates that if 4 events were to change from positive to negative, the conclusion of the trial would become mathematically non-significant. This index is comparable to many other published trials of established agents; the median FI across the reported literature appears to be 2. In conclusion, the trial results reported by Gautret et al are statistically robust, assuming that data quality is not compromised; however, the study was an open-label trial with non-homogenous groups, with analysis conducted per-protocol. Additionally, SARS-CoV-2 Reverse Transcriptase-PCR (RT-PCR) testing was not conducted in a systematic way amongst the two groups. Further analyses of this trial and future trials of antiviral agents with potential activity against SARS-CoV-2 should be performed with complementary epidemiologic and statistical techniques to determine whether the trial's results are clinically important and/or should be explored in depth. Given the statistically robust results reported by Gautret et al, despite the study's inherent methodological and analytical flaws, hydroxychloroquine should be studied as a potential agent against COVID-19 in larger clinical trials. Coronaviruses primarily target the human respiratory system. Over the past two decades, these viruses were responsible for three epidemics of infection. Prior to 2019, the two coronavirus public health threats were caused by the Severe Acute Respiratory Syndrome (SARS-CoV) and Middle East Respiratory Syndrome (MERS-CoV) coronaviruses [1] . In December 2019, an outbreak of a novel coronavirus was reported in Wuhan, China [2] and subsequently rapidly spread around the world. By March 12, 2020, the World Health Organization (WHO) officially declared COVID-19 a pandemic. At the time of this writing in mid-April 2020, there are over 165,000 deaths worldwide [3] . Currently there is no effective treatment although the scientific community is racing to find possible active agents for COVID-19 [4] . One of the first purportedly positive trials was published by Gautret et al [5] . Earlier versions were released due the urgent need to provide data to health care providers as the virus was spreading rapidly across continents. This includes the publication in medRxiv [6] which had slightly different results denoted; as these were not peer reviewed, we only used data from the final peer reviewed publication in the International Journal of Antimicrobial Agents [5] . Gautret et al treated 20 cases with hydroxychloroquine in Marseille as the first large surge of cases reached the critical care departments of Europe; this study found that the drug was significantly associated with viral load reduction in COVID-19 patients [5] . After 6 days of treatment, 100%, 57.1% and 12.5% of the hydroxychloroquine plus azithromycin group, hydroxychloroquine alone group, and control group had negative viral RT-PCR tests, respectively. This observation was statistically significant at p<0.01. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Such findings are especially important during a public health crisis, where scientific discoveries need to be both rigorous and timely. Positive trial results could motivate the scientific community to use the agents in infected patients, or commit funding and clinical resources to explore this drug regimen further. However, it is important to pause for a moment to ensure that the purportedly positive trial results are indeed robust. Making inferences about this regimen without robust results may take away precious time and resources related to other promising therapies. Furthermore, the benefits of a new regimen need to be weighed against its potential harms; therefore, hydroxychloroquine use needs to be carefully weighed against potential harms such as prolonging of the QTc interval which can be life threatening. There have been several critical appraisals of this trial, but appraisals are in many ways subjective and in this urgent and politicized subject, we sought to more quantitatively assess the value of this trial. To assess for robustness of a positive trial, a Fragility Index (FI) could be calculated. As described by Walsh et al [7] , this index complements the p-value by informing the robustness of results. The index was developed by leaders in the field of evidence-based medicine, including Guyatt and Sackett, to provide information beyond the much criticized, but commonly reported p-value. The FI is most applicable and useful in trials with small numbers of events where the p-values can often erroneously suggest that a trial is positive; this may be the case for the small-sample trial published by Gautret et al. The index adds information beyond that provided by historically used measures such as p-value, number of events, confidence interval and sample size. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020. . https://doi.org/10.1101/2020.05.06.20093237 doi: medRxiv preprint The aim of this paper is to determine the fragility of the trial by Gautret et al, one of the first purportedly positive trials exploring hydroxychloroquine for the treatment of COVID-19. The FI is the minimum number of patients that needs to change from a negative to a positive event, counterfactually, in the non-favoured group for a study's positive result to become negative/non-significant. The higher the FI, the more robust a positive trial's results. The median FI of published positive trials is reported to be between 2 and 3 [8] [9] [10] [11] . For example, in a trial where there are 10 patients in each the control and experimental arms and 5 patients with a positive event in the experimental arm, the results of 1 patient in the control arm needs to change from a negative to a positive event for a non-significant difference [ Table 1 ]. As the number of events that needs to be changed is 1, the FI is therefore 1. The number of patients with negative nasopharyngeal Reverse Transcriptase-PCR (RT-PCR) results among the number of patients per hydroxychloroquine and control arms, per day, were analyzed. To calculate the FI, the event number was incrementally increased by positive integer values until there was no statistical difference between the two trial arms. Fisher's exact test was calculated for each counterfactual simulation, using Stata 15. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. At day 3, 10 of 20 hydroxychloroquine patients had a virological cure, compared to 1 of 16 patients receiving placebo. A minimum of 2 additional control patients theoretically needed to have a virological cure in order to observe a nonsignificant difference. The FI for day 3 was 2 [ Table 2 ]. The FI for virological cure by day 4 is 1, 2 for day 5, and 4 for day 6 [Tables [3] [4] [5] . [11] . In the field of ophthalmology, the median index is 2 across 156 published trials [10] . Oncology trials are reported to have a median index of 2, according to a 36-trial review by Del Paggio and Tannock [12] . Perhaps the most pertinent assessment with respect to the COVID-19 trial by Gautret et al is a calculation of the FI in 57 critical care trials by Ridgeon All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020. Also of note is the lack of systematic RT-PCR testing between the groups consequently time bias. The hydroxychloroquine group was, on average, more severely ill with COVID-19. Therefore, we would expect their RT-PCR tests to become negative quicker, even without treatment, due to a time bias. Furthermore, RT-PCR cannot even be used as a test of virological All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020. . https://doi.org/10.1101/2020.05.06.20093237 doi: medRxiv preprint All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 11, 2020. . https://doi.org/10.1101/2020.05.06.20093237 doi: medRxiv preprint The epidemiology and pathogenesis of coronavirus disease (COVID-19) outbreak Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and coronavirus disease-2019 (COVID-19): The epidemic and the challenges Situation Report -89 [Internet]. Geneva: World Health Organization Covid-19 -The Search for Effective Therapy Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label randomized clinical trial Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index The fragility of statistically significant findings from randomized trials in spine surgery: a systematic spine The fragility index in multicenter randomized controlled critical care trials No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity Fragility of results in ophthalmology randomized controlled trials: a systematic review The fragility of statistically significant results in pediatric orthopaedic randomized controlled trials as quantified by the fragility index: a systematic review The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis A rush to judgement? Rapid reporting and dissemination of results and its consequences regarding the use of hydroxychloroquine for COVID-19 No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity