key: cord-275110-safr9z37 authors: Alexander, Paul Elias; Debono, Victoria Borg; Mammen, Manoj J.; Iorio, Alfonso; Aryal, Komal; Deng, Dianna; Brocard, Eva; Alhazzani, Waleed title: COVID-19 research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine date: 2020-04-21 journal: J Clin Epidemiol DOI: 10.1016/j.jclinepi.2020.04.016 sha: doc_id: 275110 cord_uid: safr9z37 What is new? KEY FINDINGS: Clinical decision-makers must be informed by the best, most trustworthy, highest-quality, robust evidence. This translates into how much confidence we can have in the research findings and thus be optimally informed for decision-making. The estimates of effect in clinical research depends on the underlying research methodology. COVID-19 disease is presenting global health systems, clinicians, and patients grave challenges. No treatment or prophylaxis currently exists for COVID-19. The overall body of COVID-19 research is very flawed methodologically. An examination of hydroxychloroquine-azithromycin research findings due to the recent media focus revealed very low-quality methodology underpins the research. Vast amounts of time and resources are being allocated to COVID-19 research, and being potentially squandered. WHAT THIS ADDS TO WHAT WAS KNOWN: Flawed methodology and sub-optimal reporting of research findings could lead to biased estimates of effect. This could lead to treatment decisions that are not optimal based on biased estimates which could harm the patient. This article provides specific suggestions for improving on the COVID-19 methods and reporting with a focus on the issues that researchers must consider in their methodology and reporting if we are to have confidence in the estimates of effect. Failure to consider harms in research could be detrimental to the patient. This article focuses on the potential harms when therapeutic agents such as hydroxychloroquine, are being considered. WHAT IS THE IMPLICATION AND WHAT SHOULD CHANGE NOW: Research thus far on finding an optimal therapeutic agent (s) for COVID-19 could be hampered by methodologically flawed research. COVID-19 researchers must immediately and acutely focus on improving their methodology and reporting. No current treatment or prophylaxis has proven to be effective in coronavirus disease 2019 , and patients receive either symptomatic treatment for milder presentations or more advanced life-support strategies in moderate to severe cases. As the global community eagerly awaits credible scientific solutions for this pandemic, researchers and scientists are under much pressure to identify effective therapeutic and preventive strategies for COVID -19. This commentary focuses on the quality of current COVID-19 research. We used published clinical studies on chloroquine/hydroxychloroquine as an example to demonstrate some of the methodological concerns around research currently conducted in the field. In recent weeks, academic journals and public media published and disseminated information on the use of quinine derivatives (i.e., chloroquine and hydroxychloroquine) (1, 2) for the treatment of COVID-19. Chloroquine has long been and still is used to prevent and treat malaria. Whereas, hydroxychloroquine was first used to prevent and treat malaria and is currently used to treat rheumatoid arthritis, some symptoms of systemic lupus erythematosus, juvenile idiopathic arthritis, and other autoimmune diseases. Although ongoing clinical trials are testing the efficacy and safety of several treatments for COVID-19, including chloroquine and hydroxychloroquine, there is not yet credible evidence from clinical trials on the efficacy and safety of those agents in COVID-19. Most of the data released or published thus far on chloroquine/hydroxychloroquine, and COVID-19 research in general, are imprecise and at high risk of biased estimates of effect. Here lies our concern. This pandemic emergency is fraught with tremendous uncertainty about the evidence on treatment or prophylaxis. There are many unknowns, and the massive demand for evidence on the treatment of a novel disease like COVID-19 may be unintentionally affecting studies' design and conduct. Furthermore, it may inadvertently affect the peer-review and publication process, leading to significant methodology gaps and overall lower quality evidence on COVID-19. These gaps lead to less informative studies, loss of precious time, and valuable resources. Therefore, current research should balance feasibility and efficiency against methodological rigor and carefully address methodology gaps as much as possible. In order to support universal clinical decision-making and minimize harm, the research community should focus on conducting and publishing trust-worthy evidence. Hypothesis generating studies are welcome and essential but should be explicitly framed as such, acknowledging their limitation, and not be used for critical decisions, at the national or global level. Furthermore, experimental studies aiming to confirm or refute these observations should be supported. Until there is a certainty, patients deserve, for themselves and for the sake of future patients, to be randomized in ethical and well-designed clinical trials. Our concerns on the state of the chloroquine and hydroxychloroquine research thus far expand to the quality of research methodology we see spanning across COVID-19 research. We sought to draw attention to this. To optimally inform our appraisal of the quality of COVID-19 research, we searched the MEDLINE and EMBASE electronic databases to identify clinical studies on the use of these agents in COVID-19 (2019 to April 3 rd , 2020), finding 564 initial citations that were screened for eligibility, yielding six clinical studies judged eligible (5) (6) (7) (8) (9) (10) . Based on the quality of the evidence to date, can a clinical practice guideline issue any recommendations on the use of chloroquine or hydroxychloroquine alone or combination with other treatments? The case has not been made either way for these quinine derivatives. The body of evidence thus far has been from both in vitro studies (2) (3) (4) and clinical studies (5) (6) (7) (8) (9) (10) of sub-optimal methodological quality. The emerging clinical studies such as case-series, cohort studies, and randomized control trials (RCTs), are all inconclusive; while some may suggest some benefit, the entire body of evidence does not reach the level of certainty and confidence that is required to justify the use of quinine derivatives as a treatment for COVID-19 (5) (6) (7) (8) (9) (10) . As exciting as some of these results may appear, the underlying research methodologies are often flawed, and as such, the reported results cannot be trusted. A most recent French prospective study (10) on the use of chloroquine in combination with azithromycin completely contradicts a prior French study asserting the benefit of the combination (8, 9) . When critically appraised using the appropriate risk of bias tools, all studies (5) (6) (7) (8) (9) (10) have been classified as high risk of biased estimates of effect (11, 12) (Table 1) . Overall, the methodologies of the published studies are not robust, and the results are tempered by selection bias and residual confounding bias. At the design level, most studies lack the randomization and blinding/masking needed to generate sound evidence, when they are RCTs and not observational uncontrolled single arm case-series. At the analysis level, they lack the standard steps taken to minimize confounding such as prospective design, statistical adjustment for prognostic factors, (propensity) matching, or stratification (13) . The reporting is very sparse, and patient-important outcomes needed for decision-making are often not studied or not reported. We do understand the urgency to identify effective treatments, as well as the barriers to perform rigorous research in health care settings overwhelmed by an unprecedented workload and a novel deadly disease. However, these unprecedented and unfortunate circumstances do not transform flawed data into sound results. The adequately powered, comparative, rigorous effectiveness research that is needed for optimal evidence-informed decision-making remains absent. Researchers need to prioritize minimizing bias by randomizing a large enough number of patients and masking the treatment allocation as much as it is feasible, and fully accounting for all the patients enrolled in the study following the intention-to-treat analysis principle ( Table 2 ). These critical components of high-quality, trustworthy research are required to generate high confidence that the estimates of effect reflect the true effects. At this time, more than ever, the high-quality, robust, comparative evidence from ethical randomized controlled trials (RCTs) is urgently needed to assess patient-important outcomes, including mortality, morbidity, need for life support, safety, and toxicity, informing on the safe use of chloroquine or hydroxychloroquine (with and without azithromycin) in people with COVID-19. Again, we make this clarion call across the breadth of COVID-19 research, and across all research in general. Of course, these studies will need to be fast, and even better, conducted in a flexible framework (such as adaptive trials) able to accommodate the adding of and switching to different treatments as soon the ones under study are proven ineffective or more promising alternatives are suggested. Consideration of master protocols (harmonization of efforts) and adaptive trial designs become very important (16) . The World Health Organization (WHO) has advocated for randomized multi-center adaptive clinical trials to evaluate the efficacy and safety of investigational therapeutic agents in combination with standard-of-care for the treatment of hospitalized patients with novel coronavirus disease (COVID-19) (17) . Researchers aligning their methodology to WHO's master protocol will surely improve quality of the COVID research (16, 17) . Additionally, global research groups conducting clinical trials would help in disseminating their results by adhering to the CONSORT (Consolidated Standards of Reporting Trials) checklist for optimal clinical trial reporting (15) and the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) statement that outlines the guidance for optimal reporting of observational studies (14) . This is critically important and especially in this COVID-19 urgency when very serious national and global public health decisions are being made based on what information is shared (or not shared). Evidence exists to show that clinical trial results are biased when the trials utilize inferior methodology or report findings without satisfactory description of the methods used. Failing to conceal allocation of the generated sequence has been linked to an exaggeration of the effectiveness of 30% or greater (15) . The urgency of the COVID-19 situation would also make it appropriate to be creative and move beyond the classical modalities and boundaries of academic research: what if a mobile app was made available by a respectable institution to allow randomizing any small number of consenting patients, collecting a small set of relevant covariates (age, sex, days since diagnosis, relevant comorbidities), by any doctor willing to participate in a chloroquine or hydroxychloroquine trial wherever the treatment is available for compassionate prescription (i.e. most of the world)? What if mortality in the two groups, masked as being treatment and control, was posted on a website every 100 patients reaching the outcome (recovered or dead) to transparently show if equipoise persists? What if the dataset at 1000 patients, or every 1000 patients if needed, was made publicly available for highly skilled statisticians to propose their interpretation? We would get 1000 patients every few days, and we would be receiving clinically sound results faster than any traditional study framework. Of course, this flexibility may not warrant publication in a top tier journal but could save thousands of lives. We need to use the most optimal methodology and not compromise on rigor but be willing to think outside of the box. The outcomes being reported in the COVID-19 research thus far are informative but are not ideally patient-important that could help in patient and clinician decision-making. In addition, research question gaps are glaring and future methodologically strong comparative research of quinines for COVID-19 (as well as other drug treatments) should assess the following: 1) the net benefit ratio of chloroquine or hydroxychloroquine alone or in combination with other interventions; 2) specific subgroups classified by age, stage and severity of illness, and other potential effect modifiers; and 3) optimal dosing and timing of dosing. We do believe there is still full equipoise justifying the continued investigation of the role of chloroquine/hydroxychloroquine in studies in patients hospitalized with COVID-19. Caution is urged regarding large scale uptake of the treatment as open-label use of the drug. Why extreme caution is urged in using chloroquine, hydroxychloroquine, or any drug? We could be doing more harm than good. Primum non nocere -first, do no harm. Potential adverse effects, toxicities, and medication interactions must remain key considerations when using any drug, and chloroquine or hydroxychloroquine, in combination with azithromycin, are no exception. Indeed, the evidence suggests that both drugs prolong the QT interval leading predisposing patients to serious arrhythmias (18) (19) (20) (21) . This is of particular concern when co-administering macrolides (azithromycin) (22) , also known to affect cardiac electrophysiology. Indeed, the use of doxycycline (22) has been proposed in azithromycin's place. The U.S. Food and Drug Administration (FDA) has even warned the public that azithromycin (Zithromax or Zmax) can potentially cause irregular changes in the electrical activity of the heart and could lead to a potentially fatal irregular heart rhythm (23) . In this regard, researchers recently looked at 84 COVID-19 infected patients who were administered a hydroxychloroquine/azithromycin combination (24) . They found that the QTc was prolonged maximally after 3-4 days from the beginning of treatment, and in 25 patients the QTc increased more than 40ms. They also found that in 9 patients (11%), the QTc increased to >500 ms, indicative of a high-risk group for arrhythmia. This issue of potential harmful effects and the urgent need for high-quality, methodologically robust studies also comes from a recent pre-publication (not yet peer-reviewed) of in vitro activity of hydroxychloroquine or chloroquine in combination with metformin (used in treatment of type 2 diabetes to lower blood sugar in humans) in mice (25) . Researches reported that when hydroxychloroquine or chloroquine was combined with metformin as a possible anti-cancer drug, 30-40% of all mice died. Until more high-quality evidence is available and particularly that excludes harm, caution is urged especially with combination therapy outside RCTs. There must also be acute pharmacovigilance and monitoring of adverse drug reactions with regards to these and other drugs in COVID-19 patients given so much in unknown. Some countries have chloroquine or hydroxychloroquine readily available, in some cases as an overthe-counter medicine as they are required to treat common conditions. National authorities should be mindful of these situations and take measures to govern the use of these medicines and prevent dangerous self-medication. On the equity side, there is a concern of massive purchasing, including for both personal use and research, leading to possible shortage of supply. This can lead to treatment shortages for malaria and autoimmune diseases where there is confirmed benefit. The use of existing drug treatments such as chloroquine and hydroxychloroquine outside of current guidelines and recommendations may result in adverse effects, including serious illness and death, affect patients with other diseases who may benefit from its use, and hinder the ability to conduct clinical trials if there are high demands to sue these agents by clinicians and patients. Toxic results from in vitro studies may not translate to toxicity in humans, but we caution that care must be exercised in extrapolating in vitro results before establishing clinical efficacy and safety (26) . In summary, while the massive effort in generating and disseminating evidence globally in response to COVID-19 has to be applauded, the accumulated body of evidence thus far can at best be considered hypothesis generating due to the methodological flaws across COVID clinical research. The methodological quality of ongoing and planned clinical research has to be urgently upgraded if the vast amount of time and resources are not to be squandered in this emergency. Until availability of sound results from one or more COVID-19 clinical trial (s) showing a favorable risk-benefit, caution is urged in considering indiscriminate use of these two drugs, alone or in association. Public health authorities are urged to prioritize resources on those interventions that are currently recommended as standard of care and ensure that empirical clinical use or research on these medication does not endanger adequate supply for the patients that need it for conditions in which the efficacy is already known. Declaration of interest: None to declare but note, most of the co-authors are heavily involved in guideline development as well as the GRADE Working Group, including the GUIDE Methods Group out of McMaster University, Hamilton, Ontario. Roles in the study and manuscript: None for the project What is new? Clinical decision-makers must be informed by the best, most trustworthy, highest-quality, robust evidence. This translates into how much confidence we can have in the research findings and thus be optimally informed for decision-making. The estimates of effect in clinical research depends on the underlying research methodology. COVID-19 disease is presenting global health systems, clinicians, and patients grave challenges. No treatment or prophylaxis currently exists for COVID-19. The overall body of COVID-19 research is very flawed methodologically. An examination of hydroxychloroquine-azithromycin research findings due to the recent media focus revealed very low-quality methodology underpins the research. Vast amounts of time and resources are being allocated to COVID-19 research, and being potentially squandered. Flawed methodology and sub-optimal reporting of research findings could lead to biased estimates of effect. This could lead to treatment decisions that are not optimal based on biased estimates which could harm the patient. This article provides specific suggestions for improving on the COVID-19 methods and reporting with a focus on the issues that researchers must consider in their methodology and reporting if we are to have confidence in the estimates of effect. Failure to consider harms in research could be detrimental to the patient. This article focuses on the potential harms when therapeutic agents such as hydroxychloroquine, are being considered. Research thus far on finding an optimal therapeutic agent (s) for COVID-19 could be hampered by methodologically flawed research. COVID-19 researchers must immediately and acutely focus on improving their methodology and reporting. 1 None to declare but note, most of the co-authors are heavily involved in guideline development as well as the GRADE Working Group, including the GUIDE Methods Group out of McMaster University, Hamilton, Ontario. Roles in the study and manuscript: None for the project Chloroquine and hydroxychloroquine as available weapons to fight COVID-19 Breakthrough: Chloroquine phosphate has shown apparent efficacy in treatment of COVID-19 associated pneumonia in clinical studies In Vitro Antiviral Activity and Projection of Optimized Dosing Design of Hydroxychloroquine for the Treatment of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) A pilot study of hydroxychloroquine in treatment of patients with common coronavirus disease-19 (COVID-19) Efficacy of hydroxychloroquine in patients with COVID-19: results of a randomized clinical trial url: med Rxiv preprint doi Pre-publication. Not peer-reviewed Treating COVID-19 with chloroquine Hydroxychloroquine and azithromycin as a treatment of COVID 19: results of an open label non randomized clinical trial Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: an observational study; prepublication No Evidence of Rapid Antiviral Clearance or Clinical Benefit with the Combination of Hydroxychloroquine and Azithromycin in Patients with Severe COVID-19 Infection Risk of bias in RCTs Risk of bias in non-randomized studies Confounding and adjustment /chapter_13/13_5_2_2_confounding_and_adjustment.htm (Accessed on The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials The need for novel trial designs, master protocols, and research consortia in transplantation WHO R&D Blueprint novel Coronavirus COVID-19 Therapeutic Trial Synopsis Suspected hydroxychloroquine-associated QT-interval prolongation in a patient with systemic lupus erythematosus torsade de pointes and cardiorespiratory arrest induced by azithromycin in a patient with congenital long qt syndrome Azithromycin and the risk of cardiovascular death Azithromycin and risk of sudden cardiac death: guilty as charged or falsely accused? Long Island doctor tries new twist on hydroxychloroquine for elderly COVID-19 patients Drug Safety Communication: Azithromycin (Zithromax or Zmax) and the risk of potentially fatal heart rhythms The QT Interval in Patients with SARS-CoV-2 Infection Treated with Hydroxychloroquine/Azithromycin. Pre-publication (non-peer reviewed) Fatal toxicity of chloroquine or hydroxychloroquine with metformin in mice Chloroquine is effective against influenza A virus in vitro but not in vivo. Influenza Other Respir Viruses