key: cord-0330156-362k0czq
authors: Ioannidis, J. P. A.
title: High-cited favorable studies for COVID-19 treatments ineffective in large trials
date: 2022-01-14
journal: nan
DOI: 10.1101/2022.01.11.22269097
sha: febfcb59c3e4fd3658899b40dc349a6b041af5a5
doc_id: 330156
cord_uid: 362k0czq

Importance. COVID-19 has resulted in massive production, publication and wide dissemination of clinical studies trying to identify effective treatments. However, several widely touted treatments failed to show effectiveness in large well-done randomized controlled trials (RCTs). Objective. To evaluate for COVID-19 treatments that showed no benefits in subsequent large RCTs how many of their most-cited clinical studies had declared favorable results for these interventions. Methods. Scopus (last update December 23, 2021) identified articles on lopinavir-ritonavir, hydroxycholoroquine/azithromycin, remdesivir, convalescent plasma, colchicine or interferon (index interventions) that represented clinical trials and that had received >150 citations. Their conclusions were assessed and correlated with study design features. The ten most recent citations for the most-cited article on each index intervention were examined on whether they were critical to the highly-cited study. Altmetric scores were also obtained. Findings. 40 articles of clinical studies on these index interventions had received >150 citations (7 exceeded 1,000 citations). 20/40 (50%) had favorable conclusions and 4 were equivocal. Highly-cited articles with favorable conclusions were rarely RCTs while those without favorable conclusions were mostly RCTs (3/20 vs 15/20, p=0.0003). Only 1 RCT with favorable conclusions had sample size >160. Citation counts correlated strongly with Altmetric scores, in particular news items. Only 9 (15%) of 60 recent citations to the most highly-cited studies with favorable or equivocal conclusions were critical to the highly-cited study. Conclusion. Many clinical studies with favorable conclusions for largely ineffective COVID-19 treatments are uncritically heavily cited and disseminated. Early observational studies and small randomized trials may cause spurious claims of effectiveness that get perpetuated.

clinical studies. A retracted paper and its retraction notice were excluded, and another 3 were excluded because the favored treatment (baricitinib, arbidol) was not an index intervention. The correlation of the number of citations was stronger with the number of news items (r=0.81) and more modest with the number of tweets (n=0.47).

Favorable papers did not have higher media and social media mentions than other papers.

For example, Altmetric values in the top-2000 of all science occurred in 9/20 favorable, 4/4 equivocal and 11/16 unfavorable papers (exact p=0.10 for the comparison of papers with favorable conclusions versus other papers).

Only 9 (15%) of 60 recent citations to the most highly-cited studies with favorable or equivocal conclusions were critical to the highly-cited study ( Table 2 ). Citing papers uncommonly (8/60, 13%) cited the respective RECOVERY or SOLIDARITY results.

This analysis demonstrates that many highly-cited clinical studies favor treatments that have shown no benefits in large, well-powered randomized trials. Most favorable studies are not randomized or are even uncontrolled, but they can still exercise strong, persistent influence on the scientific literature. Citation counts track well with strong presence of these studies in media and social media. The most highly-cited studies on these interventions have either entirely or partially favorable conclusions and the citations that they continue to receive are rarely critical of them.

Citations are a measure of the influence of a research paper across the broad scientific literature. Various manifestations of citation bias have been demonstrated in other clinical and scientific fields well before the COVID-19 era. [52] [53] [54] [55] [56] [57] In principle, studies with "positive" results tend to be more heavily cited than studies with "negative" results on the same topic. The citation bias creates a distorted picture for the perception of the scientific literature at large. Repeated . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint mention of the most favorable results gives the allusion that they are more likely to represent the truth, while this may not be the case. The COVID-19 scientific literature is also unique in terms of the massive volume of papers produced 58, 59 (and thus also citations generated) within a very limited timeframe. Very few studies in the history of medicine have ever received the number of citations received by the most highly-cited COVID-19-related papersincluding those whose promises could not be validated by large, well-powered RCTs.

The highly polarized and charged situation surrounding the COVID-19 pandemic may have further complicated matters and intensified the bias. Several of these treatments have received tremendous attention not only in the scientific literature, but also in the wider society and they have both strong supporters and strong critics. Many of the highly-cited papers analyzed here have also reached astronomical Altmetric scores, due to their massive discussion in media and social media. Altmetric scores correlated well with citation counts. The correlation was more prominent when news items were considered, while tweets had a more modest correlation with citation count. Altmetric score analyses have shown 60 that media and social media attention may remain high even for fully retracted papers.

Some caveats need to be acknowledged. The most important limitation of this analysis is that the large trials may not necessarily be a perfect gold standard. No single clinical study can claim to possess the perfect truth, no matter how well it is conducted and how well it is protected from bias. The CIs of the large trials cannot exclude very small benefits on survivalor small harms. These trials have also shown no benefit also on other outcomes. However, small benefits (or harms) for these outcomes are also not possible to exclude with perfect certainty. Moreover, beneficial effects for some treatments may still exist in circumscribed, special circumstances, with different dosing regimens, and in specific patient subgroups that may have been outside the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 14, 2022. eligibility criteria of the large RCTs or may have been under-represented in these large RCTs.

However, similar concerns and speculative counter-arguments may be raised almost in any clinical topic, especially by those who still believe that a treatment may have merits despite its poor performance in very large trials. 61 It should also be acknowledged that not all guidelines have removed these treatments from the list of interventions that they recommend. Remdesivir is probably the most notable example in this regard. It is not recommended by the European Respiratory Society 62 and the World Health Organization has issued a conditional recommendation against its use. 63

Conversely, the US National Institutes of Health (NIH) list remdesivir very prominently among the very few treatments that they recommend. 64 Subjective interpretation of the evidence is obvious in these differing opinions. Moreover, there may be overlap or connection between the researchers and institutions who perform the clinical studies and those that issue the recommendations and guidelines. For example, the most-cited favorable trial on remdesivir was spearheaded by NIH. 17 Moreover, a consequence of trusting a treatment as being effective is that it becomes attractive, if not necessary, to use it as background treatment or as a comparator when a new intervention is to be evaluated. In the case of remdesivir, NIH-spearheaded trials have already done this. E.g. they have compared interferon+remdesivir versus remdesivir alone and claimed to find no benefit for interferon; 65 and baricitinib+remdesivir versus remdesivir alone and claimed to find a benefit for the addition of baricitinib. 66 Retrospectively, these study designs are poor choices if remdesivir is indeed ineffective. They would be even highly misleading if remdesivir happens to be harmful.

Acknowledging these caveats and some residual uncertainty, it is more likely that these treatments overall don't save lives eventually. The most recent meta-analyses are also consistent . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint with this interpretation. [67] [68] [69] [70] [71] For hydroxychloroquine, a recent collaborative meta-analysis even shows nominally statistically significantly increased mortality. 72 Prior experience from the in-depth analysis of persistent high citations of non-validated papers in other fields may offer us insights on why this situation arose also for COVID-19 treatments. One empirical evaluation 61 assessed in-depth the citation patterns of extremely highly-cited papers on the benefits of beta carotene for cancer prevention, on estrogens for Alzheimer's dementia and on alpha tocopherol for cardiovascular disease. Despite the emergence of large RCTs with unfavorable results for these interventions, the observational studies that made the original promises continued to be heavily cited long after the "negative" RCT results had been published. Their citations were either ignoring the refuting RCTs (for beta carotene), or raising numerous counterarguments against them (for estrogen and alpha tocopherol).

Similarly, in psychology, where several major claims had been found to be irreproducible in preregistered reproducibility assessments done by many teams, frequent citation of the original claims has continued unperturbed after their non-reproduction. 73 The citing articles uncommonly take a critical stance against the original claims. 73 In other areas where evidence is heavily centered on biological considerations, citation networks citing some preferred papers may create a perpetuated distortion of what is the established knowledge. This has been seen in empirical evaluations in genetics of amygdala activation 74 and in pathology of inclusion body myositis. 75 Another empirical evaluation even found evidence of so-called "affirmative citation bias". 52 The citations to the critical articles that had thoroughly debunked prior beliefs were mostly affirmative, in favor of the original beliefs that had been debunked. The authors concluded that even criticism itself may paradoxically reinforce the establishment of debunked prior beliefs. 52 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint These observations suggest that often science is organized in cliques or schools of thought that are recalcitrant to the provision and acceptance of contrarian evidence. This may have happened also in the case of the early promising COVID-19 treatments. Moreover, in COVID-19, given the vast attention devoted to the topic, the citation rates of these non-validated treatment benefits are even more extraordinary. Furthermore, the overall impact of their dissemination reverberates across wider societal circles, not just scientific groups. The advent of very large RCTs did not suffice to perturb much this intense dissemination.

In conclusion, one should avoid putting much trust to highly promising results from early observational studies and small randomized trials of new or repurposed treatments. For serious diseases, like COVID-19, evidence on mortality endpoints should be sought. Pilot studies should not be abandoned or dejected, and they do have some value in offering early insights. However, they should be seen with great caution and with tempered enthusiasm. Large trials with flexible designs that allow obtaining large-scale rigorous evidence in a timely manner have been a major success during the pandemic [3] [4] [5] and their use should be promoted further.

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint HCQ: hydroxychloroquine; AZ: azithromycin; SOC: standard of care; LPV/r: lopinavirritonavir; MITT: modified intention-to-treat *presenting results from one of the two large randomized trials (RECOVERY and SOLIDARITY)

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint Table 2 . Qualitative analysis of recent citations to the most highly-cited article for each index treatment that reached favorable or equivocal conclusions.

Highly- . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint threatening COVID-19 did not result in significant improvement and was discontinued early" and "the clinical effect of this CP intervention has not yet been determined, since patients could have recovered due to other treatments administrated in parallel"

Hung is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint Altmetric score

Total citations received . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. ;

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted January 14, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted January 14, 2022. ; https://doi.org/10.1101/2022.01.11.22269097 doi: medRxiv preprint

The worldwide clinical trial research response to the COVID-19 pandemic -the first 100 days

COVID-19: A catalyst for transforming randomized trials

Lessons learned from COVID-19 trials: should we be doing trials differently?

Making trials part of good clinical care: lessons from the RECOVERY trial

Making trials part of good clinical care: lessons from the RECOVERY trial

Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial

Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis

Citation bias and other determinants of citation in biomedical research: findings from six citation networks

Don't let the truth get in the way of a good story: an illustration of citation bias in epidemiologic research

How a torrent of COVID science changed research publishing -in seven charts

The rapid, massive growth of COVID-19 authors in the scientific literature

Media and social media attention to retracted articles according to Altmetric

Persistence of contradicted claims in the literature

European Respiratory Society COVID-19 Task Force. The impact of therapeutics on 63

Efficacy of interferon beta-1a plus remdesivir compared with remdesivir alone in hospitalised adults with COVID-19: a double-bind, randomised, placebo-controlled, phase 3 trial

Baricitinib plus Remdesivir for Hospitalized Adults with Covid-19

Efficacy and safety of remdesivir in COVID-19 caused by SARS-CoV-2: a systematic review and meta-analysis

The safety, tolerability and mortality reduction efficacy of remdesivir

observational and case studies reported safety outcomes: an updated systematic review and meta-analysis. Ther Adv Drug Saf

Efficacy and safety of current treatment interventions for patients with severe COVID-19 infection: A network meta-analysis of

Association between convalescent plasma treatment