key: cord-0923501-j7vb8jb4 authors: Bellos, Ioannis title: A meta-research study revealed susceptibility of Covid-19 treatment research to white hat bias: First, do no harm date: 2021-03-26 journal: J Clin Epidemiol DOI: 10.1016/j.jclinepi.2021.03.020 sha: 57e9c6b1e96b3c37fb1b853e4992159d0bd428e0 doc_id: 923501 cord_uid: j7vb8jb4 OBJECTIVE: : To investigate the presence of white hat bias in Covid-19 treatment research by evaluating the effects of citation and reporting bias. STUDY DESIGN AND SETTING: : Citation bias was investigated by assessing the degree of agreement between evidence provided by a remdesivir randomized controlled trial and its citing articles. The dissimilarity of outcomes derived from non-randomized and randomized studies was tested by a meta-analysis of hydroxychloroquine effects on mortality. The differential influence of studies with beneficial over those with neutral results was evaluated by a bibliometric analysis. RESULTS: : The articles citing the ACTT-1 remdesivir trial preferentially presented its positive outcomes in 55.83% and its negative outcomes in 6.43% of cases. The hydroxychloroquine indicated no significant effect by randomized studies, but a significant survival benefit by non-randomized ones. Citation mapping revealed that the study reporting survival benefit from the hydroxychloroquine-azithromycin combination was the most influential, despite subsequent studies reporting potential harmful effects. CONCLUSION: : The present study raises concerns about citation bias and a predilection of reporting beneficial over harmful effects in the Covid-19 treatment research, potentially in the context of white hat bias. Pre-registration, data sharing and avoidance of selective reporting are crucial to ensure the credibility of future research. • Citation bias is evident by the preferential reporting of positive over neutral outcomes of the same study. • The example of the ACTT-1 remdesivir trial indicated that the majority of citing articles opted for presenting only its positive outcomes. • Non-randomized evidence about hydroxychloroquine use supported benefit from its use, which was not confirmed by randomized studies. • Conventional and alternative metrics demonstrated that research claiming positive effects of the hydroxychloroquine-azithromycin combination had higher influence than studies providing evidence of harm. • The quality of Covid-19 treatment research may be negatively affected by white hat bias, which comes from personal beliefs and the urgent need of an effective intervention. • Non-randomized studies are prone to white hat bias due to the possibility of selective analysis and reporting. • Pre-registration and public data sharing should apply for both randomized and observational studies. The peer-review process should ensure the avoidance of citation bias. • Meta-analyses of observational studies should consider routinely implementing the credibility ceiling test to explore skepticism about the presence of potential biases. The rapid spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has provoked a pandemic of coronavirus disease 19 (Covid-19)-related publications. Within 1 year after its emergence, more than 85,000 research papers about have been published in the areas of health, life, physical and social sciences [1] . Research interest has focused on the evaluation of novel anti-viral treatments, with more than 4,300 intervention studies being registered in the Clinicaltrials.gov database. The need of rapid knowledge dissemination has led to a surge of fast-track pre-prints, resulting in a complex network of academic and non-peer-reviewed publications [2] . The situation is further complicated by the existence of multiple versions of a single study in open archives and as journal articles, often presenting contradictory outcomes [3] . As a result, the process of decision-making and identification of credible evidence remains a challenge both for the researcher and the clinician. White hat bias is defined as the distortion of evidence towards the promotion of what is widely perceived as fair ends. This type of bias has been first described in nutritional epidemiology and emanates from the motivation of scientists to practice beneficence and improve human health [4] . White hat bias differs from bias due to financial conflicts of interest since it has been assumed that scientists without industry funding tend to ignore negative or insignificant results about interventions regarded as beneficial to health. In this context, publication bias risk is augmented, potentially leading to the preferential reporting of significant outcomes [5] . Concerns about white hat bias may be also raised when citation bias is suspected with citing papers inaccurately describing the available original evidence. This effect may be extended in institutional reports and press releases, with distorted presentation and "spinning" of study outcomes leading to false claims about the effectiveness of interventions [6] . As the clinical management of Covid-19 patients remains largely supportive, there is increasing demand for the development of a novel treatment in order to alter the natural history of the disease. The aim of the present meta-research study is to evaluate the potential presence of white hat bias in the Covid-19 treatment research; to achieve this, different aspects of reporting, citation and publication bias are assessed by examining the examples of remdesivir, hydroxychloroquine and azithromycin administration. In addition, the potential harmful effects of interventions claimed as effective are evaluated. The potential presence of citation bias was assessed by judging the degree of agreement between the results of a highly influential paper and the statements of its citing articles. Specifically, the article reporting the outcomes of Adaptive Covid-19 Treatment Trial (ACTT-1) [7] was chosen. This study was a randomized controlled trial aiming to evaluate the effectiveness of remdesivir administration in Covid-19 patients. Both positive and negative outcomes were reported; remdesivir was associated with significantly lower duration of hospital stay (11 vs. 15 days, p-value <0.001) and higher rates of clinical improvement at day 15 (odds ratio: 1.5, 95% confidence intervals: 1.2-1.9). On the other hand, no significant benefit was observed in the outcomes of survival (hazard ratio: 0.73, 95% confidence intervals: 0.52-1.03) and cumulative recovery among critically ill patients. The ACTT-1 was chosen as the study of interest since it is a highly influential randomized controlled trial that exerted significant effect on clinical practice by guiding the approval of remdesivir. In addition, mixed results were presented, thus rendering it suitable for investigating the potential presence of citation bias. The full reference list was accessed via NEJM.org up to 25 December 2020. The citing papers were classified as accurate (category A) when both significant and insignificant results were reported. Category B was assigned to articles presenting only positive outcomes and category C to those stating only the negative ones. Category D referred to articles with misleading statements reporting outcomes with distorted statistical significance. Papers were unscorable (category E) in case they did not report the significance of outcomes or provided ambiguous statements. The citing studies were classified depending on their design as reviews, meta-analyses, non-randomized, randomized and basic research studies (including in vitro, animal and simulation studies). Letters to the editor, commentaries, non-English papers and category E articles were excluded. The categories (A-D) were compared among different study designs (review, meta-analysis, non-randomized, randomized, basic research) using multinomial logistic regression analysis. The category A served as the reference one. Analysis was conducted in R-3.6.3 ("nnet" package [8] ). To compare the outcomes derived from observational and randomized studies, a metaanalysis was conducted regarding the effects of hydroxychloroquine treatment on survival rates of Covid-19 patients. An update of a recent meta-analysis [9] was performed by systematically searching Medline using the following search algorithm: ) AND "Hydroxychloroquine"[Mesh])". All randomized and observational (both prospective and retrospective) studies that evaluated mortality rates among Covid-19 patients receiving hydroxychloroquine were selected. The literature search process was crosschecked by two independent researchers and any possible conflict was resolved through discussion. Meta-analysis was conducted by fitting a random-effects model (restricted maximum likelihood -REML), which provided estimated of odds ratio (OR) and 95 % confidence intervals (CI). Subgroup analysis was performed based on study design. In the subgroup of non-randomized studies, the credibility ceiling test was used to test the robustness of outcomes [10] . The ceilings of 5%, 10% and 15% were implemented. As a sensitivity analysis, the summary odds ratio was estimated using 2 one-stage generalized linear mixed models (modified Simmonds-Higgins and hypergeometricnormal model) [11] . Meta-analysis was performed in R-3.6.3 ("metafor" package [12] ). The combination of hydroxychloroquine with azithromycin has been proposed early in the course of the pandemic an effective Covid-19 treatment, leading to its widespread prescription. However, this effect was not confirmed by further studies, indicating potentially increased risk of mortality associated with this intervention [13] . To compare the differential influence of studies evaluating the efficacy of hydroxychloroquineazithromycin combination, the PlumX Metrics tool was used providing information about citations, mentions in news/blogs and social media coverage [14] . Specifically, the evaluated articles included the study of Gautret et al. [15] presenting positive outcomes about the hydroxychloroquine-azithromycin combination and 3 subsequent influential studies [16] [17] [18] with negative outcomes. The main outcome of Gautret et al. [15] al. [17] reported no significant effect on in-hospital mortality risk (hazard ratio: 1.35, 95% CI: 0.76-2.40). Moreover, a citation map of articles in the field was constructed using the VOSviewer software (version 1.6.16) [19] . To achieve this, the Scopus database was searched using the algorithm "(hydroxychloroquine OR chloroquine) AND (covid-19 OR coronavirus OR sars-cov-2)", limited to original articles. The process of citing articles selection is schematically depicted in Figure 1 . Literature search resulted in 5 new studies [20] [21] [22] [23] [24] ; hence, the meta-analysis was based on a total of 20 studies (16 non-randomized and 4 randomized The outcomes of citation analysis are presented in [25, 26] . On the contrary, citations were remarkably lower in studies reporting the harmful effects of hydroxychloroquine-azithromycin combination due to potential QT prolongation [27] [28] [29] . The outbreak of Covid-19 has put clinicians and researchers in a desperate need of identifying effective interventions aiming to limit disease severity and in-hospital mortality. To this end, drug repurposing has emerged as an attractive option by studying drugs with already known safety profile [30] . [32] . However, the potential beneficial effects of this intervention were not confirmed by the subsequent large-scale Solidarity trial, reporting that remdesivir was not associated with significant effects on hard clinical outcomes [33] . As a result, the latest World Health Organization (WHO) guidelines [34] recommend against the use of remdesivir due to uncertainties for efficacy in critical outcomes, longterm safety and effects on specific patient subgroups. It should be also noted that remdesivir constitutes a costly intervention, with an official cost of $2340-3120 [35] for a 5-day treatment course; therefore, the feasibility of further large-scale trials aiming to prove any tiny significant survival benefit in certain patient subgroups remains problematic. It has been hypothesized that independently funded non-randomized studies may be biased by the prior beliefs of the researchers about the possible beneficial effects of the investigated intervention [4] . In addition, the publication probability of an observational study reporting negative outcomes may be lower than that of a randomized controlled trial due to the prospective registration of the latter [36] . The results of the present metaanalysis indicated that non-randomized studies demonstrated that hydroxychloroquine was associated with significantly lower death risk, while no significance survival benefit was evident by the pooling of randomized controlled trials. Importantly, the direction of the summary odds ratio of non-randomized studies is in the opposite direction compared to that of randomized studies. The credibility ceiling test was implemented as a statistical way to model skepticism about the potential biases of observational evidence. Statistical significance was lost by applying even the lenient ceiling of 5%, questioning the precision of the meta-analysis of non-randomized studies. This result can be translated as that the effects become insignificant if we consider that any observational study cannot provide more than 95% certainty that the intervention is beneficial or harmful. Regarding the role of funding, it may be assumed that industryfunded studies may avoid disseminating spuriously positive results since they typically opt for a randomized study design with prospective protocol registration, limiting the possibility of selective reporting. Actually, one randomized controlled trial was funded by the industry, reporting no significant clinical improvement from the administration of hydroxychloroquine [16] . The combination of hydroxychloroquine and azithromycin is one of the most representative examples of an intervention initially proposed as beneficial that was subsequently shown to be ineffective and even harmful. Specifically, the claims of a research team [15, 25, 26] about the remarkable effects of this treatment combination on viral clearance and mortality rates enjoyed wide acceptance by the media, providing policy makers and healthcare stakeholders a potentially life-saving tool to confront the pandemic. Therefore, as indicated by the present results, the study of Gautret et al. [15] had the highest influence on the research community, compared to subsequent welldesigned studies reporting negative outcomes. Interestingly, studies demonstrating the increased risk of QT prolongation and arrhythmia by the hydroxychloroquineazithromycin combination received remarkably less citation impact [27] [28] [29] . Importantly, the COALITION II trial [37] which was partially funded by the industry indicated no significant benefit by the use of azithromycin compared to standard care. It should be also noted that a pharmacovigilance study reported that the majority of adverse effects by the use of drugs for the management of Covid-19 came from the administration of azithromycin with or without hydroxychloroquine, mainly consisting of electrocardiographic changes, hepatitis and diarrhea [38] . The present study has several strengths. Selective reporting was evaluated by systematically searching the reference list in of a highly influential providing a large sample of full-text citing articles. The credibility ceiling test was used in the metaanalysis of hydroxychloroquine aiming to quantify the effects of skepticism due to potential biases of observational studies. Moreover, the greater influence of studies with positive outcomes was confirmed by both conventional and alternative metrics, while a citation density map was constructed to visualize the relative impact of studies in the field of hydroxychloroquine-azithromycin combination. On the other hand, the safe recognition of white hat bias remains problematic since no validated tool exists for its evaluation; hence, the assessment of parameters remains partially subjective. In addition, white hat bias may be complicated and difficult to distinguish from several type of biases associated with observational studies, such as cherry-picking of significant results and selective analysis. Regarding the meta-analysis, the small size of studies in conjunction with the rarity of the event (i.e. death) may challenge the assumption of normality [39] . To address this issue, two one-stage models were applied as a sensitivity analysis aiming to avoid within-studies approximations [11] . However, these generalized linear mixed models use maximum likelihood estimation and may also introduce bias by underestimating the between-study variance [40] . It should be also noted that the present study addresses only specific examples of information distortion that provide potential evidence of white hat bias; therefore, it may serve as a pilot one in the field and further studies are needed to investigate the magnitude of this type of bias in other proposed Covid-19 treatments. In conclusion, this meta-research study presented evidence of reporting and citation bias potentially associated with white hat bias in the Covid-19 treatment field. Personal beliefs in conjunction with an urgent need of the scientific community for effective interventions may have led to a predilection for propagating beneficial over neutral or harmful outcomes. In this context, the rapidly expanding body of Covid-19-related evidence is at risk of creating a medical misinformation mess and producing more noise than clinically useful information. To strengthen the credibility of evidence, both nonrandomized and randomized studies need to opt for pre-registration and public data sharing aiming to reduce the degrees of freedom in the analysis and reporting of outcomes. It is crucial to disclose both financial and non-financial conflicts, while authors should be vigilant in avoiding "spinning" during reporting of evidence in order to refrain from disseminating misleading results. A Bibliometric Analysis of COVID-19 across Science and Social Science Research Landscape Characteristics of academic publications, preprints, and registered clinical trials on the COVID-19 pandemic Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: metaresearch study White hat bias: Examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting Source of bias in sugarsweetened beverage research: A systematic review White hat bias: A threat to the integrity of scientific reporting Remdesivir for the Treatment of Covid-19 -Final Report Effect of hydroxychloroquine with or without azithromycin on the mortality of coronavirus disease 2019 (COVID-19) patients: a systematic review and metaanalysis Synthesis of observational studies should consider credibility ceilings A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio Conducting Meta-Analyses in R with the metafor Package Infectious Diseases Society of America Guidelines on the Treatment and Management of Patients with COVID-19 n Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial Hydroxychloroquine with or without Azithromycin in Mild-to-Moderate Association of Treatment with Hydroxychloroquine or Azithromycin with In-Hospital Mortality in Patients with COVID-19 in New York State Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19 VOSviewer -Visualizing scientific landscapes n Use of hydroxychloroquine in hospitalised COVID-19 patients is associated with reduced mortality: Findings from the observational multicentre Italian CORIST study Impact of Azithromycin and/or Hydroxychloroquine on Hospital Mortality in COVID-19 Low-dose hydroxychloroquine therapy and mortality in hospitalised patients with COVID-19: a nationwide observational study of 8075 participants Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19 Effect of Hydroxychloroquine on Clinical Status at 14 Days in Hospitalized Patients with COVID-19: A Randomized Clinical Trial Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: A pilot observational study Early treatment of COVID-19 patients with hydroxychloroquine and azithromycin: A retrospective analysis of 1061 cases in Marseille, France Risk of QT Interval Prolongation Associated With Use of Hydroxychloroquine With or Without Concomitant Azithromycin Among Hospitalized Patients Testing Positive for Coronavirus Disease 2019 (COVID-19) Effect of Chloroquine, Hydroxychloroquine, and Azithromycin on the Corrected QT Interval in Patients with SARS-CoV-2 Infection The QT interval in patients with COVID-19 treated with hydroxychloroquine and azithromycin Drug repurposing approach to fight COVID-19 Letter of Authorization -chloroquine phosphate and hydroxychloroquine sulfate FDA Approval of Remdesivir -A Step in the Right Direction Repurposed Antiviral Drugs for Covid-19 -Interim WHO Solidarity Trial Results A living WHO guideline on drugs for covid-19 Remdesivir for COVID-19 in Europe: will it provide value for money? Challenges in meta-analyses with observational studies Azithromycin in addition to standard of care versus standard of care alone in the treatment of patients admitted to the hospital with severe COVID-19 in Brazil (COALITION II): a randomised clinical trial Global Safety Database Summary of COVID-19-Related Drug Utilization-Safety Surveillance: A Sponsor's Perspective When should meta-analysis avoid making hidden normality assumptions? Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies The author wish to thank Dr. Georgia Fitrou (National and