key: cord-0796441-c4icv3d5 authors: Kapp, P.; Esmail, L.; Ghosn, L.; Ravaud, P.; Boutron, I. title: Transparency and reporting characteristics of COVID-19 randomized controlled trials. date: 2022-02-04 journal: nan DOI: 10.1101/2022.02.03.22270357 sha: 7ba11410c8591facf1b1de8a58f812bd0469c61c doc_id: 796441 cord_uid: c4icv3d5 Randomized controlled trials (RCTs) are essential to support clinical decision making. We assessed the transparency, completeness and consistency of reporting of 244 reports (120 peer-reviewed journal publications; 124 preprints) of RCTs assessing pharmacological interventions for the treatment of COVID-19 published the first 17 months of the pandemic (up to May 31, 2021). Transparency was poor. Only 55% of trials were prospectively registered; 39% made their full protocols available and 29% provided access to their statistical analysis plan. Only 6% completely reported the most important information. Primary outcome(s) reported in trial registries and published reports were inconsistent in 47% of trials. Of the 124 RCTs published as preprint, 76 were secondarily published in a peer-reviewed journal. There was no major improvement after the peer-review process. Lack of transparency, completeness and consistency of reporting is an important barrier to trust, interpretation and synthesis in COVID-19 clinical trials. The results of the search are detailed in Figure 1 . Of the 47,061 records screened, 244 reports of randomized trials evaluating pharmacological interventions for the treatment of COVID-19 were identified and assessed. Overall, 120 (49%) RCTs were first published in peer reviewed journals while 124 (51%) were initially available as preprints. Of the 124 preprints, 76 (61%) were subsequently published in a peer-review journal. Table 1 A trial protocol was available in 39% (n = 95) of trials, 4 (2%) were not in English. A statistical analysis plan was accessible in only 29% of the trials (n = 71). Overall, 231 trials (95%) were registered; 88 (36%) were retrospectively registered of which 37/88 (42%) had a delay of more than 30 days between the trial start date and registration date. For nine trials (4%) we could not determine if the trial was prospectively posted, due to unclear or missing information in the registry or report. When the option to post trial results was available (n = 158, 65%), only 27 (17%) posted their results in clinical trial registries (status last checked October 7, 2021) . In addition, one trial provided result summary in the registry although posting was not required. Overall, 165 trials (68%) made a data sharing statement available in the report. Of those, 150/165 trials (91%) stated their willingness to share data, 14/165 trials (9%) stated that they were not willing to share their data, while one trial (1%) reported to be undecided. Of the 150 trial reports that intended to share data, authors reported that they would share data upon email request (n = 103, 69%) or on an online repository (n = 32, 21%), while 15 trials (10%) did not report how data would be made available. When a data sharing statement was reported in the registry (n = 170), we identified discrepancies between the data sharing statement in the registry and trial report for 55 trials: from data sharing willingness 'No' or 'Undecided' in registry to 'Yes' in the report (n = 51) and from 'Yes' in the registry to 'No' in the report (n = 4). . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Results are detailed in Table 2 and S2 Table. Overall, reporting was poor. Only 78 (32%) of the reports completely reported the pre-specified primary outcome; 201 (82%) described the methods used to generate the random allocation sequence and 139 (57%) the process of allocation concealment. Of the studies that reported to be blinded (n = 107, 44%), less than half (n = 43, 40%) clearly described who was blinded and how. About half of the trials (n = 129, 53%) provided a complete description of the participant flow, either as diagram or in text form. Regarding the study results, 124 trials (51%) reported the primary outcome(s) completely. Harm was adequately described in only 15% of the trials (n = 36). In particular, information regarding the mode of harm data collection (i.e., how data was collected) (n = 134, 55%) and the time frame of observation (n = 126, 52%) was insufficiently reported (S2 Table) . Fifty-four trials (22%) did not report any results on harms and most trials (n = 147, 60%) did not highlight whether harms resulted in withdrawals or trial discontinuations (S2 Table) . Overall, only 6% (n = 15) completely reported the 10 most important CONSORT items. Most trials reported information on funding (n = 231, 95%). Two hundred-thirty (94%) disclosed information on conflicts of interest. All except four trials (98%) reported ethical approval. For most items, reporting did not differ between reports first published as preprint and reports first published in a peer-reviewed journal article (Table 2) . Results were statistically significant only for the reporting of harm which was better in peer-reviewed journal publication (23% vs 7%, p = 0,001); the reporting was better in preprints for the trial registration number (97% vs 86%, p = 0,003) and funding source (96% vs 92%, p = 0,048) . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. *Thirteen trials were not registered (12 peer-reviewed journal publications and one preprint) and nine trials did not present the registration number in the report (Five peer-reviewed journal publications and three preprints) Checklist Reporting consistency (i.e., primary outcome(s) switching) Among all registered trials (n = 231), 206 trials (89%) identified their primary outcomes in the report and registry. Of those, 108 (52%) reported primary outcomes as pre-defined in the trial registry. Primary outcome(s) switching between registered and published primary outcome(s) was identified in 97 trials (47%) ( Table 3 ). Switches comprised completely changed primary outcome(s) between report and registry (n = 35, 36%); reports that removed one or several primary outcome(s) (n = 18, 19%); reports that added one or several primary outcome(s) (n = 8, 8%) and reports that added and removed one or several primary outcome(s) (n=16, 17%). In addition, twenty trials (21%) changed the time frame or metric, while the primary outcome variable stayed the same. Twelve trials comprised both, changes in time frames or metrics, as well as added, removed or changed primary outcome(s) . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; Percentages may not add up to 100% due to rounding. * Twelve additional trials comprised both, changes in time frames or metrics, as well as added, removed or changed primary outcome(s). Those trials were counted only a single time within the added, removed or changed outcome(s) switching domains. Registry: 1. Time to clinical improvement Report: 1. Death Comparison between preprint report and subsequent peer-reviewed journal publication Of 124 preprints included in our analysis, we identified 76 corresponding peer-reviewed journal publications. The median time between preprint and publication in a peer reviewed journal was 95 days (IQR: 59-171) (range: 5 -505). The protocol and the statistical analysis plan were added in 13 (17%) and 11 (15%) peer-reviewed journal publications respectively. However, the protocol and the statistical analysis plan were also removed respectively in 5 (7%) and 3 (4%) reports. The detailed differences between preprint and peer-reviewed journal publication are described in Table 4 and S3 Table. Information was rarely added: allocation concealment (n = 6, 8%), the persons who were blinded (n = 5, 7%), the mode of harm data collection (n = 6, 8%) as well as the time frame of harm surveillance (n = 6, 8%). Information that was removed from the preprint in the peer-reviewed journal publication included the mode of harm data collection (n = 4, 5%), the description of the primary outcome (n = 3, 4%) or the registration number (n = 3, 4%). Overall, 40 trials (53%) had at least one change within all CONSORT sub-items. Only two (3%) of the trials had changed their overall CONSORT assessment from partially to completely reported. . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Method of sequence generation Who was blinded 24 (32%) 6 (8%) 5 (7%) 0 (0%) 41 (54%) † Flow chart Participants lost to follow-up 75 (99%) 1 (1%) 0 (0%) 0 (0%) 0 (0%) Participants who discontinued intervention Participants analyzed 74 (97%) 2 (3%) 0 (0%) 0 (0%) 0 (0%) . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Percentages may not add up to 100% due to rounding. * Eight publications did not clearly specify the primary outcome (either in preprint or journal publication). Due to that the full assessment of CONSORT section 6a was (item 2-7) was not possible. † Forty-one trials were unblinded and could not be assessed. Two additional trials did not use placebo control and could not be assessed for the item similar characteristics of the intervention (CONSORT section 11b, item 3). ‡ Twenty-eight trials did not present the difference in estimated effect measure (CONSORT section 17a, item 2). Precision was therefore not assessable for 28 trials (item 3) Of the 76 assessed trials, nine trials could not be compared for differences in primary outcome(s) switching, due to the missing definition of the primary outcome in the pre-print (n = 3), peer-reviewed journal publication (n = 2), or both (n = 4). There was no change in primary outcome(s) switching between preprint and peer-reviewed journal publication. Only one trial added a justification for primary outcome(s) switching in the peer-reviewed journal publication. Registration number 73 (96%) 0 (0%) 0 (0%) 3 (4%) 0 (0%) . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint Our results raise important concerns about the transparency, completeness and accuracy of the reporting of RCTs assessing pharmacological interventions for the treatment of COVID-19. Despite a considerable delay (median 95 days) between the publication of the results on a preprint platform and the publication in a peer-reviewed journal, the peer-review process had no impact on transparency, completeness and accuracy of reporting. Our results are consistent with other studies assessing reporting characteristics of RCTs reports [11] [12] [13] [14] . An analysis of more than 20.000 RCTs included in Cochrane reviewed showed important deficiencies in reporting which is a strong barrier for risk of bias assessment and the extraction of outcomes needed to conduct systematic reviews and meta-analyses [11, 14] . Primary outcome switching also raises important concerns, of which our results showed higher prevalence compared to other studies performed in other fields [15, 16] . This difference could be explained by the novelty of the disease and the rapid increase in knowledge over time which may have requested important changes to the protocol. Nevertheless, the lack of transparency related to these changes is concerning. To our knowledge, this is the first study extensively assessing reporting characteristics of all COVID-19 RCTs assessing pharmacologic treatment published as preprint or peer-reviewed journal article the first 17 months of the pandemic. Our sample has been included in a large living network meta-analysis and is comprehensive. Furthermore, we assessed various dimensions of transparency, reporting and consistency between reports and registry records. Finally, to our knowledge it is the largest study comparing preprint and subsequent publications. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Our study has nevertheless some limitations. First, we focused on randomized controlled trials and cannot extrapolate to other study designs. Nevertheless, RCTs are considered the gold standard for therapeutic evaluation. Second, most trials were assessed by a single researcher, however a random sample was extracted in duplicate and showed good reproducibility. Our results have important implications. There is an urgent need for high quality evidence to guide the management of COVID-19 patients. It is consequently essential to improve reporting and transparency and increase adherence to the CONSORT statement. As part of the COVID-NMA living review, we are already systematically contacting investigators to request the missing data. Further, we plan to inform investigators of their results in terms of reporting and transparency to help them improve the content of their reports. The publication of results on preprint servers became an essential mean of communication. It was adopted considerably by the research community during this pandemic mainly because it shortened delays between the production of reports and their dissemination to the community. In our sample, half of the trials decided to communicate first through preprint. Overall, it reduced the delay of accessing results by a median of 3 months. Some researchers, decision makers, funders, editors raised concerns related to the risk of disseminating reports that were not peer-reviewed [6] [7] [8] . However, our results do not support the hypothesis that peer-reviewed journal publications are of better quality compared to preprints. We found no difference in terms of transparency and reporting between the preprint and the peer-reviewed report [17, 18] . Finally, our results question the role of the peer-review process in improving reporting and transparency. Our results are consistent with other studies comparing completeness of reporting of the submitted report to the published report focusing on RCTs [19] . We need to develop specific interventions and tools to increase the detection and improvement of reporting in publications. Some tools such as the CobPeer tool have been proposed and evaluated [20] . Other is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint interventions targeting preprints could be useful to inform trialists of reporting deficiencies and help them improve their report prior publications. In conclusion, lack of transparency, completeness and consistency of reporting is an important barrier to trust, interpretation and synthesis in COVID-19 clinical trials. Peer-reviewed publications were not better than preprints in this regard. Furthermore, the peer-review process failed to improve the deficiency in reporting. Trial authors as well as editors and funders must apply higher standards of methodological rigor and transparency to ensure the generation of the highest level of evidence to inform decisionmaking and curb the pandemic. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint This study is part of the COVID-NMA initiative [2, 21, 22] . The two first pillars of this initiative are a living mapping and living evidence synthesis of all randomized controlled trials assessing treatments and preventive interventions for COVID-19. All results are updated weekly and made available open access on a platform (https://covid-nma.com) [2, 21, 22] . The third pillar of this initiative described in this manuscript is a monitoring of the transparency, completeness and consistency of reporting in the trial reports. We included all RCTs assessing pharmacologic interventions for the treatment of COVID-19. Trials assessing nonpharmacologic interventions (e.g., prone position, physiotherapy), pharmacological treatment of long-COVID and preventive interventions as well as vaccines were excluded. Early-phase clinical trials, single-arm trials, non-randomized studies and modelling studies of interventions for COVID-19 were excluded. There was no restriction on language. The search strategy was developed in the context of the COVID-NMA initiative for the living systematic review of all treatment interventions for COVID-19 [21] . In brief, we searched electronic databases daily to identify all COVID-19 randomized controlled trials. The search strategy is detailed in S1 Table. The last search was conducted on May 31, 2021. For all trials published as a preprint, we systematically searched weekly for a subsequent publication in peer-reviewed journals using a preprint tracker (last search October 7, 2021) [23] . is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint publication corresponding exactly to the content of the preprint in terms of methods and results reported. Two reviewers screened all retrieved titles, abstracts in duplicate independently using Rayyan [24] . Discrepancies were resolved by consensus between the two reviewers. A third reviewer was involved to resolve disagreements when necessary. We developed a standardized data extraction form covering general trial characteristics, transparency indicators, completeness and consistency of reporting. To avoid errors during the extraction and ensure calibration, two reviewers were trained and assessed separately 20 studies each. The reviewers discussed the meaning of each assessment item and reached consensus for the 20 studies. Subsequently, all included trials were extracted by a single reviewer. The inter-rater agreement between the two reviewers was good with 96.6% agreement, with a Kappa Coefficient of 0.87. We extracted the trial design, number of arms, sample size, setting, number of centers and blinding. We considered the following indicators of transparency: 1) Access to the trial documentation: We checked whether we had access to the protocol and statistical analysis plan and if it was available in English. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint ISRCTN registry, DRKS -German Clinical Trials Register, jRCT -Japan Registry of Clinical Trials, ANZCTR -Australian New Zealand Clinical Trial registry) [25] . 3) Data sharing statement: We recorded whether, when and how investigators planned to share data based on information in trial registration and the report. We evaluated systematically whether the trial report and protocol, if available, adheres to the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement [26] . We decided to focus on 10 CONSORT items which were deemed most important because they are frequently incompletely reported and are necessary for conducting a systematic review to evaluate the risk of bias and record the outcome data [20] . The completeness of reporting was assessed using the COBPeer tool (Table 2 and S2 Table) , which considers the CONSORT-items associated with sub-items eliciting what should be reported for each item extracted as stated in the CONSORT 2010 Explanation and Elaboration Explanation paper [20, 27] . Reviewers had to indicate for each sub-item if the requested information was reported (yes/no). Finally, each item was rated as "completely reported", if all sub-items were adequately reported, "partially reported", if at least one sub-item was missing and "not reported", if all-items were missing. For the assessment of the CONSORT items, we systematically considered the primary outcome of the report. If the primary outcome was not clearly identified, we considered the outcome reported in the objective and if none were reported, we assessed the completeness of reporting of all outcomes reported in the publication and recorded the least adequately reported. In addition to CONSORT related items, we assessed if authors reported information on funding, conflicts of interest for the primary investigators or trial statistician and ethical approval. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint We assessed if the first report publicly available was consistent between what was planned and reported in the registry and what was actually reported in the publication. Particularly, we checked for primary outcome(s) switching between the registration and the report. Primary outcome(s) switching was defined as adding, removing or changing an outcome (i.e., the variable of interest, time frame or metric). Trials that failed to provide any timing information in the report or trial registration, were assessed only for a change in the variable of interest. For the assessment of outcome switching, all available registration platforms were used. If the trial registration was modified after the study start date, we considered the latest registration entry before trial start, if available. We checked whether outcome switching was disclosed in the report. Explanations and justifications were considered as valid, as soon authors indicated the changed primary outcomes in the report (e.g., introduction or discussion section of the report). For preprints secondarily published in a peer-reviewed journal, we compared the reporting of the first preprint report available to peer-reviewed publication. Changes between preprint and peer-reviewed journal publication were classified as "added" information (i.e., information missing in the preprint report but reported in the publication), or "subtracted" information (i.e., information reported in the preprint report but removed in the publication) [19] . is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint The descriptive analysis consisted of frequencies, percentages, and medians with interquartile range. Fisher's exact test at p < 0.05 was used to compare the reporting between preprint and peer-reviewed journal publication. . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint The copyright holder for this this version posted February 4, 2022. ; https://doi.org/10.1101/2022.02.03.22270357 doi: medRxiv preprint RCT studies on preventive measures and treatments for COVID-19. Zenodo The COVID-NMA Project: Building an Evidence Ecosystem for the COVID-19 Pandemic The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape Pandemic publishing: Medical journals strongly speed up their publication process for COVID-19 Reporting quality of randomized controlled trials in otolaryngology: review of adherence to the CONSORT statement Opinion: The Problem with Preprints: The Scientist Magazine® Rapid publications risk the integrity of science in the era of COVID-19 Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility? The quality of reporting of randomised controlled trials in asthma: a systematic review COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine Avoidable waste of research related to outcome planning and reporting in clinical trials Methodological quality of COVID-19 clinical research Quality of adverse event reporting in clinical trials of remdesivir in patients with COVID-19 Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study Outcome switching in randomized controlled oncology trials reporting on surrogate endpoints: a cross-sectional analysis Comparison of registered and published outcomes in randomized controlled trials: a systematic review Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation COVID-19 randomized controlled trials in medRxiv and PubMed Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study Interventions for the prevention and treatment of COVID-19: a living mapping of research and living network meta-analysis Research response to coronavirus disease 2019 needed better coordination and collaboration: a living mapping of registered trials Day-to-day discovery of preprint-publication links Rayyan-a web and mobile app for systematic reviews Timing and completeness of trial results posted at ClinicalTrials. gov and published in journals statement: updated guidelines for reporting parallel group randomised trials explanation and elaboration: updated guidelines for reporting parallel group randomised trials This study as part of the COVID-NMA initiative which received funding from Université de Paris, Assistance Publique Hôpitaux de Paris (APHP), Inserm, Cochrane France (Ministry of Health), the French Ministry of Higher Education and Research, Agence Nationale de la Recherche (ANR), the World Health Organization (WHO).