key: cord-0879820-djazppmx
authors: McErlean, Mairead; Samways, Jack; Godolphin, Peter J.; Chen, Yang
title: The reporting standards of randomised controlled trials in leading medical journals between 2019 and 2020: a systematic review
date: 2022-03-03
journal: Ir J Med Sci
DOI: 10.1007/s11845-022-02955-6
sha: aa79d75d622fa42787f69664b4fc15bee26513fd
doc_id: 879820
cord_uid: djazppmx

Randomised controlled trials (RCTs) are the gold standard study design used to evaluate the safety and effectiveness of healthcare interventions. The reporting quality of RCTs is of fundamental importance for readers to appropriately analyse and understand the design and results of studies which are often labelled as practice changing papers. The aim of this article is to assess the reporting standards of a representative sample of randomised controlled trials (RCTs) published between 2019 and 2020 in four of the highest impact factor general medical journals. A systematic review of the electronic database Medline was conducted. Eligible RCTs included those published in the New England Journal of Medicine, Lancet, Journal of the American Medical Association, and British Medical Journal between January 1, 2019, and June 9, 2020. The study protocol was registered on medRxiv (https://doi.org/10.1101/2020.07.06.20147074). Of a total eligible sample of 498 studies, 50 full-text RCTs were reviewed against the CONSORT 2010 statement and relevant extensions where necessary. The mean adherence to the CONSORT checklist was 90% (SD 9%). There were specific items on the CONSORT checklist which had recurring suboptimal adherence, including in title (item 1a, 70% adherence), randomisation (items 9 and 10, 56% and 30% adherence) and outcomes and estimation (item 17b, 62% adherence). Amongst a sample of RCTs published in four of the highest impact factor general medical journals, there was good overall adherence to the CONSORT 2010 statement. However there remains significant room for improvement in areas such as description of allocation concealment and implementation of randomisation. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11845-022-02955-6.

Randomised controlled trials (RCTs) are the gold standard study design used to evaluate the safety and effectiveness of healthcare interventions [1] . When published in academic journals, many years of work, often at significant financial cost, may be summarised in fewer than 5000 words. It is therefore imperative that the reporting quality of an RCT is transparent and clear, allowing readers to appropriately analyse and understand the study design and results that may change clinical practice. In an attempt to improve trial reporting, the Consolidated Standards of Reporting Trials (CONSORT) statement was developed in 1996 [2] . Since its inception, there have been significant updates in 2001 and 2010 [3, 4] . The current iteration consists of a 25-item checklist for correct trial reporting.

Previous studies have shown poor reporting standards in RCTs, particularly in areas concerning trial methodology [5] [6] [7] [8] . More recently, a 2016 systematic review of reporting

What is already known about this subject? The reporting quality of randomised controlled trials has improved since the inception of the CONSORT Statement and its 2010 update.

In 2019-2020, the standard of reporting in four of the highest impact factor general medical journals is high though common areas of poorer reporting quality persist, particularly in the description of allocation concealment and implementation of randomisation. How might it impact clinical practice in the foreseeable future? Subsequent reporting of RCTs requires renewed focus from authors and journals to allow readers greater confidence in assessing the rigour of trial methodology and results. standards of RCTs within Cardiology reinforced this pointadherence to CONSORT items was only 63.8% (SD 18.1%) [9] .

This study aims to assess the reporting standard of a representative sample of RCTs published between 2019 and 2020 four of the highest impact general medical journals [9] . Given the dramatic changes that the COVID-19 pandemic has brought to academia, including greater use of pre-prints and scrutiny of the data veracity of studies, the importance of robust reporting of RCTs is more salient that ever.

The protocol for this systematic review was registered before data extraction, as a preprint on the medRxiv database https:// doi. org/ 10. 1101/ 2020. 07. 06. 20147 074, following an initial attempt to register on PROSPERO. This was declined on the grounds of not fulfilling the scope of PROSPERO, despite similar works being registered [10] . This manuscript has been prepared according to the updated 2020 guidelines issued by the PRISMA group [11] . A checklist is available in the Supplemental Appendices along with a list of protocol deviations and justifications.

Studies were eligible for inclusion if they (1) described the primary results of a randomised trial; (2) 

We performed a comprehensive search of the terms 'New England Journal of Medicine', 'Lancet', 'Journal of the American Medical Association' and 'British Medical Journal' on MEDLINE only, given that these target journals are all MEDLINE indexed. To identify RCTs for inclusion, the Cochrane Highly Sensitive Search Strategies for identifying randomised trials filter was used [12] . A time filter was applied to obtain results from January 1, 2019, to June 9, 2020 (the date of the search execution).

After removal of duplicates and clearly irrelevant records, two independent reviewers (MM, JS) screened the titles and abstracts of the search results. The full texts of the remaining results were individually assessed by both reviewers for inclusion with arbitration by a third author if necessary (YC).

Data from eligible studies was extracted from study reports independently by three reviewers (MM, JS and PG) including the general characteristics of the RCTs.

A random sample of 50 papers was selected to score against the CONSORT checklist using a random number list generated by one reviewer (PG). The software package Stata SE version 16.0 (StataCorp, College Station, TX, USA) was used to generate the random number list. Papers were scored independently by three authors (MM, JS and PG) against the 25-item 2010 CONSORT statement.

Each item was given an equal weighting-12 items were divided into A and B parts giving a total of 37 points scored per paper. Each item was subdivided as outlined in the CON-SORT statement [4] . Any differences in scores were resolved through consensus. The CONSORT Extension statement was used for RCTs that included designs other than a parallel 2-arm comparison [13] [14] [15] . Two papers from each scorer were selected at random and audited independently by the corresponding author (YC). If auditing revealed significant discrepancies, a re-evaluation of the original scoring was triggered.

We did not conduct a risk of bias assessment nor a quantitative synthesis.

A total of 1,413 records were retrieved by electronic searches, last updated on June 9, 2020 (see Fig. 1 ). After removal of duplicates, 568 full texts were assessed of which 70 were excluded. A final list of 50 papers was chosen at random for full analysis against the CONSORT statement.

Of the 50 RCTs selected for full CONSORT scoring, their general characteristics are summarised in Table 1 . Their representativeness compared to the 497 eligible RCTs is described, along with a full table of characteristics of all eligible studies, in the supplementary materials. The overall adherence to CONSORT items was high-across all the studies, the average adherence was 90% (SD 9%), and only 4 RCTs (8%) scored less than 75% adherence to relevant CONSORT 2010 items (Table 2 ). Compliance was variable between different CONSORT 2010 items. This is shown in Table 3 and was poorest for items relating to title, item 1a (35/50, 70%); allocation concealment, item 9 (28/50, 56%); implementation, item 10 (15/50, 30%); and outcomes and estimation, item 17b (21/34, 62%).

Our study has three principal findings. Firstly, the general reporting standard of RCTs in high impact medical journals in 2019-2020 is strong. Nine papers had either full or all but one item fully adherent to the CONSORT 2010 guidance. Second, there were common and important areas where performance was suboptimal-namely in allocation concealment and implementation of randomisation. Third, there was a significant number of cluster and factorial study designs-14% of the studies analysed were scored using their relevant CONSORT extension document.

Our findings demonstrate a marked improvement from previous systematic reviews [7, 8] which admittedly included RCTs from a more heterogeneous group of journals. More broadly, whilst the adoption and use of reporting guidelines are prominent in RCTs, there remains work to be done across other study designs and areas of medicine, notably highlighted in a recent paper examining high impact rehabilitation journals [16] . Other groups have considered additional extensions to CONSORT-in particular a more detailed description of the interventions used in RCTs [17] . How this becomes blended together with other initiatives to achieve a consistently high standard of RCT reporting remains a challenge for future work.

In current RCT reporting published within journals that represent the pinnacle of academia, areas of concern remain. The reporting of important steps in the randomisation process such as allocation concealment and implementation are areas of weakness. Whilst editors are confined by limited journal space, there may be a role for 100-200 words of protected text to allow for adequate description of the main strength of an RCT design-the component of randomisation. As an editorial described at the time of release of CON-SORT, if 'the whole of medicine depends on the transparent reporting of clinical trials' [1] , then the cost of an extra couple of paragraphs or a clearly signposted and structured appendix to find detailed descriptions of the randomisation process should be prioritised, given that robust methodology is held in as high regard as the result of the trial itself. The reporting of some basic CONSORT items remains consistently poor across time; for instance, omission of the word 'randomised' in the title was noted in nearly half of CONSORT abstracts of the same medical journals in a 2012 analysis [18] . There is no clear explanation for why this remains the case though the definition of quality of reporting is not simply limited to CONSORT checklist adherence. Indeed one could argue that for journals such as the NEJM, article titles do not require explicit conformity given the entry criteria for publishing original research on their platform necessitates robust methodological design.

Lastly, the word count limits imposed by the journals vary considerably-ranging from 2700 words in the NEJM to an unrestricted upper end in the BMJ (see supplementary materials). We noted that whilst many of the sections are interchangeable across the different journal platforms, there were advantages to the more liberal constraints of the BMJ, particularly with regard to mandatory reporting of patient and public involvement, and thus focusing readers' attention in thinking about the value of the study that they are reading with reference to the views and input of patients who may benefit from such results. Whilst a counter-argument to restricted space is that there is almost always co-publication of supplementary material which can contain helpful additional information to the reader, what remains unknown is the number of times such material is ever downloaded and read.

Our findings must be considered in the light of several limitations. First, there is inherent subjectivity to the way that adherence to CONSORT items are judged. Although we used independent reviewers and adjudication to reach consensus, there may nevertheless remain small variation in findings if results were to be replicated. Second, we chose only 50 papers to be scored-this was to balance against feasibility and comprehensiveness, and our sample size is similar to existing reviews which have examined reporting standards [7, 8] . Random sampling represented the most robust method to select papers from our initial eligible list, though this may not capture the extent to which CONSORT scores may vary depending on different study designs or subject areas. If a larger sample was assessed, then possible associations could be uncovered. Of note, there is also a significant difference between the number of articles contributed by each journal-although this is broadly representative of the full 497 studies identified, the relative contribution of each journal is unequal and should temper overall conclusions. Lastly, the choice of restriction to four of the highest impact general medical journals has substantially limited the number of RCTs that were possible to be analysed, of which many have a wide readership within specialty subject areas-for example, the European Heart Journal. Our contention is that by focusing on the journals with the largest academic circulation and readership, the assumption is that the reporting standards of the published RCTs are likely to be amongst the best within the research community. RCTs in other journals will likely require just as much if not more focus on ensuring that their reporting quality remains as close to what is advocated by the CONSORT group as possible-a standard that has garnered almost universal support amongst quality medical journals.

Amongst RCTs published between 2019 and 2020 within four of the highest impact factor general medical journals, there is strong adherence to the CONSORT 2010 statement. Specific components still have room for improvement, with allocation concealment and implementation of the randomisation process representing areas for particular focus.

The online version contains supplementary material available at https:// doi. org/ 10. 1007/ s11845-022-02955-6.

Author contribution MM and YC conceived the study idea. All authors contributed to the study protocol. YC created the search strategy. MM, PG and JS collected the data. YC wrote the first draft of the article. All authors contributed to critical revision of the article for important intellectual content and approved the final version. YC is the guarantor.

Ethics approval Not applicable.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.

CONSORT revised-improving the reporting of randomized trials

Improving the quality of reporting of randomized controlled trials. The CONSORT statement

The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trails

CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials

Reporting in randomized trials published in International Journal of Cardiology in 2011 compared to the recommendations made in CONSORT 2010

Epidemiology and reporting of randomised trials published in PubMed journals

An assessment of the reporting quality of randomised controlled trials relating to anti-arrhythmic agents

Reporting trends of randomised controlled trials in heart failure with preserved ejection fraction: a systematic review

Clarivate (Online) 2022. Available at: https:// jcr. clari vate. com/ jcr/ browse-categ ory-list

PROSPERO International prospective register of systematic reviews. A systematic review of reporting standards of anaesthetic interventions in randomised controlled trials. PROSPERO 2019 CRD42019141670 (Online). Available at

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration

Consort 2010 statement: extension to cluster randomised trials

Reporting of noninferiority and equivalence randomized trials. Extension of the CONSORT 2010 statement

The Adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design

Declaration of use and appropriate use of reporting guidelines in high impact rehabilitation journals is limited: a meta-research study

Intervention reporting of clinical trials published in high-impact cardiology journals: effect of the TIDieR checklist and guide BMJ Evidence-Based Medicine Published Online First

Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations