key: cord-0308268-6hiuwl41 authors: KATAOKA, Y.; Banno, M.; Tsujimoto, Y.; Ariie, T.; Taito, S.; Suzuki, T.; Oide, S.; Furukawa, T. A. title: The impact of retracted randomised controlled trials on systematic reviews and clinical practice guidelines: a meta-epidemiological study date: 2022-02-01 journal: nan DOI: 10.1101/2022.01.30.22270124 sha: 0642d01aaf432d133a9bf56ac3abe3f4de4930a4 doc_id: 308268 cord_uid: 6hiuwl41 Objectives To investigate whether and when the correction is done in Systematic Reviews (SRs) and Clinical Practice Guidelines (CPGs) when their included Randomised Controlled Trials (RCTs) have been retracted. Design A meta-epidemiological study. Data sources The Retraction Watch Database. Eligibility criteria for selecting studies SRs and CPGs citing the retracted RCTs on Web of Science. Review methods We investigated how often the retracted RCTs were cited in SRs and CPGs. We also investigated whether and when such SRs and CPGs corrected themselves by visually inspecting their current web pages. We summarized the proportion of correction and the time from retraction to correction. Results We identified 98 retracted RCTs as well as 360 articles (335 SRs and 25 CPGs) citing them. Among the 360 articles, 157 (44%) were published after the retraction, 203 (56%) were published before retraction. Among 77 articles published citing already retracted RCTs in their evidence synthesis without caution, none corrected themselves after publication. Of 203 articles published before retraction, 149 included RCTs that were later retracted in their evidence synthesis. Among them, one SR was retracted due to plagiarism. Only 5% of SRs (6/130) and 11% of CPGs (2/18) corrected their results. Conclusions A large number of SRs and CPGs included already retracted RCTs without caution and never corrected themselves. When SRs and CPGs had included RCTs which were later retracted, only a small minority corrected their evidence syntheses. The scientific community, including publishers and researchers, should make systematic and concerted efforts to remove the impact of retracted RCTs. To investigate whether and when the correction is done in Systematic Reviews (SRs) and Clinical Practice Guidelines (CPGs) when their included Randomised Controlled Trials (RCTs) have been retracted. A meta-epidemiological study. The Retraction Watch Database. SRs and CPGs citing the retracted RCTs on Web of Science. We investigated ho w often the retracted RCTs were cited in SRs and CPGs. We also investigated whether and when such SRs and CPGs corrected themselves by visually inspecting their current web pages. We summarized the proportion of correction and the time from retraction to correction. We identified 98 retracted RCTs as well as 360 articles (335 SRs and 25 CPGs) citing them. Among the 360 articles, 157 (44%) were published after the retraction, 203 (56%) were published before retraction. Among 77 articles published citing already retracted RCTs in their evidence synthesis without caution, none corrected themselves after publication. Of 203 articles published before retraction, 149 included RCTs that were later retracted in their evidence synthesis. Among them, one SR was retracted due to plagiarism. Only 5% of SRs (6/130) and 11% of CPGs (2/18) corrected their results. A large number of SRs and CPGs included already retracted RCTs without caution and never corrected themselves. When SRs and CPGs had included RCTs which were later . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. What this study adds -A considerable number of SRs and CPGs cited already retracted RCTs and none corrected themselves later. -Only a small minority of SRs (5%, 6/130) and CPGs (11%, 2/18) which cited RCTs that were later retracted corrected their findings after the retraction was announced. -The results indicate that publishers and researchers should make efforts to remove the impact of retracted RCT. Systematic Reviews (SR) and Clinical Practice Guidelines (CPG) aggregating randomised controlled trials (RCT) are vital sources of information for clinical decision making [1] . There are guidelines on ho w to report SRs [2] and to create CPGs [3] in a rigorous scientific manner. One important point to remember is that all these methodologies assume that the data used in their evidence synthesis is valid [4] . In recent years, much attention has been paid to the retraction of papers due to scientific misconduct [5, 6] . Notable recent examples include a RCT of ivermectin [7] and a cohort study of hydroxychloroquine, both for COVID-19 [8]. An increasing number of studies have investigated the fate of such retracted studies: some studies have evaluated the impact of retracted articles on social media [9] or reported on the ongoing citations of retracted articles in some specialties such as radiation oncology, dentistry or COVID-19 [6, [10] [11] [12] . The Cochrane has developed a new policy to address potentially problematic studies including retraction [4] . The policy set out ho w we should manage the retracted articles. Ho wever, to the best of the current authors' kno wledge, no studies have comprehensively examined how retracted RCTs managed in the evidence synthesis in SRs and CPGs. In this study we therefore investigated whether and when the correction is done in SRs and CPGs when they included retracted RCTs. This meta-epidemiological study was conducted and reported in accordance with a guideline for reporting meta-epidemiological methodology research [13] . (Table S1) The protocol of the present study has been published in OSF [14] . . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 1, 2022. ; https://doi.org/10.1101/2022.01.30.22270124 doi: medRxiv preprint We searched the Retraction Watch Database (RWD) for retracted RCTs on 27 th July 2021 using the term "random*" [15] . RWD systematically retrieves retracted biomedical articles since 1756 including more than 28000 entries [15] . The database contains titles of articles but not abstracts. Two independent reviewers then identified RCTs from the full text articles retrieved based on the search results. An RCT is defined as "a work that reports on a clinical trial that involves at least one test treatment and one control treatment, concurrent enrollment and follo w-up of the testand control-treated groups, and in which the treatments to be administered are selected by a random process" [16] . We did not include quasi-randomised controlled trials. Disagreements were resolved through discussion. We searched each retracted RCT in the Web of Science (WOS) on 18 th Oct 2021 [17] to identify references which cited the retracted RCTs. The articles not indexed in the WOS were excluded. We placed no restrictions on date or language. We included SRs and CPGs citing the identified retracted RCT articles. The SR was defined as "a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies" [3] . We included any SR, with or without meta-analysis. The definition of the CPG was "statements that include recommendations intended to optimize patient care. They are informed by systematic reviews of evidence and an assessment of the benefits and harms of alternative care options" [3] . From the searched titles and abstracts, two independent revie wers selected SRs and CPGs. Disagreements were resolved through discussion. We then retrieved the full text articles and two independent reviewers finally identified SRs and CPGs. If . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; https://doi.org/10.1101/2022.01.30.22270124 doi: medRxiv preprint necessary, a third reviewer acted as arbiter. We excluded articles published in journals without websites because judgement of correction of SR or CPG in such journals was difficult. We calculated the proportion of correction among SRs and CPGs, respectively. For SRs, the numerator was the number of SR articles which corrected the results. The denominator was the number of SR articles which cited the retracted article in the results section for evidence synthesis. For CPGs, the numerator was the number of CPGs which corrected the explanation of recommendation considering the cited retracted RCT. We presented the outcome separately for articles published before and after retraction. We extracted the following data from the search results of the WOS of the included articles: the number of authors, country of the authors, publication date, journal impact factor in 2020, number of citations as of 13 th Nov 2021, funding (for-profit, non-profit, none, unclear), and research areas [18] . We used the selenium package version 3.141.0 and the ChromeDriver version 96.0.4664.45 under python 3.7 to extract these data. Two independent reviewers evaluated where the retracted articles were cited in the full text of SRs or CPGs. In addition, one reviewer inspected the web page of the included SR and CPG to determine whether their texts were corrected and if corrected, when they were corrected. Then another reviewer confirmed the results of the inspection. We resolved the disagreements through discussion. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; We used descriptive statistics to summarise. For survival analysis, we used the publication and retraction date of RCTs from RWD. We estimated time to correction using the Kaplan-Meier method. We used R ver. 4.1.2 (R Foundation for Statistical Computing, Vienna, Austria). We used publicly available data only, and ethical considerations for participants were unnecessary. Patients and members of the public were not involved in the research because it was designed to investigate the current methodological practice in the SR and CPG. We did not conduct univariate and multivariable analyses, because of the small number of corrected SRs or CPGs. Search results: (Table S2) . By citation search of the 98 retracted RCT articles in the WOS, we found 4822 records . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; https://doi.org/10.1101/2022.01.30.22270124 doi: medRxiv preprint citing these retracted RCTs. By title and abstract screening, we selected 448 articles for full text examination. After full text screening, we finally included 335 SRs and 25 CPGs (Table S3) . A total of 149 articles cited RCTs in their evidence synthesis, which were later retracted. Among them, one SR was retracted due to plagiarism. Results were corrected in 5% of SRs (6/130) and 11% of CPGs (2/18). Figure 2 sho ws the time from the retraction to correction of these 148 pre-retraction-published articles. The median days from retraction to correction was 962.5 (IQR: 164,1621.5). . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. This study presents the most rigorous, comprehensive and up-to-date investigation of the impact of retracted RCTs in the clinically most sensitive literature of SRs and CPGs. We found 77 SRs and CPGs which cited already retracted RCT without cautions, and none of them corrected themselves during a median observation period of more than two years. Of 149 SRs and CPGs which included RCTs that were later retracted in their evidence syntheses, less than one in ten corrected themselves. Much attention has been paid recently to the fate and impact of retracted research. They found that the findings of a third of SRs and CPGs were likely to change if RCTs with concerns were excluded [21] . Hamilton investigated continued citation of retracted radiation oncology articles. He found several SRs and CPGs cited retracted articles without caution [12] . Our results indicate that most inappropriate citations in SRs and CPGs were not corrected regardless of the timing of retractions. Self-correction in scientific community has not improved despite repeated warnings. Not to disseminate inappropriate information, both researchers and publishers should be more careful of the possibility of retracted articles. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. To decrease the number of SRs and CPGs citing already retracted RCTs, researchers should pay attention to the included articles whether they were retracted. While a previous interrupted time-series analysis has sho wn that retraction does reduce the number of citations of RCTs [22] , the present results indicate that inappropriate citations still prevail. Indeed, retraction is sometimes not reflected in PubMed for a long time [23] , and in some cases, it is hard to identify retraction information even in journal web pages [24] . Researchers should check carefully not only the searched abstracts and retrieved full text files but also the journal web pages [25] . Simultaneously, publishers should make the retraction information more clearly discoverable in their web pages. They should also contact databases indexing abstracts There are several limitations in our study. First, we searched retracted RCTs in clinical medicine through title searches. If there are studies that did not declare themselves to be randomized trials in their titles, the actual number of SRs and CPGs inappropriately citing retracted RCTs may be greater than the present results. Further study is needed . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; to evaluate the influence of retracted RCTs in a more comprehensive manner. Second, we used the WOS to find citing SRs and CPGs. There is a possibility that we underestimated the impact among those not indexed in the WOS. Third, we did not investigate whether the conclusions and recommendations of SRs and CPGs will change when the retraction RCTs are excluded. Retraction is a concept with a certain range from minor to major [28] . A preliminary analysis including various types of study designs sho wed that meta-analysis including retracted studies due to issues with data tend to overestimate the effect size [29] . Ho wever, we must point out that continuing to cite and to include retracted, hence false, RCTs after their retraction is in itself tantamount to scientific misconduct. Many SRs and CPGs cited and included RCTs that had been retracted before their own publication and never corrected themselves since. A great majority of SRs and CPGs that included RCTs which were later retracted did not correct themselves and continued to be available even after one or more of their included RCTs were retracted. The whole scientific and medical community including publishers and researchers should make efforts to remove the impact of retracted RCTs. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; Authors wish to thank Dr. Akihiro Shiroshita for retrieving articles. Not applicable. TAF reports grants and personal fees from Mitsubishi-Tanabe, personal fees from SONY, grants and personal fees from Shionogi, outside the submitted work; In addition, TAF has a patent 2020-548587 concerning smartphone CBT apps pending, and intellectual properties for Kokoro-app licensed to Mitsubishi-Tanabe. The other authors have no conflict of interest. Not required. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. Dataset available from the corresponding author upon the request. The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. We plan to present our findings at a national scientific meeting. We also plan to use social media outlets to disseminate findings. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. ; T a b l e 2 . C h a r a c t e r i s t i c s o f S R a n d C P G a r t i c l e s w h i c h c i t e d a l r e a d y r e t r a c t e d R C T T a b l e 3 . C h a r a c t e r i s t i c s o f S R a n d C P G a r t i c l e s w h o s e c i t e d R C T s w e r e l a t e r r e t r a c t e Cited in the discussion, N = 19 1 Correction Corrected 9 (5%) 2 0 (0%) 9 (5%) 2 0 (0%) 0 (0%) Excluded due to the concern about study group 4 (2%) 0 (0%) 0 (0%) 4 (19%) 0 (0%) Excluded due to exclusion criteria 10 (5%) 0 (0%) 0 (0%) 9 (43%) 0 (0%) No or unclear 130 (64%) 9 (64%) 104 (70%) 9 (43%) 8 (42%) 1 n (%); Median (IQR); 2 One SR article was retracted due to plagiarism; 3 There were 20 journals without Journal impact factor. CPG, clinical practice guideline; SR, systematic review . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted February 1, 2022. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 1, 2022. ; A Manual for Evidence-Based Clinical Practice The PRISMA 2020 statement: an updated guideline for reporting systematic reviews Clinical Practice Guidelines We Can Trust When beauty is but skin deep: dealing with problematic studies in systematic reviews Temporal Trends, Characteristics, and Citations of Retracted Articles in Cardiovascular Medicine Ongoing Citations of a Retracted Study Involving Cardiovascular Disease The continued citation of retracted publications in dentistry Continued Citation of Retracted Radiation Oncology Literature-Do We Have a Problem? Guidelines for reporting meta-epidemiological methodology research OSF | The influence of retracted randomised controlled trials on systematic reviews and clinical practice guidelines: a meta-epidemiological study protocol Retraction Watch Database Randomized Controlled Trial -MeSH -NCBI Does retraction after misconduct have an impact on citations? A pre-post study An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature Assessing the impact of retraction on the citation of randomized controlled trial reports: an interrupted time-series analysis What time-lag for a retraction search on PubMed? Challenges in Identifying the Retracted Status of an Article Policy for managing potentially problematic studies: implementation guidance