key: cord-0255274-3bym5zfx authors: Hutchinson, N.; Moyer, H.; Zarin, D.; Kimmelman, J. title: The Proportion of Randomized Controlled Trials That Inform Clinical Practice: A Longitudinal Cohort Study of Trials Registered on ClinicalTrials.gov date: 2022-05-13 journal: nan DOI: 10.1101/2022.05.12.22275021 sha: 89bf1a67b8c5b6ee83696a93a4a72419b914989f doc_id: 255274 cord_uid: 3bym5zfx Background Prior studies suggest that clinical trials are often hampered by problems in design, conduct and reporting that limit their uptake in clinical practice. We have described "informativeness" as the ability of a trial to guide clinical, policy or research decisions. Little is known about the proportion of initiated trials that inform clinical practice. Methods We created a cohort of randomized interventional clinical trials in three disease areas (ischemic heart disease, diabetes mellitus and lung cancer), that were initiated between 1 January 2009 and 31 December 2010 using ClinicalTrials.gov. We restricted inclusion to trials aimed at answering a clinical question related to the treatment or prevention of disease. Our primary outcome was the proportion of clinical trials fulfilling four conditions of informativeness: importance of the clinical question, trial design, feasibility, and reporting of results. Results Our study included 125 clinical trials. The proportion meeting four conditions for informativeness was 26.4% (95% CI 18.9 - 35.0). Sixty-seven percent of participants were enrolled in informative trials. The proportion of informative trials did not differ significantly between our three disease areas. Conclusions Our results suggest that the majority of clinical trials designed to guide clinical practice possess features that may compromise their ability to do so. This highlights opportunities to improve the scientific vetting of clinical research. The ultimate goal of clinical research is to produce evidence that supports 78 clinical and policy decisions. Numerous analyses suggest that a substantial proportion 79 of clinical trials aimed at informing clinical practice are marred by flaws in design, 80 execution, analysis and reporting. 1-8 The initial research response to COVID-19 81 illustrated the fact that existing oversight mechanisms fail to prevent the initiation of 82 flawed trials. 9 While unexpected events can stymie well-conceived and implemented 83 studies, trials that have features rendering them unlikely to inform clinical practice may 84 do harm by misleading potential participants of their benefits, and by diverting patient-85 participants from otherwise informative research efforts. 10 86 We have previously described five conditions that trials should fulfill to support 87 clinical or policy decision-making. 10,11 First, trials must ask an important and clinically 88 relevant question that is not yet resolved. Second, trials must be designed to provide a 89 meaningful answer to that question. Third, trials must be feasible, with achievable 90 enrollment goals and timely primary outcome completion. Fourth, outcomes must be 91 analyzed in ways that support valid interpretation. Last, trial results must be made 92 accessible in a timely fashion. 93 In what follows, we created surrogate measures for four conditions of 94 informativeness: trial importance, design quality, feasibility, and reporting (the fifth 95 condition, analytical integrity, did not lend itself to objective, dichotomous assessment, 96 and is not assessed below). We then evaluated the proportion of "clinically directed 97 randomized controlled trials" in three common disease areas meeting these four 98 5 were as follows: trial importance (determined by citation of reported trial results in high 126 quality clinical synthesizing documents; the premise of this surrogate is that these 127 documents focus on questions of clinical importance); trial design quality (assessed 128 using a modified Cochrane risk of bias (ROB) tool which is designed to identify threats 129 to study internal validity); trial feasibility (established based on ability to achieve 130 adequate participant enrollment and timely primary outcome completion); and 131 reporting (based on accessibility of primary outcome results via deposition on 132 ClinicalTrials.gov or in journal publications). 133 Clinical Trial Sampling 136 137 We identified all trials registered on ClinicalTrials.gov in our three disease areas 138 We included randomized trials i) evaluating interventions of any type; ii) aimed at 145 the treatment or prevention of ischemic heart disease, diabetes mellitus or lung cancer; 146 iii) with at least one site in the United States (most of which will thus have a regulatory 147 requirement for results reporting); 12 and iv) interventions that were FDA approved, that 148 advanced to FDA approval, or interventions not subject to FDA approval (e.g. cardiac 149 rehabilitation). We did not include trials that we deemed unlikely to be targeted at CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. Our primary outcome was the proportion of trials that met all four conditions of 243 trial informativeness. We provided a 95% binomial confidence interval for our primary 244 outcome. We performed a sensitivity analysis on our primary outcome excluding small, 245 pilot-type studies that would not have been designed to inform clinical decision-246 making. These were identified based on an anticipated participant enrollment below 247 the lowest quartile of target enrollment for our cohort of trials. Due to concern that 248 phase 2 trials are less likely to inform clinical practice than trials of a higher phase, we 249 performed a second sensitivity analysis on our primary outcome excluding phase 1/2 250 and phase 2 trials. 251 As secondary outcomes, we estimated the proportion of trial participants who 252 were enrolled in informative trials, as well as the proportion of informative trials in each 253 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. of our three disease areas. We also report the proportion of trials advancing across 254 each condition of informativeness. We provided 95% binomial confidence intervals for 255 these secondary outcomes. 256 We compared the proportion of informative trials between disease categories 257 and by trial sponsor using the Chi-square test (chisq.test function in R) and provided 258 binomial confidence intervals for each stratum. We used the fisher.test function in R to 259 perform a two-sided Fisher's Exact test assessing the proportion of informative trials 260 by type of intervention and trial phase and provided exact confidence intervals for 261 each. We calculated inter-rater agreement rates using Cohen's kappa (eTable 2). We 262 defined p < 0.05 as statistically significant. All analyses were performed using R version 263 Over half of the 125 interventional trials in our cohort were studies of drug or 275 biologic interventions (77 trials; 61.6%). The majority were Phase 2 (24 trials, 19.2%) or 276 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. Phase 3 trials (50 trials, 40.0%). Trial status was "Completed" in 99 of 125 trials 277 (79.2%) and "Terminated" in 15 trials (12.0%) (Table 1) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. Our primary outcome, the proportion of trials that informed clinical practice, was 291 26.4% (95% CI 18.9 -35.0) ( Figure 2 ). As a sensitivity analysis, we re-analyzed our 292 primary outcome excluding the 35 trials in the lowest quartile for target enrollment. This proportion of ischemic heart disease trials that was informative was 27.5% (95% CI 306 14.6 -43.9); the proportion for diabetes mellitus trials was 31.6% (95% CI 19.9 -45.2), 307 and the proportion for lung cancer was 14.3% (95% CI 4.0 -32.7) (Figure 3) . 308 Proportions did not vary significantly by disease area (p value = 0.23) (Figure 4 ). Each 309 surrogate measure contributed considerably to the stepwise decline in the proportion 310 of informative trials (eTable 3). 311 Studies sponsored by industry were significantly more likely to fulfill all four 312 conditions of informativeness than those not sponsored by industry (50.0% vs. 6.0%, p 313 value < 0.001) (Figure 4 ). Using the two-sided Fisher's exact test, there was a non-314 random association between trial phase and informativeness, and type of intervention 315 and informativeness (Figure 4) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. This study provides the first assessment of the proportion of randomized trials 359 fulfilling four key conditions of informativeness. In our analysis, just over one fourth of 360 trials demonstrated adequacy for study feasibility, reporting, importance, and design. 361 The remaining 73.6% contained a limitation in design, conduct or reporting that 362 compromised their ability to inform clinical decision-making. that can be implemented to increase the likelihood that trials will be informative. Many 376 methodological weaknesses in trial design can be corrected at minor cost. 8 377 The proportion of informative trials did not differ significantly between ischemic 378 heart disease, diabetes mellitus and lung cancer, indicating shared challenges in 379 design, implementation, and reporting. Our study also demonstrated that each 380 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint condition of informativeness goes unfulfilled in roughly equal proportions (eTable 2), 381 suggesting that vigilance is required throughout the life cycle of a trial. Our estimates 382 for the fraction of studies fulfilling criteria for recruitment feasibility are in line with prior 383 studies. 14,21-23 The fraction of trials at low risk of bias is similar to prior estimates. 8,24,25 384 Our estimate for the fraction of studies fulfilling reporting requirements (90.0%) is in line 385 with prior studies that evaluated both ClinicalTrials.gov results deposition and 386 publication, 26,27 both of which were deemed acceptable means of results reporting in 387 our study. To our knowledge, our study is the first to apply these conditions jointly to a 388 sample of trials, in addition to assessing importance via citation in clinical synthesizing 389 Our results also indicate that certain types of trials may be at greater risk for 391 having their informativeness compromised. Phase 4 trials fared worse than Phase 3 392 trials, with only 2 of 19 fulfilling all 4 conditions of informativeness (eTable 4). Trials 393 sponsored by industry funders were far more likely to fulfill all four conditions than 394 those with non-industry sponsorship (50.0% vs. 6.0%). This is in keeping with prior 395 research demonstrating greater recruitment challenges for non-industry funded trials, 14 396 in addition to diminished compliance with timely results reporting on 397 These results suggest that funding bodies and academic medical centers may 399 not provide adequate resources for fulfilling the clinical mission of the trials they 400 support. Several recent initiatives aim at improving various aspects of informativeness, 401 including increased consideration given to the importance and clinical relevance of the 402 research question, the evidentiary basis for proposed research, study registration and 403 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. informative trials as at risk of being uninformative (e.g. trials that evaluate disease 417 management in niche populations that are not addressed in practice guidelines or 418 systematic reviews). It may also have misclassified some trials as informative (e.g. trials 419 addressing already resolved clinical hypotheses, which might nevertheless be cited in 420 systematic reviews). To the former, none of the 18 trials not fulfilling the importance 421 condition involved niche populations (eTable 5). We also acknowledge that some trials 422 may inform clinical practice despite failing our criteria. The DAPT Study (NCT00977938) 423 was a large Phase 4 study that was deemed at high risk of bias in several high-quality 424 systematic reviews. 32,33 However, this study has had an important impact on the clinical 425 management of antiplatelet therapy following drug-eluting stent placement. 34 Our 426 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint 20 metrics are best understood as capturing factors that seriously (but not fatally) 427 compromise a trial's prospects of informing practice, and that are rectifiable. Second, 428 we applied strict inclusion/exclusion criteria when identifying out cohort of "clinically 429 directed randomized controlled trials," thus limiting generalizability to other types of 430 trials, including those involving diagnostics or interventions that do not advance to FDA 431 approval. The latter would require different criteria, given their primary goal of informing 432 CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint Flow Diagram for Ischemic Heart Disease Interventional Trials 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. Conditions for Informativeness 1 Designed to inform clinical decision-making, but at elevated risk of bias . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. Assessment 811 812 Assessment of citation of trial results in high quality systematic reviews (SRs) was 813 independently performed by two authors (NH & HM) . This first involved a search for 814 sources that are well known for producing high quality SRs: Cochrane SRs on the 815 Cochrane Database of Systematic Reviews 4 and Agency for Healthcare Research and 816 Quality (AHRQ) SRs 5 . If trials were not included in Cochrane or AHRQ reviews, 817 additional SRs for published studies were identified using the Scopus database 6 818 citation analysis search function or via Google Scholar 7 for unpublished studies. SRs 819 identified through Scopus or Google Scholar that included trial results in review results 820 were assessed for quality using a modified AMSTAR scoring system: 821 822 Operationalization of modified AMSTAR 8,9 scoring system 823 Yes -the authors stated that methods were established prior to conducting the review or provided a link to a registered protocol record No -the authors stated that there's no protocol available or no information is provided Yes -at least two individuals independently performed study selection and data extraction; the method for reaching consensus in the setting of disagreement was reported No -only one person performed either study selection or data extraction Can't answer -no information about independent study selection and/or data extraction was provided □ Yes (1) Yes -at least two electronic sources were searched; keywords or MESH terms were provided No -only one database was searched; no keywords or MESH terms were provided Can't answer -partial or no information reported □ Yes (1) . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. No -a list of included and excluded studies was not provided □ No (0) and documented with a study-specific quality score provided? Yes -risk of bias or another quality metric was used and reported No -no risk of bias or quality metric was used Can't answer -the authors state that a quality metric was done, but do not provide additional information CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. Quality of CPGs was assessed using a modified AGREE II scoring system: 842 843 Operationalization of modified AGREE II 10 scoring system 844 1. Were systematic methods used to search for evidence and criteria for selection of evidence clearly described? Yes -the authors described electronic databases/sources where search was performed, time periods searched, key terms used; inclusion/exclusion criteria for evidence selection were outlined No -no description was available/no systematic search for evidence conducted/no criteria for selection of evidence described Yes -description of the tools used to assess quality of evidence provided (e.g. GRADE framework) or explicit discussion of the quality of the entire group of included trials provided No -no evaluation of quality □ Yes (1) 3. Were the methods for formulating recommendations clearly described? Yes -description of the recommendation development process was included (e.g. voting procedures) and level of consensus . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. Yes -guidelines were externally reviewed and reviewers were not involved in the guideline development group No -no external review performed or reviewers not independent of guideline information Can't answer -insufficient information to evaluate external review process □ Can't answer (0) Yes -a description of competing interests was provided and their potential impact on guideline development discussed; guideline developers were independent from funding body / funding body did not influence final recommendations No -no competing interests described, or impact on guideline development not assessed, or unclear if funding body has influenced guideline development . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint eMethods 10 -Operationalization of modified Cochrane Risk of Bias score 858 859 We employed a modified 2011 version of the Cochrane Risk of Bias Assessment Tool 860 (detailed criteria for judging risk of bias provided in Table 8 .5d of the Cochrane 861 Handbook for Systematic Reviews of Interventions version 5.1) 12 . When available, ROB 862 scores were extracted directly from high-quality SRs identified during the assessment 863 of trial importance. When not available, assessment was independently carried out by 864 two authors (NH & HM), with differences resolved by a third (JK). Assessment included 865 the following elements: i) random sequence generation; ii) allocation concealment; iii) 866 blinding of participants and personnel; iv) blinding of outcome assessment; v) 867 incomplete outcome data; and, vi) selective reporting. Trials were deemed of sufficient 868 design quality if all elements were deemed to be "low risk of bias" or if a minority of 869 elements were deemed of "unclear risk" and the remaining were "low risk." Any "high 870 risk" of bias element equated with poor trial design. 871 872 Of the 63 trials assessed for trial design, 36 ROB scores were extracted directly from 873 SRs, the remaining 27 trials were assessed by the study team. 874 875 876 877 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint eMethods 11 -Deviations to the Study Protocol 878 879 1. For our feasibility assessment, we first evaluated feasibility of goal patient enrollment 880 and planned date of primary completion based on the first record available on 881 ClinicalTrials.gov (this was independently double-coded). However, we repeated this 882 assessment using the final registration record prior to trial start date as this was felt to 883 provide a better evaluation of feasibility, allowing investigators to adjust enrollment 884 plans and primary outcome timeline prior to trial start. The latter method was single 885 coded and did not produce any change in the results of our feasibility assessment. 886 887 2. We performed two additional sensitivity analyses on the primary outcome: i) 888 excluding all trials in the lower quartile of goal patient enrollment; and, ii) excluding all 889 phase 1/2 and phase 2 trials from the assessment. 890 891 3. We excluded evaluation of primary outcome integrity from our assessment of trial 892 informativeness, given concerns that there can be scientifically valid reasons for 893 altering a primary outcome. For example, a primary outcome might be changed due to 894 evolving clinical practice or in response to new data from outside the trial. 895 896 4. We excluded trials of interventions that were subject to FDA regulations, but were 897 never approved for any indication, or were FDA approved after trial start, but did not 898 have CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 13, 2022. ; https://doi.org/10.1101/2022.05.12.22275021 doi: medRxiv preprint Harms From Uninformative Clinical 937 Trials reviews) 940 5. Agency for Healthcare Research and Quality AMSTAR is a reliable and valid measurement 945 tool to assess the methodological quality of systematic reviews An overview 948 of reviews evaluating the effectiveness of financial incentives in changing 949 healthcare professional behaviours and patient outcomes. Cochrane Database 950 Syst Rev AGREE II: advancing guideline 952 development, reporting and evaluation in health care Cochrane Handbook for Systematic Reviews of 956 Interventions version 5.1.0