key: cord-0857589-ox0j0e84
authors: Gaudino, Mario; Robinson, N. Bryce; Di Franco, Antonino; Hameed, Irbaz; Naik, Ajita; Demetres, Michelle; Girardi, Leonard N.; Frati, Giacomo; Fremes, Stephen E.; Biondi‐Zoccai, Giuseppe
title: Effects of Experimental Interventions to Improve the Biomedical Peer‐Review Process: A Systematic Review and Meta‐Analysis
date: 2021-07-19
journal: J Am Heart Assoc
DOI: 10.1161/jaha.120.019903
sha: f2cf5e3b6b410c239da52878c8636e3dbe7219bf
doc_id: 857589
cord_uid: ox0j0e84

BACKGROUND: Quality of the peer‐review process has been tested only in small studies. We describe and summarize the randomized trials that investigated interventions aimed at improving peer‐review process of biomedical manuscripts. METHODS AND RESULTS: All randomized trials comparing different peer‐review interventions at author‐, reviewer‐, and/or editor‐level were included. Differences between traditional and intervention‐modified peer‐review processes were pooled as standardized mean difference (SMD) in quality based on the definitions used in the individual studies. Main outcomes assessed were quality and duration of the peer‐review process. Five‐hundred and seventy‐five studies were retrieved, eventually yielding 24 randomized trials. Eight studies evaluated the effect of interventions at author‐level, 16 at reviewer‐level, and 3 at editor‐level. Three studies investigated interventions at multiple levels. The effects of the interventions were reported as mean change in review quality, duration of the peer‐review process, acceptance/rejection rate, manuscript quality, and number of errors detected in 13, 11, 5, 4, and 3 studies, respectively. At network meta‐analysis, reviewer‐level interventions were associated with a significant improvement in review quality (SMD, 0.20 [0.06 to 0.33]), at the cost of increased duration of the review process (SMD, 0.15 [0.01 to 0.29]), except for reviewer blinding. Author‐ and editor‐level interventions did not significantly impact peer‐review quality and duration (respectively, SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for quality, and SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for duration). CONCLUSIONS: Modifications of the traditional peer‐review process at reviewer‐level are associated with improved quality, at the price of longer duration. Further studies are needed. REGISTRATION: URL: https://www.crd.york.ac.uk/prospero; Unique identifier: CRD42020187910.

Meta-Analysis on Peer-Review however, have led to unequivocal and significant improvements. 5 In this systematic review and network meta-analysis we aimed to quantitatively evaluate the effect of the different interventions tested in randomized trials focusing on improving the quality or efficiency of the peer-review process.

A systematic review and meta-analysis of all published or registered randomized trials assessing interventions aimed at improving quality of the biomedical peer-review process was performed, after formal design disclosure (PROSPERO ID: CRD42020187910). The data that support the findings of this study are available from the corresponding author upon reasonable request.

A medical librarian (M.D.) performed a comprehensive search to identify contemporary randomized trials on peer-review (no language restrictions). Searches were run on December 2019 in Ovid MEDLINE and updated on June 12, 2020. The full search strategy is available in Table S1 .

Two independent reviewers (N.B.R. and I.H.) screened retrieved studies; discrepancies were resolved by the senior author (G.B.Z.). Titles and abstracts were reviewed against predefined inclusion/exclusion criteria. Articles were considered for inclusion if they were randomized trials reporting comparisons between different peer-review interventions at author-, reviewer-, and/or editor-level, aimed at improving quality of the peer-review process by exploring at least one of the following outcomes: acceptance/rejection rate, quality of the manuscript, quality of the review, duration of the peer-review process, number of errors detected. Case reports, conference presentations, editorials, expert opinions, and studies not comparing review processes were excluded.

Full texts of the selected studies were examined for a second round of eligibility screening. Reference lists for articles selected for inclusion were also searched for relevant articles (backward snowballing). All studies were reviewed by 2 independent investigators and discrepancies were resolved by the senior author. The full Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram outlining the study selection process is available in Figure S1 .

Interventions were classified based on the process level at which they operated (author-level, reviewerlevel, editor-level). The following variables were extracted for each study: study level data (sample size, year, country of origin, journal), interventions tested, main outcomes assessed, level of intervention (author-, reviewer-, editor-level), assessors of review quality, timing of the assessment, assessment method, main findings, and summary of the effects of the interventions. For studies with multiple interventions, data were separately collected for each intervention. Two investigators performed data extraction independently; the extracted data were verified by a third investigator for accuracy.

The quality of the included studies was assessed using the Cochrane Collaboration's Tool for assessing Risk of Bias in randomized trials (Table S2) .

The main outcome assessed was the quality of the peer-review process. The differences between traditional and intervention-modified peer-review were pooled as standardized mean difference (SMD) in quality based on the definitions used in the individual studies. Duration of the peer-reviewing process, defined as time-to-decision, was also compared. Randomeffects network meta-analysis was performed using the generic inverse variance method with the netmeta statistical package in R with the study control groups serving as the reference. The Cochran's Q statistic was used to assess inconsistency. Rank scores with probability ranks of different treatment groups were calculated.

Small study effects and publication bias were assessed with comparison-adjusted funnel plots and CLINICAL PERSPECTIVE What Is New?

• While most interventions on the traditional peerreview process do not significantly improve its quality, modifications at reviewers-level are associated with improved quality at the price of a longer duration of the review process.

What Are the Clinical Implications?

• Further investigation into interventions aimed at improving the peer-review process is needed.

MD mean difference SMD standardized mean difference Gaudino et al Meta-Analysis on Peer-Review regression tests. Leave-one-out sensitivity analysis and a sensitivity analysis based on fixed effect methods were also performed. Statistical significance was set at the 2-tailed 0.05 level, without multiplicity adjustment. All statistical analyses were performed using R (version 3.5.2, R Foundation for Statistical Computing, Vienna, Austria).

Searches across the chosen databases retrieved 622 studies. After results were de-duplicated, a total of 575 studies were retrieved of which 24 studies met inclusion criteria (Table 1) . There were 13 studies originating from the United States, 6 from the United Kingdom, 3 from Spain, 1 from India, and 1 from Denmark. There were 9 studies published before the year 2000 and 15 after 2000.

There were 8 studies evaluating the effect of interventions at the author-level, 16 at the reviewer-level, and 3 at the editor-level. Three studies evaluated interventions at more than one level (Tables 1 and 2) .

The outcomes assessed were reported as mean change in review quality in 13 studies (evaluated by means of either a 5-point scale [ 

Eight studies investigated the impact of author-level interventions on the quality of the review; van Rooyen analyzed 467 manuscripts submitted to British Medical Journal where one reviewer was blinded to author identity, and the other was not. The authors found no significant differences in review quality between the groups, as measured by mean total quality score (mean difference [MD], 0.02; 95% CI, −0.11 to 0.14); however, the author highlighted that their results were likely not generalizable to other journals. 25 Fisher et al assigned 57 manuscripts to reviewers blinded or unblinded to author identity and found that while there was no difference in mean rating scores (scores 1 to 5 [1=accept; 5=reject]), but unblinded reviewers gave higher priority scores to authors with more published articles (Spearman rank correlation coefficient [r]=−0.45 for blinded group versus r=−0.14 for unblinded group). 13 In a similar study, Alam et al found no differences in the rates of acceptance (37.5% versus 32.5%), revision (48.75% versus 47.4%), or rejection (13.75% versus 20.0%) between blinded and unblinded reviewers (P=0.32). 6 Godlee and colleagues performed a randomized trial in which reviewers were given a paper accepted to publication with 8 known errors. The reviewers were either blinded or unblinded to authors identity, and then asked to either sign or not sign their comments. Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had effect on rate of detection of errors. However, reviewers who were blinded to author identity were more likely to recommend acceptance (odds ratio (OR), 0.5; 95% CI, 0.3-1.0). 14 Okike et al evaluated the impact of blinding of reviewers on acceptance rate and identification of 5 known errors. They found that both recommendation for acceptance (MD, 1.28; 95% CI, 1.06-1.39; P=0.02) and attribution of higher scores to the manuscript (MD, 1.35; 95% CI, 0.56-2.13; P<0.001) were more likely in the unblinded group. 21 John et al found no effect on the quality of the review by providing the authors' conflict of interest disclosures to the reviewers (MD, 0.04; 95% CI, −0.05 to 0.14). 16 McNutt et al reported that blinding improved review quality (MD, 0.41; P=0.007); no difference was reported in terms of acceptance rate and time to review. 19 Justice et al found no difference in review quality in a trial of 118 manuscripts randomized to a control group (where journals followed their usual practice) or to an intervention group (where one reviewer knew authors identities, and the other was blinded). 18 

Sixteen studies tested the impact of reviewer-level interventions. van Rooyen and colleagues in 2 separate analyses reported that revealing reviewer identity did not significantly impact the quality of review, although it increased the amount of time for the review to be written. 26, 27 Godlee et al found that revealing reviewers' identity did not affect the rejection rate (OR, 0.5; 95% CI, 0.3-1.0) 14 and McNutt and associates reported that revealing the reviewer's identity did not change the quality of the reviews. 19 Walsh et al found a significant difference in mean quality between blinded and unblinded reviewers (3.35 versus 3.14, P=0.02), but the small absolute difference Gaudino et al Meta-Analysis on Peer-Review does not seem to support a clear advantage of one approach over another. 29 Das Sinha reported no difference in mean review score when reviewers were informed that a copy of their comments would be sent to other reviewers working on the same manuscript 12 and Vinther and colleagues found no difference between blinded and unblinded reviewers (mean quality score 3.34 for unblinded reviewers versus 3.29 for blinded reviewers, P=0.51). 28 Schroter et al investigated the impact of reviewer training. Reviewers underwent either a face-to-face training, a self-taught module, or no training and were sent 3 papers with deliberate errors added. A slight improvement in quality was seen in the self-taught group (MD, 0.29; 95% CI, 0.14-0.44; P=0.001) and face-toface group (MD, 0.16; 95% CI, 0.02-0.3; P=0.025) when compared with controls. This improvement, however, was transient, as disappeared upon review of the third paper. 24 Callaham and Schriger invited reviewers to attend a 4-hour formal workshop. While most (81%) found the workshop helpful and 85% of attendees felt that the quality of their review would improve, the authors did not find a significant difference in the mean quality of review between attendees and controls (MD, 0.11; 95% CI, −0.25 to 0.48 for controls versus MD, 0.10; 95% CI, −0.20 to 0.39 for intervention group). 8 Houry et al tested the efficacy of pairing new reviewers with senior reviewers and found that the quality of the review did improve significantly (effect size, 0.1; 95% CI, −0.4 to 0.6). 15 Three studies evaluated interventions at reviewerlevel aimed at decreasing turnaround time. Two of these studies investigated the impact of contacting reviewers before manuscript assignment. Neuhauser and Koran found that this strategy increased turnaround time (from 37.7 to 44.2 days) 20 while, Pitkin and Burmeister found a significant reduction in review turnaround time (21.0 days versus 25.0 days, P<0.001) but not in the overall manuscript processing time (24.7 days versus 25.9 days, P=0. 19) , in large part because of the high rate (15%) of reviewers who declined in the ask-first group. 22 Provenzale and co-authors found a significant decrease in turnaround time when reviewers were given 1 instead of 3 days to accept the invitation to review (total turnaround time 27.9 days in 1-day decision group versus 31.5 days in 3-day decision group, P=0.04). 23 Two studies investigated the impact of the addition of a statistical reviewer on review quality. Cobo et al found that addition of a statistical reviewer improved the quality of the review (MD, 5.5; 95% CI, 4.3-6.7), 10 while, Arnau and colleagues reported no effect (MD, 

Three studies investigated the impact of editor-level interventions. Callaham et al asked editors to give written feedback to poor-quality and average-quality reviewers and found that the review quality did not significantly change, (MD, −0.13; 95% CI, −0.49 to 0.23 in the poor quality and 0.06, 95% CI, −0.19 to 0.31 in the average quality group). 9 Cobo and associates investigated the use of checklists such as CONSORT and STROBE and found that it improves manuscript quality, although the observed effect was small (MD, 0.33; 95% CI, 0.03-0.63 for the comparison "as reviewed"). 11 Johnston et al tested the effect of in-house editorial screening before external review and found that it significantly decreased review time (from 48 days to 18 days, P<0.001). 17

Twenty-four studies were included in the network meta-analysis for the outcome of peer-review quality ( Figure 1, Figure S2 , and Tables S3 and S4). Compared with traditional process, reviewer-level interventions were associated with a significant improvement in the quality of peer review (SMD, 0.20; 95% CI, 0.06-0.33). There was no significant improvement associated with author-(SMD, 0.10; 95% CI, −0.11 to 0.30) and editorlevel interventions (SMD, 0.01; 95% CI, −0.32 to 0.34) ( Table 3) . Reviewer-level interventions ranked as the best intervention (rank score for reviewer-level 0.88 versus 0.57 for author-level, and 0.34 for editor level) ( Figure 1 ).

The level of evidence was high for all comparisons (Table S2 ). Heterogeneity/inconsistency and netsplit analyses are shown in Tables S3 and S4. Egger test for a regression intercept indicated no evidence of publication bias (P=0.18) ( Figure S3 ). Leave-oneout analysis confirmed the solidity of the results ( Figure S4 ). Sensitivity analysis based on fixed-effect methods confirmed the main analysis (Figures S5 and  S6, Tables S5 and S6 ).

The impact of the interventions at different levels (author-, reviewer-, and editor-level) on the duration of the peer-review process was also tested (Figure 2 , Figures S7 through S9, Tables S7 through S9 ). Interventions at reviewer-level were associated with a significant increase in the length of the peer-review process (SMD, 0.15; 95% CI, 0.01-0.29), while author-and editor-level interventions were not (SMD, 0.17; 95% CI, −0.16 to 0.51 and SMD, 0.19; 95% CI, −0.40 to 0.79, respectively) ( Table 3 ). Sensitivity analysis confirmed these results ( Figures S10 and S11 , Tables S10 and S11).

Among the different reviewer-level interventions tested, unblinding reviewer's identity was the only modality that did not significantly impact duration of the peer-review process (SMD, 0.01; 95% CI, −0.17 to 0.19) (Figures S12 through S15, Tables S12 through S14).

In the present quantitative synthesis, we found that among the different interventions proposed to improve the process of peer-review, those directed at reviewer level were associated with improved review quality when compared with traditional methods. However, reviewer-level interventions were also associated with increased duration of the peer-review process, with the only exception of revealing the identity of the reviewers. In individual studies, the only interventions found to have a significant effect on the peer-review process were the addition of a statistical reviewer, the use of appropriate checklists/guidelines, the editorial pre-screening of manuscripts, the assignment of a shorter deadline to accept the invitation to review, and the blinding of the reviewers to authors' identity (Table 2) . No effect was demonstrated for all the other strategies. Larger P values signify larger standardized mean difference vs control and larger intervention effect on peer-review quality. SMD indicates standardized mean difference.

With almost 30 000 journals indexed in PubMed and scientific publication guiding medical practice, the importance of peer-review in medical journals cannot be underestimated. 30 However, only limited research on it has been published to date. In 2012 Larson and Chung 31 performed a systematic review of articles on peer-review of scientific manuscripts and found that out of 37 included papers, the great majority (78%) were editorials or commentaries that did not include original data.

In the only other systematic review and metaanalysis on the topic, Bruce et al found that the addition of a statistical reviewer and the use of open peer-review were associated with an increase in the quality of review. 5 Compared with their work, we have included 2 additional trials, grouped the intervention by their level in the process, and used a network meta-analysis to allow for direct and indirect comparisons and increase analytic power because of the relatively low number of available studies. It is concerning to note how, over the course of 3 decades, only 24 trials, mostly small, were performed to investigate a process that has immense implications for the medical community and the society at large. We believe that the most important finding of our analysis is that much more evidence is needed on such a crucial topic. This is even more important as new concerns with regard to the integrity and quality of the peer-review process have recently emerged. A serious threat to good practice is represented by "predatory publishing", ie, an exploitive academic publishing business model based on journals that charge authors article processing fees and hijack the traditional peer-review processes by either manipulating peerreviewer choice or fabricating reviews reports. 32 The dissatisfaction with the peer review system has led to an increasing use by authors of preprint servers, which however, raise concerns because of the absence of evaluation or certification of the published work (with the risk of unverified information being disseminated). 33 A key issue rests with open review process, ie, the disclosure of reviewers' identity. While this approach may increase transparency and accountability, it may undermine the objectivity and thoroughness of reviewers, especially junior ones without tenure appointments. Also, during the current COVID-19 pandemic the traditional mechanisms of control that major scientific journals use have been stressed to their limits, and have sometimes failed. 34 Indeed, there is a clear conflict between the need to timely revise and possibly publish manuscripts 

The present analysis has several limitations. First, this review, as any similar work, provides more accurate estimates of effect than each included primary study, but cannot generate additional insights. Furthermore, it must be noted that the concept of "quality" of peer-review process is subjective by definition. There were important differences in interventions, journals, publishing models, as well as medical fields and outcomes among the included trials. While attempts were made to standardize the outcome definitions, heterogeneity between the studies remained. Most importantly, review quality is not necessarily related to manuscript quality and clinical importance. Because the number of studies for the individual interventions is limited some of the comparisons are underpowered. Finally, no trial included had a specific cardiovascular focus, but it seems likely that their results can be effectively applied to cardiovascular peer-review.

Limited information is available on the efficacy of interventions aimed at improving the peer-review process. Actions at reviewer-, rather than author-or editor-level seem to be the most effective, but further investigation into this important area is crucially needed. Figure S1 . Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of our analysis. Figure S2 . Net graph for the main outcome of peer-review quality. Figure S3 . Funnel plot for the assessment of publication bias for the main outcome of peerreview quality. Figure S4 . Leave-one-out analysis for standardized mean difference for the main outcome of peer-review quality (random effects model).

. Figure S5 . Network forest plot for quality of the peer-review process among the different interventions (fixed effect model).

Peer review in scientific publications: benefits, critiques, & a survival guide

Peer review: a flawed process at the heart of science and journals

Duration and quality of the peer review process: the author's perspective

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis

Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi-rater study: blinded vs. unblinded peer review

Efecto de la revisión estadística en la calidad de los manuscritos publicados en MEDICINA CLÍNICA: estudio aleatorizado

Effect of structured workshop training on subsequent performance of journal peer reviewers

Effect of written feedback by editors on quality of reviews: two randomized trials

Statistical reviewers improve reporting in biomedical articles: a randomized trial

Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial

Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews

The effects of blinding on acceptance of research papers by peer review

Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial

Does mentoring new peer reviewers improve review quality? A randomized trial

Effect of revealing authors' conflicts of interests in peer review: randomized controlled trial

Early editorial manuscript screening versus obligate peer review: a randomized trial

Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators

The effects of blinding on the quality of peer review. A randomized trial

Calling medical care reviewers first. A randomized trial

Single-blind vs double-blind peer review in the setting of author prestige

Identifying manuscript reviewers: randomized comparison of asking first or just sending

A shorter invitation period for AJR manuscript reviewers: impact on time to completion of reviews

Effects of training on quality of peer review: randomised controlled trial

Effect of blinding and unmasking on the quality of peer review: a randomized trial

Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial

Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial

Same review quality in open versus blinded peer review in

Open peer review: a randomised controlled trial

A systematic review of peer review for scientific manuscripts

Ethical issues in publishing in predatory journals

Preprint servers: challenges and consequences

Two coronavirus studies retracted after questions emerge about data

Author contributions: Gaudino and Biondi-Zoccai were responsible for study design, data analysis, writing, critical review, and gave final approval. Robinson, Di Franco, Hameed, Naik, Demetres, were responsible for study selection, data extraction, writing, critical review, and gave final approval. Girardi, Frati, Fremes participated in study design, data analysis, manuscript drafting, critical review, and gave final approval.

None.

None.

Tables S1-S14 Figures S1-S15 *"Peer Review"/ or *"Peer Review, Research"/ 2 (peer adj3 (review or reviewed or reviewing or reviewer or reviewers)).ti. 3

(blind review or blind reviewed or referee* or post-publication review or cascading review or third party review or author suggested reviewers or editor suggested reviewers or manuscript reviewer*).ti. 4or/1-3 5"randomized controlled trial".pt. 6 (random$ or placebo$ or single blind$ or double blind$ or triple blind$).ti,ab. 7 (retraction of publication or retracted publication).pt. 8or/5-7 9(animals not humans).sh. 10 ((comment or editorial or meta-analysis or practice-guideline or review or letter) not "randomized controlled trial").pt. 11(random sampl$ or random digit$ or random effect$ or random survey or random regression).ti,ab. not "randomized controlled trial".pt. 12 8 not (9 or 10 or 11) 13