key: cord-0021555-o85pzc5p
authors: London, Barry
title: Reviewing Peer Review
date: 2021-07-19
journal: J Am Heart Assoc
DOI: 10.1161/jaha.121.021475
sha: f0f9c70497af1747520d2bf8c4efe820287dbc49
doc_id: 21555
cord_uid: o85pzc5p

nan

M ultiple elements are included in the best practices for scientific publishing. 1 These include the following: (1) establishing a fair and uniform peer review system that assures accuracy and promotes scientific debate; (2) guarding against conflicts of interest and bias; (3) making decisions and publishing manuscripts in a timely manner; (4) requiring transparency about funding sources and author roles; (5) avoiding redundancy in publications and plagiarism; (6) requiring clinical trial registration and maintaining confidentiality of research subjects; (7) reporting research misconduct when it is discovered; and (8) maintaining editorial independence and avoiding commercial/ financial motivations. Of these, effective peer review is the most difficult to institute, and its success is the hardest to demonstrate. 2, 3 After the submission of a scientific manuscript to most journals, the editors first decide whether to proceed with a peer review or to reject the manuscript without review. One or more established scientists with expertise in the research field are then asked to evaluate and rate the strengths and weaknesses of the manuscript with respect to experimental design, methods, data quality, novelty, mechanistic insight, impact on the field, and priority. The reviewers also provide confidential comments to the editors, comments to be transmitted to the authors, and a recommendation to the editors to reject, reject with a de novo chance for resubmission, solicit revision (major, moderate, or minor), accept with revision, or accept the manuscript. The editors then formulate a decision. If a revision is allowed and submitted, the manuscript goes through the process again until an accept or reject decision is reached. Variation between journals and article types includes, among other things, the number of reviewers, the requirements for a statistical and/or technical review, completion of a plagiarism check, blinding of the identity of the authors to the reviewer, and maintaining confidentiality of the reviewers to the authors.

In this issue of the Journal of the American Heart Association (JAHA), Gaudino et al perform a systematic review and network-level meta-analysis of 24 randomized controlled trials that sought to evaluate modifications that would improve the peer review process. 4 The interventions were grouped into author level (blinding the authors' identity to the reviewers; revealing the authors' conflicts of interest to the reviewers), reviewer level (requiring international reviewers; modifying the process of soliciting reviewers; providing reviewer guidelines or specific training; revealing the reviewers' identity to the authors; mandating a statistical review), and editor level (prescreening of manuscripts by the editors; editorial review to identify missing elements; feedback from editors to the reviewers on the quality of the review). The outcomes measured included review quality, manuscript quality, duration of the peer review process, acceptance/rejection rate, and number of errors detected. In the individual studies, the interventions with significant effects on the peer review process included the addition of a statistical reviewer, the use of checklists and guidelines, editorial prescreening of manuscripts, the assignment of a shorter deadline to accept the invitation to review, and the blinding of the reviewers to authors' identity. In the meta-analysis groupings, Gaudino et al found that reviewer-level interventions led to a small improvement in quality measures at the expense of increased review time.

Reviewing Peer Review Identifying ways to evaluate and improve peer review is certainly important. The significance of the studies included in this review is, unfortunately, somewhat less clear. The criteria used to measure review and manuscript quality are subjective, and these outcomes cannot predict an accurate assessment of data quality, novelty, and mechanistic insight by the reviewers. In addition, they will not necessarily predict acceptance of the findings by other scientists and impact on the field.

The peer review of this manuscript is illustrative. Reviewer 1 thought that the quality outcomes were subjective, that the quality of the review is more important than duration of the review process, and that the results were largely expected. Reviewer 2 similarly thought that the quality outcomes were subjective, that the duration of the review process is not an index of quality, and that the heterogeneity and small numbers of studies limit the conclusions. Of interest, both reviewers gave identical scores of top 25% for experimental design, top 25% for data quality, top 50% for originality, and top 25% for overall priority. Of note, reviewer 1 liked the manuscript and recommended minor revision, whereas reviewer 2 thought that the findings were unlikely to be of interest to the readership of JAHA and recommended rejection. This scenario is common, with each reviewer identifying similar strengths and weaknesses but weighing study design, novelty, and/or importance differently. Ultimately, moving forward with this manuscript and others like it requires a decision by the editors, often following discussion during a weekly teleconference, and is based on their subjective overall assessment of priority coupled with the publishing capacity of the specific journal.

Peer review is also central to the evaluation of the grant proposals that fund science. The subjective nature of the process is apparent to all who participate. A common joke refers to reviewer 3, who rants against the proposal and prevents the funding of an otherwise stellar submission that was liked by the first 2 reviewers. I, like many, have submitted essentially the same grant proposal to 2 funding agencies, with one agency triaging the grant (rejecting it without discussion) and the other funding it with a top priority score.

Peer review is heavily dependent on the ability of a journal to solicit expert reviews. Manuscript review is time-consuming and generally not reimbursed. Some reviewers will agree to evaluate manuscripts only for top journals. Several reviewers actively seek to promote the advancement of science in their field, whereas others may seek to stifle any work that may compete with their own. Here again, the onus falls on the journal editors to identify fair, qualified, and thoughtful reviewers and weigh their reviews appropriately.

The limitations of peer review have fueled consideration of other options. These include open online reviews and publication on preprint servers, such as bioRxiv and medRxiv. 5 Although these options have been accepted in other scientific fields, clinical medicine poses additional challenges. Specifically, most submitted manuscripts have some flaws and require revision or correction before their publication, especially if the results will direct patient care.

As Editor-in-Chief of JAHA for the past 6 years, manuscript submissions have increased more than 5-fold, and I have overseen the review of ≈17 000 manuscripts. In attempting to improve peer review, JAHA and the American Heart Association family of journals have instituted mandatory plagiarism checks and statistical reviews, established minimum review standards based on article types, increased the number and diversity of editors and editorial board members, and added additional layers of scrutiny for manuscripts with potentially controversial implications. Despite this, I know firsthand that peer review remains imperfect. It is equally clear to me, however, that there is no better alternative at the present time. We must continue to strive to improve the process and identify novel ways to measure whether our changes are working. The article of Gaudino et al and the studies that it evaluates are small steps in that direction. They also highlight that much more work needs to be done.

Division of Cardiovascular Medicine, University of Iowa, Iowa City, IA.

Best practice guidelines on publication ethics: a publisher's prospective

Editorial peer review: let us put it on trial

Editorial peer review for improving the quality of reports of biomedical studies

Effects of experimental interventions to improve the biomedical peer-review process: a systematic review and meta-analysis

The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape

Thank you to the reviewers who granted permission to include their comments in this editorial.

None.