What’s So Important about Peer Review? Editorial 678 There has been a lot of dialogue about peer review recently. There were a couple of presentations and a number of discussions about it at ALA’s Annual Conference in Orlando. Since then, it is been a topic that has caught my eye whenever it comes up—which seems to be more often. Lately, there has been commentary popping up in scholarly communication prompted by offbeat perspectives of peer review: A Russian Sociologist has gone so far as to fund a monument to peer reviewers in Kickstarter—what he says will ultimately be a sculpture of a rolling dice with the various traditional outcomes of peer review on each side (https://www.kickstarter. com/projects/972533097/monument-to-an-anonymous-peer-reviewer). It is even receiving some recognition in such venues as Nature (http://www.nature.com/news/ moscow-monument-proposed-to-immortalize-peer-review-1.20578) and the Chronicle of Higher Education among others. This particular example demonstrates the perceived randomness of peer reviewer feedback. There is also the recent maneuver by the largest scientific publisher in the world to patent their online peer review process, in such a broadly worded fashion that the Electronic Frontier Foundation awarded it the “Stupid patent of the Month” (https:// www.eff.org/deeplinks/2016/08/stupid-patent-month-elsevier-patents-online-peer- review). The response to this event has been extreme in some cases, with conjecture that this is the publisher’s effort to shut down other journals. Despite the ongoing rhetoric about the publisher’s desire for world domination, I believe it is unlikely that this will adversely impact other scholarly efforts but it has certainly prompted some strong opinions. Even in the academy, peer review, oddly, prompts some controversy and in some cases, outright disdain. A university provost recently made the statement that “peer review is the gold standard for academia”—while a bit of a truism, this declaration met with a fair bit of consternation from faculty. Some responded that they felt like peer review was out of touch with the real world, underscoring the divide between academia and practice, between knowledge and the application of knowledge to make the world a better place. In addition, some scholars maintain that traditional peer review is the only standard—that journal-based publication is all that should be considered for quality (and cited reference the indicator for impact). Certainly, many guidelines for promotion and/or tenure consider journal quality and impact primary metrics. Tenure and promotion guidelines are what send the signal to researchers about how their work is acknowledged and rewarded, and ultimately, determine who remains in universities to do research. As these guidelines may be slow to change, research, and in turn, peer review, may also be slow to evolve. The peer review process itself does seem to engender some aggravation. Certainly the formal process of peer review can appear as a barrier to publishing new discover- ies in a timely and unmediated manner. There is some aggravation for the perceived arbitrariness of decisions and reviews or, at times, the downright contradictory feed- back from reviewers. In a simplified model with 2 peer reviewers, there are 6 possible permutations in the outcome; of these, the reviewers would theoretically agree 50% of the time. The reality is that they disagree more than 50% of the time. A quick analysis What’s So Important about Peer Review? doi:10.5860/crl.77.6.678 https://www.kickstarter.com/projects/972533097/monument-to-an-anonymous-peer-reviewer https://www.kickstarter.com/projects/972533097/monument-to-an-anonymous-peer-reviewer http://www.nature.com/news/moscow-monument-proposed-to-immortalize-peer-review-1.20578 http://www.nature.com/news/moscow-monument-proposed-to-immortalize-peer-review-1.20578 https://www.eff.org/deeplinks/2016/08/stupid-patent-month-elsevier-patents-online-peer-review https://www.eff.org/deeplinks/2016/08/stupid-patent-month-elsevier-patents-online-peer-review https://www.eff.org/deeplinks/2016/08/stupid-patent-month-elsevier-patents-online-peer-review Editorial 679 of the papers submitted to College & Research Libraries and reviewed from 2015 through 2016 indicates complete agreement between 2 peer reviewers only 31% of the time; in some cases, one reviewer recommends Revise and Resubmit and the other chooses Reject or Accept but in 16% of the cases, the reviewers are diametrically opposed with one saying Accept while the other recommends Reject. A simplified illustration of the review and decision points: Submitted Papers Reviewed by Editor (and Editorial Team) Reviewed by Peer Reviewers Final Decision 36% Desk Reject 29% Acceptance rate The disparate feedback does not indicate that either one is right or wrong; in fact, having the various perspectives is very valuable as they may point out different fac- tors or nuances and because they also represent how readers may approach a paper differently. However, editors work to reconcile this variation by balancing and clarify- ing comments as well as providing additional guidance, and possibly seeking another reviewer. It is also an opportunity to improve documentation of the journal standards as well as selection and onboarding of peer reviewers. College & Research Libraries invites scholars to be peer reviewers through a couple of methods. The most formal process is through the solicitation for committee interest or participation that ALA sends out to the membership once a year. Service on the C&RL is one of the opportunities listed and individuals can self-nominate for inclusion on the Editorial Board: however, editorial preference is that members of the Editorial Board already have some experience serving as peer reviewers for the journal so they gain experience with the journal operations and can bring that perspective to the Board. That list of individuals is then passed along to the Publication Coordinating Committee of ACRL, the C&RL Editor and the Editorial Board who are then considered, a number of whom may not be selected for the board but may be selected as reviewers. The other way in which reviewers are chosen for the journal is through the discretion of the editor which may be a positive response to a scholar inquiring about getting involved with the journal or an invitation to an expert with a specific expertise that fills a niche for the 680 College & Research Libraries November 2016 journal or corresponds with a submission that may have a very rare or specialized focus. In considering people as peer reviewers, the Editor and Editorial will review the names, their areas of expertise, professional experience and record of research and publication. Overall, priority is given to emerging subject areas, representation across types, or libraries, geography and other factors that will provide a broad and diverse expertise. For all of the questioning of the peer review process in scholarly journals, the concept is sound: at its foundation is the ideal that acknowledged and experienced experts within a discipline have a significant role to play in reviewing new research in an objective manner that is consistent with the standards of the discipline. In doing so, they assess the research question, validate the methodology, consider the findings and the way in which they contribute to knowledge or innovate practice. Traditionally, this has been done through the literature with a blind peer review process. It can only be conjectured that at least some of the concern in the academy is about the very traditional model of peer review: but, like so many other aspects of scholarly communication and publishing, peer review can, and absolutely should, evolve. To my mind, peer review refers broadly to the evaluation of knowledge, innovation or practice by someone with recognized expertise in the discipline. Taking an equally broad view of scholarship, I find Boyer’s model provides a cohesive way to look at different kinds of scholarship: • Scholarship of discovery is original research and what is often considered the traditional model which takes the form of scholarly books, journal articles, reference works. • Scholarship of integration involves synthesis of information across disciplines, across topics within a discipline or identifies trends over time. Interdisciplinary research projects or scholarly conferences are such examples. • Scholarship of application or engagement is the application of knowledge to solve real world problems. The “Grand Challenges” that the US President charged higher education to address is symbolic of this type of research. This may take the form of collaboration with community, industry or service or- ganizations, development of policy or educational outreach or services or the contribution of processes or products to improve the world. • Scholarship of teaching and learning looks at the scholarship and transmis- sion of knowledge around pedagogy and learning within a discipline or more broadly. This may be formally published materials, innovative teaching materi- als or the use of emerging technology for instruction. A number of colleges and universities have framed their promotion and tenure documents around Boyer’s Model, allowing the advancement of knowledge to inform practice and the educational process, cross disciplinary boundaries and contribute to society. This inclusive definition of scholarship is one that lends itself to emerging models of scholarly communication that breaks down barriers. The next few editorials in College & Research Libraries will explore evolving models of peer review, from the process itself to the application of expert review in new areas. Therefore, there will be several editorial in the coming issues that will address dif- ferent aspects and kinds of peer review, especially related to the emerging scholarly environment. • Megan Hodge and colleagues will be addressing the process of peer review used in ALA’s PRIMO (Peer-Reviewed Instructional Materials Online) project. This effort models peer review of best practices—a term which is widely used but rarely backed by a formal process for identifying such practices. • In this era of data-driven decision and emphasis on big data, it is ironic that research data itself is rarely reviewed. In most cases, the article is reviewed as a proxy for the dataset. Morten Wendelbo will discuss the necessity of peer Editorial 681 review for datasets and the expertise needed to do so effectively. Editors from the Journal of Peace Research, a premier journal in its field and one of the first to apply a peer review model to data (not to mention including the dataset with the publication of the article). • Emily Ford will be discussing open peer review and the models that a more trans- parent process facilitates, including the benefits of developmental peer review. • Ideally, we would like to also include editorials on Peer review of professional skills, Peer review of grant/funding proposals and Peer review of digital schol- arship. However, we are still identifying scholars with the related experience willing to write on these topics. • Lastly, Sarah Potvin asks the question “Who will review the reviewers?” The foundation of the quality of peer review rests firmly on the expertise and com- petence of those doing the reviewing: the process of identifying and selecting reviewers and the standards to which they are held are critical. These guest editorials will also, once revised and expanded, serve as anchoring chap- ters for a collection to be published by ACRL that will address the evolving models of peer review. Additional contributions will be solicited through the Call for Papers below. Wendi Arant Kaspar Texas A&M University CFP: Evolving Models of Peer Review (Monograph collection to be published by ACRL in 2018) With emerging environments in scholarly communication and initiatives such as open access impacting research activity and venues, the process of peer review plays a critical role in assessing value and quality. However, it is necessary for models of peer review to align with new scholarly efforts and formats, maintaining the validation by experts but demonstrating the flexibility needed for emerging research. We invite submissions of papers examining best practices and innovative models in peer review for inclusion in a monograph collection. While studies within the field of librarianship are preferred, compelling and original cases outside of the discipline will be considered (i.e., Journal of Peace Research’s process for peer review of data). Submissions should focus on specific cases, applications of models or best practices. Note the scope of the guest editori- als: similarly innovative venues, formats or subjects of review are encouraged. Deadlines • November 30, 2016: Submission of proposed paper topic, 300 to 400 words. Submissions will be reviewed as received and selected authors will be notified by January 15. • March 30, 2017: Submission of final papers. Please use the Instruc- tions for Authors from College & Research Libraries. • May 2018: Final collected manuscript is due. Inquiries and submissions may be made to Wendi Arant Kaspar at warant@tamu.edu with the subject line: Peer Review Collection. _GoBack