For what it’s worth – the open peer review landscape This is a repository copy of For what it’s worth – the open peer review landscape. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/90139/ Version: Accepted Version Article: Tattersall, A. (2015) For what it’s worth – the open peer review landscape. Online Information Review, 39 (5). 649 -663. ISSN 1468-4535 https://doi.org/10.1108/OIR-06-2015-0182 eprints@whiterose.ac.uk https://eprints.whiterose.ac.uk/ Reuse Unless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy solely for the purpose of non-commercial research or private study within the limits of fair dealing. The publisher or other rights-holder may allow further reproduction and re-use of this version - refer to the White Rose Research Online record for this item. Where records identify the publisher as the copyright holder, users can verify any specific terms of use on the publisher’s website. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. mailto:eprints@whiterose.ac.uk https://eprints.whiterose.ac.uk/ For What It’s Worth - The Open Peer Review Landscape Andy Tattersall ScHARR University of Sheffield a.tattersall@sheffield.ac.uk Structured Abstract Purpose - The aim of this paper is twofold, firstly to discuss the current and future issues around pre and post publication open peer review. Secondly to review some of the main protagonists and platforms that encourage open peer review, pre and post publication. Approach - The first part of the paper aims to discuss the facilitators and barriers that enable and prevent academics engaging with the new and established platforms of scholarly communication and review. These issues are covered with the intention of proposing further dialogue within the academic community that ultimately address researchers' concerns, whilst continuing to nurture a progressive approach to scholarly communication and review. The paper will continue to look at the prominent open peer review platforms and tools and discuss whether in the future it can become a standard model. Findings – The paper identifies several problems, not exclusive to open peer review that could inhibit academics from being open with their reviews and comments of other’s research, whilst highlighting the opportunities to be had by embracing a new era of academic openness. Practical Implications – The paper summarises key platforms and arguments for open peer review and will be of interest to researchers in different disciplines as well as the wider academic community wanting to know more about scholarly communications and measurement. Keywords - Open Peer Review; Open Access; Peer Review, Post Publication Peer Review; Altmetrics Introduction The purpose of this paper is to discuss the development of open peer review and the various platforms that are exploring and championing this method of measuring and communicating research. Open peer review is not a new idea and was trialled in the late 1990s by (Smith 1999) and (Godlee et al. 1998) in high profile journals, BMJ and JAMA. Other notable publications followed including Nature in 2006. Whilst peer review in all its guises as a practice within academia is one that pre-dates the invention of the scholarly journal, originating with the formation of the national academies in the 17th Century (Fitzpatrick 2011). This paper explores the different approaches taken by ten of the leading academic open peer review platforms. Other social tools, including non-academic ones such as Twitter are potential platforms for open peer review, but for the purpose of this paper we will explore the formal academic examples. What was once on the surface a simple, mostly blinded pre-publication model, peer review of research is now exploring new methods of assessing research quality. Some of these methods revolve around openness and discourse and can be referred to as several things such as post publication review and open peer review. (Ford, 2013) literature review of open peer review found there was no established definition of the term accepted by the scholarly research and publishing community. However (Ford, 2013) found several common open peer review characteristics that describe the openness of the review process: signed review, disclosed review, editor-mediated review, transparent review, and crowdsourced review. Three additional characteristics describe review timing, similar to traditional peer review: prepublication review, synchronous review, and post-publication review. This paper will focus on this evolving method of open peer review, whether that be pre or post publication. Whilst much focus will be on the various platforms that are now exploring open peer review and comment. There are several models of peer review currently in practice with the standard models being single or double blind. Single blind is where usually the author’s identity is revealed to reviewers and double-blind is where all identities are kept hidden. Blind peer review is very much focused around pre-publication of research and acts as quality control. This however is not always successful due to various issues including bias (Lee et al. 2013) as well as plagiarism, self-citation, conflict of interests and the holding back of competing research. In addition there are fictitious pieces of research (Baxt et al., 1998) which can also on occasion be set up to sting peer reviewers and editors. Finally there is the simple factor of bad research reviewed by bad or inappropriate reviewers. Some of these problems could be negated by open peer review which allows authors and reviewers to be aware of each other’s identities. Open peer review can take place pre and post publication with the idea being that it creates a long tail of communication and knowledge exchange between researchers. There may be variants on this model as reviewers and reviewees may be given the option to supply their name and details to each other. For some platforms there has to be agreement before the review begins that it is a fully transparent process, warts and all. Platforms like PLOS ONE make their pre-publication review process open as an option, whilst post publication is open. Whilst platforms like Peerage of Science takes the approach of encouraging open peer review than enforcing it. As (Smith 2006) points out, people have a great many fantasies about peer review, and one of the most powerful is that it is a highly objective, reliable and consistent process, yet in reality, many are discontent with the model of traditional blind peer review, thinking of it only in negative terms- lacking in rewards, slow in return, inconsistent, and occasionally open to fraud and bad behaviour. A systematic review conducted by (Jefferson et al. 2002) found that the practice of peer review is based on faith in its effects, rather than facts. Peer review does not have the best reputation within some areas of academia and is treated with a degree of distrust. Equally so, open peer review could open up new feelings of discomfort in equal measure if not handled properly. The feeling of discomfort is partly due to a change in the academic landscape that is happening on a variety of different fronts. MOOCs, the growing impact agenda, social media, open access, altmetrics and big data are affecting the how universities and research centres operate. All of these changes have leanings towards openness and may be seen as a threat by those academics who prefer to carry out their work in isolation and with a degree of anonymity. Open post-publication review is nothing new as the BMJ and other research publications have accepted letters, email communications and Rapid Responses about research published in their journals for some time. Tools such as blogging and social media have more recently created platforms for researchers to discuss other’s work. More recently websites such as The Conversation have enabled academics to publish their ideas, thoughts and research to wider audiences, all of whom can pass comment on their work directly. Other discussions take place very publicly on news sites and forums or in member-only forums. It would not take long for many researchers to find some mention of their work on the web that goes beyond citations. Traditional Peer Review Despite its key, idealised role in the history of scholarship, peer review has at times been subject to criticism (Sullivan 2014), whilst the traditional academic publishing model has been criticised for lagging behind the rest of the modern publishing industry. Much of this criticism is fair, as a piece of research which can take over a year to complete can then take even longer to be published. After such time, work in that area may have moved on, new methods, technologies and ideas may have surfaced. Open peer review could help highlight these problems and may make researchers aware of potential future collaborators or similar research already being undertaken. We have to weigh up the benefits of open peer review that can encourage collaboration, wider input and knowledge transfer with the negative costs. These include bias, trolling, abusive behaviour and misinformation. Whilst these negative problems may seem barriers it would be foolish to think they do not already exist within academia, nor that they are exclusive to that of the office space and traditional peer review. For peer review to be truly beneficial it has to be open. Websites such as YouTube have allowed aliases and therefore trolls to flourish due to the anonymity it can afford them. Whilst trolling may not seem like the behaviour of an intelligent, logical person, such as an academic, it can be (Klempka & Stimson 2014). Citations and Peer Review Not every piece of research published commands an open peer post-publication review, as not every piece of research gets cited. Figures bandied around the web on the percentage of papers that never get cited range wildly from about 12% in medicine (Larivière & Gingras 2009) to an unsubstantiated claim of 82% for the humanities within the first years of publication. Whilst figures of up to 90% have been shared across academic websites and blogs with no evidence, it seems impossible to get a true figure. Nevertheless it should follow that a large number of research papers will never get commented on in open post publication review platforms. Whilst we have to remember that some areas of research are less reliant on the journal publishing model, it does not mean open peer review is not beneficial to their advancement- it could provide new untapped opportunities. Where some papers were never destined to get cited, they may receive post publication comments and potentially useful insight. Academic debate using the many social and open peer review tools freely available has so far been embraced by a small number of academics. Papers are frequently shared using tools such as Twitter, Google+ and LinkedIn rather than being discussed on these platforms, but that is reflective of social media content on the web in general. It is far easier and less time- consuming to share content on the web than to review it. Proper reviewing takes time and requires more considered thought than most other content shared on the web such as music and film, which is often more subjective. The Evolving Publication In 2004 the term Web 2.0 started to be popularised around the Internet. Originally coined by (DiNucci 1999) some five years earlier, it was popularised by Tim O’Reilly of O’Reilly Media who also had pushed the term ‘open source’ in 1998. Web 2.0 was the point where the web could be manipulated by wider audiences without the need for web authoring and publishing skills such as HTML coding. This new era opened up the possibility for anyone to publish, catalogue, communicate, share and network on the web, including academia. In the decade or so since little has changed in academia, research is often conducted in private, findings are published in journals and presented at conferences. Many research papers, such as those in health research, conclude with the suggestion that more research is needed (Phillips 2001). Therefore the idea of supplanting some of this published research with open post review, new data and insights rather than wholly new inconclusive publications could prove beneficial. Open peer review could not just add new insights to existing research but also form new collaborations. Systematic reviews could be periodically updated and enhanced with previous versions publically available. This would not work for all research but could serve as a way to bring experts together to solve problems, forge networks and create a knowledge transfer economy. A Fear of Openness Many researchers can feel uncomfortable speaking about other’s research (Smith 1999). For those who put their heads over the open peer review parapet it is the fear they will be humiliated by their peers or more senior colleagues. This is especially so of junior researchers and their senior peers (Walsh et al. 2000). This is understandable as bullying and intimidation happen face to face within universities, so it should inevitably extend to the web. Despite the open public face of the web it does not deter people behaving horribly to others. Given that the established peer review system has for years shielded reviewers from a right to reply for their comments; it should follow that by going public it may require some to change their tact. It is no different to how some lecturers feel uncomfortable giving feedback to their students, especially when that feedback is negative. No one likes to receive bad news, just as no one likes to deliver it, certainly face to face. It is understandable that no researcher wants to hear negative comments about their own hard work, especially via open peer review when delivered on a public platform. Throughout the history of research, whether as we know it now or going back hundreds of years there has always been some element of fear. This includes the fear of failure, not receiving acceptance, being wrong or too radical. Add the often mentioned ‘publish or perish’ and academia can be an intimidating arena. Imagine for example one of history’s scientific greats such as Galileo and consider if he lived today. How would open peer review respond to his support of the idea that the Earth revolved around the Sun and not the other way round as was commonly believed? Four hundred years ago he was opposed by astronomers and the Catholic Church for supporting such revolutionary ideas. Could open peer review publicly lead to the humiliation and quashing of such incredible minds and their ideas? Yet if we think about the period of history in which Galileo and his peers lived, it was the fear of imprisonment, punishment and even death, far worse than that of comments posted on the web. As the nursery rhyme goes, ‘sticks and stones will break my bones, but words will never harm me’. Nevertheless we know this to be untrue for some that have gone public with their ideas on the web. At present the decision lies with individual academic as there will always be those maverick enough to voice their ideas publicly and take the flack, and those who do not. Researchers may not feel the desire to review a peer’s work publicly, but they may struggle to resist reading what others have said about their own work. Whilst most researchers may not be aware of this shift towards open peer review in their own fields of work, they may be aware of it in others that extend beyond academia. This existing world of review and comment is where much of the anxiety is likely to come. Popular websites such as YouTube are full of hostile and negative comments, Twitter is renowned for trolling behaviour and comments on stories published in the media will be enough to concern academics. These comments can become personal, malicious and toxic. That is not to say open peer review will stoop to these lows, it is however still possible. The problem for some academics is that the more vociferous and aggressive among them could draw others into public arguments that serve no one well. This already happens between researchers and the public over controversial research, so it should follow it can happen between peers. The thorny issue we have now is that everyone’s opinion can extend to public platforms on the web to voice it. The web has facilitated an opinion culture to the point where ‘trolling’ is now an acknowledged and serious problem. Academics are more than culpable for making barbed comments, but making unjustified ones online will help no one, especially in the advancement of knowledge via discourse. Despite this potential for negativity it is important this issue is discussed openly and maturely as the pattern is quite clear, research in whatever format is becoming more open not less so. This is through public engagement, social and traditional media, open access, blogging and news curation sites. The modern web that was predicted in the last century (DiNucci 1999) is now one where anyone can publish and comment on any platform from just about anywhere. Newspapers, YouTube and Twitter are just three platforms that are rife with a multitude of polarised, mischievous, hostile and un-evidenced comments. Despite these issues there is a real need for academics to be mature about open peer review and embrace the openness of the web that in the longer term has real potential to carry their research much further afield than ever before. The aforementioned platforms, which are often poorly or totally un-moderated exist in an open environment. There are no requirements to be an expert, whilst communities that populate them can be diverse with polarised agendas. That is not to say academics are any different, yet their professional communities are more homogenised, many know each other personally and professional reputation plays an important role. Two problems with any kind of commenting and reviewing model (open or blind) is that you do not always know who you are talking to. For all someone knows, they may as well being talking to a dog over the web (Adrian, 2008). You may know their name, you may have information about where they work, what they do for a living and what they do in their personal time. However, if you have never met them in person it can be hard to gauge the tone they are communicating with you in. This is true of any kind of textual communication and can lead to misunderstandings. The second problem is that of online personality changes as (Joinson 2007) showed that individuals can behave differently when engaging with others over the web, whilst at the other end of the spectrum one study (Buckels et al. 2014) highlighted cyber-trolling as an internet manifestation of sadism. Those capable of behaving badly in person will continue to do so over the internet. Open peer review is much the same as social media and how it has been applied within universities -the reality is some academics will get it horribly wrong. There have been various incidences of university teaching staff being suspended or having contracts cancelled as a result of demeaning comments posted openly online about colleagues or students. The key for academics who are confronted by abusive or inappropriate open peer review is to either make the following communications private or report it. The worst thing to do is get drawn into a public argument online, as rarely either side comes out looking good. There is a clear difference between reviewing, discussing and commenting, something Kent Anderson in The Scholarly Kitchen touched on. (Anderson 2014) summarised that today’s commentators seem to have many axes to grind. Far too often, commentary forums degrade into polemical attacks with win or lose dynamics at their heart. The pursuit of knowledge and science isn’t the goal. Capitulation of one combatant to another is. Anderson questioned the validity of comments being championed by publications and websites, that they could never be considered in the same light as peer-review. Not everyone gets it right first time Open peer review is unlikely to iron out the imperfections of blind peer reviewed research, as this can never happen. There are various factors that will always work against a utopian publishing model. Predatory journals will publish poor quality research, authors will attempt to hoodwink reviewers and editors with previously published, fictional and poor quality work. (Rowland 2002) adds the issues of salami publishing (producing too many articles out of one piece of research) or duplicate publication, and also omission or down-grading of junior staff by senior authors who effectively steal their subordinate’s work. Not exclusive to open peer review but potentially more humiliating for the reviewer and reviewee, than the blind model, is that reviewers can misinterpret research or get things wrong. They may misunderstand the research findings or have inadequate knowledge on the topic they are reviewing (Rowland 2002), whilst some may more purposefully steal author’s unpublished work or deliberately delay competing work (Rennie 1998). By opening up peer review to a totally transparent process complete with timelines the opportunities for gamekeepers to go poaching will diminish- this meaning those involved in editorial and blind peer review roles will have less opportunity to steal. This could add a higher level of responsibility and accountability that extends not just to author and reviewer but also commentators and post publication reviewers (Rennie 1998). Research carried out by the BMJ (Rooyen et al. 1999) into the effects of informing reviewers that reviews might be posted publicly on the web had no discernible effects on the review process. Potential negative implications from this study found that open peer review could reduce the number of willing reviewers and increase the time taken to review. Naturally a researcher’s time is very precious and there are increasing pressures on them to expand into other avenues of work such as impact evidence. Nevertheless, by improving peer review it can only serve to benefit the quality of research. Open peer review, post publication could give a right to reply for any authors once their work is public. This may be little compensation once the negative comments have been left on a public platform that is then shared across the web. A system of moderation can help and the option to remove inaccurate, biased and malicious comments that are posted by those not involved in the formal peer review. Anyone leaving comments about a piece of research or responding to them must then think carefully before they hit the send button. As with most things on the web these days, it is a very public place, so once a comment is posted, it could be some days, weeks or even years before it is removed or corrected. A Review of the Platforms As open peer review, pre or post publication continues to gain traction the number of platforms, aligned to research publications, individuals or groups will undoubtedly continue to grow in line with other research technologies. The web is very good at reacting to supply and demand with more academics and aligned professionals employing online technologies in greater numbers. The platforms below are not an exhaustive list but do account for a lot of the current activity and discussion around open peer review, whether that be pre or post publication. The websites that are reviewed were chosen as the leading established platforms in the area of open peer review and its variants. Many of them are firmly established not just as review platforms, but as databases, social networks and journal publishers in their own right. F1000Research Faculty of 1000 combines different strands, all committed to publishing research and communicating its findings. Firstly there is F1000Prime, which is a personalised recommendation system for biomedical research articles from F1000. Like PLOS ONE, F1000Research is an open science journal that tries to speed up publishing turnaround times with a transparent referee model. The final component is F1000 Posters which is an open repository for conference posters and slide presentations. F1000Research’s approach to peer review is totally open and is one where there are published referee comments and subsequent replies by the authors. The commenting system is no different than you would see in a newspaper or blog post where reader comments are replied to on an individual basis by the original authors. As with blind peer review, articles are ‘approved’ at once or ‘approved with reservations’ or ‘not approved’. This absolute open approach not only ensures the author’s research is not only revealed to the wider world but also the competencies of the reviewer. Each comment is date stamped and allows for a right to reply by the authors. Visitors to F1000Research can track the conversation and even discuss the article at the foot of the page. This gives a good snapshot of the research publishing timeline, with the entire process, paper, review and discussion taking place on one webpage. Even referee’s reports can be cited in F1000Research and published under a Creative Commons By Attribution License. A DOI (digital object identifier) is assigned to every referee report, so it can be cited independently from the article. http://f1000research.com/ Open Review Open Review is part of the popular academic social network ResearchGate. Open Review gives researchers the ability to publish an open and transparent review of any paper they have read, worked with, or cited. ResearchGate’s approach is to try and look at the evaluation of research in a different way and ask if this research is reproducible. Registered users select an article that is listed on ResearchGate and then can go through a simple review process. This process asks simple yes and no questions relating to the research’s methodology, analyses, references, findings and conclusions. Supporting resources can be attached and the reviewer can leave free text statements supporting each field. The completed review can be seen with each aspect scored which given over time would collate further reviews. Reviewers can add the names of other colleagues involved in the review process but they must consent to their admission. http://www.researchgate.net/publicliterature.OpenReviewInfo.html Peer J Peer J is an open access peer-reviewed scientific journal with a focus on publishing research in the biological and medical sciences. It received substantial financial backing of $950,000 from O'Reilly Media whose founder Tim O’Reilly famously popularised the term Web 2.0. It is also part of the same publishing company co-founded by publisher Peter Binfield (formerly at PLOS ONE) and CEO Jason Hoyt (formerly at Mendeley), bringing with them a lot of experience in scholarly communications. Peer J operates a points system for authors and commentators as their incentive to publish and comment on research. A reviewer can gain anything from 100 points for being an editor or an author on a PeerJ article to just one point for receiving an ‘up vote’ for a reply to a question or comment. The website hosts tables showing the top authors and reviewers which can be filtered by topic area, publication date and those who have asked the most questions and given the most answers. The questions and answers aspect is different to the commenting approach as witnessed on other platforms. It does potentially open up further dialogue between authors and commentators, although as with other similar platforms there is not a lot of activity at present in this area. As for this points ranking system, it will appeal to some researchers, especially those with a competitive edge, but on the flipside it may make others equally http://creativecommons.org/licenses/by/4.0/ http://f1000research.com/ http://f1000research.com/ http://www.researchgate.net/publicliterature.OpenReviewInfo.html http://www.researchgate.net/publicliterature.OpenReviewInfo.html uncomfortable. Academics like to see their research measured in different ways, some captured qualitatively, others quantitatively, and some both. It is an interesting approach towards existing peer review models that is certain to split the academic community. https://peerj.com/ Peerage of Science This website Peerage of Science is not explicitly an open peer review platform but does give authors who submit content the option for reviewers to see their details. The website does encourage authors to remain anonymous but it is not compulsory. The main purpose of this platform is to offer authors an opportunity to have their manuscripts reviewed by qualified, non-affiliated peers. There are some merits for authors to submit their manuscripts to such a site, especially researchers who do not have contact with peers in the field there are submitting. Whilst for reviewers the purpose of Peerage of Science is to build their reputation as a reviewer of research. It operates like an agency that matches reviewers with manuscripts. The concern with such a model is that reviewers risk building a reputation based on a quantity of reviews, not the quality. That said, given the problems some authors have in getting opinions on their work, the benefits could outweigh the risks. https://www.peerageofscience.org/ PLOS ONE PLOS ONE is an open access, mostly traditional peer-reviewed scientific journal published by the Public Library of Science. Whilst the pre-publication submissions are usually a blinded review, unless a reviewer indicates otherwise, the open peer review happens post publication. Registered PLOS ONE members are able to comment and discuss published research. It is the world’s largest journal based on the number of papers it publishes and has given itself a mandate to make research easier to reach and discuss by speeding up the publication process whilst ensuring authors retain copyright. PLOS ONE allows users to comment on research it publishes in the same way newspapers and blogs allow visitors to comment on their news articles. Registered users are required to make comments with the purpose of adding to the research or by making clarifications. This involves identifying and linking to materials that will lead to threaded discussions with regards to the content of the published research. There is no limit as to what a commenter can post as it can be as simple or in-depth as they wish. They may want to just focus on part of the research, the results, the methodology or the conclusion and write just a few lines. Whilst others may take more time to write a longer, in-depth review about the whole paper. Anyone wishing to comment on papers in PLOS ONE must be a registered user and identify any competing interests. The rules are quite simple and say that anyone commenting on someone else’s research must not post content as stated below: 1. Remarks that could be interpreted as allegations of misconduct 2. Unsupported assertions or statements 3. Inflammatory or insulting language Those who break these rules are removed and their account disabled, although this does not prevent them from creating new accounts. This is not a problem exclusive to open peer review websites. http://www.plosone.org/ PubMed Commons http://www.plosone.org/ http://www.plosone.org/ PubMed is a huge publicly accessible search engine that accesses the Medline database of references and abstracts on life sciences and biomedical topics. PubMed Commons was launched as a platform to enable authors who are eligible to post comments on research that was accessible via PubMed. Eligibility is based on being an author of a publication in the database, therefore hopefully preventing just anyone from going onto the site and leaving spurious or mischievous comments. Emails of eligible authors are collected from the National Institutes of Health, the Wellcome Trust and author’s emails within PubMed and PubMed Central. In addition, authors can ask a colleague who is already on the system to send them an invite. PubMed Commons has tighter controls than such as PLOS ONE and other such sites. Anyone wishing to leave comments must use their real name and disclose any conflicts of interest. http://www.ncbi.nlm.nih.gov/pubmedcommons/help/guidelines/ http://www.ncbi.nlm.nih.gov/pubmedcommons/ Publons Publons applies a different approach to open peer review by switching the focus more towards the reviewer. The primary aim of Publons is to highlight and aid researchers and their reviewing activity. Peer review is often regarded as a necessary chore for academics that is rarely acknowledged as part of their public profile and kudos. Publons aims to give credit for their peer review work. Whilst working for peer-reviewed journals have often been seen by researchers as a way of adding to their growing workloads for the benefit of others, that being financial for the publisher and recognition for the authors. Reviewing is very much part of the academic’s profile building exercise but given the hidden element of this role it is not always as easily quantifiable as that of editor or author when applying for jobs or promotion. Peer reviewing may have rewards with regards to the researcher’s CV and promotion prospects in addition they get to see emerging research but it is so much harder given the existing anonymous culture. However, it is no less part of the system that is the research publishing cycle. Publons aim is to work with reviewers, publishers, universities, and funding agencies to turn peer review into a measurable research output. This is done by collecting peer review information from reviewers and publishers, and using the data to create reviewer profiles. This information is verified by publishers so that researchers can add these contributions to their CV. This allows reviewers to control how each review is displayed on their profile, whether that be blind, open, or published. Reviewers can add both pre- publication reviews they do for journals and post-publication reviews of any article. https://publons.com/ PubPeer PubPeer’s is an online journal club, one that allows users to search for papers via DOIs, PMIDs, arXiv IDs, keywords and authors amongst other options. The purpose of this to create an online community of academics that comments and discusses the publication of research results. Researchers can comment on almost any scientific article published with a DOI or preprint in the arXiv. They can also browse an extensive list of journals with comments, although the majority of titles only have one or two comments at present. Unlike some of the other tools mentioned, PubPeer also allows for anonymous commenting, which can be accomplished without the user signing up. These comments are moderated first and how quick an anonymous comment is posted depends on the number of items there are in the queue for moderation. This model, as with any kind of anonymous commenting is always susceptible to trolling and abusive behaviour as reviewers feel an extra level of protection http://www.ncbi.nlm.nih.gov/pubmedcommons/help/guidelines/ http://www.ncbi.nlm.nih.gov/pubmedcommons/help/guidelines/ http://www.ncbi.nlm.nih.gov/pubmedcommons/ from what they say. One researcher filed a lawsuit over anonymous comments which they claim caused them to lose a job offer after accusations of misconduct in their research. https://pubpeer.com/topics/1/3F5792FF283A624FB48E773CAAD150#fb15527 https://pubpeer.com/ ScienceOpen ScienceOpen is an independent publishing platform that operates an open peer review system with a full transparency of reviewers and comments. ScienceOpen makes their referee reports available under a Creative Commons By Attribution Licence and is part publishing platform, part social network. As with other platforms this allows reviewers to build a public collection of reviews with the aim of showcasing researchers not just as authors but critical reviewers. Once users register for an account is can be automatically synchronised with their ORCiD profile. https://www.scienceopen.com/home The Winnower The Winnower is another platform committed to open research that extends to a long tail of discovery and dialogue. The Winnower state that; “The Winnower is founded on the principle that all ideas should be openly discussed, debated and archived.” One of the smaller platforms for open peer review, The Winnower is very much in the mould of so many of the new academic start-ups in that it began life thanks to a PhD student. A small platform may seem less appealing to researchers wishing to review other’s work, especially compared to large established sites such as PLOS ONE and PubMed. This can matter when trying to attract a larger audience but it is useful to remember that from acorns oak trees grow. Take for example Mendeley which started life thanks to three early career researchers and was reportedly acquired by Elsevier for $100m in 2013. The Winnower provides an interesting angle that extends beyond successfully published research to that which was retracted with its own ‘Grain’ and ‘Chaff’ page. The 'grain' features publications with more than 1000 citations or a Altmetric score above 250, whilst the 'chaff' includes papers that were pulled from publication offering authors an opportunity to talk about their research rather than just providing a ‘name and shame’ list. Despite being a fledgling academic start-up and only having a handful of reviews, The Winnower does provide another take on the open peer review landscape. https://thewinnower.com/ Platform Open Pre or Post Publication Review/ Comment Level of Openness Owner Year Established Key Audience Other Services Creative Commons Licence F1000Research Post Open Faculty of 1000 2002 as Faculty of Biology (Now F1000 Prime) Life Sciences F1000Prime F1000Posters F1000Specialists F1000Journal Clubs NA Open Review Post Open ResearchGate 2008 as ResearchGate 2014 Open Review Non-Specific ResearchGate NA Peer J Pre and Post Open Review encouraged Jason Hoyt Pete Binfield 2012 Biology Medicine PeerJ Computer Science PeerJ PrePrints CC-BY-4.0 Peerage of Science Pre Open - Onymous Janne Kotiaho, Mikko Mökkönen, Janne- Tuomas Seppänen 2012 Science NA PLOS ONE Pre and Post Optional for pre- publication. Open for post comment The Public Library of Science 2006 Medicine Science CC BY 4.0 PubMed Commons Post Open U.S. National Library of Medicine 2015 Biomedicine PubMed CC BY 3.0 Publons Pre and Post Optional Andrew Preston, Daniel Johnston 2012 Non-Specific CC BY 4.0 PubPeer Post Optional NA 2013 Non-Specific NA ScienceOpen Pre and Post Open ScienceOpen 2013 Non-Specific ScienceOpen Research ScienceOpen Posters CC BY 4.0 The Winnower Pre and Post Open Josh Nicholson 2012 Non-Specific CC BY 4.0 Table 1. Comparison of pre and post publication open review and comment platforms The previous table indicates some of the issues that the development of open peer review must address. That there are many facets and many different opinions on how best to improve scholarly measurement and communication via peer review, open or otherwise. Not only do we have the option of pre and post open peer review, but also the ability for researchers and reviewers to agree the level of openness. Some of the platforms are backed by larger entities such as ResearchGate and the U.S. National Library of Medicine, whilst others are start-ups involving just a handful of individuals. Whilst this list is not exhaustive, more platforms will appear, and with it potentially more iterations on open peer review. The websites in table 1 feature peer review, commenting, discussion, points systems in addition to question and answers. Some options will be more popular than others depending on the researcher’s personal beliefs, as well as peer and subject area influence. Researchers and reviewers will have their own agendas and bias as to why open up the peer review process. Some may feel they have nothing to hide, whilst others may feel hard done to by blind peer review. Whilst choosing the correct and most rewarding platform will cause some researchers and reviewers concern and confusion. Other notable mentions It is worth mentioning the various platforms that also explore and have explored the pre and post open peer review landscape. PaperCritic worked in conjunction with Mendeley to monitor papers in your reference collection and via your Mendeley contacts list. It ceased posting updates on its various social media platforms in early 2013, which is never a good sign. Chapter Swap aimed to democratise the peer review system by offering authors a grassroots approach to peer review and inviting authors to swap draft copies of their work for review. Chapter Swap’s target is postgraduates and postdocs working within the discipline of the arts and humanities. Again like PaperCritic, the previously active Twitter feed ceased in 2013 indicating that the service was no longer active. Libre is an open peer review platform hosted by Open Scholar C.I.C that operates solely within the academic community. It aims to put authors in charge of the reviewing process which is open and published under a Creative Commons Licence. At the time of writing this paper Libre was still in a testing phase and users could sign up in readiness for the first stable release. Science Open Review, not to be confused with the aforementioned Science Open or SciOR is based at Queens University in Canada. Its remit is to connect authors with reviewers in author-led non-blind peer review. Again with some of the previous tools there appears to be little activity in the year prior to publishing this paper. Finally the Journal of Visualized Experiments (JOVE) is the leading online video journal with a remit to aid the replication of published research. The pre- publication model is anonymous as is part of the post publication comment. Its inclusion is based on it allowing commentators to leave comments that include their first name and the initial from their surname, possibly enough for recognition. A Mixed Model Approach At present we still have an entrenched system for how we measure and understand the scope of research through citations, indexes and impact scores. Add to that Altmetrics, Snowball Metrics and similar systems of measurement and communication through open and blind pre and post publication review and we should be in a good place to sort out the wheat from the chaff. A novel idea that has been suggested is that of giving contributors digital badges for their role in a piece of research (Cantor & Gero 2015) to create an R-index scale of reviewer recognition. A formalised approach to this is certainly more attractive to most than independent review sites such as Rate My Professor, whilst Shit My Reviewers Say operates as a way for researchers to share reviewer comments and “aims to collect the finest real specimens of reviewer comments since 1456.” As with open access, the purpose is to remove access barriers not quality filters (Suber 2012). (Ford 2013) notes that whilst open access and open peer review go hand in hand, open peer review does not need to occur only in open access journals. Open peer review could remove some of the barriers discussed earlier in this paper and improve filters. The evolving publication model could in time encourage more academics to discuss their work more openly on the web, rather than operate in silos. This may take place via other public discussion forums but also by blogging and social media platforms such as Twitter, ResearchGate and Piirus to name but a few. Digital scholarship can only have meaning if it marks a radical break in scholarship practices brought about through the possibilities enabled in new technologies. This break could encompass a more open form of scholarship (Pearce et al. 2012). The benefits of this new openness are also highlighted by David Goldstein, Director of Duke University Centre for Human Genome Variation (Mandavilli 2011). Open review and commenting on published research can help identify incorrect findings. Goldstein (Mandavilli 2011) said; “When some of these things sit around in scientific literature for a long time, they can do damage: they can influence what people work on, they can influence whole fields.” Conclusion At present most of the post publication, open peer review platforms have just a few comments for some research articles, the majority have none. Despite the sheer amount of published research, navigating and responding to them is still quite manageable. As more researchers begin to comment and review work openly it opens up more conversation and communication which could lead to a cacophony of noise if not properly moderated. (Shirky 2008) suggests that our problem is not one of information overload, it’s filter failure. Whether that is the case for some, there still appears a genuine problem of information and communication overload for many, academics included. Responding and leaving comments can be another potential disruptive interruption to their workflow as comments go back and forth between various parties. As with social media and discussion forums there is that temptation to continually peek back to see if anyone has responded to your own review or comment. As with the problem of email and social media, there may be too strong a temptation to get the last word in. Despite that, for open peer review to thrive it needs researchers to leave comments, constructive ones at that, although that is perhaps too much to expect for all of them. It also needs engagement and discussion that brings with it tangible benefits, most importantly being the advancement of knowledge within that research area. Before the advent of the web, researchers worked in deeper silos which meant that it was often not until publishing work and speaking at a subsequent conference they became aware of similar such research taking place. Now researchers using tools such as Twitter or Mendeley can get a feel for what research is going on around them that is of interest and has some overlapping features. The debate on which is the best way forward for open peer review will continue as will other topics that look at the measurement of research. There appears to be no single solution, with at present a collection of websites and tools, sometimes operating in silos and all offering to solve a problem of how better to improve the quality of published research. As with predatory journals and conferences we are likely to see similar ventures in open peer review. The key to improving this process is more active participation by researchers, reviewers and editors in the discussion of how to negate the various problems related to traditional peer review. Peer-review may not be perfect, but as the social web becomes more useful as a formal and informal platform for discussion and knowledge sharing within the academic community it makes sense that these options are explored. The co-existence of blind and open peer review, alongside post publication review can help shape a better system. This is similar to the argument for Altmetrics, first seen by detractors as wholly alternative to the traditional measurement of citations and now more eloquently argued as an alternative indicator, rather than total measurement. Open peer review platforms need to be explicit in their aims and any considerations and explain that clearly to readers and reviewers. Like social media, it is unlikely that we will see every researcher using these unless they became standardised, formalised and part of the research cycle. It is optional, as with academic discussion lists, where some the most insightful and on occasion barbed communications take place. Whilst many authors may feel vulnerable by making their research open for comment, they have to realise that this happens already. Certainly a sizable chunk of unqualified reviewing happens courtesy of the general public when research makes local or national news. Given how the web has evolved into a democratic and social platform it has given anyone connected via the Internet a voice. Open peer review has been theorised and trialled for some time, but as yet it remains the junior partner to the traditional model of peer review. For platforms covered previously there is still some way to go for a majority acceptance, and key to this is a support mechanism for reviewers and authors that is structured, aided by moderation and authentication. If not, as (Van Noorden 2014) asks; “will online comments look more like a scattered hodgepodge of reviews, comments and discussions across websites unlinked to original publications?” Nevertheless, the world of open post publication peer review is happening right now and someone may have already commented on your research, whether you respond remains your choice. References Anderson, K., 2014. Stick to Your Ribs: The Problems With Calling Comments “Post- Publication Peer-Review.” The Scholarly Kitchen. Available at: http://scholarlykitchen.sspnet.org/2014/04/15/stick-to-your-ribs-the-problems-with- calling-comments-post-publication-peer-review/ [Accessed May 28, 2015]. Anonymous. 2015. Shit My Reviewers Say. @yourpapersucks [Twitter] Available from: https://twitter.com/yourpapersucks [Accessed 20 May 2015] Adrian, A. (2008) No one knows you are a dog: Identity and reputation in virtual worlds. Computer Law & Security Review [online]. Available from: http://www.sciencedirect.com/science/article/pii/S0267364908000290 Baxt, W. et al. (1998) Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Annals of emergency …. 32 (3), 310–317. [online]. Available from: http://www.sciencedirect.com/science/article/pii/S019606449870006X Buckels, E.E., Trapnell, P.D. & Paulhus, D.L., 2014. Trolls just want to have fun. Personality and Individual Differences, 67, pp.97–102. Available at: http://www.sciencedirect.com/science/article/pii/S0191886914000324 Cantor, M. & Gero, S., 2015. The missing metric: quantifying contributions of reviewers. Royal Society Open Science, 2(2). Available at: http://rsos.royalsocietypublishing.org/content/royopensci/2/2/140540.full.pdf. DiNucci, D., 1999. Fragmented Future. Print JulyAug, 4(July/Aug), pp.32–34. Fitzpatrick, K., 2011. Planned obsolescence: Publishing, technology, and the future of the academy, New York: NYU Press. Available at: https://books.google.co.uk/books?hl=en&lr=&id=wF4ry3m9ulMC&oi=fnd&pg=PR7&d q=Planned+Obsolescence:+Publishing,+Technology,+and+the+Future+of+the+Academ y&ots=KT_-wEb8UD&sig=FwZqj4RM-HfM2dKEc9d_RQQlFGM [Accessed June 17, 2015]. Ford, E., 2013. Defining and Characterizing Open Peer Review: A Review of the Literature. Journal of Scholarly Publishing, 44(4), pp.311–326. Available at: http://muse.jhu.edu/journals/journal_of_scholarly_publishing/v044/44.4.ford.html Godlee, F., Gale, C. & Martyn, C., 1998. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Jama, 280(3), pp.237–240. Available at: http://jama.jamanetwork.com/article.aspx?articleid=187748 Jefferson, T. et al., 2002. Effects of editorial peer review: a systematic review. Jama. Available at: http://jama.jamanetwork.com/article.aspx?articleid=194989 Joinson, A.N., 2007. Disinhibition and the Internet 2nd ed. J. Gackenbach, ed., San Diego, CA: Elsevier Academic Press. Available at: https://books.google.co.uk/books?hl=en&lr=&id=_cyPiIDZy4YC&oi=fnd&pg=PA75& dq=disinhibition+and+the+internet&ots=MBnm_mx2iW&sig=usBcKwSOMyd5CloTm vNhoEI5BYI Klempka, A. & Stimson, A., 2014. Anonymous Communication on the Internet and Trolling. Concordia University Saint Paul. Available at: https://comjournal.csp.edu/wp- content/uploads/sites/16/2013/12/TrollingPaper-Allison-Klempka.pdf. Larivière, V. & Gingras, Y., 2009. The decline in the concentration of citations, 1900–2007. Journal of the American Society for Information Science and Technology, 60(4), pp.858–862. Available at: http://onlinelibrary.wiley.com/doi/10.1002/asi.21011/abstract Lee, C., Sugimoto, C. & Zhang, G., 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 99(4), pp178-182. Available at: http://onlinelibrary.wiley.com/doi/10.1002/asi.22784/full Mandavilli, A., 2011. Trial by twitter. Nature. Available at: http://www.axeleratio.com/news/trial_by_twitter_nature2011.pdf [Accessed March 31, 2015]. Van Noorden, R., 2014. The new dilemma of online peer review: too many places to post? Nature Newsblog. Available at: http://blogs.nature.com/news/2014/03/the-new- dilemma-of-online-peer-review-too-many-places-to-post.html [Accessed May 30, 2015]. Pearce, N. et al., 2012. Digital scholarship considered: How new technologies could transform academic work. in education. Available at: http://ineducation.couros.ca/index.php/ineducation/article/view/44 [Accessed March 31, 2015]. Phillips, C. V, 2001. The economics of “more research is needed”. International journal of epidemiology, 30(4), pp.771–6. Available at: http://www.ncbi.nlm.nih.gov/pubmed/11511601 Rennie, D., 1998. Freedom and responsibility in medical publication: setting the balance right. JAMA, 280(3), pp.300–2. Available at: http://www.ncbi.nlm.nih.gov/pubmed/9676691 Rooyen, S. Van et al., 1999. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. Bmj. Available at: http://www.bmj.com/content/318/7175/23.short Rowland, F., 2002. The peer-review process. Learned publishing. Available at: http://www.ingentaconnect.com/content/alpsp/lp/2002/00000015/00000004/art00002 Shirky, C., 2008. It’s Not Information Overload. It's Filter Failure. In Web 2.0 Expo. New York: Web 2.0 Expo. Available at: http://www.web2expo.com/webexny2008/public/schedule/detail/4817. [Accessed May 29, 2015]. Smith, R., 1999. Opening up BMJ peer review: A beginning that should lead to complete transparency. British Medical Journal, 318(January), pp.4–5. Smith, R., 2006. Peer review: a flawed process at the heart of science and journals. Journal of the royal society of medicine. Available at: http://jrs.sagepub.com/content/99/4/178.short Suber, P., 2012. Open Access First., Cambridge MA: MIT Press. Available at: http://mitpress.mit.edu/sites/default/files/9780262517638_Open_Access_PDF_Version. pdf. [Accessed March 29, 2015]. Sullivan, M., 2014. Peer Review and Open Access. OpenStax. Available at: http://cnx.org/contents/1137808f-5dd6-40d0-b60f-3ddb518122c2@2 [Accessed March 31, 2015]. Walsh, E. et al., 2000. Open peer review: a randomised controlled trial. The British Journal of Psychiatry. Available at: http://bjp.rcpsych.org/content/176/1/47.short