key: cord-1047230-qukunizg
authors: Gehanno, J.-F.; Grosjean, J.; Darmoni, S. J.; Rollin, L.
title: Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals
date: 2022-02-17
journal: nan
DOI: 10.1101/2022.02.16.22271068
sha: 17af93b227efa870be4989e0d080224a94b3e703
doc_id: 1047230
cord_uid: qukunizg

Background : Preprints have been widely cited during the COVID-19 pandemics, even in the major medical journals. However, since subsequent publication of preprint is not always mentioned in preprint repositories, some may be inappropriately cited or quoted. Our objectives were to assess the reliability of preprint citations in articles on COVID-19, to the rate of publication of preprints cited in these articles and to compare, if relevant, the content of the preprints to their published version. Methods : Articles published on COVID in 2020 in the BMJ, The Lancet, the JAMA and the NEJM were manually screened to identify all articles citing at least one preprint from medRxiv. We searched PubMed, Google and Google Scholar to assess if the preprint had been published in a peer-reviewed journal, and when. Published articles were screened to assess if the title, data or conclusions were identical to the preprint version. Results : Among the 205 research articles on COVID published by the four major medical journals in 2020, 60 (29.3%) cited at least one medRxiv preprint. Among the 182 preprints cited, 124 were published in a peer-reviewed journal, with 51 (41.1%) before the citing article was published online and 73 (58.9%) later. There were differences in the title, the data or the conclusion between the preprint cited and the published version for nearly half of them. MedRxiv did not mentioned the publication for 53 (42.7%) of preprints. Conclusions : More than a quarter of preprints citations were inappropriate since preprints were in fact already published at the time of publication of the citing article, often with a different content. Authors and editors should check the accuracy of the citations and of the quotations of preprints before publishing manuscripts that cite them.

3 47 Keywords: Quality of reporting, Preprints, Peer review, Publication, MedRxiv, Medical 48 journals.

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 65 However, preprints posted on preprint platforms are not subject to peer-review, and therefore 66 critical appraisal, until they are submitted to peer-reviewed journals. Only a small proportion will 67 be converted to scientific publications and the share of converted preprints shows a declining 68 trend over time [4] [5] [6] [7] . For those finally passing the peer-review process, peer-reviewers' 69 comments and critiques lead authors to revise their manuscripts, substantially at times [4, 6, 8] . In 70 that case, the preprint is not the latest version of the work anymore and identifying if it has been 71 published, and referring to the published version instead of the preprint is a major issue as it 72 provides readers with the latest version of a now certified work. 73 However, matching preprints to subsequent peer review publications is a challenge. Yet, once a 74 preprint is published in a peer-reviewed venue, the preprint server is supposed to update its web 75 page, adding a prominent hyperlink leading to the newly published work. Nevertheless, it has 76 been observed that the preprint server MedRxiv reports only 39.7% of all existing publication 77 links [9] . It is therefore probable that some studies are inadequately cited as preprints, although 78 the work cited has been already published, and sometimes with a different content. 79 The first objective of our study was to assess the reliability of preprint citations in articles on 80 COVID-19 published by the world leading medical journals. The second objective was to assess . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint 6 81 the rate of publication of preprints cited in articles appearing in theses journals and the last 82 objective was to compare, if relevant, the content of the preprints to their published version.

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. Figure 1 ). None of the preprints cited were withdrawn from medRxiv server.

115 Figure 1 : Flow chart of preprint citations and publications.

117 However, among those 124 "preprints", 51 (41·1%) had already been published when the citing 118 article was published online, at least 2 months before for 33 of them, and were, in fact, not 119 preprints anymore at that time.

120 Among these 51 superseded preprints, differences between the preprint version that was cited 121 and the published version were observed in the data or the conclusions for 21 (40%) of them and 122 in the title for 23 (45%) of them (Table 2) .

123 Table 2 : differences between the preprint and its published version . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 144 Overall, the median interval between the first posting of a preprint in MedRxiv and its 145 publication in a peer-reviewed journal was 94 days.

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 165 The fact that a large proportion of preprints cited in articles published in peer-reviewed journals 166 were not preprint anymore at the time of publication of the citing article has not been reported in 167 the literature, to our knowledge. This would not be a problem if the contents of the preprint and 168 its published version were similar. However, we observed differences in the data and even a 169 different conclusion for nearly half and 10% of them, respectively. Therefore, searching for a . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint 170 potential peer-reviewed article before citing a paper as a preprint would better reflect the latest 171 evidence available in many cases. However, identifying the subsequent publication of a preprint 172 can be difficult since we observed, that MedRxiv often does not mention the publication of the 173 preprints in a peer-reviewed journal, although it is supposed to display link to journal publication 174 within a month [14] , which confirms previous studies [9, 15] . Furthermore, in our study, nearly is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint 211 For the preprints which were finally published, the differences between the preprints and their 212 published version were frequent and close to those reported in another study on 139 preprints, 213 with changes in the study results or the abstract conclusions between the first preprint version to 214 journal article for 38% and 29% of preprints, respectively [6]. However, when studying 67 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint 15 236 In light of that, in our study, when combining the 58 preprints that were published and the 56 that 237 appeared finally in a peer-reviewed journal, but with different results or conclusions, we can 238 consider that 62.6% of the quotations of preprints may not be fully reliable. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) 284 Acknowledgements: authors thank Anna Gehanno for her help in editing the manuscript.

. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 17, 2022. ; https://doi.org/10.1101/2022.02.16.22271068 doi: medRxiv preprint

However, preprints have been widely used during the COVID-19 pandemic where much of the 194 preliminary evidence has been made available through preprints at the time of WHO declaring 195 the epidemic a public health emergency. Only a fraction of these preprints is subsequently 196

In our study, nearly half of the preprints which had not been published in a peer-reviewed journal 198 before being cited as a preprint, were not subsequently published in peer-reviewed journals

among the preprints that were not finally published, all but one had been posted in MedRxiv 201 before mid-August 2020 and we searched for publication more than one year after the posting. A 202 subsequent publication is therefore improbable, considering the usual median delay of 203 publication reported in other studies

The rate of subsequent publications of preprints cited in the four world-leading medical journals 207 is higher than the rate of publication of preprints reported in other studies

However, the rates of publication reported in those studies are probably 209 underestimated since they relied on information about subsequent publication in MedRxiv, 210 which we observed to be largely inaccurate

215 studies on Covid-19 prevention or treatment, Bero et al. found only 23 articles (34%) that had no 216 discrepancies in results reporting between preprints and journal publications

217 The changes in the conclusions were less frequent than the changes in studies' result, mostly 218 because conclusions were usually broad, such as "among patients hospitalized with Covid-19, 219 those who received hydroxychloroquine did not have a lower incidence of death at 28 days than 220 those who received usual care

These changes can be the consequence of publishing preliminary results in a preprint and final 222 results in a peer-reviewed journal, of the fact that some preprints receive comments that allow 223 authors to improve their manuscript prior to submission to a journal

Preprints on COVID-19 posted in preprint servers are shorter and contain less references than 226 non-COVID-19 preprints

Concerning gradual quality improving, although preprint servers offer authors the opportunity to 228 post updated versions of a preprint, enabling them to incorporate feedback, correct mistakes, or 229 add additional data and analysis, the majority of preprints on COVID-19 in preprint servers exist 230 in fact as only a single version [1]. A study on bioRxiv preprints showed that less than 10% of 231 preprints received at least one comment

Although little empirical evidence is available to support the use of editorial peer review as a 234 mechanism to ensure quality of biomedical research [24], the peer review process is a still

The evolving role of 287 preprints in the dissemination of COVID-19 research and their impact on the science 288 communication landscape

The swift knowledge development path of COVID-19 research: the first 291 150 days

Preprint servers: a 'rush to publish' or 'just in time 293 delivery' for science?

The development of preprints during the COVID-19 295 pandemic

Submissions and 297 Downloads of Preprints in the First Year of medRxiv

Changes in evidence for studies 300 assessing interventions for COVID-19 reported in preprints: meta-research study

Systematic 325 examination of preprint platforms for use in the medical and biomedical sciences setting

Using 328 preprints in evidence synthesis: Commentary on experience during the COVID-19 329 pandemic

Maslove DM. Medical Preprints-A Debate Worth Having

Guidelines Are Urgently Needed for the Use of 334 Preprints as a Source of Information

Proliferation of Papers and Preprints During the 337

Pandemic: Progress or Problems With Peer Review? Advances 338 in Chronic Kidney Disease

Characteristics of academic publications, 340 preprints, and registered clinical trials on the COVID-19 pandemic

Assessment of 343 Concordance and Discordance Among Clinical Studies Posted as Preprints and

rate and citation counts for preprints released during the COVID-19 pandemic: the good, 304 the bad and the ugly. PeerJ. 2021;9: e10927. doi:10.7717/peerj.10927 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 17, 2022.