key: cord-0923076-3ma14hkz
authors: Raynaud, Marc; Goutaudier, Valentin; Louis, Kevin; Al-Awadhi, Solaf; Dubourg, Quentin; Truchot, Agathe; Brousse, Romain; Saleh, Nouredine; Giarraputo, Alessia; Debiais, Charlotte; Demir, Zeynep; Certain, Anaïs; Tacafred, Francine; Cortes-Garcia, Esteban; Yanes, Safia; Dagobert, Jessy; Naser, Sofia; Robin, Blaise; Bailly, Élodie; Jouven, Xavier; Reese, Peter P.; Loupy, Alexandre
title: Impact of the COVID-19 pandemic on publication dynamics and non-COVID-19 research production
date: 2021-11-22
journal: BMC Med Res Methodol
DOI: 10.1186/s12874-021-01404-9
sha: 504e98598d69a4479045b4971ce4cc9acd1b5335
doc_id: 923076
cord_uid: 3ma14hkz

BACKGROUND: The COVID-19 pandemic has severely affected health systems and medical research worldwide but its impact on the global publication dynamics and non-COVID-19 research has not been measured. We hypothesized that the COVID-19 pandemic may have impacted the scientific production of non-COVID-19 research. METHODS: We conducted a comprehensive meta-research on studies (original articles, research letters and case reports) published between 01/01/2019 and 01/01/2021 in 10 high-impact medical and infectious disease journals (New England Journal of Medicine, Lancet, Journal of the American Medical Association, Nature Medicine, British Medical Journal, Annals of Internal Medicine, Lancet Global Health, Lancet Public Health, Lancet Infectious Disease and Clinical Infectious Disease). For each publication, we recorded publication date, publication type, number of authors, whether the publication was related to COVID-19, whether the publication was based on a case series, and the number of patients included in the study if the publication was based on a case report or a case series. We estimated the publication dynamics with a locally estimated scatterplot smoothing method. A Natural Language Processing algorithm was designed to calculate the number of authors for each publication. We simulated the number of non-COVID-19 studies that could have been published during the pandemic by extrapolating the publication dynamics of 2019 to 2020, and comparing the expected number to the observed number of studies. RESULTS: Among the 22,525 studies assessed, 6319 met the inclusion criteria, of which 1022 (16.2%) were related to COVID-19 research. A dramatic increase in the number of publications in general journals was observed from February to April 2020 from a weekly median number of publications of 4.0 (IQR: 2.8–5.5) to 19.5 (IQR: 15.8–24.8) (p < 0.001), followed afterwards by a pattern of stability with a weekly median number of publications of 10.0 (IQR: 6.0–14.0) until December 2020 (p = 0.045 in comparison with April). Two prototypical editorial strategies were found: 1) journals that maintained the volume of non-COVID-19 publications while integrating COVID-19 research and thus increased their overall scientific production, and 2) journals that decreased the volume of non-COVID-19 publications while integrating COVID-19 publications. We estimated using simulation models that the COVID pandemic was associated with a 18% decrease in the production of non-COVID-19 research. We also found a significant change of the publication type in COVID-19 research as compared with non-COVID-19 research illustrated by a decrease in the number of original articles, (47.9% in COVID-19 publications vs 71.3% in non-COVID-19 publications, p < 0.001). Last, COVID-19 publications showed a higher number of authors, especially for case reports with a median of 9.0 authors (IQR: 6.0–13.0) in COVID-19 publications, compared to a median of 4.0 authors (IQR: 3.0–6.0) in non-COVID-19 publications (p < 0.001). CONCLUSION: In this meta-research gathering publications from high-impact medical journals, we have shown that the dramatic rise in COVID-19 publications was accompanied by a substantial decrease of non-COVID-19 research. META-RESEARCH REGISTRATION: https://osf.io/9vtzp/. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-021-01404-9.

Background With a total of 3,541,881 deaths among 170,360,315 confirmed cases [1] as of May 31st, 2021, the coronavirus disease 2019 (COVID-19) pandemic has placed a strain on health systems worldwide. According to the Organization for Economic Co-operation and Development, an estimated 7 billion dollars, dedicated to COVID-19 research, were unlocked worldwide in the first 9 months of 2020 [2] . Linked to that, major collaborative efforts have been launched to urgently address COVID-19 related medical issues [3, 4] , sometimes at the expense of non-COVID-19 research [5] . Some medical fields have experienced a decrease in funding allocation and publications [6] , which had potentially affected patient care outside of COVID-19. For instance, the pandemic has seriously impacted cancer patients with treatment delays and reduced access to healthcare [7] . Similarly, it has had detrimental effects on organ allocation and transplantation worldwide [8] , with a significant reduction in the number of transplanted organs per day, with disastrous consequences for patients whose lives depend on getting transplanted.

Overall, there has been a substantial redistribution of resources which has significantly impacted the non-COVID-19 medical research worldwide [9] , including clinical trials [10, 11] . In addition, leading scientists have voiced concerns about science expediency [12] and the lowering of scientific standards [13, 14] . Together, these phenomena could have played a significant role on the dynamics of publication and worldwide medical research.

Moreover, recent research has reported a rising number of authors in COVID-19 publications [15] , especially in case reports [16, 17] , which may also reflect a lowering in scientific standards [12, 13] . In medical science, there has been a constant rise in the number of authors since the 1950s [18] . This phenomenon has been highlighted in numerous medical specialties [19] [20] [21] and may be the aggregate consequence of multiple forces, such as the growing complexity [22] and interdisciplinarity of medical research [23] , increasing academic and career pressure, increasingly limited funding and the rising number of collaborations [23] . However, the impact of a pandemic on the number of authors has not been investigated.

We made the hypothesis that a worldwide pandemic such as COVID-19 may impact the medical research and in particular non COVID-19 scientific production [15] . Therefore, to address these questions, we conducted a meta-research to comprehensively investigate the effects of the COVID-19 pandemic on the medical research publication dynamics and the impact of COVID-19 research on non-COVID-19 research.

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement to design and report our meta-research, where applicable (see protocol). A literature search of PubMed was performed between January 1st 2019 and January 1st 2021, for articles published in medical, broad journals and journals specializing in infectious disease and public health with an impact factor greater than 8. It hence included the ten following journals: New England Journal of Medicine, Lancet, Journal of the American Medical Association, Nature Medicine, British Medical Journal, Annals of Internal Medicine, Lancet Global Health, Lancet Public Health, Lancet Infectious Disease and Clinical Infectious Disease.

Two researchers (VG, KL) independently implemented the search strategy and did the data extraction to ensure that the same references were identified. The references of the included medical articles and relevant reviews were scanned for potentially relevant medical articles that may have been missed in the literature search. We also requested potentially eligible medical articles from content experts. The search strategy is presented in the supplementary methods with the study protocol, which integrating COVID-19 publications. We estimated using simulation models that the COVID pandemic was associated with a 18% decrease in the production of non-COVID-19 research. We also found a significant change of the publication type in COVID-19 research as compared with non-COVID-19 research illustrated by a decrease in the number of original articles, (47.9% in COVID-19 publications vs 71.3% in non-COVID-19 publications, p < 0.001). Last, COVID-19 publications showed a higher number of authors, especially for case reports with a median of 9.0 authors (IQR: 6.0-13.0) in COVID-19 publications, compared to a median of 4.0 authors (IQR: 3.0-6.0) in non-COVID-19 publications (p < 0.001).

Page 3 of 10 Raynaud et al. BMC Medical Research Methodology (2021) 21:255 has been retrospectively registered, at https:// osf. io/ 9vtzp/.

We included all English-language publications with original data, comprising original articles, research letters (and corresponding synonyms) and case reports. Publications without original data (editorial, perspective, viewpoint, narrative reviews, etc.) were excluded.

All references were screened according to the titles, abstracts and full texts by 20 reviewers (MR, VG, KL, SA, QD, AT, RB, NS, AG, CD, ZD, MD, SN, EB, BR, AC, JD, SY, ECG, FT). The following data from each article were extracted: (1) basic information: journal, title, publication date, name of first author, (2) publication type, (3) number of authors (for consortia, we considered the total number of authors), (4) whether the publication was COVID-19 related or not, (5) whether the publication was based on case series, (6) number of patients for case reports and case series. Uncertainty in the categorization was resolved through a weekly discussion with all members. After the screening completion, three independent reviewers (MR, VG, KL) randomly checked 20% of the references for each reviewer -except theirs. If more than 3% of inconsistencies were observed for one given reviewer, a re-evaluation with re-adjudication was conducted for all the references of the reviewer.

We aimed at estimating the overall trend of the publication dynamics and the associations with the publications type and journals. To do so, we used a locally estimated scatterplot smoothing (LOESS) method using the smooth function in R. We represented the weekly number of publications over time, and we used the Wilcoxon test to compare the weekly number of publications between periods of time.

We aimed at investigating the publication type in COVID-19 studies and non-COVID-19 studies, which are characterized by case report, research letter, original article. The Chi 2 test was used to assess the difference in proportions of these publication types.

We aimed at investigating the number of authors and the associations with the type of publications. A Natural Language Processing algorithm was specifically designed to calculate the number of authors for each publication.

We aimed at estimating the overall trend of the number of authors dynamics and the associations with the publications type. To do so, we used the LOESS method described above. We represented the weekly-estimated, median number of authors for the number of authors dynamics. We further compared the difference in the median number of authors between COVID-19 and non-COVID-19 publications, stratified by publication type, with the Wilcoxon test.

We aimed at estimating the number of non-COVID-19 studies that could have been published during the pandemic period. First, using the LOESS method described above, we extrapolated the publication dynamics observed from January 1st 2019 to the start of the pandemic which was set on the 30th January 2020, following the official declaration of the World Health Organization [24] . Based on this trend, we then simulated the number of non-COVID-19 studies that could have been published, from 31st January 2020 to 31st December 2020, if the pandemic had not occurred. We then subtracted the number of non-COVID-19 studies that were actually published to obtain the final, simulated number of unpublished non-COVID-19 studies. All analyses were performed with Endnote (Endnote X9, Thomson Reuters), NoteExpress (Version 3.2, Beijing Aegean Software Co., Ltd.,) and R (version 3.2.1, R Foundation for Statistical Computing) software. Data are available upon reasonable request.

A total of 22,525 references were identified in the top ten medical journals, of which 3663 (16.3%) were COVID-19 related publications and 18,862 (83.7%) were non-COVID-19 related publications (Fig. 1 ). After removing duplicates and publications which did not include original data (editorial, perspective, viewpoint, narrative reviews, etc.), 6319 publications with original data remained for the final analyses. One thousand twenty-two were related to COVID-19 (16.2%), and 5297 (83.8%) were not.

In the year 2020, COVID-19 publications accounted for 1022 (25.9%) of the total number of publications while We present in the supplementary Fig. 1 the publication dynamics for all journals.

As shown in The Clinical Infectious Disease journal presented with a distinct pattern, and increased its production of non-COVID-19 publications in 2020, while starting to additionally publish COVID-19 studies: its median, weekly number of non-COVID-19 publications was of 12.0 (IQR 10.5-13.5), and 28.5 (IQR 24.0-33.0) in 2019 and 2020 respectively (p < 0.001).

Based on the publication dynamics in the time period from January 1st 2019 to January 30th 2020, we extrapolated what the publication dynamics of non-COVID-19 studies could have been from February 1st 2020 to December 31st 2020 (see methods), if the pandemic had not occurred (supplementary Fig. 2) . We removed the studies published in the Clinical Infectious Disease journal, given its very specific distribution that would bias the simulation.

Based on the simulation, we estimated that 1632 non-COVID-19 studies could have been published without the pandemic, in the nine selected journals, from February 1st 2020 to December 31st 2020. Since 1344 non-COVID-19 studies were published, this represents 288 unpublished non-COVID-19 studies, which thus corresponds to an estimated decrease of 18% in the production of non-COVID-19 research.

Among the 1022 COVID-19 publications, original articles, research letters and case reports accounted for 490 (47.9%), 424 (41.5%) and 108 (10.6%) publications respectively. Among the 5297 non-COVID-19 publications, original articles, research letters and case reports accounted for 3779 (71.3%), 800 (15.1%) and 718 (13.6%) publications respectively. (P < 0.001 for difference) (Fig. 3) .

The number of authors dynamics are presented in Fig. 4 . (Fig. 5D) .

To consider the influence that the number of patients in case report and case series may have on the number of authors, we calculated the ratio of the number of authors to the number of patients in case report and case series. In publications based on case report, the ratio was 6.0 (IQR: 3.0-10.0) and 4.0 (IQR: 3.0-5.0) for COVID-19 and non-COVID-19 publications respectively (p < 0.001) (supplementary Fig. 3A ). In original articles and research letters based on case series, the ratio was 0.5 (IQR: 0.2-2.0) and 0.7 (IQR: 0.2-3.2) for COVID-19 and non-COVID-19 publications respectively (p = 0.057) (supplementary Fig. 3B ).

In this meta-research gathering 22,525 publications in 10 major medical journals between 2019 and 2020, 6319 publications with original data (gathering original articles, research letters and case reports) were identified after reviewing. This study revealed the effects of the COVID-19 pandemic on the publication dynamics and the impact of COVID-19 research on non-COVID-19 research.

The exponential rise of COVID-19 publications began in February 2020, as the World Health Organization declared COVID-19 as a global public health emergency [24] . Only 4 months later, almost half of published medical research in top medical journals was dedicated to COVID-19, illustrating the commitment of editorial leadership to promote research related to the virus and provide time-sensitive data to scientifically address the problematics related to the pandemic [25, 26] . As a consequence, based on our simulation, there was a 18% decrease in the production of non-COVID-19 research, revealing to what extent the COVID-19 research has deeply impacted the non-COVID-19 research, a phenomenon highlighted and criticized by many researchers and key opinion leaders [6, 7, 9] .

Overall, although most journals have created a specific section dedicated to the COVID-19 research [27, 28] , they responded differently to the pandemic. Two main editorial strategies were identified based on the publication dynamics. First, we identified the journals that maintained the production of non-COVID-19 research while incorporating COVID-19 research. In these journals, as more published studies imply that editors worked on more studies, it results that they probably dedicate, in average, less time to conduct the editorial review. Therefore, it is possible that the scientific standards may have been, in some situations, considered with lower attention because of the need to provide timely scientific advances related to the pandemic. Second, we identified the journals that decreased the production of non-COVID-19 research while incorporating COVID-19 research, thus maintaining their overall production. In these journals, the editors rejected more papers than usual [29] , supporting adherence and commitment to high scientific standards [13] . The number of authors significantly differed between COVID-19 and non-COVID-19 studies in original articles and case reports, which corroborates with a recent study by Zdravkoric et al. that focused on the top three medical journals [15] . This difference was especially noteworthy in case reports. Interestingly, when considering publications based on case series, there was no significant difference.

In addition, we showed a change in the publication type, that was mainly driven by the high proportion of research letters at the expense of original articles in COVID-19 publications, a phenomenon that has been previously illustrated [15] and might reflects how willingness to provoke immediate impact and provide novel insights could possibly have affected the quality of the medical research worldwide [30] . Given the current rapid change and adaptation in the medical research and resources [11, 31] , such results would be of high interest for health researchers, public health officials and practitioners who are focused on controlling the pandemic while also sustaining the pace of non-COVID-19 research.

Nevertheless, these findings should be interpreted in the light of the efforts made around COVID-19, in particular the commitment around the estimation of COVID-19 cases and related deaths worldwide [1] , the implementation of massive, international collaborations [4] , the fast-tracking process of COVID-19 medical research [32] , the vaccine development [33, 34] , or the will to provide researchers with the most up to date information with, for instance, living systematic reviews on COVID-19 research [35] [36] [37] . Overall, innovations and discoveries have been brought and may help advancing medical research.

Although the worldwide population is progressively getting vaccinated, the COVID-19 pandemic still exerts a very harmful effect on many countries [38, 39] . In addition, the rise of many variants may challenge the efficacy of vaccines [40, 41] and delay the decrease in the number of cases and deaths. Accordingly, the medical research beyond COVID-19 is likely to be impacted in the long run. As such, we urge researchers to help continuing the evaluation on how health systems, medical research and resources are managed in pandemic time, as we attempted to accomplish in the present study.

Several limitations should be acknowledged. First, due to the very high number of studies, the references were not assessed by two independent reviewers. However, after the screening completion, three reviewers randomly checked 20% of the references for each reviewer, and a second screening was performed if more than 3% inconsistencies were observed. Second, for the same reason, we had to restrict the analyses on the highly cited medical journals only. This may have induced a selection bias, as the editorial strategies might be different in lower-impact journals. However high-impact journals likely reflect and drive the trends in publications and are therefore relevant examples to analyze the impact of the pandemic on medical research. Third, we investigated the impact of the COVID-19 pandemic on non-COVID-19 research by focusing on the publication dynamics, the publication type, and the phenomenon of author multiplicity. Critically appraising all studies would have been ideal and enhanced our demonstration; but this would have been a gigantic work that cannot be accomplished in such study design.

To conclude, in this meta-research gathering original articles, research letters and case reports published in high-impact medical journals, we have shown the heterogeneity in the publication dynamics, and measured the impact of the COVID-19 pandemic in the production of non-COVID-19 studies. This study revealed how medical journals adapted to the pandemic, as some maintained the production of non-COVID-19 studies, and some decreased the production of non-COVID-19 studies. Last, we have identified an author multiplicity phenomenon in COVID-19 studies.

COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). 2020

STIP COVID-19 WATCH

Global coalition to accelerate COVID-19 clinical research in resourcelimited settings

Covid-19 makes the future of UK clinical research uncertain

The future of cancer research after COVID-19 pandemic: recession? Fut Oncol

Covid-19 is shattering US cancer care

Organ procurement and transplantation during the COVID-19 pandemic

Impact of the COVID-19 pandemic on clinical research

COVID-19 and readjusting clinical trials

Dramatic drop in new cancer drug trials during the COVID-19 pandemic

Against pandemic research exceptionalism

Waste in covid-19 research

COVID-19-related medical research: a meta-research and critical appraisal

Scientific quality of COVID-19 and SARS CoV-2 publications in the highest impact medical journals during the early phase of the pandemic: a case control study

Clinical and virologic characteristics of the first 12 patients with coronavirus disease 2019 (COVID-19) in the United States

Coagulopathy and antiphospholipid antibodies in patients with Covid-19

The rising trend in authorship

Changing authorship patterns and publishing habits in the European journal of pediatric surgery: a 10-year analysis

A comprehensive analysis of authorship in radiology journals

Authorship in radiation oncology: proliferation trends over 30 years

Effects of research complexity and competition on the incidence and growth of coauthorship in biomedicine

The impact of research collaboration on scientific productivity

COVID-19 Public Health Emergency of International Concern (PHEIC) Global research and innovation forum

Defining the epidemiology of Covid-19 -studies needed

A planetary health perspective on COVID-19: a call for papers

The COVID-19 pandemic and the JAMA network

Coronavirus disease 2019: the harms of exaggerated information and non-evidence-based measures

Research funding after COVID-19

Scope, quality, and inclusivity of clinical guidelines produced early in the covid-19 pandemic: rapid review

Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK

Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine

Drug treatments for covid-19: living systematic review and network meta-analysis

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal

The COVID-NMA initiative -A living mapping and living systematic review of Covid-19 trials

Why is India having a covid-19 surge?

No end in sight for the Brazilian COVID-19 crisis

Experts discuss COVID-19-variants and vaccine efficacy, immunosuppressed patients, and more

Will SARS-CoV-2 variants of concern affect the promise of vaccines?

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Not applicable.

The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s12874-021-01404-9. 

This work was supported by INSERM-Action thématique incitative sur programme Avenir (ATIP-Avenir) and the Fondation Bettencourt Schueller. Both funding was used for the data collection, analysis and interpretation.

The data of the manuscript are available upon reasonable request. To obtain the data, please contact the corresponding author.

Ethics approval and consent to participate Not applicable.

Not applicable.

The authors declare that they have no competing interests.