key: cord-0860642-b6cg0qql
authors: Nye, Jonathan; D'Souza, M. Patricia; Hu, Dale; Ghosh, Dolan
title: Research productivity and collaboration of the NIH-funded HIV vaccine trials network: A bibliometric analysis
date: 2021-01-22
journal: Heliyon
DOI: 10.1016/j.heliyon.2021.e06005
sha: 6ce047a8b185ad4412f68fb998315e6c4ea61906
doc_id: 860642
cord_uid: b6cg0qql

The HIV Vaccine Trials Network (HVTN) is the world's largest publicly funded, multi-disciplinary international collaboration facilitating the development of vaccines to prevent HIV/AIDS and has conducted the vast majority of HIV/AIDS clinical trials since its inception in 1999. Although scientific findings from the program have been published in scholarly journals, the impact of a large scientific research network such as the HVTN on the HIV/AIDS vaccine field has not been assessed. This paper describes and elucidates the productivity, influence, and collaboration among HVTN researchers over the last two decades. Our analyses indicate that the HVTN has funded a large number of HIV/AIDS vaccine safety and efficacy clinical trials through a strong global network of clinical sites. In addition, several metrics indicate HVTN researchers also published original research articles that are influential in the HIV vaccine field. Scientific research collaboration is critically important in a complex and multidisciplinary field such as HIV vaccine development as it allows improved sharing of knowledge and expertise as well as the pooling of resources and data. We found that collaboration in the HIV vaccine field increased during this time period and collaboration among HVTN authors increased even more. Combining these productivity, influence, and collaboration metrics with research outcomes can provide a comprehensive assessment of large complex programs such as the HVTN.

Human immunodeficiency virus (HIV) and acquired immune deficiency syndrome (AIDS) remain public health concerns with an estimated global prevalence of 36.9 million HIV-infected persons worldwide and 1.8 million new infections per year [1] . Remarkable progress in treating HIV/AIDS has been made after almost four decades of active research since the first cases of AIDS were reported. Prevention and treatment have dramatically improved as a result of increased testing and treatment with anti-retroviral therapy (ART) [2] . However, there are still no licensed vaccines to prevent HIV infection, even though a vaccine will likely be essential to achieve a long-lasting end to the global pandemic [3] .

Several HIV vaccine efficacy trials were conducted between 2004 and 2009. One of these trials, known as RV144, resulted in the first vaccine regimen to exhibit a protective effect, suggesting that an effective vaccine might be achievable [4] . Since then, researchers around the world have worked to build on these findings in hopes of developing a more effective and durable immune response capable of preventing HIV infection. The National Institutes of Health (NIH) funds the large majority of research on HIV/AIDS vaccines in the world. Indeed, in 2018, 85% of funding for HIV vaccine research worldwide was contributed by only two major funders-the NIH and the Bill and Melinda Gates Foundation [5] . Within the NIH, one institute in particular-the National Institute of Allergy and Infectious Diseases (NIAID), through its Division of AIDS (DAIDS)-has led the effort to develop a safe and effective vaccine. DAIDS has supported a robust body of HIV vaccine-related research from preclinical and translational research to clinical trials. In addition, it has established and supported several large networks dedicated to conducting HIV/AIDS clinical trials both within the United States and globally.

Since it was established in 1999, one of these networks-the NIAIDsupported HIV Vaccine Trials Network (HVTN)-has conducted the majority of clinical trials of preventive HIV vaccines worldwide [6] . The HVTN comprises an international group of scientists, educators, and community members whose mission is to support the development of a safe and effective vaccine for prevention of HIV infections. It conducts all phases of clinical trials, from testing safety and immunogenicity of vaccine candidates to evaluating vaccine efficacy. It is made up of three parts: the Laboratory Center, the Statistical and Data Management Center, and the Leadership and Operations Center [7] . All three of these centers work closely with the clinical research and trial sites. As the federal government funder of non-governmental networks like the HVTN, DAIDS plays a major collaborative role in scientific and protocol development, trial and safety monitoring, and laboratory and other support, in addition to serving as the regulatory sponsor. The HVTN's trial sites are located at research institutions around the world while the vaccine products come from various developers including both for-profit and academic investigators. This structure allows the HVTN to streamline HIV vaccine testing and to reach populations severely impacted by the HIV/AIDS epidemic in both the US and abroad [8] .

Although the HVTN is one of the largest and oldest HIV research programs, its productivity and impact has not been well-documented in the literature. While previous studies have examined research outputs [9] , expansion of subject areas [10] , and collaborations [11] , they have been relatively limited in scope in terms of geographic region or time [12, 13, 14, 15] . Moreover, despite the growing importance of scientific collaborations [16] , studies examining collaborations within HIV clinical trials networks have been limited to research conducted during only a few years. For example, in one study, a high degree of coauthorship and interdisciplinarity has been found in papers produced by NIAID HIV extramural clinical trials networks, but the sample included publications in only three years, 2006-2008 [11] . Previous work has outlined the scientific achievements over the first decade of the HVTN, but a bibliometric analysis of the program has yet to be done [7] .

Our study seeks to build on previous work by providing a comprehensive bibliometric analysis of the HVTN from 1999-2019. We include an overview of the international network of clinical trial sites utilized by the program and an in-depth examination of research outputs such as clinical trials and number of publications in combination with advanced field normalized metrics to assess the influence of this work. We also show how collaboration has evolved in the HIV vaccine field as a whole as well as among HVTN investigators. Our primary hypotheses are that HVTN investigators produced more publications, their publications had greater influence, and they collaborated to a greater degree than other investigators in the vaccine field. Together, this work provides an overview of the productivity and influence of the HVTN since it was first established 20 years ago.

Both publicly available and internal NIH databases were used to gather data for the study. Each of these databases is described below, and a summary is presented in Table 1 . All analyses and visualization were carried out using the R programming language. While there are more established bibliometric tools available, new tools continue to be added to the R software environment including those for bibliometric analyses. R offers certain advantages over established tools, such as a broad array of statistical algorithms and mathematical functionality in addition to visualization capabilities [17] . R is more flexible than some of the established tools in the data formats accepted and ability to integrate into a workflow that involves non-bibliometric analyses of the same dataset. However, this flexibility comes at a cost-a steep learning curve for non-programmers. But, this is being addressed with related R packages such as biblioshiny, an interface to bibliometrix for non-coders [18] .

A comprehensive list of clinical trials that were attributed to the HVTN was obtained from the ClinicalTrials.gov database [19] through 2019 by searching for trials with keyword "HVTN." From this list we kept only trials with an HVTN/identifier listed in the Other Study ID or the Sponsor Collaborators fields. We further selected trials which listed "HIV" as one of the Conditions and only selected Interventional trials using the Study Type identifier field. This resulted in a final list of 79 clinical trials which was vetted by NIAID program staff to ensure that they were associated with the HVTN. The list of these trials will be available at 10.6084/m9.figshare.12962762.

Mapping of global clinical trial sites and HIV prevalence was done using the ggplot2 package [20] . Of the 79 HVTN clinical trials, 77 had location information associated with them. Data on HIV prevalence among people ages 15-49 in 2017 was obtained from the World Health Organization [21] . Counties and states in the US comprising more than 50% of new HIV infections were obtained from the Ending the HIV Epidemic initiative [22] .

We also provide information on the geographical distribution of HVTN-supported publication authors. This distribution was based on the location of authors' institutional affiliations appearing in the papers and extracted from the Web of Science database. Authors' countries were manually entered for 14 papers that did not appear in the Web of Science collection.

We searched for all publications that acknowledged HVTN grant funding using the iSearch platform. iSearch is a suite of tools available to NIH staff that, through a single interface, provides access to a comprehensive, curated, extensively linked data set of global grants, patents, publications, clinical trials, and FDA-approved drugs. The iSearch Publications tool utilizes the National Library of Medicine PubMed database and adds to it records from the NIH Scientific Publication Information Retrieval and Evaluation System (SPIRES). The SPIRES database contains verifiable mappings between scientific publications and NIH grant numbers. NIH's publicly available iCite tool [23] was used to distinguish research articles from derivative or non-research articles. The iCite article type classification is based on the PubMed "Publication Type" tag. iCite also provides article-level citation data and the Relative Citation Ratio (RCR) [23] for each publication. The RCR quantifies the influence of a research article by using its co-citation network to field-normalize the number of citations it has received. The RCR of a reference article (RA) is expressed as the ratio of the yearly average number of citations the RA has received to an expected number of citations based on a "field citation rate"-the yearly average number of citations to articles published in the same year as the RA in journals in which the RA's co-cited articles appear. The expected number of citations is determined from a reference set of articles-those supported by R01 research grants from the NIH.

The data gathered from NIH's internal iSearch database can be approximated using publicly available data sources. A public version of SPIRES that includes the PMIDs of publications linked to HVTN grant numbers is available through the link tables on the NIH ExPORTER website [24] . From these PMIDs, the publicly available iCite tool can be used to obtain RCR values for these articles. The PMIDs of articles used in this study is available at 10.6084/m9.figshare.12964919.

In all, there were 465 publications citing HVTN grant support; 416 (89.5%) were original research articles, 34 (7.3%) were reviews, and 15 (3.2%) were other types of articles. We decided to focus specifically on the 416 original research articles when assessing the performance and influence of the HVTN because these best reflect the network's scientific contributions. The HVTN publications found in the SPIRES system include only those publications that acknowledge support from NIAID funding in the paper or have been linked to a grant by grantees using the My NCBI system [25] . Not all HVTN-supported publications contain such citations or have had these linkages created.

iSearch was also used to identify a larger set of publications encompassing the HIV vaccine field, using the following search terms applied to publication titles and abstracts: (HIV* AND VACCIN*), (AIDS AND VACCIN*), (Antibodies AND Neutralizing AND (HIV* OR AIDS)). This resulted in 16,643 publications published between 2000 and 2019. From this dataset, 12,426 were identified as research articles using the iCite article type classification described above. In addition, to enrich for articles that were specific to the HIV vaccine field, we excluded articles where HIV/AIDS was not the primary focus of the article. Therefore, we removed publications that contained the following keywords or parts of keywords in the title: tuberculosis, hepatitis, influenza, papilloma, pneumococc, meningococc, herpes, streptococc, HBV, HCV, or yellow fever. We further checked this dataset and eliminated articles that did not contain HIV/AIDS/Acquired Immunodeficiency Syndrome as a MeSH term. This left us with a final list of 10,462 HIV vaccine research articles including 281 acknowledging HVTN funding. All PMIDs are available at 10.6084/m9.figshare.12964907.

From the HIV vaccine publication dataset, we created undirected and unweighted coauthorship networks using the igraph [26] and ggplot2 [20] packages in R. We built two coauthor networks, one spanning the years 2000-2009 and another 2010-2019 to capture how collaboration in the field has changed over time. The network layouts were generated using the Kamada-Kawai force-directed algorithm [27] . Authors publishing under different name variations is a challenge in creating coauthor networks, so to ensure the quality of our results we used a simple script to aid in author disambiguation. By calculating the number of name variations associated with each specific last name first initial we were able to separate each author into categories of high or low confidence based on this value. For authors with last name, first name and middle name or initial, author names were collapsed based on last name, first and middle initials. For authors with only last name and first name or initial, names were collapsed based on last name and first initial. Low confidence names (e.g. Smith, J) with many different name variations associated with it (e.g. Smith, John; Smith, James; Smith, Jack) were manually reviewed and corrected while high confidence names, those that were unique or had few name variants were left unchanged. Only the 150 most prolific authors in each time period were used in our networks. This allowed us to see how collaboration among the most prolific investigators in the HIV vaccine field evolved while avoiding overly dense and crowded networks. Furthermore, we identified all investigators on publications acknowledging HVTN grant funding and highlighted nodes representing these HVTN-associated investigators and the edges connecting them.

Of the 79 trials funded by the HVTN through 2019, 61 were Phase I, 6 were Phase I/Phase II, 10 were Phase II, 1 was Phase II/Phase III, and 1 was a Phase III trial (Figure 1 ). In total, over 26,000 participants were enrolled in the 79 trials over this time period, the largest portion coming from the large proof-of-concept and efficacy trials which enrolled 18,658 participants. The smaller Phase I and II trials testing safety and immunogenicity enrolled 7,978 participants. Table 2 shows a list of top 10 journals in which these articles appeared, and Table 3 shows the most frequent author locations (countries) based on their organizational affiliations. Our results are consistent with previous observations on the distribution of HIV/AIDS research outside the U.S. [29, 30] ; several lowand middle-income countries appear in Table 3 , led by South Africa. In addition to productivity, we wanted to gauge the performance and influence of these publications using citation-based metrics, including multiple field normalized indicators (Table 4 ). We found that HVTN research articles had been cited a total of 12,521 times, with a median number of citations per paper of 10 and a mean of 30.1 (range 0-1088). In addition, 99.1% were cited at least once after 5 years compared to 88% of all articles in the iSearch database. In order to determine how these publications performed relative to their field of study, we used two separate metrics. The first is the RCR, a field normalized metric that measures a research article's influence [23] . We found that HVTN publications had a mean RCR of 1.8 (range 0-47.4) meaning that, on average, they were cited 1.8 times more often than publications in their co-citation network.

In addition, we used the InCites database, which organizes research articles by publication year and subject area based on journal category [31] , to analyze the percentile rank of 399 HVTN publications found in their collection. We found, after normalizing for time and subject area, 22.1% of the HVTN publications are among the top decile of the citation distribution, indicating that these publications were represented more than twice as highly as expected among the most-cited publications in their respective fields. Moreover, we found that 5% of the HVTN publications were among the 1% most-cited papers, meaning that HVTN-supported papers appeared among these most highly cited papers 5 times as often as expected. Their representation in this highly cited group also was greater than that of all NIH-supported papers, of which 2.8% have been shown to be in the top 1% of the citation distribution [32] . 1-56) . When we compared the number of authors on HVTN publications to non-HVTN publications, we found no differences in the first time period, but HVTN publications had a significantly higher (Welch's two sample t test, p-value < 0.001) number of authors per paper during the second time period, suggesting that collaboration among HVTN authors increased to a greater extent when compared to all authors in the HIV vaccine field.

As an additional measure of collaboration, we created two coauthor networks; one for each of the two different time periods (Figure 4 ). Such networks have been shown to be a very useful tool for the analysis of collaboration within a field [33, 34] . We restricted the networks to the 150 most prolific authors in each time period. These authors were responsible for more than a third of the publications and they had the most coauthors. Author names were disambiguated as described in the Methods section. Each node in the networks represents an author while a connection (edge) between these nodes indicates coauthorship. Edges connecting authors on HVTN publications have been highlighted.

Next, we calculated the degree of each investigator, which refers to the number of authors an individual has published with and is equal to the number of edges in the network connected to that person. The network analysis revealed that collaboration increased over time in the HIV vaccine field, with the average degree rising from 19.4 in 2000-2009 to 63.0 in 2010-2019. Among only the HVTN-associated authors, we found a similar trend in which the mean degree increased from 27.6 to 66.0. This increase was significantly greater than for non-HVTN authors (Welch's two sample t test, p-value < 0.01), whose mean of 17.5 in 2000-2009 increased to a mean of 52.1 in 2010-2019. Finally, our analysis revealed that the number of the HVTN-associated investigators represented in the networks more than tripled over these two time periods, increasing from 28 to 116 individuals.

The development of a safe and effective HIV vaccine is entering a very exciting phase with four efficacy trials underway, more than at any other time in the history of HIV vaccine development. These developments represent the culmination of many years of preclinical research and clinical trials, with most of this research funded by NIH, making this a perfect time for assessing the HVTN program. Clinical sites are an essential component of the network, so the HVTN must support a robust global network capable of handling a large number of clinical trials. Indeed, we found that the HVTN supported clinical sites in 23 countries and 94 cities worldwide. Many of these countries have been the hardest hit by the AIDS epidemic including South Africa, Zimbabwe, and Botswana among others. Our studies indicate that the HVTN is furthering its program goals of reaching populations severely impacted by the HIV/ AIDS epidemic in both the US and abroad [8] .

In addition to carrying out a large number of vaccine clinical trials, the HVTN has increased productivity over time, publishing more in the most recent ten-year period. This increase in publications may be due in part to new insights gained from trials conducted by the HVTN and the RV144 trial in which the first partially effective HIV vaccine was tested. The HVTN also expanded to different regions in the world which might have enabled additional capacity and output from the network. Furthermore, we found that HVTN publications were high impact/influence as shown by multiple field-normalized citation metrics including RCR and the percentage of publications in the top 1% or 10% in their respective fields of study. Many of these articles summarize the major accomplishments throughout the life of the network, such as the development and analyses of numerous new vaccine approaches, products, and adjuvants [35, 36, 37, 38, 39, 40, 41, 42] . Additionally, our analyses revealed that the number of HVTN-associated PIs among the 150 most prolific authors in the field from 2010-2019 more than tripled when compared to the previous 10-year period.

Previous research found a relatively high degree of coauthorship among papers produced by NIAID HIV clinical trial networks, but the short time window studied precluded analyses for trends [11] . By examining papers published over a period of more than 20 years, our analysis of research articles in the HIV vaccine field revealed that collaboration increased significantly during the assessed time period as indicated by an increase in the mean number of authors per publication. Moreover, this increase was higher for HVTN-associated investigators. Our coauthor network analysis of the 150 most prolific authors showed that collaboration among all authors increased substantially from 2000 to 2019 as indicated by the tripling of the average degree. In addition, we found that HVTN-associated investigators had a significantly higher degree of collaboration compared to non-HVTN investigators. This difference was likely driven in part by increased publication frequency but also by larger team sizes. Those with in-depth knowledge of the program believe that the HVTN's unique structure creates an environment that fosters collaborations to stimulate interdisciplinary clinical research. Anecdotal information such as this could be evaluated with more detailed information on authors' disciplinary backgrounds, but such information is currently difficult and expensive to obtain.

Scientific research collaboration is critically important in a complex and multidisciplinary field such as HIV vaccine development as it allows improved sharing of knowledge and expertise as well as the pooling of resources and data. Increasingly sophisticated technologies and the massive amount of data that is being generated means that more and more researchers must specialize and focus their resources. In turn, increasing specialization of research scientists means that successful research requires increasingly larger, multidisciplinary collaborations and sharing of knowledge. This trend has been documented across many disciplines including science and engineering, and it is certainly true for a field as specialized as HIV vaccine development [16] . Therefore, HVTN's focus on data sharing and collaboration may help researchers to capitalize on the knowledge gained from its different teams to carry out multidimensional analyses. The utility of these collaborations is illustrated by the HTVN's ability to pivot its platform to help develop vaccine candidates against SARS CoV2, the virus causing the emergence of the current COVID-19 pandemic [43] .

Metrics based on the larger HIV research field and its investigators provided relevant benchmarks against which to compare outcomes of the HVTN. However, this study is limited in ways common to any observational study in that strong causal inference is more difficult than it is in experimental studies. Gathering similar data from other formal research networks, along with information on how these networks systematically differ from the HVTN, might aid in interpreting the HVTN outcomes and strengthen conclusions surrounding the HVTN's impact.

Beyond the productivity, influence, and collaboration measured in this study, the NIH values work that culminates in advances to human health, a process that historically takes decades. Metrics have quantified the diffusion of knowledge from basic research toward human health studies by considering the type, rather than merely the number, of citing articles. Insights into how to accelerate this process may come from such quantitative analysis [44] . Comprehensive evaluation programs will need to incorporate additional metrics that can capture outcomes such as the value of innovation, clinical outcomes, novel vaccine platforms, research enabling vulnerable populations, global collaborations, and training the next generation of scientists.

Jonathan Nye: Conceived and designed the analysis; Analyzed and interpreted the data; Wrote the paper.

M. Patricia D'Souza, Dale Hu, Dolan Ghosh: Conceived and designed the analysis; Wrote the paper.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. 

The authors declare no conflict of interest.

No additional information is available for this paper. 

Global AIDS Update 2019 -Communities at the centre

Antiretroviral therapy for the prevention of HIV-1 transmission

An HIV vaccine is essential for ending the HIV/AIDS pandemic

Vaccination with ALVAC and AIDSVAX to prevent HIV-1 infection in Thailand

The complex challenges of HIV vaccine development require renewed and expanded global commitment

HIV Vaccine Trials Network, HVTN mission

HIV Vaccine Trials Network: activities and achievements of the first decade and beyond

National Institutes of Health, NIH Guide to Grants and Contracts, Notice RFA AI-05-001

Bibliometric analysis of aids literature

The literature of acquired-immunodeficiencysyndrome (AIDS) -continuing changes in publication patterns and subject access

Evaluating research and impact: a bibliometric analysis of research by the NIH/NIAID HIV/ AIDS clinical trials networks

Bibliometric analysis of AIDS literature in central Africa

A comparative study of the literature on HIV/AIDS in Kenya and Uganda: a bibliometric study

HIV/AIDS in Nigeria: a bibliometric analysis

HIV/AIDS research in India: a bibliometric study

The increasing dominance of teams in production of knowledge

Bibliometric analysis using Bibliometrix an R package

Biblioshiny: the Shiny Interface for Bibliometrix, 2020. Available from

Elegant Graphics for Data Analysis

Ending the HIV Epidemic: about Ending the HIV Epidemic: Plan for America

Relative citation ratio (RCR): a new metric that uses citation rates to measure influence at the article level

ExPORTER Data Catalog

electronic Research Administration

An algorithm for drawing general undirected graphs

MMWR Morb Mortal Wkly Rep ; HIV Testing in 50 Local Jurisdictions Accounting for the Majority of New HIV Diagnoses and Seven States with Disproportionate Occurrence of HIV in Rural Areas

HIV/ AIDS research in Africa and the Middle East: participation and equity in North-South collaborations and relationships

Equity in global health research in the new millennium: trends in first-authorship for randomized controlled trials among lowand middle-income country researchers 1990-2013

InCites: an Objective Analysis of People, Programs and Peers

Publication Impact of NIH-Funded Research -A First Look

Co-authorship network analysis: a powerful tool for strategic planning of research, development and capacity building programs on neglected diseases

International scientific collaboration in HIV and HPV: a network analysis

Subtype C ALVAC-HIV and bivalent subtype C gp120/MF59 HIV-1 vaccine in lowrisk, HIV-uninfected

Phase 1 HIV vaccine trial to evaluate the safety and immunogenicity of HIV subtype C DNA and MF59-adjuvanted subtype C Env protein

Research collaboration in universities and academic entrepreneurship: the-state-of-the-art

The safety and immunogenicity of an interleukin-12-enhanced multiantigen DNA vaccine delivered by electroporation for the treatment of HIV-1 infection

Broadly neutralizing antibodies for HIV prevention

Prophylactic HIV vaccine: vaccine regimens in clinical trials and potential challenges

Pox-Protein Public Private Partnership program and upcoming HIV vaccine efficacy trials

Landscapes of binding antibody and T-cell responses to pox-protein HIV vaccines in Thais and South Africans

HIV Vaccine Trials Network, Skilled Partners in the Conduct of HIV Prevention Trials Join COVID-19 Response

Predicting translational progress in biomedical research

We would like to thank Mary Marovich, Ariel Zane, Marie Parker and James Onken for their helpful guidance and feedback.