Using Evidence in Practice
Using Bibliometrics to Illustrate the Impact of the Library on Military Medical Research
Holly Rose
Beverley
Medical
Librarian
Cadence Group
contract at the Walter Reed Army Institute of Research
Silver
Spring, Maryland, United States of America
Email: holly.r.beverley.ctr@health.mil
Maureen
Humphrey-Shelton
Command
Librarian
United States
Army Medical Research and Development Command and
United States
Army Medical Research Institute of Infectious Diseases
Frederick,
Maryland, United States of America
Email: maureen.p.humphrey-shelton.civ@health.mil
Elisia George
Medical
Librarian
United States
Army Medical Research Institute of Chemical Defense
Aberdeen
Proving Ground, Maryland, United States of America
Email: elisia.a.george.civ@health.mil
Olivia Briere
Librarian
Cadence Group
contract at the Department of Justice
Washington, D.C.,
United States of America
Email: okbriere1207@gmail.com
Received: 9 Mar.
2023 Accepted: 17 Apr.
2023
2023 Beverley, Humphrey-Shelton, George, and Briere. This is an Open Access article
distributed under the terms of the Creative
Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work is
properly attributed, not used for commercial purposes, and, if transformed, the
resulting work is redistributed under the same or similar license to this one.
Data Availability: Data associated with this article
cannot be made publicly available because they are proprietary information to
the United States Army.
DOI: 10.18438/eblip30345
Setting
In this
article, the authors describe how bibliometrics were used as a novel method of
demonstrating use and impact of specialized biomedical library collections. By
exploring an approach used previously for collection development, the authors
were able to illustrate library impact on military medical research beyond the
traditional approach of return-on-investment investigations and statistical
presentations.
The U.S. Army
Medical Research and Development Command (MRDC) libraries consist of the
headquarters virtual library and five physical subcommand libraries, all
staffed by master’s degree-holding librarians. Information professionals in the
MRDC libraries consortia are often solo librarians or small library teams who
assist highly experienced researchers with electronic access and literature
searching. MRDC libraries are present throughout the research workflow
providing effective virtual libraries and end-user training.
MRDC
libraries support basic science and translational research. The MRDC pursues
medical research and testing that pertains to human treatments, like
neurological effects of exposure and how illnesses or trauma manifest. Animal
and human research conducted at our organizations frequently translates into
military products and devices, as well as treatments for military and
civilians. MRDC libraries facilitate the research of illnesses and their
causation while also focusing on global military health. The focus on military
health encompasses both physical and psychological causes and prevention.
Problem
The MRDC
Library Program is a multifaceted consortium overseen by a command librarian in
the headquarters library and five subcommand libraries located at the United
States Army Medical Research Institute of Chemical Defense (USAMRICD), Walter
Reed Army Institute of Research (WRAIR), United States Army Medical Research
Institute of Infectious Diseases (USAMRIID), United States Army Institute of
Surgical Research (USAISR), and the United States Army Aeromedical Research
Laboratory (USAARL). Besides providing an onsite location for usage, each
subcommand library offers additional resources and services specifically
tailored to their users. The goal of the library program is to keep the service
member at the forefront of our collections, services, and trainings to
contribute to the MRDC mission of ensuring that the U.S. Armed Forces are
equipped to protect themselves from disease and injury.
Historically,
MRDC was a subordinate organization under the U.S. Army Medical Command and had
access to the Army Medical Department (AMEDD) virtual library resources (Figure
1) as well as maintaining their local physical resources. Recently, the Defense
Health Agency (DHA) was established to bring together all the medical
departments of all military branches under one umbrella to consolidate
resources when possible. As MRDC transitions to DHA, those clinical library
resources are now accessible through the MRDC virtual library. However, as MRDC
librarians know from first-hand experience, DHA clinical library resources
alone would not be sufficient for MRDC researchers because of differing usage
needs from their clinical colleagues.
DHA clinical
library resources are designed to meet the needs of medical professionals
working in military treatment facilities (e.g., clinicians, nurses, physician
assistants, and other clinical staff directly responsible for patient care). It
is more likely that clinical library users would need evidence-based treatment
summaries, patient care guidelines, anatomy or procedure textbooks, clinical
decision support, manuals or electronic tools to check
for contraindications or drug interactions. Maintaining access to older years
of scientific literature coverage is rarely considered best practice in
clinical libraries. This is especially true if clinical information services
are implemented in a point-of-care setting where clinicians only need access to
the most recent research as it relates to patient treatment. Information from
the last five years is often sufficient for these purposes.
Military
medical research library collections support scientific staff in the discovery
and testing of novel therapeutics, preventative measures, or treatments for
diseases and conditions that affect humans. The MRDC libraries provide access
to the full body of knowledge within a given discipline, in applicable subject
areas since such content might contribute to growth in the field of military
medical research. Some of these subject areas encompass infectious diseases,
combat casualty care, military operational medicine, chemical defense, clinical
and rehabilitative medicine, and entomology, among others because of their
prevalence in the field (U.S. Army Medical Research and Development Command,
n.d.). Because the researchers frequently read and cite content from older
scientific journal literature, the MRDC librarians also purchase backfiles to
maintain journal access to many years of coverage. Lercher and Smolinsky (2016)
extrapolate, “if the use of older scientific literature is increasing, then one
might argue that the value of archives of such literature is likewise
increasing” (pp. 1219–1220). Analysis of the increased citations and usage
trends for older scientific literature supports the value of more extensive
archives in research libraries, though the frequency of citing older articles
can vary by discipline as can the definition of “older” articles (Verstak et
al., 2014).
Though the
librarians maintain an ongoing awareness of Army medical researchers’ needs and
advocate for the resources that these specialized library users require, new
leadership or new funding agencies would also benefit from this information
especially since DHA is largely composed of clinical military treatment
facilities, whereas MRDC is composed of military medical research facilities.
As a result, this project emerged to simultaneously advocate for the resources
needed by MRDC researchers and staff while illustrating the differing subject
expertise among researchers supported by the MRDC Library Program in
preparation for the transition to DHA.
Evidence
Federal
medical library surveys are conducted regularly by clinical libraries, but
using a survey as the research instrument was not sufficient due to the
clinical library-oriented questions in the survey instrument and the lack of
statistical proof of library impact desired by the MRDC librarians. Though MRDC
was invited to participate in a federal library survey initiated by clinical
libraries, the points of data collection did not align with the mission and
services provided by the medical research libraries. A working group consisting
of four of the MRDC librarians developed questions for a survey in an earlier
phase of this project, but ultimately chose not to use it.
Several
different approaches to demonstrating value were explored through a literature
review. Literature regarding “return on investment” in library settings uses
different conceptual models like social value analysis, value co-creation,
perceived value, long term impact of library service use, and economic impact
(Urquhart, 2020). Literature on topics surrounding library value and approaches
to impact studies often come from academic or clinical settings (De Groote et
al., 2020; De Groote & Scoulas, 2022). Additionally, literature which
addresses “value studies” focuses heavily on financial data and justifying
ever-increasing costs of operation rather than focusing on deliverables and the
greater impact of library services. None of these options were found to be
suitable for this project.
Very limited
research was discovered on the topic of using bibliometric data in federal
research libraries. Though identifying literature that would illuminate an
applicable strategy for MRDC libraries proved challenging, Belter and Kaske’s
2016 article in College & Research Libraries provided an example of
how citation metadata could be used in federal research libraries for
collection development. After reviewing the article, the working group
determined that using the citation metadata of research publications authored
by MRDC researchers could also be used to investigate whether MRDC or DHA
provided the most access to cited articles in MRDC researchers’ publications.
Cited references would provide a compelling foundation for why specific
resources are required and demonstrate the library’s impactful role in the
research process.
Though Belter
and Kaske used Web of Science, the MRDC librarians used Scopus for their
analysis. Scopus is an abstract and citation database that covers thousands of
book and journal titles. The majority of the journals
are peer-reviewed and cover such fields as life sciences, social sciences,
physical sciences, and health sciences. Scopus also enables affiliation-based
searching, which helps identify publications within a database that were
written by individuals employed by or associated with a particular agency. From
professional experience, the MRDC librarians knew that Scopus content coverage
was more comprehensive and captured more research authored by MRDC affiliated
researchers than Web of Science.
After
determining how to proceed, the working group utilized the bibliometric tools
in Scopus to analyze the citation metadata of all publications written by MRDC
researchers from 2017–2021. All MRDC locations were included in the analysis of
3,631 total scientific journal articles (Figure 1). The librarians analyzed
130,256 cited references used to create the published articles. MRDC libraries
currently receive subscription resources funded by DHA, MRDC, and subcommand
libraries. Free or open access resources and interlibrary loan (ILL) services
were evaluated as well. Further analysis determined the library that most
likely provided the cited source. Coverage of cited references from the
articles retrieved in the affiliation searches are presented in Figures 2 and
3, using one subcommand’s affiliation search as an example. These data were
compared to the MRDC virtual library and subcommand library holdings.
Figure 1
Coverage of all citations from MRDC researcher-authored publications
between 2017–2021.
Figure 2
Example of affiliation search for one of the five subcommand libraries
(WRAIR) evaluated in this study: What was published by WRAIR-affiliated authors
between 2017–2021?
Figure 3
Example of publications' cited references for one of the five
subcommand libraries evaluated in this study: What was cited by WRAIR-affiliated
authors in their written publications between 2017–2021?
To identify
the library that provided each cited source, librarians evaluated their
holdings for each cited reference listed in MRDC-affiliated publications to
determine the level of access provided. For every resource considered, current
coverage was required at a minimum, and the archival coverage was also
evaluated. In the instance that a resource was covered by MRDC and DHA, credit
was given to the organization that provided the most extensive full text access
and the most extensive archive (Figure 4). For example, each author went
through the list of individual journal titles with articles that were cited by
MRDC researchers between 2017–2021 and searched their online library catalog to
confirm whether the MRDC command or their subcommand library provided current
access to each journal’s content. If a journal could be found in both a
DHA-provided resource and an MRDC-provided resource, and current access was
provided by both, the distinguishing criteria would be the extensiveness of the
archive or the years of coverage that were provided for that journal within
that resource (e.g., ClinicalKey or ScienceDirect). So, when comparing the
level of access provided for a particular journal in the DHA-provided resource
ClinicalKey versus the MRDC-consortium provided ScienceDirect, both DHA and
MRDC had current coverage for the journal. However, MRDC was given credit
because that agency had a more extensive archive for that journal.
ILL
statistics were also incorporated to account for gaps in coverage and to highlight
the MRDC subcommand libraries’ contributions to connecting researchers with
resources. ILL services facilitate unique literature requests from researchers
and scientists to enable the work they do advancing their field of research and
creating deliverables for military service members. Though outside libraries
were used to fill ILL requests during this timeframe, most ILL requests were
filled by other research libraries within the MRDC consortia, thus reinforcing
the fact that even if one of our libraries does not have access to a particular
resource, local subscriptions throughout the consortia can often facilitate the
request.
Figure 4
Example of evaluating one subcommand library’s holdings: Which
organization provided access (MRDC, DHA, Free = open access, or blank if
covered by MRDC and DHA without a difference in archival coverage) to journals
cited by researchers at this subcommand?
Implementation
This project
determined that the resource collections put together by the MRDC library program
and subcommand libraries are extremely comprehensive and effective at
fulfilling the needs of researchers. Analysis of citation metadata can help
justify continued funding for research-specific resources as libraries continue
to partner with military medical researchers to fulfill the Department of
Defense (DoD) needs for experimental research. Loss of resources will greatly
impact the ability of military medical researchers to obtain research needed to
further the missions of the DoD and could have potentially grave impacts for
warfighters down the road.
These data
have been presented in annual reports and multiple presentations discussing the
importance of libraries to headquarters leadership.
Outcome
The majority of the topics covered in MRDC-authored publications are not clinical in
nature and are clearly bench or experimental research; therefore,
clinical-leaning resources would not satisfy the needs of the MRDC researchers.
Each location determined that most of the references needed to generate their
research were available through MRDC-provided subscriptions, either by the
command-wide subscriptions or through locally subscribed resources. Further
determination found that ILL provided by the subcommand libraries (Figure 1)
was also used a small percentage of the time.
PowerPoint
presentation slides were used to highlight the purchasing agency with the most
expansive resource coverage at each MRDC institution based on citations from
the last five years. The presentation was shared with MRDC leadership and local
libraries to demonstrate library impact and advocate for comprehensive resource
funding. Initial slides provided a description of how the data were gathered,
the years included in our publication set, and significant revelations from the
data. Namely, that most publications generated by MRDC organizations referenced
materials provided by the MRDC Virtual Library or subcommand libraries. MRDC
library subscriptions encompass a range of medical research topics designed to
align with the organizational mission. In no instance did any library
rely primarily on the clinical resources provided by DHA.
Conveying
budgetary requirements to non-library stakeholders called for rethinking ways
to distinguish clinical resources and MRDC research resources, and which
subscriptions best support the needs of our users. Specialized backgrounds of
non-library stakeholders ought to be considered when communicating with
leadership, especially when contemplating how to best present library
information and metrics that will most effectively illustrate the role of the
library in the mission. The most effective metrics for each library may vary
depending on the type of library and its deliverables.
Reflection
Analysis of
citation metadata can help libraries justify budgetary costs, make collection
development decisions, and affirm the libraries’ alignment with critical
research activities. Bibliometrics should be used to advocate the impact of
libraries within their organizations. Although MRDC libraries differ from
academic and clinical libraries, this evidence-based methodology is applicable
in other types of library settings. While surveys can be beneficial for
gathering feedback from the patron, information professionals should also
consider using citation metadata to illustrate the role that the library plays
in research, publication, and medical product development.
Many
stakeholders are concerned with the cost of library operations, leading to an
increase in library literature focused on value studies and return on
investment. However, librarians in the MRDC library consortia recognize that
the most important “currency” in our libraries is research. The bibliographies
of MRDC publications prove that MRDC libraries provide access to the primary
cited resources in our researchers’ publications. Libraries must consider the
backgrounds of library stakeholders and leadership when evaluating what
evidence would most effectively communicate their impact.
Author Contributions
Holly Rose
Beverley: Data
curation (equal), Formal analysis (equal), Project administration (lead),
Supervision (lead), Writing – original draft (lead), Writing – review &
editing (lead) Olivia Briere: Data curation (equal), Formal analysis
(equal) Elisia George: Data curation (equal), Formal analysis (equal),
Writing – review & editing (equal) Maureen Humphrey-Shelton: Conceptualization
(lead), Data curation (equal), Formal analysis (equal), Methodology (lead),
Supervision (equal), Writing – review & editing (equal)
Disclaimer
Material has
been reviewed by the Walter Reed Army Institute of Research and the United
States Army Medical Research and Development Command. There is no objection to
its presentation and/or publication. The opinions or assertions contained
herein are the private views of the authors and are not to be construed as
official or as reflecting true views of the Department
of the Army or the Department of Defense.
References
Belter, C. W., & Kaske, N. K.
(2016). Using bibliometrics to demonstrate the value of library journal
collections. College and Research
Libraries, 77(4), 410–422. https://doi.org/10.5860/crl.77.4.410
De Groote, S. L., Akusa Dunya, B., Scoulas, J. M., & Case, M. M. (2020).
Research productivity and its relationship to library collections. Evidence Based Library and Information
Practice, 15(4), 16–32. https://doi.org/10.18438/eblip29736
De Groote, S. L., & Scoulas, J. M. (2022). The impact of the academic library
on students' success, in their own words. portal:
Libraries and the Academy, 22(2),
355–374. https://doi.org/10.1353/pla.2022.0021
Lercher, A., & Smolinsky, L. (2016).
Persistent value of older scientific journal articles. Scientometrics, 108(3), 1205–1220. https://doi.org/10.1007/s11192-016-2011-5
Urquhart, C. (2020). Assessing
return on investment in health libraries requires lateral thinking. Health Information and Libraries Journal, 37(1), 1–4. https://doi.org/10.1111/hir.12298
U.S. Army Medical Research and
Development Command. (n.d.). Program
areas: Medical research & development. Retrieved November 15, 2022 from https://mrdc.health.mil/index.cfm/program_areas/medical_research_and_development
Verstak, A., Acharya, A., Suzuki,
H., Henderson, S., Iakhiaev, M., Lin, C. C. Y., &
Shetty, N. (2014). On the shoulders of
giants: The growing impact of older articles. arXiv.
https://doi.org/10.48550/arXiv.1411.0275