Evidence Summary
Librarians’ Reported Systematic Review Completion Time Ranges Between 2
and 219 Total Hours with Most Variance due to Information Processing and
Instruction
A Review of:
Bullers, K., Howard, A. M., Hanson, A., Kearns, W. D., Orriola, J. J.,
Polo, R. L., & Sakmar, K. A. (2018). It takes longer than you think:
Librarian time spent on systematic review tasks. Journal of the Medical Library Association, 106(2), 198-207. https://doi.org/10.5195/jmla.2018.323
Reviewed by:
Peace Ossom Williamson
Director for Research Data Services
University of Texas at Arlington Libraries
Arlington, Texas, United States of America
Email: peace@uta.edu
Received: 16 Nov. 2018 Accepted:
7
Jan. 2019
2019 Williamson.
This is an Open Access article distributed under the terms of the Creative
Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.
DOI:
10.18438/eblip29525
Abstract
Objective – To
investigate how long it takes for medical librarians to complete steps toward
completion of a systematic review and to determine if the time differs based on
factors including years of experience as a medical librarian and experience
completing systematic reviews.
Design –
Survey research as a questionnaire disseminated via email distribution lists.
Setting – At
institutions that are members of the Association of Academic Health Sciences
Libraries (AAHSL) and librarians at Association of American Medical Colleges
(AAMC) or American Osteopathic Association (AOA) member institutions.
Subjects –
Librarians of member institutions who have worked on systematic reviews.
Methods – On
December 11, 2015, AAHSL library directors and librarian members of AAMC and
AOA were sent the survey and the recommendation to forward the survey to
librarians on staff who have worked on systematic reviews. Reminders were sent
on December 17, 2015, and the survey closed for participation on January 7,
2016. Participants who had worked on a systematic review within the past five
years were asked to indicate experience by the number of systematic reviews
completed, years of experience as a medical librarian, and how much time was
spent, in hours, on the following: initial consultations/meetings; developing
and testing the initial search strategy; translating the strategy for other
databases; documenting the process; delivering the search results; writing
their part of the manuscript; other tasks they could identify; and any
instruction (i.e., training they provided to team members necessary for
completion of the systematic review). Participants also further broke down the
amount of their time searching, by percentage of time, in various resources,
including literature indexes/databases, included studies’ references, trial
registers, grey literature, and hand searching. Participants were also given
space to add additional comments. The researchers reported summary statistics
for phase one and, for phase two, excluded outliers and performed exploratory
factor analysis, beginning with principal components analysis (PCA), followed
by a varimax rotation, to determine if there was a relationship between the
time on tasks and experience.
Main Results – Of
the 185 completed responses, 105 were analyzed for phase one because 80
responses were excluded due to missing data or no recent experience with a
systematic review. The average respondent had between 1 and 6 years of
experience: 1-3 years in librarianship (49.5%) and 4-6 years (23.8%). The time
reported for completion of all tasks ranged from 2 to 219 hours with a mean of
30.7 hours. Most of the variance (61.6%) was caused by “information processing”
and “interpersonal instruction/training” components. Search strategy
development and testing had the highest average time at 8.4 hours. Within that
category, databases accounted for 78.7% of time searching, followed by other
searching methods. For remaining systematic review tasks, their averages were
as follows: translating research (5.4 hours), delivering results (4.3 hours),
conducting preliminary consultations (3.9 hours), instruction (3.8 hours),
documentation (3.0 hours), additional tasks that were written-in by respondents
(2.2 hours), and writing the manuscript (1.8 hours). The most common written-in
tasks were development of inclusion/exclusion criteria, critical appraisal, and
deduplication. Other write-ins included retrieving full-text articles,
developing protocols, and selecting a journal for publishing the systematic
review.
For
the second phase of analysis, 12 responses were excluded as extreme outliers,
and the remaining 93 responses were analyzed to detect a relationship between
experience and time on task. Prior systematic review experience correlated with
shorter times performing instruction, consultation, and translation of
searches. However, librarian years of experience affected the percentage of
time on task, where greater years of experience led to more time spent
consulting and instructing than the percentage for librarians with fewer years
of experience. Librarians with greater than 7 years of experience skewed trends
toward shorter time on task, and, with their data excluded, years of experience
showed weak positive correlation with instruction and consultation.
Conclusion –
Because the average librarian participating on systematic review teams has had
few prior experiences and because the times can vary widely based on assigned
roles, duties, years of experience, and complexity of research question, it is
not advised to establish expectations for librarians’ time on task. This may be
why library administrators have disparate expectations of librarians’
involvement in systematic reviews and find it difficult to allocate and
anticipate staff time on systematic review projects. While it may not be
possible to set specific overarching guidelines for librarians’ expected time
on systematic review tasks, librarian supervisors and library directors
planning for their staff to offer systematic review services should work to
develop extensive understanding of the steps for conducting and assessing
systematic reviews in order to better estimate time commitments.
Commentary
Librarians
typically serve in searching and data management roles on systematic review
teams, and many provide expertise in question design and content evaluation
(Dudden & Protzko, 2011; Spencer & Eldredge, 2018). Two recent studies
investigated total time for systematic reviews and meta-analysis (Gann &
Pratt, 2013; Saleh, Ratajeski & Bertolet, 2014); however, this is the first
study to investigate the librarian’s time on each task contributing to
systematic reviews. The study involved a survey questionnaire emailed to
various lists, and the evidence could be stronger if the research were
supported by an observational study of librarians, where they tracked their
time and effort while conducting one or several systematic reviews. The
variation in reporting may also be reduced by stratifying times by systematic
review quality. There may be a relationship between time on task and quality of
the final product if, for example, fewer databases were searched, search
strategies did not include subject headings and were not customized for each
database, additional relevant terms were missed, and grey literature was not
explored.
The
critical appraisal checklist developed by Boynton and Greenhalgh (2004)
includes claims of validity and reliability in the reporting of survey research,
and the study includes these attributes when explaining the methods of
exclusion, quantitative analysis, and interpretation. The authors were
transparent in their research by providing the instrument that was used.
However, the researchers did not describe what were the statistical criteria
for exclusion of outliers and thresholds for factor loadings and whether these
were preestablished prior to data analysis, and, in Figure 2 and Table 3, there
are numbered components upon which the data relies; however, there is no
explanation of what factors (other than factors 1 and 2) with which those
numbers align.
The
article did not report how the survey was piloted, although the survey was
relatively brief with questions that were straightforward. The options for
selection and text entry were appropriate. This allowed for clean data and
clear structure for analysis. Open-ended commentary elicited further
information from participants. One notable strength in the survey was that it
allowed for participants to input other tasks that weren’t accounted for in the
original list. This allowed the researchers to catch common systematic review
steps, like developing a protocol and conducting appraisals, that they would
have otherwise omitted.
Ultimately,
this study introduces a new way of thinking and compartmentalizing of tasks,
and it is clear that the vast majority of time spent by the librarian on
systematic reviews is time spent searching. Training on how to search more
efficiently may be one of the most impactful ways of reducing time spent on
systematic reviews, allowing librarians to better involve themselves in
systematic reviews while still performing other job duties.
References
Boynton,
P. M., & Greenhalgh, T. (2004). Hands-on guide to questionnaire research:
Selecting, designing, and developing your questionnaire. BMJ: British
Medical Journal, 328(7451), 1312-1315. https://doi.org/10.1136/bmj.328.7451.1312
Dudden,
R. F., & Protzko, S. L. (2011). The systematic review team: Contributions
of the health sciences librarian. Medical
Reference Services Quarterly, 30(3), 301-315. https://doi.org/10.1080/02763869.2011.590425
Gann,
L. B., & Pratt, G. F. (2013). Using library search service metrics to
demonstrate library value and manage workload. Journal of the Medical Library Association, 101(3), 227-229. https://doi.org/10.3163/1536-5050.101.3.015
Saleh, A. A., Ratajeski, M. A., & Bertolet, M.
(2014). Grey literature searching for health sciences systematic reviews: A
prospective study of time spent and resources utilized. Evidence Based Library and Information Practice, 9(3), 28-50. https://doi.org/10.18438/B8DW3K
Spencer,
A. J., & Eldredge, J. D. (2018). Roles for librarians in systematic
reviews: a scoping review. Journal of the Medical Library Association, 106(1),
46-56. https://doi.org/10.5195/jmla.2018.82