Commentary
Systematic Searching in a Post-Pandemic World: New Directions for Methods, Technology, and People
Paul Levay
Information
Specialist
National
Institute for Health and Care Excellence
Manchester,
United Kingdom
Email: paul.levay@nice.org.uk
Jenny Craven
Information
Specialist
Manchester,
United Kingdom
Email: cravenj@btinternet.com
Received: 5 Aug.
2023 Accepted: 13 Oct.
2023
2023 Levay and Craven. This is an Open Access
article distributed under the terms of the Creative
Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work is
properly attributed, not used for commercial purposes, and, if transformed, the
resulting work is redistributed under the same or similar license to this one.
DOI: 10.18438/eblip30415
Introduction
In
January 2019 we concluded our book, Systematic Searching: Practical Ideas
for Improving Results, by asking the question, Where do we go from here?”
We would like to begin to answer this question by assessing how the search
landscape has changed since our book was published (Levay
& Craven, 2019). The COVID-19 pandemic has accelerated change and
led us in new directions, as well as confirmed some of the issues that we
already anticipated.
This
commentary focuses on the challenges facing information specialists and others
who are engaged in searching to support systematic reviews and other evidence
syntheses in healthcare, briefly reviewing the main developments that have
occurred in relation to methods, technology, and people. Throughout this
commentary, we refer to information specialists as a general term to describe
any information professional involved in systematic searching. This article is
intended to prompt discussion about how the information profession might need
to adapt, and it is not a comprehensive summary of the recent literature. In
addition, we have not covered information about how to conduct systematic
searching, as a new guide to that topic is available (Foster
& Jewell, 2022).
Methods
We
have to ensure that we keep our methods up to date to
meet new challenges so that we continue delivering the right evidence at the
right time.
Living Reviews and Living Guidelines
The
importance of keeping systematic reviews and guidelines up to date became
vitally important during a global pandemic caused by a new disease. Living
reviews, which are continually updated to incorporate relevant new evidence as
it becomes available, had been established several years before the pandemic
(Elliott et al., 2017). COVID-19 meant we had to accelerate the process of
developing living guidelines to incorporate the findings of living reviews into
evidence-based recommendations. A living guideline contains one or more
recommendations that are “kept current by an optimized guideline-updating
process that accounts for potentially consequential evidence as soon as or
shortly after it becomes available” (El Mikati et
al., 2022, p. 1155).
A
key question when developing living guidelines is whether to search broadly to
cover a whole topic or to run several, targeted searches (McDonald et al., 2023). The decision is affected by
how often the evidence is likely to change, the number of databases or other
search techniques required, the frequency of the searching, the current level
of uncertainty in the evidence, and the time available for processing the
search results. Reporting the searches for a living guideline is challenging as
the recommendations could be based on different strategies, date limits, and
sources (Metzendorf et al., 2022).
COVID-19
has shown that it is feasible to establish living reviews that support living
guidelines. The challenge now is to extend the living approach in topics other
than COVID-19. This will require thinking about the issues we explore in the
following sections relating to types of evidence, using technology efficiently,
and developing new skills.
Searching for the Right Types of Evidence
As
information specialists, we had to identify the
appropriate types of evidence for the COVID-19 pandemic. There is an
ongoing debate about how systematic reviews can incorporate mechanistic
evidence, which is derived from studies that explain the factors, interactions,
and other mechanisms that are responsible for a phenomenon (Greenhalgh et al., 2022). For example, reviews on
the effectiveness of face masks for stopping the spread of COVID-19 might
consider a wide range of study designs from various disciplines, including in
vitro experiments, imaging data, aerosol science, and engineering research (Greenhalgh et al., 2022). Search strategies
focusing on data from clinical trials would miss significant areas of this
evidence. These are new subject areas for many information specialists in
healthcare settings, requiring us to explore the appropriate sources and search
techniques to retrieve mechanistic evidence.
The
international focus on mechanistic evidence highlighted a long-standing issue
on how we identify and synthesize evidence to understand how complex
interventions operate (Greenhalgh & Peacock,
2005). It was even more important than ever that we discussed with
review teams the types of evidence required instead of relying on our familiar
processes. We need to apply this approach in all systematic reviews rather than
relying on standardized methods. We must choose the appropriate search
approaches (Cooper et al., 2022) and sources (Levay et al., 2022a) for the topic and the type of
evidence required. We know that using different search approaches and a wide
range of sources takes longer and requires more resources (Briscoe et al.,
2022). We have to resolve these logistical challenges
with review teams or else we risk missing relevant evidence.
Incorporating Preprints
Accessing the
latest evidence was crucial when millions of lives were potentially at risk
from COVID-19. Preprints, which are full manuscripts of papers made available
before or in parallel with the peer review process (Clyne
et al., 2021), were required when our core sources could not provide the
up-to-date studies needed to understand COVID-19.
We have to devise methods to deal with the challenges that we
face when incorporating preprints into evidence synthesis (Khalil et al., 2021). It is important to update
processes so that we can identify whether the preprints included in a
systematic review changed or were retracted after they had been peer reviewed (Brierley et al., 2022). Our reference management
practices need updating, as a preprint and the linked final article are not
strictly duplicates, although they must not be double counted in a meta-analysis.
Preprint servers are also of varying scope and quality (Kirkham
et al., 2020), often lacking the sophisticated interfaces needed to
write precise search strategies. Some of these technical issues have been
resolved since preprints became available on Europe PubMed Central (Rosonovski et al., 2023).
Technology
We
must deploy technology effectively to be able to handle increasing complexity,
higher volumes, and different types of evidence.
Study-Based Registers
Study-based
registers (databases in which all references to a study are available in a
single record) have been available for several years to make evidence easier to
find (Shokraneh & Adams, 2019). For
example, a study-based register covering clinical trials might bring together
references to the protocol, main results, and sub-group analyses into a single
record. COVID-19 gave fresh impetus to these registers, as they facilitated
rapid searches, while reducing research waste and duplication between teams.
A
number of open access COVID-19 study-based registers
have been established, such as the Cochrane COVID-19 Study Register (Metzendorf & Featherstone, 2021) and
Epistemonikos COVID-19 L·OVE (Verdugo-Paiva et al.,
2022). They have been reviewed favourably in terms of completeness and
timeliness (Butcher et al., 2022; Pierre et al.,
2021).
We
would benefit from having guidance to help us identify when to use registers in
place of separate databases. Barriers to uptake include technical ones (such as
how easy it is to export the results) or personal ones relating to confidence
with using unfamiliar databases and other search tools, such as study-based
registers. Registers are time consuming and expensive to maintain so extending
this approach would require the major producers of systematic reviews to invest
in the infrastructure.
Machine Learning Classifiers
A
feature in facilitating up to date and comprehensive study-based registers and
living systematic reviews was the widespread adoption of machine-learning
classifiers. Machine learning deploys algorithms that learn to perform a
specific task in order to make predictions based on
the training data that has been provided (Thomas et
al., 2021). In the context of systematic reviews, the training data is
often the decisions on which papers to include or exclude. The data enables the
algorithm to “learn” which words and phrases are more likely to lead to a paper
being included, in comparison to those indicating that the paper should be
excluded (O’Mara-Eves et al., 2015).
Machine
learning requires large quantities of training data, and this takes time to
acquire, validate, and process (Stansfield et al.,
2022). It would be fruitful to share machine-learning algorithms across
topics or domains so that each review team does not have to start afresh.
Machine learning has been used to identify randomized controlled trials (Thomas et al., 2021) and to populate the Cochrane
COVID-19 Study Register (Shemilt et al., 2022).
As machine learning becomes more fully incorporated into workflows for
screening search results (Chappell et al., 2023), we
may be able to provide broader, less precise, strategies for some
reviews. Information specialists can advise review teams, and we should be
involved in decisions about how and when to use machine learning.
Developing Search Strategies
Technological
developments are driving change even in areas where we have well-developed
practice. Numerous tools are available to help us design and deliver searches,
as listed on the Systematic Review Toolbox website (Johnson
et al., 2022). More changes are coming that will affect how we do this
work.
Automated
indexing will affect the sensitivity and precision of our strategies,
encouraging us to review how we develop and test searches. To aid selection of
controlled vocabulary terms, the National Library of Medicine (NLM) has had a
fully automated process for indexing MEDLINE records with Medical Subject
Headings (MeSH) since April 2022 using the Medical Text Indexer (MTI) (National Library of Medicine, 2022). There are
initial indications that MTI, compared to human indexers, will be responsible
for applying more MeSH terms to each record, omitting age-related check tags,
and choosing headings from different levels in the hierarchy (Chen et al., 2023). This increasing automation
might influence the effectiveness of particular search
strategies. As a result, we might need to review any search strategies,
including validated search filters, written before fully automated indexing was
implemented.
Another
area worth exploring is the potential of using search visualization to replace
the familiar form-based method of inputting queries into services such as
PubMed. For example, there has been some success in testing a visual interface
for creating and editing searches, such as the one provided by 2Dsearch (Svarre & Russell-Rose, 2022). We will benefit
from collaborating with computer scientists and software engineers to develop
the tools we need. These conversations can be facilitated by using design principles
relevant to systematic searching (MacFarlane et al.,
2022).
Can
we use artificial intelligence (AI) to generate search strategies?
Text-generation systems are already being rolled out to question-answering
services in familiar search engines, such as Bing and Google. We are now seeing
attempts to apply generative AI to evidence synthesis with mixed results (Qureshi et al., 2023). ChatGPT-3.5, launched in
November 2022, can generate seemingly plausible PubMed strategies, if prompted
with the right question (Wang et al., 2023).
These strategies would not currently pass through our peer-review checklists,
as they can contain serious errors, such as subject headings that do not
actually exist in MeSH (Wang et al., 2023).
We
should not, however, over-emphasize the fact that strategies generated by
ChatGPT-3.5 are currently “unusable” (Qureshi et al.,
2023, p. 2). The use of third-party plug-ins will improve the accuracy
of ChatGPT-4.0, which is currently available to subscribers. Generative AI will
probably be incorporated into bibliographic databases, with trials imminent in
Scopus and Web of Science, among others (van Noorden,
2023). The technology is going to improve massively and very quickly!
As
large language models are fundamentally based on prediction, the quality of
their training data is vitally important. We know, however, that many published
systematic reviews are based on low-quality searches (de
Kock et al., 2020). Any strategy generated from this poor data is likely
to be flawed. Longer term, we should take steps to ensure that AI systems are
learning from high-quality training data. The most effective AI systems for use
in evidence synthesis will be those that incorporate the recommendations from
the International Collaboration for the Automation of Systematic Reviews
(ICASR) (Beller et al., 2018).
AI
cannot be ignored and so information specialists must be ready to lead the
transformation of working practices. It is probable that we will see
human-in-the-loop systems develop, rather than purely automated evidence
syntheses. We should grasp the opportunity to expand our roles into
troubleshooting, user education, evaluation of sources, and validation of
results.
People
Several
chapters in our book explored the benefits of effective communication and
collaboration, and these skills have never been as important as they were
during the COVID-19 pandemic.
Collaboration
Effective
collaboration might be between information specialists and the wider team, or
it might feature groups of information professionals in local, national, and
international networks (Waffenschmidt & Hausner,
2019). Collaboration between information specialists was particularly
valuable during COVID-19, which Caroline De Brún (2022)
has helpfully summarized as involving:
·
Supporting other librarians
·
Reducing duplication of effort
·
Sharing best practices
·
Problem solving and local support
These
principles were demonstrated by the Librarian Reserve Corps (LRC), a voluntary
network of medical, health sciences, and public health librarians, who came
together to provide an evidence-based response to the international emergency (Callaway, 2021). The LRC has published a valuable
guide that draws on the lessons of COVID-19 to guide searching during future
emergencies (Brody et al., 2023). As another
example, the European Association for Health Information and Libraries (EAHIL)
Evidence Based Information Special Interest Group (EBI-SIG) is working on a
project to create a living open access library of search strategy resources (EBI-SIG, 2023). We have also seen the launch of the
searchRxiv website for sharing, archiving and
identifying search strategies (CABI Digital Library,
2023), which demonstrates the value of large-scale collaborative efforts.
Networking
and developing relationships are vital for keeping up to date with other
information specialists and organizations.
Communication
Communication
is a key skill for facilitating collaboration. Many of us have been using Zoom,
Microsoft Teams, and other platforms far more than we had ever envisaged five
years ago. This certainly helped to develop international links. We ought not
overlook how these networks helped to overcome isolation and promote wellbeing
in a time of lockdowns and other pandemic restrictions (De
Brún, 2022).
Communication
skills are as important as technical knowledge in training to become an
information specialist (Levay et al., 2022b).
We must be able to demonstrate that our searches are reliable, transparent, and
reusable (Sampson, 2019). The Preferred
Reporting Items for Systematic reviews and Meta-Analyses literature search
extension (PRISMA-S) is a guide to reporting search histories in a replicable
and transparent way (Rethlefsen et al., 2021).
The checklist is also a useful communication tool, as it encourages us to share
our work clearly and completely. We would encourage all information specialists
to integrate PRISMA-S into their processes.
Skills and Training
Skills
and training were a central theme in our book to ensure information specialists
could lead change and drive improvements. The evidence on the value of expert
searchers has continued to accumulate since then (Ramirez
et al., 2022).
Information
specialists need to develop the skills to promote equality, diversity, and
inclusion through our work in systematic reviewing. For example, with
appropriate experience, writing a search protocol is an opportunity to help
tackle health inequalities, if we select appropriate data sources, ensure the
search strategies cover diverse populations, and deal with underrepresentation
in the literature (Naicker, 2022). Important
work has been done to show how we might discuss outdated, discriminatory, and
other potentially inappropriate terminology with review teams, and then
consider how we should report these search terms carefully when they are
required in a strategy (Townsend et al., 2022).
There
are opportunities to learn from the experiences of data librarians, who are
involved with managing the research lifecycle, data curation, data analysis,
and visualization (Ashiq & Warraich, 2022).
Data management skills would be useful for managing data generated during a
systematic review, incorporating real-world evidence, advocating for open data,
and gaining a better understanding of technical challenges.
In
addition, all information specialists would benefit from acquiring data
literacy and computational-thinking skills to help us to solve problems with
technology. Having this awareness opens up new
opportunities to introduce technology into our work, through learning how to
code, understanding programming languages, developing apps, or using packages
from Github. Bibliometrix (Aria & Cuccurullo,
2017) and similar packages perform well-established tasks quickly and
effectively with powerful results, such as identifying
phrases in a set of text, checking citation networks, and topic modelling.
Evaluating these tools and knowing how and when to use them appropriately will
be essential for everyone, including those who do not learn to code themselves.
The
purpose of developing these skills is to improve systematic reviews through the
value information specialists add to teams. It is not just about doing the
searches, it is about educating review teams on automation, showing them how to
deploy technology and improving their processes. We have the skills and
experience to be agents of cultural change as we help teams to integrate AI and
other technology into existing processes. We will do this effectively where we
can show that the new ways of working uphold the values of accuracy,
transparency, and accountability (Arno et al., 2021).
Conclusion
The
COVID-19 pandemic has clearly had a significant impact on the methods and
technology we use for systematic searching, accelerating some trends and
introducing new challenges. In terms of methods, searching for evidence on
COVID-19 has focussed on living reviews and guidelines, the use of mechanistic
evidence, and new sources, such as preprints. The increasing volume and
complexity of evidence necessitates better use of technology, such as
study-based registers, machine-learning classifiers, and visualization
software. International collaboration was valuable during the pandemic, and it
was facilitated through good communication. We can promote equality, diversity,
and inclusion through our searches. New skills, such as data management and
coding, will become increasingly valuable. Automation will not result in our
spending less time on systematic searching, but it may change how we focus our
efforts.
Fundamentally,
we stand by the conclusion we drew in our book, Systematic Searching:
Practical Ideas for Improving Results, that these methods and technologies
will only be deployed effectively if information specialists are involved and
set the agenda. The five years since 2019 have shown that systematic searchers
need to be flexible, creative, and at the forefront of innovation: we expect
these trends will intensify in the coming years.
Author Contributions
Paul Levay: Conceptualization (equal), Writing –
original draft (lead), Writing – review and editing (equal) Jenny Craven:
Conceptualization (equal), Writing – original draft (supporting),Writing – review and editing (equal)
Acknowledgements
The
authors would like to thank Amy Finnegan, Tom Hudson, Catherine Jacob, Caroline
Miller, Marion Spring, Nicola Walsh, and Riz Zafar for their comments on
earlier versions of this paper.
Disclaimer
The
views expressed in this paper are those of the authors and not necessarily
those of the National Institute for Health and Care
Excellence (NICE).
References
Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for
comprehensive science mapping analysis. Journal of Informetrics, 11(4),
959–975. https://doi.org/10.1016/j.joi.2017.08.007
Arno, A., Elliott, J., Wallace, B., Turner, T., & Thomas, J.
(2021). The views of health guideline developers on the use of automation in
health evidence synthesis. Systematic Reviews, 10(1), 16. https://doi.org/10.1186/s13643-020-01569-2
Ashiq, M., & Warraich, N. F. (2022). A systematized review on
data librarianship literature: Current services, challenges, skills, and
motivational factors. Journal of Librarianship and Information Science, 55(2),
414–433. https://doi.org/10.1177/09610006221083675
Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund,
H., Ouzzani, M., Thayer, K., Thomas, J., Turner, T., Xia, J., Robinson, K.,
& Glasziou, P. (2018). Making progress with the automation of systematic
reviews: Principles of the International Collaboration for the Automation of
Systematic Reviews (ICASR). Systematic Reviews, 7(1), 77. https://doi.org/10.1186/s13643-018-0740-7
Brierley, L., Nanni, F., Polka, J. K., Dey, G., Pálfy, M., Fraser,
N., & Coates, J. A. (2022). Tracking changes between preprint posting and
journal publication during a pandemic. PLOS Biology, 20(2), e3001285. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8806067/
Briscoe, S., Abbott, R., & Melendez‐Torres, G. J. (2022).
Expert searchers identified time, team, technology and
tension as challenges when carrying out supplementary searches for systematic
reviews: A thematic network analysis. Health Information & Libraries
Journal. Advance online publication. https://doi.org/10.1111/hir.12468
Brody, S., Loree, S., Sampson, M., Mensinkai, S., Coffman, J.,
Mueller, M., Askin, N., Hamill, C., Wilson, E., McAteer, M. B., & Staines,
H. (2023). Searching for evidence in public health emergencies: A white paper
of best practices. Journal of the Medical Library Association, 111(1),
566–578. https://doi.org/10.5195/jmla.2023.1530
Butcher, R., Sampson, M., Couban, R. J., Malin, J. E., Loree, S.,
& Brody, S. (2022). The currency and completeness of specialized databases
of COVID-19 publications. Journal of Clinical Epidemiology, 147, 52–59. https://doi.org/10.1016/j.jclinepi.2022.03.006
CABI Digital Library. (2023). searchRxiv. Retrieved October
12, 2023, from https://www.cabidigitallibrary.org/journal/searchrxiv
Callaway, J. (2021). The Librarian Reserve Corps: An emergency
response. Medical Reference Services Quarterly, 40(1), 90–102. https://doi.org/10.1080/02763869.2021.1873627
Chappell, M., Edwards, M., Watkins, D., Marshall, C., &
Graziadio, S. (2023). Machine learning for accelerating screening in evidence
reviews. Cochrane Evidence Synthesis and Methods, 1(5), e12021. https://doi.org/10.1002/cesm.12021
Chen, Y. Y., Bullard, J., & Giustini, D. (2023). Automated
indexing using NLM's Medical Text Indexer (MTI) compared to human indexing in
Medline: A pilot study. Journal of the Medical Library Association, 111(3),
684–695. https://doi.org/10.5195/jmla.2023.1588
Clyne, B., Walsh, K. A., O'Murchu, E., Sharp, M. K., Comber, L.,
O’ Brien, K. K., Smith, S. M., Harrington, P., O'Neill, M., Teljeur, C., &
Ryan, M. (2021). Using preprints in evidence synthesis: Commentary on
experience during the COVID-19 pandemic. Journal of Clinical Epidemiology,
138, 203–210. https://doi.org/10.1016/j.jclinepi.2021.05.010
Cooper, C., Booth, A., Husk, K., Lovell, R., Frost, J.,
Schauberger, U., Britten, N., & Garside, R. (2022). A Tailored Approach: A
model for literature searching in complex systematic reviews. Journal of
Information Science. Advance online publication. https://doi.org/10.1177/01655515221114452
De Brún, C. (2022, June 1–3). Knowledge makes the world go
round: Librarians working together to fight the COVID infodemic [Poster
session]. European Association for Health Information and Libraries, Rotterdam,
Netherlands.
de Kock, S., Stirk, L., Ross, J., Duffy, S., Noake, C., &
Misso, K. (2020). Systematic review search methods evaluated using the
Preferred Reporting of Items for Systematic Reviews and Meta-Analyses and the
Risk of Bias in Systematic reviews tool. International Journal of Technology
Assessment in Health Care, 37(1), E18. https://doi.org/10.1017/S0266462320002135
EBI-SIG. (2023). Library of Search Strategy Resources. Retrieved
October 12, 2023, from https://sites.google.com/view/searchresourceslib/home
El Mikati, I. K., Khabsa, J., Harb, T., Khamis, M., Agarwal, A.,
Pardo-Hernandez, H., Farran, S., Khamis, A. M., El Zein, O., El-Khoury, R.,
Schünemann, H. J., & Akl, E. A. (2022). A Framework for the development of
living practice guidelines in health care. Annals of Internal Medicine, 175(8),
1154-1160. https://doi.org/10.7326/M22-0514
Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A.,
McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., Tovey, D.,
Shemilt, I., & Thomas, J. (2017). Living systematic review: 1. Introduction—the why, what, when, and how. Journal of Clinical
Epidemiology, 91, 23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010
Foster, M. J., & Jewell, S. T. (2022). Piecing together
systematic reviews and other evidence syntheses. Rowman & Littlefield.
Greenhalgh, T., Fisman, D., Cane, D. J., Oliver, M., &
Macintyre, C. R. (2022). Adapt or die: How the pandemic made the shift from EBM
to EBM+ more urgent. BMJ Evidence-Based Medicine, 27(5), 253–260. https://doi.org/10.1136/bmjebm-2022-111952
Greenhalgh, T., & Peacock, R. (2005). Effectiveness and
efficiency of search methods in systematic reviews of complex evidence: Audit
of primary sources. BMJ, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68
Johnson, E. E., O’Keefe, H., Sutton, A., & Marshall, C.
(2022). The Systematic Review Toolbox: Keeping up to date with tools to support
evidence synthesis. Systematic Reviews, 11(1), 258. https://doi.org/10.1186/s13643-022-02122-z
Khalil, H., Tamara, L., Rada, G., & Akl, E. A. (2021).
Challenges of evidence synthesis during the 2020 COVID pandemic: A scoping
review. Journal of Clinical Epidemiology, 142, 10–18. https://doi.org/10.1016/J.JCLINEPI.2021.10.017
Kirkham, J. J., Penfold, N. C., Murphy, F., Boutron, I.,
Ioannidis, J. P., Polka, J., & Moher, D. (2020). Systematic examination of
preprint platforms for use in the medical and biomedical sciences setting.
BMJ Open, 10(12), e041849. https://doi.org/10.1136/bmjopen-2020-041849
Levay, P., & Craven, J. (2019). Conclusion: Where do we go
from here? In P. Levay, & J. Craven (Eds.), Systematic searching:
Practical ideas for improving results (pp. 289–292). Facet Publishing.
Levay, P., Heath, A., & Tuvey, D. (2022a). Efficient searching
for NICE public health guidelines: Would using fewer sources still find the
evidence? Research Synthesis Methods, 13(6), 760–789. https://doi.org/10.1002/jrsm.1577
Levay, P., Walsh, N., & Foster, L. (2022b). The National
Institute for Health and Care Excellence information specialist development
pathway: Developing the skills, knowledge and confidence to quality assure
search strategies. Health Information and Libraries Journal, 39(4),
392–399. https://doi.org/10.1111/hir.12460
MacFarlane, A., Russell-Rose, T., & Shokraneh, F. (2022).
Search strategy formulation for systematic reviews: Issues, challenges
and opportunities. Intelligent Systems with Applications, 15, 200091. https://doi.org/10.1016/j.iswa.2022.200091
McDonald, S., Sharp, S., Morgan, R. L., Murad, M. H., & Fraile
Navarro, D. (2023). Methods for living guidelines: Early guidance based on
practical experience. Paper 4: Search methods and approaches for living
guidelines. Journal of Clinical Epidemiology, 155, 108–117. https://doi.org/10.1016/j.jclinepi.2022.12.023
Metzendorf, M., & Featherstone, R. M. (2021). Evaluation of
the comprehensiveness, accuracy and currency of the
Cochrane COVID-19 Study Register for supporting rapid evidence synthesis
production. Research Synthesis Methods, 12(5), 607–617. https://doi.org/10.1002/jrsm.1501
Metzendorf, M., Weibel, S., Reis, S., & McDonald, S. (2022).
Pragmatic and open science-based solution to a current problem in the reporting
of living systematic reviews. BMJ Evidence-Based Medicine, 8(4),
267–272. https://doi.org/10.1136/bmjebm-2022-112019
Naicker, R. (2022). Critically appraising for antiracism.
Education for Information, 38(4), 291–308. https://doi.org/10.3233/EFI-220052
National Library of Medicine. (2022). Indexing FAQs. Retrieved
October 12, 2023, from https://support.nlm.nih.gov/knowledgebase/article/KA-05326/en-us
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., &
Ananiadou, S. (2015). Using text mining for study identification in systematic
reviews: A systematic review of current approaches. Systematic Reviews, 4(1),
5. https://doi.org/10.1186/2046-4053-4-5
Pierre, O., Riveros, C., Charpy, S., & Boutron, I. (2021).
Secondary electronic sources demonstrated very good sensitivity for identifying
studies evaluating interventions for COVID-19. Journal of Clinical
Epidemiology, 141, 46–53. https://doi.org/10.1016/j.jclinepi.2021.09.022
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li,
T., & Agai, E. (2023). Are ChatGPT and large language models “the answer”
to bringing us closer to systematic review automation? Systematic Reviews,
12(1), 72. https://doi.org/10.1186/s13643-023-02243-z
Ramirez, D., Foster, M. J., Kogut, A., & Xiao, D. (2022).
Adherence to systematic review standards: Impact of librarian involvement in
Campbell Collaboration's education reviews. The Journal of Academic
Librarianship, 48(5), 102567. https://doi.org/10.1016/j.acalib.2022.102567
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P.,
Moher, D., Page, M. J., & Koffel, J. B. (2021). PRISMA-S: An extension to
the PRISMA statement for reporting literature searches in systematic reviews.
Systematic Reviews, 10(1), 39. https://doi.org/10.1186/s13643-020-01542-z
Rosonovski, S., Levchenko, M., Ide‐Smith, M., Faulk, L., Harrison,
M., & McEntyre, J. (2023). Searching and evaluating publications and
preprints using Europe PMC. Current Protocols, 3(3), e694. https://doi.org/10.1002/cpz1.694
Sampson, M. (2019). Communication for information specialists. In
P. Levay, & J. Craven (Eds.), Systematic searching: Practical ideas for
improving results (pp. 249–268). Facet Publishing.
Shemilt, I., Noel-Storr, A., Thomas, J., Featherstone, R., &
Mavergames, C. (2022). Machine learning reduced workload for the Cochrane
COVID-19 Study Register: Development and evaluation of the Cochrane COVID-19
Study Classifier. Systematic Reviews, 11(1), 15. https://doi.org/10.1186/s13643-021-01880-6
Shokraneh, F., & Adams, C. E. (2019). Study-based registers
reduce waste in systematic reviewing: Discussion and case report. Systematic
Reviews, 8(1), 129. https://doi.org/10.1186/s13643-019-1035-3
Stansfield, C., Stokes, G., & Thomas, J. (2022). Applying
machine classifiers to update searches: Analysis from two case studies.
Research Synthesis Methods, 13(1), 121–133. https://doi.org/10.1002/jrsm.1537
Svarre, T., & Russell-Rose, T. (2022). Think outside the
search box: A comparative study of visual and form-based query builders.
Journal of Information Science. Advance online publication. https://doi.org/10.1177/01655515221138536
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott,
J., Mavergames, C., & Marshall, I. J. (2021). Machine learning reduced
workload with minimal risk of missing studies: Development and evaluation of a
randomized controlled trial classifier for Cochrane Reviews. Journal of
Clinical Epidemiology, 133, 140–151. https://doi.org/10.1016/j.jclinepi.2020.11.003
Townsend, W., Anderson, P., Haines, K., Hansen, S., James, L.,
MacEachern, M., Rana, G., Saylor, K. & Saylor, K. (2022). Addressing
antiquated, non-standard, exclusionary, and potentially offensive terms in
evidence syntheses and systematic searches. Retrieved October 12, 2023,
from https://doi.org/10.7302/6408
van Noorden, R. (2023). ChatGPT-like AIs are coming to major
science search engines. Nature, 620(7973), 258. Retrieved from https://www.nature.com/articles/d41586-023-02470-3
Verdugo-Paiva, F., Vergara, C., Ávila, C., Castro-Guevara, J.,
Cid, J., Contreras, V., Jara, I., Jiménez, V., Lee, M. H., Muñoz, M.,
Rojas-Gómez, A. M., Rosón-Rodríguez, P., Serrano-Arévalo, K., Silva-Ruz, I.,
Vásquez-Laval, J., Zambrano-Achig, P., Zavadzki, G., & Rada, G. (2022). COVID-19
Living Overview of Evidence repository is highly comprehensive and can be used
as a single source for COVID-19 studies. Journal of Clinical Epidemiology,
149, 195–202. https://doi.org/10.1016/j.jclinepi.2022.05.001
Waffenschmidt, S., & Hausner, E. (2019). Collaborative working
to improve searching. In P. Levay, & J. Craven (Eds.), Systematic
searching: Practical ideas for improving results (pp. 229–248). Facet
Publishing.
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023). Can
ChatGPT write a good Boolean query for systematic review literature search? SIGIR
'23: Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval, Taipei. 1426–1436. https://doi.org/10.1145/3539618.3591703