Evidence Summary

 

Additional Search Strategies May Not Be Necessary for a Rapid Systematic Review

 

A Review of:

Westphal, A., Kriston, L., Hölzel, L.P., Härter, M., & von Wolff, A. (2014). Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of Public Health Research, 3(2), 177. doi: 10.4081/jphr.2014.177

 

Reviewed by:

Joanne L. Jordan

Research Information Manager

Arthritis Research UK Primary Care Centre, Keele University

Keele, Staffordshire, United Kingdom

Email: j.jordan@keele.ac.uk

 

Received: 10 Mar. 2015  Accepted: 12 May 2015

 

 

cc-ca_logo_xl 2015 Jordan. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

Abstract

 

Objective – To evaluate the efficiency and contribution of additional searching strategies for finding randomized controlled trials (RCTs) in a systematic review.

 

Design – A methodological case study.

 

Setting – Biomedical literature.

 

Methods – A sensitive search (defined as “the ratio of the number of relevant reports identified to the total number of relevant reports in existence”) was conducted of electronic databases, Cochrane CENTRAL database, MEDLINE, EMBASE, PsycInfo, CINAHL, BIOSIS, and Web of Science databases (Science and Social Science Citation Indexes). The following additional searching strategies were conducted: hand-searching contents of relevant journals (Archives of General Psychiatry, Journal of Consulting and Clinical Psychology, and Journal of Affective Disorders), citation tracking (forwards tracking using Social Science and Science Citation Index and backwards tracking by looking through reference lists of included studies), screening reference lists of relevant systematic reviews, searching clinical trials registers (ClinicalTrials.gov and ICTRP registers), and contacting first authors of included studies to find any similar unpublished studies.

 

The number of articles identified by each of these methods was recorded and screened for inclusion in the systematic review. The authors calculated what they labelled as the ‘efficiency’ of each searching strategy (the number of included studies identified by the search method as a proportion of the full text articles screened) and the ‘contribution’ of the search strategies (the ratio of included studies identified by that method to the final number of included studies in the systematic review). The methodological quality of each included study was assessed using the Cochrane Risk of Bias Tool, which is a critical appraisal checklist used to judge the study’s value in the systematic review. The meta-analysis in the systematic review was conducted with and without the studies identified by the additional searching strategies to assess their impact on the review’s findings.

 

Main Results – In total 50 studies were identified, 42 from electronic database searches and 8 from additional search strategies. As illustrated by the results in Table 1, the most useful additional search strategy was screening reference lists of relevant systematic reviews. Journal hand-searching and contacting authors also contributed to the review.

 

Of the eight studies identified by the additional search strategies none were judged to have a low risk of bias (four had high risk of bias and four were unclear). Of the 42 included studies from electronic searches only 11 were judged to have a low risk of bias, whereas 9 studies had a high risk of bias and 22 were unclear.

 

Excluding the eight studies retrieved from additional search strategies in the systematic review meta-analysis did not influence the results on the effectiveness of the different interventions for chronic depression. These studies were found to be indexed correctly on the electronic databases, but were not identified in the initial search.

 

Conclusion – Additional search strategies, especially screening reference lists of systematic reviews and hand-searching relevant journals, retrieved a substantial number of relevant studies for a systematic review of interventions for treating chronic depression. However, results of the review’s meta-analysis did not differ when these additional studies (rated as either high or unclear risk of bias) were not included and search methods were time consuming. It might be reasonable to rely on electronic searching strategies when resources for conducting a systematic review are limited or when doing a “rapid review.” The benefits and limitations of additional search strategies should be considered particularly when resources or time for conducting a systematic review are limited.

 

 

Table 1

The numbers of articles retrieved and included studies from each of the searching strategies.

Search Strategy

Search Results

Full texts screened

Included in review

Efficiency

Contribution

Electronic database search

2417

276

42

9.8%

84%

Journal hand-search

19076

33

2

6.1%

4%

Forward citation tracking

2979

7

0

0%

0%

Backward citation tracking

1692

15

0

0%

0%

Review reference screening

1191

16

5

31.3%

10%

Clinical trial registers

2053

0

0

0%

0%

Contacting authors

16

11

1

9.1%

2%

 

 

If the electronic database search is sensitive and includes the Cochrane CENTRAL database additional search strategies may not be necessary, but these findings should be tested in other research areas.

 

Commentary

 

This study is highly relevant for information professionals who often conduct search strategies for systematic reviews. Systematic reviews frequently take 8 to 12 months to complete and studies evaluating rapid review methods that reduce production time without compromising rigour are needed.

 

The Cochrane CENTRAL database (The Cochrane Library) consists of RCTs retrieved from a variety of sources, including hand-searching journals and searches of trial registries. As the authors of the study rightly point out, by including CENTRAL in their electronic database searches, additional strategies are likely to be less influential. In this study, journals with high impact factors and those containing well-known potentially relevant studies were hand searched. The authors did not mention if they checked the list of journals already hand-searched for the CENTRAL database, which could have avoided the duplication of effort required in screening several thousand article titles and abstracts.

 

However, Cochrane review standards (Cochrane Editorial Unit, 2013) state it is mandatory for Cochrane reviews (the gold standard systematic reviews), in addition to searching CENTRAL, to search ClincalTrials.gov, WHO International Clinical Trials Registry Platform (ICTRP) trial registers, and screen reference lists of included studies and relevant systematic reviews. These standards ensure that searches for systematic reviews are extensive and sensitive. The authors found that studies indexed in the electronic databases were missed by their search strategies. By using a variety of methods systematic reviewers can be more confident about identifying all the relevant studies.

 

Studies found in this review by additional search strategies did not influence the results of the meta-analysis. However, as pointed out by the authors, where results of studies are sparse or conflicting, missing even the smaller studies will have a large impact and bias the results of the systematic review. This study needs to be repeated in a broad range of reviews to fully assess the impact of additional searching strategies. Including unpublished studies (grey literature), which were excluded in this study due to limited searching, also have the potential to affect systematic review results. Searching for grey literature requires different resources and their value needs to be assessed.

 

The authors of this study did not count studies that had already been included by a previous strategy. They comment that the order in which searching strategies were applied will have affected the level of contribution. Therefore the contribution of the searching strategies may be underestimated and the impact of this needs further exploration.

 

A criticism of the study is that the electronic search strategies are not sensitive enough and may be the reason eight studies were missed. The strategies do not include indexed terms (e.g., MeSH terms) that would help to identify relevant studies. The authors state they did not consult an expert librarian or information specialist when constructing their search strategies, which might have improved their search retrieval.

 

Reference

 

Cochrane Editorial Unit. (2013) Methodological Expectations of Cochrane Intervention Reviews (MECIR): Methodological standards for the conduct of new Cochrane Intervention Reviews. Version 2.3. Available from http://editorial-unit.cochrane.org/mecir