An Exploration of Systematic Review Publication Trends in Conservation Biology Journals
Jocelyn Boice
Assistant Professor
Colorado State University Libraries
Fort Collins, Colorado
jocelyn.boice@colostate.edu
Researchers in conservation biology and other non-medical fields are adopting systematic review as a research methodology. Since this methodology requires extensive and well-documented literature searching, it is beneficial for information professionals to understand disciplinary developments in its use. This article investigates trends in systematic review publication in conservation biology journals between 1998 and 2017 and examines the prevalence of search reporting among these systematic reviews. Results show an increase in published systematic reviews over the study period, and the majority of these include a description of the literature search. However, evidence of variable search quality and reporting indicates an important role for librarians in improving literature search strategies and documentation.
Introduction
Systematic review is a research methodology that requires a comprehensive, repeatable literature search as one of its core components (Lefebvre et al. 2011). Foster and Jewell (2017) regard "expert-level skills in searching...as well as having a great understanding of databases' content, biases, and limitations" as "essential" to systematic reviews, and note that a number of systematic review standards suggest consulting a librarian during the research process. In a recent study, Spencer and Eldredge (2018) described 18 ways in which medical librarians have participated in systematic reviews, such as guiding the development of research questions, identifying literature databases to use, creating appropriate search strategies, providing peer review for searches created by others, and documenting the research process. Accordingly, this methodology holds a special significance for librarians.
Stated simply, systematic reviews attempt "to identify as many studies as possible on a specific research question" using "a rigorous and transparent methodology" in order to decrease bias, synthesize the findings, and respond to the question (Gore & Jones 2015). In contrast to traditional literature reviews, systematic reviews generally involve searching for both published and unpublished materials, recording the steps of the search in detail, and defining in advance the criteria that will decide which studies to include or exclude from the synthesis phase (Haddaway et al. 2015). While a systematic review may include a meta-analysis, it does not have to, and other options for combined analysis may be employed (McKibbon 2006; Gough et al. 2017). Researchers may choose to conduct a systematic review for various reasons, such as distilling the research findings for a highly studied topic, integrating all available information on rare events, or attempting to resolve disputed issues (McKibbon 2006).
Although systematic review has its roots in medicine and the health sciences, researchers have increasingly adopted it for use in other fields (Gore & Jones 2015; Fournier & Ghezzi-Kopel 2016; Young & Eldermire 2017). In conservation biology, researchers began advocating for the incorporation of systematic reviews into the field's regular practice as early as 2001, as part of an argument for evidence-based conservation (Pullin & Knight 2001). Additional calls for the use of systematic reviews appeared during the early part of that decade (Pullin & Knight 2003; Fazey et. al. 2004; Sutherland et. al. 2004), culminating in Pullin and Stewart's (2006) influential article outlining suggested procedures for conducting conservation biology systematic reviews.
Organizations that provide guidance and support for the production of systematic reviews exist in several fields (Young & Eldermire 2017). The creation of the Collaboration for Environmental Evidence, a collective based upon the model of the Cochrane Collaboration in medicine, fills this role for conservation biology (Pullin & Knight 2009). Despite the availability of this outlet for conservation systematic reviews, the need for additional publication venues, especially in prestigious journals, remains a concern (Pullin & Knight 2009; Cook et al. 2013).
Given this disciplinary push to employ systematic reviews and the relevance of the methodology to librarianship, this article examines the publication of systematic reviews in conservation biology journals from 1998 to 2017. In particular, how frequently do systematic reviews appear and in which journals? What are the characteristics of the journals? Are certain authors or institutions represented more often than others? Are there any clear developments over time?
In a second line of inquiry, this article investigates how often the systematic review articles in this sample actually use systematic review methodology in the literature searching process. Anecdotally, colleagues have reported receiving requests for assistance with "systematic reviews" that turn out to simply be thorough literature reviews. Both Gore and Jones (2015) and Heimlich (2014) note that there may be some confusion on the part of researchers as to what constitutes a systematic review and point to a role for librarians in elucidating the methodology. Fournier and Ghezzi-Kopel (2016) observe that this may be especially true for fields other than medicine, and the comprehension, or lack thereof, of the phrase "systematic review" has also been described by environmental researchers (Haddaway et al. 2017). The present study attempts to examine the phenomenon within the conservation biology literature and to ascertain whether "systematic review" has various meanings or understandings within the field.
Methods
The Web of Science database was used to explore the publication of systematic reviews in conservation biology journals. Web of Science is a primary database for the field of conservation biology and provides broad coverage of relevant literature. In addition, journals indexed in the database are classified by subject and each article record includes this information as a searchable field (Web... 2018). This feature provides a straightforward way to identify journals that publish in the area of conservation biology and allows for a simple search structure. Web of Science also provides built-in tools that facilitate analysis of various aspects of a results set.
Web of Science (specifically the Science Citation Index Expanded, the Social Sciences Citation Index, and the Arts & Humanities Citation Index) was searched for the string "systematic review*", using the topic field tag to identify records where it appeared in the title, abstract, or keywords. Results were limited to items published from 1998 through 2017 in journals classified in the Biodiversity Conservation Web of Science category. This category is defined as including "resources on the conservation management of species and ecosystems...conservation ecology, biological conservation, paleobiology, natural history and the natural sciences" (Scope... 2017). Bibliographic information for the results set was exported to a spreadsheet and data from subsequent steps (described later) was recorded within that spreadsheet.
The title, abstract, and if necessary the keywords and full text were reviewed for each article to discover whether the authors stated that the article was a systematic review or that it used systematic review methodology. Conference abstracts and corrections for previously published articles were excluded. The set of articles identified as systematic reviews is referred to as simply "the SRs" going forward.
Since literature search documentation is a basic requirement for systematic reviews (Lefebvre et al. 2011), the existence of a reported search can serve as a measure of whether an article actually used a systematic review search methodology. It can also serve as a simple way to discover possible differences in understanding and usage of the phrase "systematic review" within conservation biology. Consequently, the SRs and any supplemental materials referenced in the text were examined for compliance with a modified version of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Checklist criteria #7 "Information sources" and #8 "Search" (Moher et al. 2009). PRISMA is a well-known and widely acknowledged reporting standard in medicine and related fields. An initial check of several articles suggested that few would meet the PRISMA requirements as written, and since the intent was not to evaluate the quality of the searches or reporting but rather to determine whether a report of the search existed, the following modifications were made. In order to be counted as reporting a search an article had to (a) state the name of a database or resource searched (corresponding with PRISMA #7) and (b) list or describe the search terms used (corresponding with PRISMA #8).
Aggregate authorship and organizational affiliation information for the SRs was retrieved by creating marked lists in Web of Science and using the "Authors," "Organizations," and "Countries/Regions" options of the Analyze Results feature to export the data to a spreadsheet. Authorship and country data was manually de-duplicated, but organizational data was not. Organizational affiliation and country data missing from one record was added to the spreadsheet after consulting the full text of the associated article.
A list of all journals classed as Biodiversity Conservation in Web of Science from 1998 through 2017 was also obtained. To generate the list, the "Web of Science Categories" field tag was used to search for "Biodiversity Conservation" in the three indexes noted earlier. The date range was limited to 1998 through 2017, book series were excluded, and the aggregated source titles were exported using the Analyze Results feature. Alternate titles and title changes were combined to avoid double counting. The Journal Citation Reports (JCR) database was used to retrieve all available 2017 impact factors for the journals on the Web of Science Biodiversity Conservation list, and quartiles were calculated for this set of journals using JCR's methods (InCites... n.d.).
Results
The initial database search retrieved 111 records. Of these, 79 articles met the study requirements for identification as systematic reviews. Figure 1 displays the number of these SRs published each year
Figure 1: Systematic reviews published in conservation biology journals, 1998-2017
SRs appeared in 28 different journals, with five journals accounting for 49% of the published reviews. Table 1 provides the complete list of journals and the number of systematic reviews published in each, while Figure 2 charts the trends over time.
| Table 1: Conservation biology journals publishing systematic reviews, 1998-2017 | ||
|---|---|---|
| Journal Title | # of SRs published | % of total SRs published |
| Biological Conservation | 11 | 14 |
| Conservation Biology | 9 | 11 |
| Biological Invasions | 6 | 8 |
| Ecological Indicators | 6 | 8 |
| Journal of Applied Ecology | 6 | 8 |
| Bulletin of the American Museum of Natural History | 4 | 5 |
| Diversity and Distributions | 4 | 5 |
| Global Change Biology | 4 | 5 |
| Urban Ecosystems | 3 | 4 |
| American Museum Novitates | 2 | 3 |
| Ecohealth | 2 | 3 |
| Environmental Conservation | 2 | 3 |
| Global Ecology and Conservation | 2 | 3 |
| Human and Ecological Risk Assessment | 2 | 3 |
| Journal for Nature Conservation | 2 | 3 |
| Oryx | 2 | 3 |
| Biodiversity and Conservation | 1 | 1 |
| Conservation Genetics | 1 | 1 |
| Conservation Letters | 1 | 1 |
| Endangered Species Research | 1 | 1 |
| Insect Conservation and Diversity | 1 | 1 |
| Journal of Fish and Wildlife Management | 1 | 1 |
| Journal of Insect Conservation | 1 | 1 |
| Journal of Natural History | 1 | 1 |
| Neobiota | 1 | 1 |
| Palaeobiodiversity and Palaeoenvironments | 1 | 1 |
| Southeastern Naturalist | 1 | 1 |
| Tropical Conservation Science | 1 | 1 |
| Total | 79 | 101* |
| *Sum is more than 100% due to rounding | ||
Figure 2: Conservation biology journals publishing systematic reviews and systematic review articles published, 1998-2017
The Web of Science Biodiversity Conservation category contained 66 unique journals during the study period, and 42% of the journals in this category (28 journals) published SRs. Sixty-one of the 66 journals in the category had 2017 impact factors. The five journals without impact factors did not publish any SRs. The mean and median 2017 impact factors for the remaining journals, divided into those that published SRs and those that did not, are shown in Table 2.
| Table 2: Comparison of impact factors (IF) for conservation biology journals | ||
|---|---|---|
| Mean 2017 IF | Median 2017 IF | |
| Journals that publish SRs | 2.913 | 2.303 |
| Journals that do not publish SRs | 1.389 | 1.16 |
When compared to the other journals in the set of 61, the 28 journals that published SRs generally ranked higher in terms of impact factor. Specifically, 11 journals were ranked in the first quartile by impact factor, nine were in the second quartile, five were in the third quartile, and three were in the fourth quartile. Table 3 displays the number of journals in each group by quartile.
| Table 3: Comparison of impact factor (IF) quartiles for conservation biology journals | ||
|---|---|---|
| 2017 IF Quartile | Journals that publish SRs | Journals that do not publish SRs |
| Q1 | 11 | 4 |
| Q2 | 9 | 6 |
| Q3 | 5 | 10 |
| Q4 | 3 | 13 |
A total of 348 people contributed to writing the 79 systematic review articles. Only 17 people, or 5% of the authors, wrote or co-wrote more than one article. The authors were affiliated with 204 different organizations from 36 different countries. A total of 30 organizations, or 15%, contributed authors to at least two articles.
Among the SRs, 62 (78%) met the study criteria for reporting a literature search. Of those that did not meet the criteria, three reported at least one resource name but omitted search terms, while 14 reported neither a resource name nor search terms. Figure 3 compares the total number of published SRs to those reporting searches for each year of the study period.
Figure 3: Search reporting in systematic reviews published by conservation biology journals, 1998-2017
Discussion
There is a clear upward trend in both the number of systematic review articles being published and the number of conservation biology journals publishing them over the past 20 years. Systematic reviews are concentrated in a small number of journals, with the five most prolific journals accounting for 49% of the articles. The remaining 51% of the articles are spread across 23 journals, which suggests that while some journals may be more established venues for systematic reviews, many others are willing to consider them for publication.
In terms of impact factor, it is clear that within the Web of Science Biodiversity Conservation category the journals which publish systematic reviews are more influential as a group than those that do not. Not only are the mean and median 2017 impact factors higher for this group, the ranking of individual journals by impact factor shows that journals which publish systematic reviews are primarily found in the upper half of the field. This, along with the overall growth in journals publishing systematic reviews, indicates substantial progress in addressing the need for high-impact publication outlets, as identified by Pullin and Knight (2009) and Cook et al. (2013).
It is worth noting that conservation biology systematic reviews began to consistently appear in the journal literature after 2005, following the publication of several articles that explored the possible use of the methodology within the discipline (Pullin & Knight 2001, 2003; Fazey et al. 2004; Sutherland et al. 2004). It seems reasonable to infer that these initial articles impacted the trajectory of systematic reviews in the field, although other fields have also seen an increase in the use of this method (Gore & Jones 2015; Fournier & Ghezzi-Kopel 2016; Young and Eldermire 2017).
While authorship of conservation biology systematic reviews is widely distributed among individuals, it is less distributed when considering organizational affiliation. Very few researchers have authored more than one SR, but organizations have contributed authors to multiple SRs at a much higher rate. This suggests that expertise in conducting systematic reviews may be concentrated in certain organizations, but interest in exploring the methodology is widespread.
The majority of the SRs reported a search strategy, and the publishing pattern for these articles closely follows the trajectory of the SRs overall. Perhaps the most significant variation is that the first systematic review reporting a search strategy appeared in 2007 (as opposed to 2002 for the first systematic review overall), after the publication of Pullin and Stewart's seminal 2006 article, "Guidelines for Systematic Review in Conservation and Environmental Management" which has since been cited hundreds of times. Reporting of searches may also have been influenced by the Collaboration for Environmental Evidence's systematic review standards, which are currently in their fifth version (CEE 2018).
The collection of search reporting data revealed some additional insights, most notably that the meaning of "systematic review" varied, despite the existence of the disciplinary systematic review standards mentioned previously. One common alternate meaning: a study of systematics, or the classification and naming of an organism or group of organisms. Eight of the articles that did not report a search used the term this way. This finding corresponds with Cook et al. (2013), who briefly noted this phenomenon. In other SRs where no search was reported, it appeared that the authors used "systematic review" not to refer to a specific research methodology, but rather as a way of indicating that they undertook a measured, step-by-step, or otherwise carefully planned approach to a literature review. Heimlich (2014) and Gore and Jones (2015) mention similar conclusions. Still other articles seemed to refer to the research methodology described by guidelines both within and outside the discipline, but simply failed to adequately report a search.
While this study did not specifically examine the questions of search quality or repeatability, the reported searches contained numerous examples of ambiguous reporting and incomplete literature searches. For example, multiple articles searched in only one database or made no mention of a gray literature search. This agrees with the findings of studies of systematic review searching in other fields (Golder et al. 2008; Faggion et al. 2013; Toews 2017).
Limitations
The journals considered here are not the only ones that publish conservation biology studies. Some relevant journals may be categorized in the Web of Science Ecology category or other categories related to biology, or perhaps not indexed in Web of Science at all. Also, a single investigator decided whether the articles met the criteria at all stages of the study, and decisions were not checked against a second rater for reliability.
Conclusion
This study suggests that librarians may see increased opportunities for involvement in conservation biology systematic reviews due to more frequent use of this methodology and widely distributed authorship activity. Initial evidence indicates a role for librarians in educating researchers on appropriate search and reporting methods, clarifying general systematic review methodology, and referring researchers to systematic review standards and resources as needed.
As Toews (2017) notes, "the literature search is the data collection method in a systematic review and the search results form the evidence base," making a well-planned, performed, and documented search essential. Since a recent study of medical systematic reviews found significantly better literature searches when librarians co-wrote the reviews (Rethlefsen et al. 2015), an even larger role for librarians in conservation systematic reviews may be warranted.
Further research into the characteristics of reported searches could assist librarians in all of these roles by identifying common errors, weaknesses, and misunderstandings to address during consultations, instruction, outreach, or participation on systematic review teams.
Acknowledgements
This article expands upon material contained in "A Preliminary Exploration of Systematic Reviews in Conservation Biology," a poster which the author presented at the 2018 SLA Annual Conference in Baltimore, MD.
References
CEE (Collaboration for Environmental Evidence). 2018. Guidelines and standards for evidence synthesis in environmental management. Version 5.0. Pullin, A.S., Frampton, G.K., Livoreil, B., & Petrokofsky, G., editors. Bangor (UK): Collaboration for Environmental Evidence. Available from http://www.environmentalevidence.org/information-for-authors
Cook, C.N., Possingham, H.P., & Fuller, R.A. 2013. Contribution of systematic reviews to management decisions. Conservation Biology 27(5):902-915. DOI: 10.1111/cobi.12114
Faggion, C.M., Atieh, M.A., & Park, S. 2013. Search strategies in systematic reviews in periodontal and implant dentistry. Journal of Clinical Periodontology 40(9):883-888. DOI: 10.1111/jcpe.12132
Fazey, I., Salisbury, J.G., Lindenmayer, D.B., Maindonald, J., & Douglas, R. 2004. Can methods applied in medicine be used to summarize and disseminate conservation research? Environmental Conservation 31(3):190-198. DOI: 10.1017/S0376892904001560
Foster, M.J. and Jewell, S.T. 2017. Introduction to systematic reviews. In: Foster, M.J and Jewell, S.T., editors. Assembling the pieces of a systematic review: a guide for librarians. Lanham (MD): Rowman & Littlefield. p. 1-12.
Fournier, C. & Ghezzi-Kopel, K. 2016. "I want to do a systematic review!" Presentation at: Upstate New York Science Librarians Conference. Hamilton, NY. Available from https://surface.syr.edu/nyscilib/78
Golder, S., Loke, Y., & McIntosh, H.M. 2008. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology 61(5):440-448. DOI: 10.1016/j.jclinepi.2007.06.005
Gore, G.C. & Jones, J. 2015. Systematic reviews and librarians: a primer for managers. Partnership: The Canadian Journal of Library and Information Practice and Research 10(1). DOI: 10.21083/partnership.v10i1.3343
Gough, D., Oliver, S., and Thomas, J. 2017. Introducing systematic reviews. In: Gough, D. et al., editors. An introduction to systematic reviews. 2nd ed. Los Angeles (CA): SAGE. p. 1-17.
Haddaway, N.R., Land, M., & Macura, B. 2017. "A little learning is a dangerous thing": a call for better understanding of the term 'systematic review.' Environment International 99:356-360. DOI: 10.1016/j.envint.2016.12.020
Haddaway, N.R., Woodcock, P., Macura, B., & Collins, A. 2015. Making literature reviews more reliable through applications of lessons from systematic reviews. Conservation Biology 29(6):1596-1605. DOI: 10.1111/cobi.12541
Heimlich, S.L. 2014. New and emerging roles for medical librarians. Journal of Hospital Librarianship 14(1):24-32. DOI: 10.1080/15323269.2014.859995
InCites Journal Citation Reports help: rank [Internet]. n.d. Clarivate Analytics; [cited 2018 May 31]. Available from http://ipscience-help.thomsonreuters.com/incitesLiveJCR/JCRGroup/jcrJournalProfile/jcrJournalProfileRank.html
Lefebvre, C., Manheimer, E., and Glanville, J. 2011. Chapter 6: searching for studies. In: Higgins, J.P.T., and Green, S., editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0. London (UK): The Cochrane Collaboration. Available from http://www.handbook.cochrane.org
McKibbon, A. 2006. Systematic reviews and librarians. Library Trends 55(1):202-215. DOI: 10.1353/lib.2006.0049
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group. 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 6(7). DOI: 10.1371/journal.pmed.1000097
Pullin, A.S. & Knight, T.M. 2001. Effectiveness in conservation practice: pointers from medicine and public health. Conservation Biology 15(1):50-54. DOI: 10.1111/j.1523-1739.2001.99499.x
Pullin, A.S. & Knight, T.M. 2003. Support for decision making in conservation practice: an evidence-based approach. Journal for Nature Conservation 11(2):83-90. DOI: 10.1078/1617-1381-00040
Pullin, A.S. & Knight, T.M. 2009. Doing more good than harm - building an evidence-base for conservation and environmental management. Biological Conservation 142(5):931-934. DOI: 10.1016/j.biocon.2009.01.010
Pullin, A.S. & Stewart, G.B. 2006. Guidelines for systematic review in conservation and environmental management. Conservation Biology 20(6):1647-1656. DOI: 10.1111/j.1523-1739.2006.00485.x
Rethlefsen, M.L., Farrell, A.M., Osterhaus Trzasko, L.C., & Brigham, T.J. 2015. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology 68(6):617-626. DOI: 10.1016/j.jclinepi.2014.11.025
Scope notes: Science Citation Index Expanded [Internet]. c2017. Clarivate Analytics; [cited 2018 Jun 1]. Available from http://mjl.clarivate.com/scope/scope_scie/
Spencer, A.J. & Eldredge, J.D. 2018. Roles for librarians in systematic reviews: a scoping review. Journal of the Medical Library Association 106(1):46-56. DOI: 10.5195/jmla.2018.82
Sutherland, W.J., Pullin, A.S., Dolman, P.M., & Knight, T.M. 2004. The need for evidence-based conservation. Trends in Ecology and Evolution 19(6):305-308. DOI: 10.1016/j.tree.2004.03.018
Toews, L.C. 2017. Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association 105(3):233-239. DOI: 10.5195/jmla.2017.246
Web of Science Core Collection help [Internet]. c2018. Clarivate Analytics; [cited 2018 May 31]. Available from https://images.webofknowledge.com/images/help/WOS/hp_subject_category_terms_tasca.html
Young, S. and Eldermire, E. 2017. The big picture: finding, evaluating, and applying systematic reviews across disciplines. In: Foster, M.J and Jewell, S.T., editors. Assembling the pieces of a systematic review: a guide for librarians. Lanham (MD): Rowman & Littlefield. p. 13-29.
Issues in Science and Technology Librarianship No. 91, Spring 2019.