![[Refereed]](http://www.istl.org/graphics/r2.gif){kind=small}
What Do Engineering Researchers Cite? A Citation Analysis Study of Sixteen Engineering Journals
University of Mississippi Libraries
University of Mississippi
University, Mississippi
bwyoung@olemiss.edu
Abstract
This paper focuses on the citation analysis of all research articles from one issue of 16 engineering journals from different engineering disciplines and captures data on the formats cited and the age (median and mean) of those citations. In addition to contextualizing the data by comparing to other engineering citation analyses, the study also replicates (both data source and variables) a 1996 study so that longitudinal comparisons may also be made. Implications of the results on engineering collection development are discussed.
Introduction
It is well known that the resources used by each discipline can greatly differ; specifically, humanities-based research largely uses monographs, while science and engineering-based research depends largely on journals. What is less commonly known is how dependent on particular formats each discipline, or sub-discipline, is or whether certain sub-disciplines differ greatly from the overall group.
While varying methods exist to collect data to answer these questions, the most commonly used is citation analysis. Citation analysis offers an unobtrusive method for studying the scholarly communication process of researchers in a specific field and of libraries' own patrons. Citation studies use references from a particular collection of works (journals, theses, dissertations, etc.) as their data set, and then researchers code the data based on the study's emphasis. Data may include source type, citation age, citations per publication, source subject area (call number), publishers, language, as well as other variables (Hoffmann and Doucette 2012).
Two forms of citation studies exist: local and discipline-wide studies. Local studies focus only on the behavior of researchers at a particular institution, whereas discipline-wide studies focus on a specific discipline, such as engineering or history, and analyze a designated corpus from that area. This article reports on a discipline-wide study in engineering and uses the first issue of sixteen engineering journals published in 2012 as its corpus. By analyzing this specific corpus, the study provides an additional facet not always found in citation studies--a longitudinal look at engineering citations achieved by comparing this study's data to a 1996 citation study using the same journals.
Literature Review
Citation studies have long been used as an unobtrusive method for aggregating data on the information seeking behavior of researchers. Gross and Gross (1927) conducted a citation study to evaluate periodicals in an effort to identify a core title list for chemistry collections and similar studies have been conducted by collection development librarians to evaluate holdings and create core lists for their libraries based on local researcher needs (Kayongo and Helm 2012). Local citation study results, which analyze use at a specific institution by that institution's patrons, though, cannot be generalized to larger populations. Differences in methods, including varying time frames and data sources, and variables reported, such as format, citation age, number of citations per item, and library variability further complicate comparing citation studies to each other (Hoffmann and Doucette 2011).
Some researchers have also raised valid concerns regarding citation analysis. Beile, Boote, and Killingsworth (2004) caution against using graduate student citations in theses and dissertations as a proxy for faculty research behavior. Further, MacRoberts and MacRoberts (1989; 2010) note that citation studies fail to capture the articles that, while not cited, were influential in the research process. When used to guide, and not define, collection management strategies, citation analyses can be useful tools for generating core lists and gaining perspective on research behavior.
While core lists can be beneficial to institutions as a guideline for titles to purchase and/or retain, the study presented here does not focus on this aspect of citation studies. Instead, it examines the scholarly communication process by examining the citations made by researchers. Further, it provides a longitudinal analysis using Musser and Conkling's (1996) study as an initial baseline. In their study, percent use of various formats and citation age were the two primary variables investigated.
Musser and Conkling observed that journals constituted a slim majority of citations (53%) with other formats, like conference papers (19%) and monographs (12%), accounting for the remaining citations. Rather than journals, Larivière et al. (2006) instead used the SCI index and found journal citations in engineering increased from 60% in 1981 to 68% in 2000. While no other global citation studies in engineering could be located, many local citations studies have been published. In one such example, Eckel (2009) analyzed the citations from Western Michigan engineering theses and dissertations. Extrapolating on the data presented in the study, student researchers' three most used formats were scholarly journals (37%), monographs (19%), and conference papers (17%). Bierman's (2012) longitudinal study looked at theses at the University of Oklahoma published in 1991 and 2011 and, while it revealed shifts in percent use, journals, monographs, and conference papers were also the three most used content formats in both 1991 and 2011 theses. Additionally, other researchers have reported data for their local institutions with similar results (Kayongo and Helm 2012; Williams and Fletcher 2006).
Echoing the concerns of Beile et al., the citations, by format, found in graduate students' theses and dissertations appears to differ from those found in journal articles. In contrast to the two previously mentioned studies that focused on journal articles, several studies that focused on theses and dissertations by engineering graduate students found journal article citations to be less than 50% (Bierman 2012; Eckel 2009; Kriz 1977; Williams and Fletcher 2006).
When using citation analysis as a means to understand collection use and make collection management decisions, citation age is as important to consider as percent use by format. The latter can offer insight as to where researchers typically cite and, thus, where to allocate funds when making purchasing decisions. The former is more important when developing policies regarding format retention, specifically journals and conference proceedings. Unfortunately, this measurement has been more neglected than percent use in the engineering citation studies. Additionally, how it is presented (mean/average vs. median/half-life) is not consistent, preventing direct comparisons across all studies, even when the variable is captured. Studies have found the citation half-life is 7-8 years (Sjoberg 2010; Williams and Fletcher 2006; Musser and Conkling 1996) and the average citation age is 8-12 years (Kayongo and Helm 2012; Sjoberg 2010).
Methods
Hoffmann and Doucette (2012) emphasize the importance of modeling citation studies on previously conducted research, in part, to improve comparability across studies. This study seeks to apply the same methods used by Musser and Conkling (1996) in an effort to present a longitudinal analysis of engineering citations in scholarly research. The present study primarily analyzes the current state of engineering citation behavior and, secondarily, looks at changes in citation behavior over the sixteen years.
Research articles (i.e., not editorials, technical notes, etc.) from the first 2012 issue from the same 16 journals (Table 1) used in Musser and Conkling's study were used as the data source, with each citation's format and age being documented. Musser and Conkling originally selected these journals for several reasons. By selecting one journal from each field, their study could cover the breadth of engineering fields and limiting to only U.S based society journals ensured a degree of journal prestige while also minimizing variability by isolating the data to a specific region. They then chose specific titles based on their own and their engineering faculty's expertise.
To maintain consistency, items were classified based on the prior study's format categories: conference paper, journal article, monograph, doctoral dissertation, master's thesis, standard, patent, software/software manual, technical report, product literature, unpublished material, and unknown. An additional category for web sites was created to account for the technological development and increasing acceptance by researchers since the original study was published. Like the original study, formats were categorized based on the original nature of the item (e.g., conference papers include items published in conference proceedings).
| Aeronautical Engineering | AIAA Journal |
Agricultural Engineering | Applied Engineering in Agriculture |
Architectural Engineering | Journal of Structural Engineering |
Bioengineering | Journal of Biomechanical Engineering |
Chemical Engineering | AIChE Journal |
Civil Engineering | Journal of Construction Engineering and Management |
Electrical Engineering | Proceedings of the IEEE |
Environmental Engineering | Water Environment Research |
Experimental Mechanics/Engineering | Experimental Mechanics |
Industrial/Manufacturing Engineering | IIE Transactions |
Materials Engineering | Journal of Composite Materials |
Mechanical Engineering | Journal of Heat Transfer |
Mining Engineering | Mining Engineering |
Naval/Marine Engineering | Journal of Ship Research |
Nuclear Engineering | Nuclear Technology |
Petroleum Engineering | SPE Drilling & Completion |
Table 1: Journals used in current study and Musser and Conkling (1996)
Results
Percent Use by Format
The author reviewed 6,229 references from 199 articles. Percent use by material type was calculated based on the total citations, rather than averaging the percent use for each journal in the study. For example, if monographs were 10% of all citations in Journal A and 20% of all citations in Journal B, then the percent use will not necessarily be 15%, as it will depend on the number of citations from each journal. For this specific study, normalization does not have a major impact on any calculation (within 2.5 percentage points), but attention must be given when comparing normalized data to unnormalized data.
As seen in other studies (see Table 2), journal articles were the most often cited format, accounting for 63% of all citations -- around 19% greater than the comparison study. The high percentage, though, approaches the 68% for engineering articles found by Larivière et al. (2006) in their review of publications in the Science Citation Index (SCI) in 2000. Local citation studies of engineering dissertations and theses, however, have found journal use to range from 33% to 73%. While having a much lower percentage in some instances, journal articles were still the most used resource by graduate students in most studies. The difference in resource use by engineering researchers and by engineering graduate students warrants further consideration but is outside of the scope of this study.
Corpus | Journals | Monographs | Conference Papers | Other | |
|---|---|---|---|---|---|
Young (current study) | Journals | 63% | 11% | 14% | 11% |
Bierman (2012) – 2011 data | Theses | 50% | 15% | 20% | 15% |
Kayongo and Helm (2012) – normalized | Dissertations | 73% | 13% | 7% | 7% |
Kayongo and Helm (2012) – unnormalized | Dissertations | 64% | 14% | 11% | 11% |
Eckel (2009) | D&T | 37% | 19% | 17% | 27% |
Larivière et al. (2006) | Journals | 68% | n/a | n/a | n/a |
Williams and Fletcher (2006) | Theses | 38% | 18% | 19% | 25% |
Musser and Conkling (1996) | Journals | 53% | 12% | 19% | 16% |
Bierman (2012) – 1991 data | Theses | 46% | 22% | 13% | 19% |
Kriz (1977) | Theses | 33% | n/a | n/a | n/a |
Table 2: Summary of citation data in literature review articles
Heavy use of journal articles is not necessarily widespread across all engineering fields. Journals accounted for less than half of all citations in four fields: architectural engineering (50%), electrical engineering (43%), nuclear engineering (35%), and petroleum engineering (28%). Across the disciplines, conference paper use ranged from 2% to 53% and monograph use ranged from 4% to 24%. In-depth studies for each engineering discipline, such as the study conducted by Musser (2007), would be required to determine whether the observed frequency is representative of the entire discipline. The longitudinal aspect of the study suggests a gradual shift away from non-serial formats to serials: researchers have cited journals around 19% more often in 2012 than in 1996.
Citation Age
Collection management librarians can leverage knowledge about the citation age of each format in their subject area when developing retention policies or analyzing local user data. The median citation ages (half-life) for each material followed the same trend observed by Musser and Conkling (1996): monographs have the longest half-life and conference papers have the shortest. Further, while most categories have increased slightly, the 2012 data are largely consistent with the 1994 data (see Table 3).
| 50% of all citations | 75% of all citations | 90% of all citations | Average Citation Age |
|---|---|---|---|---|
Journal articles | 9 years (+1) | 16 years (0) | 31 years (+6) | 13 years |
Conference papers | 6 years (+1) | 11 years (+2) | 17 years (+1) | 8 years |
Monographs | 13 years (+2) | 22 years (+3) | 36 years (+6) | 17 years |
Technical reports | 10 years (+3) | 18 years (+2) | 46 years (+14) | 15 years |
All formats | 9 years (+2) | 16 years (+1) | 31 years (+6) | 13 years |
Table 3: Citation age data (with differences from prior study noted parenthetically)
Despite the presence of many older works, the half-life for all formats was only nine years. The half-life is not only close to Musser and Conkling's (1996) findings but also close to the 7-8 year range found in local citation studies. Across disciplines, the half-life ranges from 3 to 13 years. Additionally, the average citation is 13 years old, slightly outside the 8-12 year range found in local citation studies. The discrepancy between the median and mean ages supports the idea that while engineering researchers cite older materials (13 citations older than 100 years in this study), the overwhelming majority are much newer.
Discussion
It is generally accepted that science and engineering research predominantly relies on serials. As shown in this study and others focused on journal article citations (Larivière et al. 2006; Musser and Conkling 1996), it is true that engineering researchers rely more heavily on serials publications than non-serials and their reliance on the journal literature appears to be increasing.
Journal use from 1996 to 2012 increased by roughly 19% (10 percentage points). A more qualitative study or in-depth look at the journal articles in the present corpus would be needed to address the reason why this possible shift has occurred, but one speculative reason is the increase in journal article access—specifically electronic access. Through the migration of journals from print to electronic, development of tools like Google Scholar, and the creation of institutional and subject-based repositories, the journal literature is more readily available than in the past.
Despite the increase in journal use, monographs were only used marginally less than in 1996. Additionally, 11% is smaller than the percent use of any engineering citation study reviewed. The median (13 years) and third quartile (22 years) suggests that collection development librarians would need circulation data from 15-20 years to have an accurate picture of their collection's use. Researchers' consistent use of monographs indicates that monographs may not be the most-used format but still remain a relevant resource for engineering research. It also suggests that engineering monograph budgets, to some degree, should be preserved despite increasing pressures from the price increases of engineering journals and databases.
In some engineering disciplines, conference proceedings should also be actively collected or have their existing budgets maintained. Based on the limited discipline-specific data, electrical and petroleum engineering researchers are the only groups who do not primarily cite other journal articles; conference papers, instead, were the most cited resource format for those disciplines. Although further investigation into each discipline is needed to confirm this finding, it would suggest that engineering librarians may choose to protect or subscribe to conference proceedings packages (e.g, IEEE conference proceedings for electrical engineering) or discipline specific databases (OnePetro for petroleum engineering) at the expense of seldom used or high cost per use journals.
Conference proceedings are generally consulted to review recent research; older conference proceedings are not necessarily as needed by engineering researchers. Only 10% of all conference proceeding citations were older than 17 years. Similarly, all other formats experience 75% of their use within two decades. Looking at the data longitudinally, this has remained consistent in the engineering literature.
Conclusion
This study confirms both anecdotal thoughts and previous findings that engineering researchers rely heavily on the journal literature when writing their own journal articles. It also suggests that this reliance on journal literature is becoming more prominent. Future discipline-wide citation studies should isolate sub-disciplines to determine not only the use of each format but also whether shifts in format use has occurred over time. Qualitative methods could be employed to investigate the reasons why faculty in each area use the format they do and why any shifts have occurred.
This study also confirms the citation age findings from Musser and Conkling's 1996 study as each category (50%, 75%, and 90%) had relatively small changes for all but one item (90% of all citations for technical reports). This information can be beneficial to engineering collection managers who are required to make decisions regarding their collections and the space required for storage. Although a holistic citation study, such as this one, cannot replace the more nuanced data of a local citation study, it can be used as a substitute when time or available data is a limiting factor.
The advantage of holistic citation studies is the generalizability of the data, which can then be used in conjunction with other research projects to better inform collection management decisions and future scholarship. One such example would be using the information regarding citation age in combination with studies that focus on database coverage (e.g., Meier and Conkling's Google Scholar study) to gauge how much impact a database's coverage from earlier decades or particular formats may have on researchers.
Based on many studies, including this one, journal articles remain the primary resource for engineering researchers. Future research into the scholarly communication process should include differences between engineering graduate students and engineering researchers, further investigation of the scholarly communication process for specific fields in engineering, and further investigation into which type (handbook, textbook, etc.) of monograph engineering researchers, at all levels, use, as well as the currency of each type. The latter, specifically, can help guide future engineering collection development.
References
Beile, P.M., Boote, D.N., & Killingsworth, E.K. 2004. A microscope or a mirror? A question of study validity regarding the use of dissertation citation analysis for evaluating research collections. The Journal of Academic Librarianship, 30(5): 347-353.
Bierman, J. 2012. A citation study of engineering masters theses at the University of Oklahoma: Comparing the years 1991 and 2011. Science & Technology Libraries, 31(4): 412-425.
Eckel, E.J. 2009. The emerging engineering scholar: A citation analysis of theses and dissertations at Western Michigan University. Issues in Science and Technology Librarianship 56. [Internet]. [Cited 11/29/13]. Available from http://www.istl.org/09-winter/refereed.html.
Gross, P.L.K. & Gross, E.M. 1927. College libraries and chemical education. Science, 66(1713): 385-389.
Hoffmann, K. & Doucette, L. 2012. A review of citation analysis methodologies for collection management. College & Research Libraries, 73(4), 321-335.
Kayongo, J. & Helm, C. 2012. Relevance of library collections for graduate student research: A citation analysis study of doctoral dissertations at Notre Dame. College & Research Libraries, 73(1): 47-67.
Kriz, H. 1977. Citation counting and the future of engineering libraries. Engineering Education, 67(7): 707-710.
Larivière, V., Archambault, É., Gingras, Y., & Vignola-Gagné, É. 2006. The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities. Journal of the American Society for Information Science & Technology, 57(8), 997-1004.
Meier, J. & Conkling, T.W. 2008. Google Scholar's coverage of the engineering literature: An empirical study. The Journal of Academic Librarianship, 34(3): 196-201.
Musser, L.R. & Conkling, T.W. 1996. Characteristics of engineering citations. Science & Technology Libraries, 15(4): 41-49.
Musser, L. 2007. A study of references in mining engineering publications. Issues in Science and Technology Librarianship 49. [Internet]. [Cited 11/29/13] Available from http://www.istl.org/07-winter/article3.html
MacRoberts, M.H., & MacRoberts, B.R. 1989. Problems of citation analysis: A critical review. Journal of the American Society for Information Science 40(5): 342-349.
MacRoberts, M.H., & MacRoberts, B.R. 2010. Problems of citation analysis: A study of uncited and seldom-cited influences. Journal of the American Society for Information Science & Technology 61(1): 1-12.
Sjoberg, D.I.K. 2010. Confronting the myth of rapid obsolescence in computing research. Communications of the ACM, 53(9): 62-67.
Williams, V.K. & Fletcher, C.L. 2006. Materials used by master's students in engineering and implications for collection development: A citation analysis. Issues in Science and Technology Librarianship 45. [Internet]. [Cited 11/29/13]. Available from http://www.istl.org/06-winter/refereed1.html
This work is licensed under a Creative Commons Attribution 4.0 International License. 