Citing of Industry Standards in Scholarly Publications

Erin M. Rowley
Head, Science and Engineering Library Services / Engineering Librarian
Lockwood Library
University at Buffalo
Buffalo, New York 14260
epautler@buffalo.edu

A. Ben Wagner
Retired Sciences Librarian
Lockwood Library
University at Buffalo
Buffalo, New York 14260
abwagner@buffalo.edu

Technical or industry standards and specifications are used across many disciplines and impact everyday life in countless and often unseen ways. They are created and designed for a variety of reasons, including to protect the health and safety of consumers, establish best practices, and facilitate trade. However, per our own experiences, many people, including some librarians, are not aware of what a standard is or how it is developed. A single definition of the term “standard” does not exist. Different organizations define the term in their own way. The European Committee for Standardization (CEN) defines a standard to be a technical document designed to be used as a rule, guideline, or definition. It is a consensus-built, repeatable way of doing something (European Committee for Standardization (CEN) n.d.). Or, as Phillips and Huber so aptly and simply state, standards “communicate agreement” (Phillips & Huber 2017).

Per Phillips and Huber (2017), standards are created by standards development organizations (standards organizations). Standards organizations can be professional societies, such as the American Society of Mechanical Engineers (ASME), national standards bodies (NSBs), such as the Standards Administration of China (SAC), or international standards organizations, such as the International Organization for Standardization (ISO) or International Electrotechnical Commission (IEC). ISO/IEC (2004) defines a standards body as a "...body recognized at national, regional, or international level, that has as a principal function, by virtue of its statutes, the preparation, approval, or adoption of standards that are made available to the public."

However, other organizations can create standards outside of these groups. For example, automotive manufacturers such as Ford Motor Company, produce their own internal test standards. These standards are typically not available publically or for purchase.

The standards produced by these organizations are largely dependent on the scope of the group. Hence, some standards organizations are more far reaching than others in terms of products, materials, and processes covered in the standards they produce. The development of standards is a rigorous process. Committees of experts in the field under examination follow formalized procedures to produce a standard. In most cases, the creation and any future updates of standards are subject to a voting process before publication (Batik 1992). Similar to other publications, standards are copyrighted documents and, in most cases, can only be obtained by purchasing the document or subscribing to an online full-text database. However, some standards are freely available, such as those through the ANSI IBR (incorporated by reference) Standards Portal at https://ibr.ansi.org/ or standards from the National Operating Committee on Standards for Athletic Equipment (NOCSAE).

Standards aim to be best practices in their particular areas, and in some cases, may even be mandatory under the law. The safety of work environments, the manufacture of certain consumer products, and the disposal of specific types of waste are all examples of areas where technical standards and specifications are mandatory in the United States (U.S.), per federal, state, and municipal level regulations. However, standards can sometimes pose an issue for library collections. Many standards can be expensive, especially in the context of a library budget. Further, standards frequently reference other standards which results in additional documents that need to be obtained and reviewed (Thompson 2001; Mathews 2006).

While standards are widely used in science, engineering, and technology fields, especially by practitioners, we were unaware of studies of how they are cited in the scholarly literature. This led us to two questions: 1) How frequently are standards being cited in scholarly publications; and, 2) What are the characteristics of the publications citing standards? For purposes of this study, characteristics was defined to mean the disciplines of the journals where the scholarly publications are published, the country or countries where the authors are located, the institutional affiliation of the authors, the funding agencies of the scholarly publications, the subject areas of standards, and open access details.

In answering these two questions, we predicted this information would be most useful to those in science, technology, engineering, and mathematics (STEM) librarianship roles and could provide clear data to verify where standards are used and needed most frequently. We anticipate this data could impact decisions regarding collections and services for faculty and students of academic institutions in science, engineering, and technology fields.

We conducted a study of publications that cite standards. Three major standards organizations – ASTM International (formerly known as the American Society for Testing & Materials), the International Organization for Standardization (ISO), and the European Committee for Standardization (CEN) – were selected for this study, as each one develops and maintains standards across many disciplines. In addition, as this study will show, standards for these organizations are recognized and used throughout the world.

Heilbrun et al. (2016) examined the citation of professional standards from organizations, such as the American Psychological Association (APA) and the American Bar Association (ABA), in law and behavioral science publications. The study aimed to understand how these professional standards were used in scholarly and professional literature, and thereby gauge the level of awareness and knowledge about these standards. Given the effort needed to develop standards and the key role they play in the practice of the professions, one might expect to see this reflected in citation rates in the scholarly publications. However, as these authors noted, it can be difficult to judge this impact. The study hypothesized that a high citation rate for these standards could equate to a high influence on the profession. Following the analysis, the study found that the converse was true; the citation rate for the professional standards studied was low, leading to the conclusion that increased awareness was needed for these standards to be seen as “relevant sources of authority” (Heilbrun et al. 2016).

Numerous citation analyses of science and engineering publications have been conducted, but few analyses have specifically mentioned standards in the results. Curtis (2011), Musser & Conkling (1996), and Zhang (2018) are some of the few citation analyses focusing on journal literature that specifically reported on standards cited. The results from these studies reveal a common thread. Each study found only a small number of publications citing standards. Zhang (2018) investigated the citation and collaboration characteristics of engineering faculty in eight engineering departments at Mississippi State University. Of the 734 publications that were examined by Zhang, roughly 1% of total works cited were standards. Curtis (2011) found that only 1.8% of citations were to standards in the civil engineering journals examined. Musser and Conkling (1996) also found that 1% of citations were to standards when studying articles from sixteen engineering journals.

Previous citation analyses involving science and engineering publications often focus on the scholarly output of a given institution, discipline, or journals, whereas approaching it holistically from cited-reference search, one finds little published research conducted in general and nothing focusing on standards in particular. Using the Cited Reference Search feature of the Web of Science Core Collection^TM (WOSCC), as we did for this study, is another method for studying citations of various information sources in science and engineering fields. Tomaszewski (2017, 2018) and Tomaszewski and MacDonald (2016) explore using the Cited Reference Search tool in WOSCC to study citations of chemical resources, chemical encyclopedias, and Wikipedia citations, respectively. In another study, Cardona & Marx (2007) used the Cited Reference Search to study citations to papers from a particular international conference.

Others have written about how to develop standards collections in academic libraries (Taylor 1999) and how libraries provide access to standards (Pellack 2005). Based on previous work (Curtis 2011; Musser & Conkling 1996; Zhang 2018), we expected that standards are cited in scholarly publications; however, one central question is whether the analysis could reveal which disciplines are most frequently citing standards and therefore validate the need for these documents as part of an academic library collection.

During the weeks of December 11 and December 18, 2017, searches were performed on the WOSCC for all 2015-2017 publications that cited standards from three prominent standards organizations:

• ASTM International (West Conshohocken, PA) – ASTM Standards
• International Organization for Standardization (Geneva, Switzerland) – ISO Standards
• European Committee for Standardization (Brussels, Belgium) – CEN/EN Standards

Our subscription to the WOSCC includes journal articles, conference papers, books, and book chapters covered in the following database components, all registered trademarks of Clarivate Analytics:

• Science Citation Index Expanded
• Social Sciences Citation Index
• Arts & Humanities Citation Index
• Conference Proceedings Citation Index- Science
• Conference Proceedings Citation Index- Social Sciences & Humanities
• Book Citation Index– Science
• Book Citation Index– Social Sciences & Humanities
• Emerging Sources Citation Index

The WOSCC was chosen because of the unique abilities to search and capture the raw citation data, even for bibliographic citations that cannot be linked to citing publications indexed in the database. As standards are not directly indexed in WOSCC, citations to standards can only be captured via the Cited Reference Search, which displays the cited references as found and parsed by WOSCC algorithms. This search query, in a sense, is misnamed, as the cited references are the starting point for the search, but the search results are the citing papers.

Looking at the raw citation data, we found that there is no consistency to the way scholars cite standards. The organization’s name was spelled out in full and abbreviated in various ways. Even the acronyms showed variations in punctuation and spacing. Some of the variants may be the result of the WOSCC citation parsing algorithms rather than from the citation in the original source. These variations required us to conduct a long series of searches to ensure the most comprehensive results possible. Standards organizations and acronyms were searched in both the Cited Author and Cited Work fields. The exact search queries used are shown below. To account for variations in the standards organizations, the WOSCC truncation symbol, the asterisk, was used in these queries.

• ASTM variants: ASTM, A S T M, Amer Soc Test*, American Society for Test*
• ISO variants: ISO, ISO*, International Standard*, International Organi*, Organizacion Internacional de Normalizacion
• CEN variants: CEN, CEN*, EN, European Standard, EN Standard, European Committee*

Although there was a high degree of overlap between the search results from the Cited Author and the Cited Work fields, there were unique items found only by searching both fields.

The Cited Reference Search does not directly produce a search set of citing papers, but rather an extensive table listing each individual cited reference, as shown in Figure 1. Even this small excerpt shows many variations in format for the cited standards.

At the time the search was conducted, up to 500 cited references (rows in the results table) could be selected at one time. If a particular set of results exceeded 500 rows, even when limited year by year, the first 500 results were selected and searched, then the search was regenerated and the next 500 were selected. As explained later in this section, the resulting citing papers were de-duped by using the WOSCC Marked List feature and EndNote™ citation management software.

After selecting rows from the cited reference search, a Finish Search button was clicked to retrieve the citing journal articles, conference papers, books, and book chapters indexed in the WOSCC. Due to system limits, each standards organization was searched separately, year by year. All citing items were transferred to a Marked List, a temporary folder of results that can be added to in subsequent searches in the same search session. This allowed the creation of a set of all the unique citing papers across all the name variants and years for each of the three standards organizations. By using the ability to save Marked Lists, a master set of results merging the citing papers for all years and all standards organizations was created.

Data analysis was done using the WOSCC built-in Analyze Results feature that generates a frequency histogram on 16 available data fields. Six fields were examined for this study: Web of Science Categories, Document Type, Country, Region, Organizations, and Funding Agency.

Analyses were run separately on the citing papers for each of the three standards organizations and then on the combined unique set of citing papers across all organizations. Each analysis was downloaded to a Microsoft Excel™ spreadsheet, from which the various graphs and figures were produced for this article. The fields analyzed are defined in the Results section of this article.

In addition to these analyses of fields in the citing-papers records, we also captured, via a copy and paste mechanism into Excel, the raw citation data of the ASTM standards (each row of the Cited Reference Search results page). Of the three standards organizations, only the ASTM standards numbering scheme uses a letter designation (A-G) to reflect a broad subject area, e.g., A designates Ferrous Metals (ASTM International 2013). ISO and CEN numbers have no inherent meaning. By capturing the cited ASTM standard numbers, we were able to analyze the distribution of cited ASTM standards across the seven broad subject areas.

The number of open-access publications was also determined. Since the Analyze Results feature did not provide an option for open access, data was gathered in the general search results screen of WOSCC and determined using the open-access filter in WOSCC. It should be noted that the WOSCC open-access filter includes several types of open-access publications (e.g., bronze, gold, and green).

As described in the Methods section, we used the Analyze Results feature to look at six data fields, independently. In total, 5,214 publications were retrieved from our search. Of these, the total number of publications citing ASTM, ISO, and CEN standards are 3002, 2023, and 189, respectively.

Web of Science Categories Analysis

Each journal in the Web of Science (WOS) is assigned to one or more of 252 WOS subject categories, e.g., Engineering Chemical and Energy Fuels. These are the same subject categories used in Clarivate’s Journal Citation Reports™ database. By extension, these subject categories are assigned to all publications appearing in a given journal. Table 1 shows the top ten WOS subject categories associated with the publications that cited standards for each of the three standards organizations. There were four categories ranked in the top ten of all four datasets: Materials Science Multidisciplinary, Engineering Civil, Environmental Sciences, and Engineering Mechanical.

Not surprisingly, the WOS subject categories shown in Table 1 are as diverse as the universe of standards are, though focused heavily on engineering and materials science disciplines. Only the Chemistry Analytical category, which ranked eighth for ISO citing papers, falls outside these two areas. Europe’s strong commitment to sustainable technologies is shown by Green Sustainable Science Technology being ranked in the top ten only for CEN citing papers. This same emphasis might possibly be reflected in the Materials Science Paper Wood category ranked eighth by CEN exclusively. The growing research in materials science is seen in a number of categories with the Materials Science Multidisciplinary category being ranked in the top five in all three datasets, the only category to hold that distinction. Three computer science categories were ranked in the top ten by ISO exclusively. Although ASTM and CEN have a significant amount of computer science-related standards, ISO computer science standards are heavily cited in the scholarly publications. As some standards are approved by multiple standards organizations (e.g., EN ISO/IEC 19762-1:2012), it is possible the authors of publications citing standards only cited the ISO portion. Of the 252 WOS subject areas, 175 areas were coded to at least two citing papers. This again shows the diverse disciplines making use of standards.

Document Type Analysis

The distribution of document types for items citing standards closely aligns with the overall distribution of materials within the total WOSCC database (http://clarivate.libguides.com/webofscienceplatform/woscc) for the 2015-2017 time period studied. Hence, it is no surprise that journal articles are by far the most common document type citing standards, as WOSCC is predominantly a journal database. However, it is interesting to note the diversity of document types that cite standards, with a total of 10 different types. The top five document types were Article (77%), Proceedings (16%), Review (4%), Book/Book Chapters (2%), Editorial Material (1%), with all others collectively less than 0.6%.

Analysis of Top Eleven Countries

The citing publications were analyzed by the country field. The data in this field is drawn from the address field for all authors. This yielded a list of 97 countries that had at least two citing publications attributed to an author from that country. Since the WOSCC includes the address with country information for all listed authors, not just the primary author, a single citing article may be coded to more than one country. The country analysis was split into two separate analyses. In this section, only data for the top 11 countries is presented, those with a total number of citing publications equal to or greater than 200 (Figure 2). In the following section, a separate regional analysis of only the 86 smaller countries is given.

The top 11 countries were led by the U.S., unsurprisingly, with more than 1,200 citing publications having at least one U.S.-affiliated author. A large gap separated the U.S. from the next country in the results, the People’s Republic of China, with just under 450 citing publications. The large gap between the two countries could be explained by the standards organizations studied. For this publication only ASTM, ISO, and CEN standards were included in the search, whereas China has its own standards organization, the Standardization Administration of the People's Republic of China (SAC). While SAC standards (most commonly cited as “GB” or “GB/T” standards) were not included in this study, they are found in WOSCC as cited documents. Had these standards been included in this study, perhaps the number of citing publications with authors from China would greatly increase.

Not surprisingly, countries with large economies take the top positions. Countries included in the top 11, beyond the U.S. and China, included Brazil and India. These latter countries are part of the “BRIC” countries, which are deemed to be at a similar stage of newly advanced economic development (O'Neill 2001). BRIC refers to Brazil, Russia, India, and China, and, interestingly, the only member of the BRIC countries not included in the top 11 is Russia. However, in some circles, the “BRIC” acronym has been extended to “BRICK,” where “K” refers to South Korea, which just made it into the top 11 countries in our study.

It should be noted, there are different iterations of the “BRIC countries” that include other nations such as Mexico or South Africa, neither of which placed in the top 25 countries. Therefore, it could be argued that it is not particularly significant that several of the BRIC/ BRICK countries are included among the top countries. Like China, Russia also maintains their own national standards, GOST (sometimes referred to as GOST R). Russian standards are rarely equivalent or identical to standards from other large international standards organizations like those analyzed in this study. Therefore, it is possible that the reason Russia is not among the top 11 countries is that Russian authors are more likely to cite GOST standards rather than those from ASTM, ISO, or CEN.

Four of the remaining countries in the top 11 are Germany, England, Italy, and Spain, all member states in the European Union (EU), although the recent decision by the United Kingdom to exit the EU is well known. The EU is the region responsible for creating CEN standards, therefore these countries would be expected to be heavy users of CEN standards. According to Eurostat data (Europa 2017), these four countries plus France, which ranked 12th, represent the top five countries by gross domestic product (GDP) in the EU, which reinforces the theory that the largest economies would be well represented in the overall country analysis. Canada and Australia round out the top 11. With strong economies and GDP’s in the top 15 in the world (World Bank 2018), their appearance in the analysis aligns with our postulation.

Regional Analysis of Remaining Countries

The top eleven countries shown in Figure 2 show the dominance of North America, China, and Europe, overall. In order to gauge the regional distribution of the remaining “long tail” of 86 countries (2 to 199 citing publications), a separate regional analysis was run that excluded data from the 11 major countries already presented in the previous section. The 86 smaller countries were grouped and analyzed by six regions: Africa, East Asia/Pacific, Europe, Mexico and Central America, the Middle East, and South America (Figure 3). Grouping countries by region is somewhat subjective. In some cases, a region is one portion of a continent, while others include countries from multiple continents. A complete breakdown of each country and its designated region can be found in the Appendix.

Once again, one geographic area cites standards more frequently than the others. In this instance, Europe cited far more than the other regions, with more than 1,700 citing publications having at least one author from the area. Note again that, if a single citing article had authors from more than one European country, each country would be counted.

One of the major takeaways when examining the data by region was the sheer international reach standards exhibit. With the exception of Antarctica, every continent was represented in our analysis of just three standards organizations over three years. Each region had a minimum of five individual countries represented.

We found ten countries from the continent of Africa represented and eight countries from South America, with almost 100 countries represented in the data in total. Even smaller, developing countries appeared in the results, such as Panama, Tunisia, and Nepal. These results indicate the worldwide impact of standards from the three major organization in our study.

Organization Analysis

WOSCC records the institutional affiliation of each author in the Address field, but in addition, organizational names are extracted and, to a degree, standardized in the Organization-Enhanced field. This field is available in the Analyze feature of WOSCC.

Each organization was hand coded by type: Academic-Foreign, Academic-US, Government-Foreign, and so forth. Figure 4 shows the distribution of authors’ institutions by type of organization for publications that cite standards.

Given that all three standards organizations have significant international reach, a significant number of publications citing standards by non-U.S. organizations could be anticipated. The difference between foreign academic and U.S. academic institutions is large, a total of 2,195 citations. Note that if a given paper has five authors from five different institutions, five citations are recorded. After reviewing the organizations by type, it was interesting to look at the top ten institutions citing standards (Table 2).

Seven of the top ten citing organizations are U.S. affiliated (Table 2), despite the fact that foreign organizations in the aggregate dominate the total number of publications citing standards as shown in Figure 4. Table 2 shows there is a fairly even split of governmental entities (four) vs. academic institutions (six). This demonstrates the value and reach of standards internationally and across many types of organizations pursuing a wide variety of scholarly activities.

Funding Agency Analysis

Many WOSCC records do not include funding-agency information. In the data analyzed for this article, funding agencies were recorded for no more than 16% of the total number of publications citing standards. The actual percentage is somewhat lower than 16%, since multiple funding agencies can be listed in a single article. Citation counts for name variants of the same funding agency were combined manually (e.g., “US DOE” and “U S Dept. of Energy” were combined under “U.S. Department of Energy”). Duplicate entries were especially common for agencies outside of the U.S., where the funding agency could be listed in the native country language or as the English translation of the name. The agency names were then hand coded to assign a country of origin, if that could be ascertained.

In some cases, the exact origin of a funding agency was unable to be determined. Some names were too generic or the abbreviations/acronyms could represent more than one agency. This was not a common occurrence, as only eight funding agencies were classified as of “unknown” origin. Of these agencies of unknown origin, they had five or less associated publications.

Of the top 20 funding agencies for publications citing standards (Figure 5), four agencies are located in China, three are located in Brazil, and only two from the U.S., as well as two from the European Union. Other countries represented by funding agencies mirror the results of the Country field analysis and include Canada, Korea, Australia, Spain, Germany, and England. The only origin countries appearing in the top 20 funding agency analysis that were not represented in the top country analysis were Turkey and Portugal.

While many individual companies appeared in the results for funding agencies, no companies were present in the top 20 funding agencies. Petrobras, a petroleum company based in Brazil, ranked highest (53rd overall), being listed as the funding agency on seven papers.

Interestingly, while the authors writing publications citing standards were overwhelming from the U.S. (Figure 2), the National Natural Science Foundation of China was identified as the funding agency more often than any other single agency (189 publications). The U.S. National Science Foundation was second, with 97 publications. Still, these numbers are quite small compared with the overall number of publications that cited standards in this study.

The total number of funding agencies represented was also noteworthy. There were 317 total funding agencies named after removing identifiable duplicates. The entire list of funding agencies included government agencies, academic institutions, companies, and non-government organizations from 52 countries and regions, not including international agencies, further demonstrating the reach of the research from publications citing standards.

ASTM Standard Designations Analysis

ASTM standards differ from ISO and CEN standards in that their designation starts with a letter corresponding to a broad technical area, whereas ISO and CEN standard numbers have no inherent meaning. The letters used in ASTM standards, A-G, correspond to groups of technical committees that create and maintain the over 12,000 standards in the ASTM catalog.

These letter groupings represent the highest level of ASTM’s classification system for standards, covering areas such as ferrous metals (A standards), or materials for specific applications (F standards). Under each letter, a list of technical subcommittees exists, tasked with a specific area of focus. For example, Technical Committee A01 is responsible for standards related to steel, stainless steel, and related alloys, while Technical Committee F24 is charged with amusement rides and devices. Unfortunately, our analysis can only be done at the top letter level. Although standards for a given committee in the A class always start with the letter A, the technical committee is not assigned to a particular range of numbers within that class. A full listing of Technical Committees and their designations are available at https://www.astm.org/COMMIT/committees_full.html.

Upon examination of the ASTM standards from the cited works, standards from all letters, A-G, were represented in the results. Figure 6 provides the complete breakdown of the standards by their letter designation. Some citations in the WOSCC database did not clearly cite a specific standard number and could not be included in Figure 6.

Letters D and E are classified as miscellaneous “catch-all” categories for materials and subjects, respectively. This large scope is likely the reason for high counts in these two categories. For example, technical committees for D-classed standards can include but are not limited to, textiles, packaging, water, plastics, air quality, petroleum products, road and paving materials, water, and engine coolants. E-classed standards are just as diverse in the areas covered. However, it was unexpected to see the high citation counts for C-classed standards that were cited. Compared to the range of topics for D and E standards, C-classed standards are quite narrow, focusing on concrete, ceramics, and masonry materials. The high counts may be a reflection of an interest in construction and engineered materials. If nothing else, the figure shows that, at least in terms of the scholarly publications within our study, metals standards (Classes A & B) are cited significantly less than C, D, and E standards.

It was also interesting how few F-classed standards were cited in scholarly publications. The scope of F standards, like D and E standards, is broad, and includes many new technologies, such as driverless automatic guided industrial vehicles, unmanned aircraft systems, and safety related standards, such as medical materials and personal protective equipment. The small number of F standards cited could be due to the comparative lack of standards that exist in these new areas. For example, Technical Committee F48 works on exoskeletons and exosuits, but was only created as a separate committee in 2017. The committee has not published any standards as of December 2018.

Open Access Analysis

Of the 5,214 citing publications, a total of 836 publications, nearly 16% of the total, were open access. While this was not an overwhelming percentage, it is a number that is in line with at least some estimates of open-access articles in scholarly publications as a whole (Gargouri et al. 2012).

When looking at the percentages of open-access publications by the type of standard cited, CEN standards had the greatest percentage at nearly 24% of the total number of publications. The percentages of publications citing ASTM standards and ISO standards were roughly in line with the overall open-access trend, with nearly 14% (418 out of 3,002) and just over 18% (373 out of 2,023), respectively.

While the overall percentage of publications that are open access is not large, nearly one quarter of publications citing CEN standards were open access. This is potentially related to the European Union’s push towards open access and preservation of scientific and technical information. In 2012, the European Commission published a formal recommendation urging all EU Member States to make publicly funded research results available to the public, in an effort to strengthen scientific research and accelerate research development (European Commission 2012).

Future Research

This study used the simple Analyze Results tools in the WOSCC subscription database. Although much useful information was derived from the analysis, the main limitation was that we were unable to run a correlation analysis to examine the relationships between the fields. This is because our data did not contain individual pairs of specific citing papers and cited standards. Researchers can request special datasets from Clarivate Analytics which would make cross tabulations possible, though this may involve a fee.

Further, it would be interesting to expand this study to include more standards organizations and publication years of citing papers. Specifically we would be curious to compare the results of this study to a similar study of select major national standards organizations, such as the Standardization Administration of China (SAC), or more specialized standards developing organizations such as the American Society of Mechanical Engineers (ASME) or SAE International.

Our data confirms the general impression of the value, international reach, and broad utility of standards in a wide range of disciplines, whether analyzing the country, region, disciplines, funding agency, or author affiliation of the citing papers. Unsurprisingly, citing papers from large economies with corresponding large investments in research and development dominate the number of standards cited. Europe is the lead citing region for standards, with North America a close second. However, all regions including Africa and Mexico/Central America are represented in our dataset, including nearly 100 countries.

There was considerable breadth in the disciplines represented by the WOS Categories and in the variations in ranking between the three different standards organizations studied. This reinforces the utility of standards for research and development activities in many areas. More in-depth studies of specific sets of standards in a given field could also provide useful results.

In short, it can be broadly concluded that standards are important to research and development in many disciplines. Standards are often referred to as “grey literature” by librarians. While standards can occasionally be hard to locate for a particular topic, they should not be discounted when conducting scientific or technical research. Further, they should not be forgotten by STEM librarians when making collections decisions, especially for academic libraries supporting scientific or engineering research.

We would like to thank Nancy Schiller and Jill Hackenberg for comments and suggestions during the review of this article.

ASTM International. 2013. ASTM International Technical Committees [Internet]. [cited 27 Feb 2019] Available from https://www.astm.org/GLOBAL/images/CommitteeList_Eng.pdf.

Batik, A. 1992. The Engineering Standard: A Most Useful Tool. Ashland (OH): Book Master/El Rancho. p. 31-57.

Cardona, M. & Marx, W. 2007. Anatomy of the ICDS series: A bibliometric analysis. Physica B: Condensed Matter 401-402, 1-6. DOI:10.1016/j.physb.2007.08.101.

Curtis, S. 2011. Informing collection development through citation examination of the civil engineering research literature. Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. Available from: https://peer.asee.org/18170.

European Commission. 2012. Commission Recommendation 2012/417/EU of 17 July 2012 on access to and preservation of scientific information [Internet]. [cited 2 Feb 2018]. Available from: https://publications.europa.eu/en/publication-detail/-/publication/48558fc9-d4c8-11e1-905c-01aa75ed71a1.

European Committee for Standardization (CEN). n.d. What is a standard? [Internet]. [cited 2 Feb 2018]. Available from https://www.cen.eu/work/ENdev/whatisEN/Pages/default.aspx.

Europa. 2017. Share in EU GDP, 2017 (%) [Internet]. [cited 23 Feb 2018]. Available from https://ec.europa.eu/eurostat/web/products-eurostat-news/-/DDN-20180511-1?inheritRedirect=true.

Gargouri, Y., Larivière, V., Gingras, Y., Carr, L., & Harnad, S. 2012. Green and gold open access percentages and growth, by discipline. arXiv preprint arXiv:1206.3664..

Heilbrun, K., Phillips, S. & Thornewill, A. 2016. Professional standards’ citations in law and behavioral sciences: Implications for policy and practice. Professional Psychology-Research and Practice 47(4): 287-294. DOI:10.1037/pro0000080.

ISO/IEC (International Organization for Standardization, International Electrotechnical Commission). 2004. Guide 2 Standardization and related activities - General vocabulary. Geneva, Switzerland: ISO.

Mathews, B. 2006. The role of industry standards: An overview of the top engineering schools’ libraries. Issues in Science and Technology Librarianship 46. DOI:10.5062/F48C9T6D.

Musser, L. & Conkling, T. 1996. Characteristics of engineering citations. Science & Technology Libraries 15(4): 41-49.

O'Neill, J. 2001. Building better global economic BRICs. Goldman Sachs, Global Economics Paper [Internet]. [cited 23 Feb 2018]; 66. Available from: https://www.goldmansachs.com/insights/archive/archive-pdfs/build-better-brics.pdf.

Pellack, L. 2005. Industry standards in ARL libraries: Electronic and on-demand. Collection Building 24(1): 20-28.

Phillips, M., & Huber, S. 2017. Standards resources for engineering and technology. Issues in Science and Technology Librarianship 87. DOI:10.5062/F4B27SJ7.

Taylor, D. 1999. Standards collection development in an acdement library. Collection Building 18(4), 148-152. DOI: 10.1108/01604959910303280.

Thompson, L. 2001. Grey literature in engineering. Science & Technology Libraries 19(3-4): 57-73. DOI: 10.1300/J122v19n03_05.

Tomaszewski, R. 2017. Citations to chemical resources in scholarly articles: CRC Handbook of Chemistry and Physics and The Merck Index. Scientometrics 112(3): 1865-1879. DOI:10.1007/s11192-017-2437-4.

Tomaszewski, R. 2018. A comparative study of citations to chemical encyclopedias in scholarly articles: Kirk-Othmer Encyclopedia of Chemical Technology and Ullmann's Encyclopedia of Industrial Chemistry. Scientometrics 117(1): 175-189. DOI:10.1007/s11192-018-2844-1.

Tomaszewski, R. & MacDonald, K. 2016. A study of citations to Wikipedia in scholarly publications. Science & Technology Libraries 35(3): 246-261. DOI:10.1080/0194262X.2016.1206052.

World Bank. 2018. Gross domestic product 2016 [Internet]. [cited 23 Feb 2018]. Available from http://databank.worldbank.org/data/download/GDP.pdf.

Zhang, L. 2018. Analyzing citation and research collaboration characteristics of faculty in aerospace, civil and environmental, electrical and computer, and mechanical engineering. College & Research Libraries 79(2): 158-178. DOI:10.5860/crl.79.2.158.

Countries Divided by Region