Economic Analyses of Federal Scientific Collections Methods for Documenting Costs and Benefits David E. Schindel and the Economic Study Group of the Interagency Working Group on Scientific Collections Smithsonian Scholarly Press WASHINGTON, D.C. 2020 Cover images (from top to bottom): Aedes aegypti, a disease-transmitting mosquito, after extracting a blood meal from a human subject; photo by James Gathany (2006), courtesy of CDC Public Health Image Library (https://phil.cdc.gov/Details.aspx?pid=8923). Guatema- lan bat, collected in 2011 to determine the prevalence of pathogens; CDC’s Division of Vector Borne Diseases worked closely with the Global Disease Detection Regional Center in Guatemala and the Universidad del Valle de Guatemala; photo courtesy of CDC Public Health Image Library (https://phil.cdc.gov/Details.aspx?pid=18013). Rodents collected during or after World War II in Formosa (now Taiwan) as part of anti-typhus efforts; vials probably contain fleas and other ectoparasites that can transmit pathogens; photo by U.S. Army Signal Corps, courtesy of National Museum of Health and Medicine, Otis Historical Archives 343 (https://www.flickr.com/photos /medicalmuseum/3543831668/in/album-72157618366141658/). CDC scientist stacks plates with human serum samples into a robotic system that detects the presence of antibodies that protect the subject from poliovirus; photo by James Gathany, courtesy of CDC Pub- lic Health Image Library (https://phil.cdc.gov/Details.aspx?pid=22912). A microbiologist in CDC’s Special Pathogens Branch lowers a rack of boxes containing cryovials into a liquid nitrogen freezer for long-term storage; photo by James Gathany, courtesy of CDC Public Health Image Library (https://phil.cdc.gov/Details.aspx?pid=10724). Published by SMITHSONIAN INSTITUTION SCHOLARLY PRESS P.O. Box 37012, MRC 957, Washington, D.C. 20013-7012 https://scholarlypress.si.edu Copyright Information This document is a work of the United States Government and is in the public domain (see 17 U.S.C. §105). Subject to the stipulations below, it may be distributed and copied with acknowledgment to the Smithsonian Institution. Copyrights to graphics included in this document where the original copyright holders or their assignees are credited are used here under the government’s license and by per- mission. Requests to use these images must be made to the provider identified in the image credits. Disclaimer The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of any partic- ipating agency.  Suggested Citation Schindel, D. E. and the Economic Study Group of the Interagency Working Group on Scientific Collections. 2020. “Economic Analyses of Federal Scientific Collections: Methods for Documenting Costs and Benefits.” Report. Washington, DC: Smithsonian Scholarly Press. https://doi.org/10.5479/si.13241612 Library of Congress Control Number: 2020044628 ISBN (online): 978-1-944466-41-1 ISBN (print): 978-1-944466-42-8 Publication date (online): 20 November 2020. Open access PDF available from Smithsonian Institution Scholarly Press at https://doi.org/10.5479/si.13241612 Printed in the United States of America c The paper used in this publication meets the minimum requirements of the American National Standard for Permanence of Paper for Printed Library Materials Z39.48–1992. https://phil.cdc.gov/Details.aspx?pid=8923 https://phil.cdc.gov/Details.aspx?pid=18013 https://www.flickr.com/photos/medicalmuseum/3543831668/in/album-72157618366141658/ https://phil.cdc.gov/Details.aspx?pid=22912 https://phil.cdc.gov/Details.aspx?pid=10724 https://scholarlypress.si.edu https://doi.org/10.5479/si.13241612 https://doi.org/10.5479/si.13241612 https://www.flickr.com/photos/medicalmuseum/3543831668/in/album-72157618366141658/ Contents Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Scientific Collections as a Marketplace Types of Federal Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Ownership and Stewardship of Federal Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Costs Related to Federal Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 9 Project Collection Costs 9 Institutional Collection Costs 10 Services Provided by Institutional Collections 12 Cost Categories and Accounting 13 Method for Reconstructing Collection Budget 14 Cost Recovery Benefits Generated by Federal Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 15 Technology/Knowledge Transfer 17 Success Stories 19 Option Value 20 Value Added by Users 23 Counter-Factual Scenarios 23 Comparison among Methods Implications for Policies and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27 27 Constraint A 28 Constraint B 28 Constraint C Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Appendix 1: Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Appendix 2: Collections Cited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 v Executive Summary Federal object- based scientific collections have been created to serve agency missions and, in a few cases, to comply with legislative and regulatory mandates. “Project collections” (those managed by the researchers who obtained them for re- stricted use) and their costs and benefits were considered too varied for standard methodologies that assess costs and benefits. In a few cases, departments and agencies are required by legislation or regulations to retain objects in long- term “institutional collections.” In most cases, decisions to retain objects are based on long- term costs relative to the perceived potential for benefits to taxpayers. Federal collections vary in their philosophies and practices of offsetting operating costs by charging users for access to their collections. Operational costs vary among institutional collections, reflecting differences in the size of collections, types of material they contain, and differences in the services they provide to the agency, extramural users, and society in general. This report describes six general services that federal institutional collections provide. Departments and agencies vary in the number of services they offer and the degree to which these services have been developed. Returns on investment are controlled, in large measure, by decisions about what is accessioned into a collection, policies concerning user fees and access, and the services provided by a collection. Those collections that offer only the basic service of accessioning objects have limited ability to generate benefits because few users will know about and have access to objects in the collection. By offering more services, collections broaden their potential use to: future generations (through proper maintenance and preservation), intramural research and by extramural users (through online documen- tation and user access programs), users in other disciplines (through data curation), and the general public (through edu- cation and outreach). The benefits generated by federal institutional collections can take many forms, both monetary and non- monetary. These benefits are usually indirect and delayed, and the value chains that connect costs to benefits are generally difficult to document. This report describes five methodologies (and their strengths and weaknesses) that are available to federal collections for describing and estimating the benefits they generate. Departments and agencies can use the methods de- scribed here for evidence- based decisions concerning policies and management practices for their institutional collections. vii Acknowledgments The Interagency Working Group for Scientific Collections (IWGSC) consists of representatives from more than 15 Federal departments and agencies, each of which owns, manages, and/or provides financial support for scientific collections. These agencies and their collections cover a wide spectrum of scientific disciplines. In developing this report, IWGSC’s Economic Study Group (ESG) considered the full range of IWGSC disciplines and organizational missions while examining collec- tions from an economic perspective. This report was developed over an 18-month period during which the ESG received and discussed dozens of online presentations devoted to case studies of costs and benefits, as well as methodologies used to evaluate them. The findings transmitted in this report were made possible by critical contributions from the following individuals. Members of the Economic Study Group The following individuals participated in ESG’s online meetings during which they received and discussed presentations on the economics and management of scientific collections. ESG members are scientists representing different disciplines, collection professionals, economists, and policy specialists. Several ESG members gave presentations to the group, and all members shared their experiences and insights. ESG’s interagency and interdisciplinary discussions, and the generous contributions of time and effort by its members, produced the findings and recommendations presented here. The members of the ESG are: Reed Beaman (NSF), Brad Bowzard (HHS/CDC), Vanessa Burrows (HHS/FDA), Jeffrey DeGrasse (HHS/FDA), Kevin Hackett (USDA/ARS), Marianne Henderson (HHS/NIH), Susan Lukacs (HHS/CDC), Gerry McQuillan (HHS/CDC), Scott E. Miller (SI), Tom Moreland (USDA/FS), Emily Pindilli (DOI/USGS), Cassidy R. Sugimoto (NSF), John P. Swann (HHS/FDA), Michael Walsh (DOC/NIST), Ellen Wann (HHS/NIH), and Paul Zankowski (USDA). Presenters The following individuals, not associated with the ESG, provided presentations that were critical in developing the report’s findings and recommendations: Lindsay Powers (USGS), Erik Lichtenberg and Lars Olson (University of Maryland), Doug- las Gollin (Oxford University), Jessica Jones (HHS/FDA Gulf Coast Seafood Laboratory), Peter Bretting (USDA/ARS Crop Production and Protection), Todd Ward (USDA/ARS Culture Collection), Kelly Day-Rubenstein and Paul Heisey (USDA/ Economic Research Service), and Abhi Rao (HHS/NIH). Other Contributors Members of the IWGSC who were not ESG members provided thorough reviews of the draft report, often contributing valuable information about their collections, which was incorporated into the report. Eileen Graham (SI) provided Ap- pendix 2: Collections Cited using information from the IWGSC Clearinghouse and its registry of U.S. Federal Scientific Collections. Keith Crane, Lauren Bartels, and Thomas Olszewski (IDA Science and Technology Policy Institute) provided support for this study under an interagency agreement with the Office of Science and Technology Policy. Deborah Paul (University of Florida) provided the benefit of her experience in natural history collection digitization, aggregation of data on collection usage, and data leadership. 1 Introduction P hysical objects form the basis of research in many scientific disciplines. Organisms, soil, medical samples, meteorites, and thousands of other types of objects provide the evidence that is central to the missions of many Federal depart- ments and agencies. These missions include research, regulatory responsibilities, and complying with legislation that serve the Nation’s interests. In many cases, departments and agencies decide to retain objects for long- term preservation in scientific collections, in anticipation of future use or in compliance with regulation or legislation. The long- term support for collections is a commitment of Federal resources without clear evidence of future returns on these investments. History has shown that some of these objects prove to be valuable, even critical, in solving mission- related challenges. They may help to cure diseases, save agricultural crops, avoid natural disasters, and provide other tangible benefits to the Nation. Many others may not have been used since being added to a long- term collection. Faced with their uncertain future value, how can Federal collection officials decide which objects to preserve and which to discard? How can departments and agencies justify the cost of creating and maintaining scientific collections? Are there evidence- based approaches that can reduce the costs and increase the benefits associated with Federal scientific collections? In 2005, the Office of Science and Technology Policy (OSTP1) created an Interagency Working Group on Scientific Collections (IWGSC)2 to explore and solve the common challenges that face Federal departments and agencies with sci- entific collections. Their 2009 report included a survey of more than 300 collections in 14 departments and agencies, documenting their size, uses, management practices and financial support (IWGSC 2009). The report also put forward recommendations, including the development of better methods for documenting the budget support needed by these col- lections. In 2010, OSTP directed all Federal departments and agencies with collections to implement this recommendation (OSTP 2010) and the America COMPETES Reauthorization Act of 20103 said: (d) Cost Projections - The Office of Science and Technology Policy, in consultation with relevant Federal agencies, shall develop a common set of methodologies to be used by Federal agencies for the assessment and projection of costs associated with the management and preservation of their scientific collections. This report presents the findings of a year- long IWGSC study to provide Federal departments and agencies with standard methods for documenting costs and benefits related to their scientific collections.4 These findings enable Federal scientific collections to do the following: • Document the costs related to operating their long- term institutional collections, whether or not they have separate line items in the budgets of their organizational structure; • Identify how these costs are allocated to the six standard services provided by institutional collections; • Review their philosophies and practices regarding user fees and other forms of cost recovery, and consider how they may be affecting collection use and the benefits stemming from this use; • Consider new methods for documenting the benefits generated by use of their collections, especially the five methods presented in the report, with examples from Federal collections: ◌ Tracing the value chains from collection use to new products, processes, or other economic activity; ◌ Describing the roles their collections have played recently in breakthrough discoveries and meeting major societal challenges; ◌ Associating collections and the services they provide with the mission- critical uses they have played in the past, and are being maintained to serve if needed again; 2 Economic Analyses of Federal Scientific Collections ◌ Documenting the pattern of collection use and the financial assets that users are investing in the collection; and ◌ Gathering data on the losses the Nation’s economy and well- being would suffer if the collections and the services they provide didn’t exist. Scientific Collections as a Marketplace Taken collectively, the collections in any discipline resemble a marketplace with providers, customers, and limited supplies of goods and capital. In this context, “goods” refers to the objects in scientific collections, not other goods and services provided by institutions with collections (e.g., jobs created, increased tourism, museum gift shop items sold; see American Alliance of Museums, 2017). Providers bear the costs of making goods available, and consumers want the highest quality goods at the lowest cost possible. The marketplace analogy is not perfect for two principal reasons. First, the prices that consumers pay for access to goods are not the result of competition or other market forces. Only a minority of collections owned and operated by Federal departments and agencies charge fees that are set to recover costs, and they are never set to generate profits. Most Federal collections charge fees that recover only a portion of costs borne by providers, and they often charge only for ship- ping—or for nothing at all. Second, the benefits generated from the goods provided by collections are generally bestowed on society, not the consumers themselves. They take the form of new knowledge in the public domain, or improved public health and security, or protection against future shocks to the environment, food supply, and public safety. Relatively few Federal collections provide goods that private companies use to generate revenue and profits. Schumann (2014) provides a clear summary of the distinctions between valuations of public and private goods; Smale and Koo (2003) provide a tax- onomy of values generated by scientific collections used for plant breeding. Scientific collections have been described by some authors as “global public goods” and have been discussed in the context of the “public commons” (e.g., Halewood, 2013 for plant genetic resources; Kothamasi, Spurlock, and Kiers, 2011 for agricultural microbial resources; and Reichman, Uhlir, and Dedeurwaerdere, 2015 for microbial culture collections). This literature focuses on intellectual property law pertaining to collections but does not explore methods for estimating their costs and benefits, and was not considered directly germane to this report. This study explores the other factors that determine the costs borne by providers and the benefits generated from the use of collection- based goods obtained by consumers. A clearer understanding of these factors and the relationships among them may equip Federal collections to make evidence- based policies and better management decisions (Graves 2003). Introduction 3 Box 1. Glossary of Terms Accessioning is the process of transferring ownership of objects into an institutional collection. Collections may incorporate the records of the objects into the registry of that collection and objects may be physically transferred into a collection at the time of accessioning, but these procedures vary. Deaccessioning is the process of ending legal ownership of objects in an institutional collection, either by transferring ownership to another institution or destroying the objects. Institutional collections are scientific collections that are made available for use by qualified researchers, companies, government agencies and other qualified users (see original definition in IWGSC 2013a). They can be non-renewable or renewable (defined below), and they are generally managed by collection professionals. Non-renewable collections consist of objects that have been collected or fabricated at a specific time and place and are no longer being produced. They cannot be replaced with identical objects so they are maintained and preserved long-term for future use. Objects in these collections may undergo some degree of destructive analysis, but they are completely consumed only under rare circumstances. Project collections are scientific collections that are managed and used by scientists who may have obtained the physical objects for specific research or another purpose, or by others working on that or related research (see original definition in IWGSC 2013a). Objects in project collections are sometimes completely consumed by destructive analyses or discarded at the end of the project. Renewable collections consist of living organisms that can generate replicas of themselves, or man-made objects that can be fabricated. As a result, objects can be completely consumed by destructive analyses without exhausting the supply. Renewable collections are made available in response to user demand and can be disestablished if demand wanes. Scientific collection is used herein as shorthand for “object-based scientific collection”: a group of physical objects that are used for research, development, education, and other activities related to disciplines in physical, life, earth, and planetary sciences as well as archaeology, physical anthropology and applied sciences such as engineering, agriculture, and veterinary science. Scientific collections may include the maps and notes related directly to physical objects but would not include libraries, document archives, or data repositories not associated with physical objects. 5 Types of Federal Collections IWGSC (2009) included a survey of Federal scientific collections that found 14 departments or agencies with approxi-mately 300 scientific collections. They cover the full spectrum of scientific disciplines and the majority of these depart-ments or agencies own and manage their collections long- term. IWGSC has since created the Registry of U.S. Federal Scientific Collections (USFSC)5 that now contains data on many hundreds of scientific collections in 19 departments or agencies. Despite the great diversity of disciplines in which Federal collections are maintained, there are only a few types of col- lections from the perspective of their operating principles (see Box 1 for definitions). Non- Federal collections can generally be assigned to one of these categories and many of them are eligible for Federal support through NSF grants and other Federal funding opportunities. Most, but not all, of the objects in Federal collections begin as objects in “project collections.” While a project is in progress, access to the collection and information about its contents are generally controlled by the researchers working on them. They make decisions about consuming an object, in part or in whole, through destructive analyses, choosing not to preserve a part for future use. Eventually, one of several things occur: the project can be completed with or without publication of the results; the project leaders can retire or pass away; or the department in which the project was conducted could be reorganized, relocated, or abolished. In each case, decisions need to be made about what to do with the collection. Many objects in project collections are accessioned into “institutional collections” (sometimes referred to as “archival collections”). IWGSC agencies have developed policies governing their institutional collections6 and these are available on the IWGSC Clearinghouse. Ownership of and responsibility for these objects passes to the institutional collection and control of the objects normally passes from researchers to professional collection managers. Their mission is to preserve the contents of the collection and to make them available for uses that serve the broader agency mission and generate benefits for society. Most institutional collections will disseminate information about the contents of the collection and will facilitate their use by qualified researchers. There may be restrictions on the use of certain objects in a collection based on national security, public safety, or the terms of collecting permits and/or material transfer agreements. Examples include informed consent given by human subjects at the time samples are collected, and biological samples collected and exported under international agreements such as the Convention on Biological Diversity, the Nagoya Protocol on Access and Benefit Shar- ing, the Food and Agriculture Organization’s International Treaty on Plant Genetic Resources for Food and Agriculture, and the Convention on International Trade in Endangered Species. Most institutional collections are “non- renewable.” They include inanimate objects, preserved dead organisms, or parts thereof (e.g., frozen tissue or body fluid samples). These collections normally have procedures by which researchers can request permission to perform destructive subsampling and analysis of a sample or specimen. Since the mission of in- stitutional collections is the preservation of objects for future study, collection managers require strong justifications based on the scientific importance of the proposed analysis. Permission is rarely granted to consume the last remaining portion of a sample or specimen. In contrast, “renewable” collections consist of organisms that reproduce or replenish themselves (e.g., cell cultures; viable microbial, plant and animal germplasm) or inanimate objects that can be replicated. Destructive sampling and analyses of these objects are common and are not issues of concern for renewable collections because replacements can be grown or manufactured. These are sometimes referred to as “research resources” though IWGSC considers them scientific collections. 7 Ownership and Stewardship of Federal Collections I n addition to directing agencies to establish adequate and sustainable operating budgets for collections, the America COMPETES Act and OSTP (2010) directed agencies to develop and disseminate policies on the management and use of Federal collections. OSTP (2014) specified the components that should be included in these collections policies and the IWGSC Clearinghouse now presents the policy documents that comply with these requirements. Taken together, the IWGSC (2009) recommendations, America COMPETES, OSTP (2010), and OSTP (2014) direct agencies to harmo- nize their collections policies, budgets, and practices with agency missions as expressed in authorizing legislation and regulations. When Federal departments and agencies obtain and add non- renewable objects to project collections, the objects normally meet some immediate mission- related need. Project collections do not encumber agency budgets beyond the resources allocated for research projects. As described above, objects in project collections can be consumed through destructive analyses when researchers decide that immediate project goals outweigh concerns about potential future use. Within the limits placed on the use of objects by collecting permits, consents, and material transfer agreements, objects in project collections can be viewed as consumable research supplies. In contrast, when agencies accession objects into their institutional collections they accept long- term responsibility and related costs for maintaining the objects long- term for future use, as stipulated in their collections policies. Complete destructive sampling of an object is permitted only under very specific conditions. Figure 1 illustrates the diverse pathways that objects can follow before they are accessioned into Federal institutional collections. Each of the pathways leading to a Federal institutional collection can be viewed as a long- term financial obliga- tion by the receiving agency. However, not all of the objects being added to Federal institutional collections begin as project collections obtained and owned by that agency. Some renewable collections provide users with living organisms for research and development of agricultural prod- ucts and industrial processes (see Boxes 2 and 3). Users rely on these collections to discover and develop traits found in nature’s diversity, so specimens are collected based on how they expand the collection’s diversity. Accessioning them into an institutional collection represents a significant financial commitment because of the costs of maintaining living popula- tions. Other renewable collections offer precisely standardized objects to users for use as calibration standards, or highly characterized and uniform organisms for controlled experiments. This second group of renewable collections goes through an initial period of planning, development, testing, and quality control before the standardized samples or specimens can be offered to users. Planning may involve market surveys and analyses or workshops to determine market demand and to determine the exact properties and characteristics that users want the samples or specimens to have. The development phase involves activities like chemical synthesis, genetic engineering, or selective breeding to produce samples or speci- mens that will satisfy user demands. Testing and quality control follow for confirmation. We consider renewable collections as project collections during this initial phase. We consider them institutional collections once testing has been completed and they have been made available to users. 8 Economic Analyses of Federal Scientific Collections FIGURE 1. Processes leading to accessioning of objects into non- renewable and renewable Federal institutional collections. (A) Typical pathway leading from collecting activities to project collections to institutional collections. This process is the same for Federal and non- Federal non- renewable collections and for non- renewable and renewable collections. (B) The collecting activities of some Federal research projects and other sources go directly into the institutional collections of that agency (e.g., the component of the Centers for Disease Control and Prevention [CDC] National Health and Nutrition Examination Surveys [NHANES7]) collections that is sent directly to the CDC Biorepository; the Department of Defense’s Serum Repository,8 and U.S. Department of Agriculture’s [USDA] renewable collections). (C) Non- Federal collecting activities conducted on certain Federal lands (e.g., National Parks) require collecting permits, many of which specify that the objects collected remain Federal property. (D) Some non- Federal institutions with project collections may choose to donate material to a Federal institutional collection when the project ends. The Smithsonian Institution and US Geological Survey receive many collections in this manner. (E) Non- Federal entities sometimes end their operations or encounter financial obstacles that leave them unable to continue maintaining an institutional collection. Federal institutional collections sometimes acquire these “orphaned” collections. (F) Agencies sometimes transfer objects in project collections to non- Federal institutions.9 9 Costs Related to Federal Collections T his study explores costs associated with institutional scientific collections and standard methods that can be used to estimate and document those costs. This approach is essential and the logical predicate to evaluating benefits gener- ated by collections—the returns on investments made by taxpayers in those collections. The evaluation of costs takes into consideration differences among institutional, project, renewable, and non- renewable collections. Project Collection Costs Project collections have a wide range of cost categories. The costs in each category, measured on a per object basis, vary over many orders of magnitude among collections. This variation results from the fact that project collections involve collecting activities, preserving the objects and preparing them for analyses, conducting a diverse array of analyses, inter- preting data, and preparing publications. The logistics for project collections are sometimes simple, involving one or a few researchers collecting in local areas with no requirements for collecting permits other than the agreement of landowners. Others are far more complex, involving inter- institutional and international agreements or treaties that govern collecting, transfer and use of research material. For example, several different authorities may have legal responsibility for issuing permits or consents to: collect threatened and endangered species; obtain samples from human subjects; or collect and transfer research material among countries or from territories beyond national boundaries (e.g., Antarctica, the open oceans, or outer space). Significant effort and expense may be needed to obtain permits or consents before the first col- lecting activity can begin or access to and use of samples can be granted. Once permission to collect has been granted, the cost of obtaining samples or specimens can be as high as NASA space missions, deep- sea drilling operations, or expeditions into remote regions like tropical forests or the deep sea, to name a few. Analytical techniques can involve a wide range of other instrumentation and techniques, including expensive infrastructure such synchrotron beams, CT scanners, DNA sequencers, and mass spectroscopy. For these reasons, project collections do not lend themselves to standard methodologies for estimating costs. There are simply too many types of costs and too many factors that affect these costs to produce cost estimates that can be compared meaningfully. For these reasons, cost estimation for project collections is not treated in this report. Institutional Collection Costs Unlike project collections, there are relatively few types of costs involved in the operation of institutional collections. The costs per object within each cost category will vary according to: • the type of preservation needed for non- renewable collections (e.g., ultra- cold versus refrigerated; dry versus alcohol- preserved at room temperature), and processes for reproducing renewable collections; • geographic location, which affects salaries (due to cost of living variation), facilities and utility expenses (due to rental, construction and utility rates); • size of the collections (affecting economies of scale); and • the services provided by the collection. The first three sources of variation are relatively straightforward and can be taken into account when comparing costs among institutional collections. The most important source of variation in per object operating costs is differences among collections in the services they provide. That is, the more different services a collection provides, and the more of each service it provides, the higher the cost per object. Cost comparisons among institutional collections will only make sense 10 Economic Analyses of Federal Scientific Collections if differences in the services provided are considered. Like costs, the types of benefits generated by institutional collections vary with the number of services they provide (see below, Implications for Policies and Management). Services Provided by Institutional Collections This study identified six different services that Federal institutional collections may elect to perform, depending on the mis- sions of their institutions and the resources available to them (see Table 1). NSF provides support for all of these services to non- Federal collections in the biological sciences. The exact nature of some services will differ between renewable and non- renewable collections, as portrayed in Table 1. For example, accessioning decisions in non- renewable collections are supply- driven: which of the available samples are needed and supportable by the collection with available resources? For renewable collections, management decisions are driven more by user demand: what type and how many items will the user community want added to the collection to improve its coverage? For some renewable collections of living organisms, both supply and demand are factors in decisions of what to accession. Managers of these collections consider (a) the poten- tial future demand for a new accession and (b) how a new accession will increase the genetic variability and the geographic, habitat, and taxonomic coverage of the collection. TABLE 1. Standard services that are commonly provided by non- renewable and renewable collections Non- Renewable Collections Renewable Collections 1. Accessioning • Taking legal ownership, though not necessarily physical possession • Verifying provenance, ownership, import permits • Receiving • Physical integration into collection • Manufacturing, growing or breeding inventory to meet user demand and to replace living contents to ensure viability of specimens/samples that are provided to users • Transferring accessions from other collections • Collecting materials in nature 2. Preserving and maintaining • Facilities and environmental controls • Security and inventory control • Object conservation 3. Documenting additions • Importing data and metadata (e.g., collector, collecting location and date) into a collection registry • Relabeling and transfer to standard containers • Detecting and correcting errors in data and metadata • Documenting collecting source and location • Documenting specimen characteristics • Data quality control 4. Providing access to users • Establishing and maintaining web- based catalog and applications • Digitizing specimen/sample records • Capturing digital images of collection contents • Creating and maintaining online databases of collection contents • Creating and managing loan and visitor programs • Communicating availability to potential users • Governing access by reviewing applications for access • Managing Material Transfer Agreements • Shipping and receiving • Creating and maintaining online catalogs • Maintaining and updating inventory of holdings • Reviewing and processing orders • Managing Material Transfer Agreements • Shipping and receiving 5. Data curation • Adjusting data and metadata format and terminology to community standards • Documenting user access to collection • Updating data records with corrections, additional information • Linking collection records to publications and datasets in public repositories resulting from use 6. Increasing public understanding through education and outreach • Public exhibits • Developing and disseminating informational material about collection contents, uses and impacts through formal and informal education, media, and other mechanisms Costs Related to Federal Collections 11 The first service, accessioning, is the core service that institutional collections perform. The accessioning function is integral to the definition of institutional collections. As one study participant said, “It’s what we do.” Accessioning is both a service and a critical decision- point for collection managers because adding objects to an institutional collection is a commitment of space and support that usually has no clear time limit. In a few cases, legislation and/or regulations require that certain objects must be accessioned into Federal scientific collections. In other cases, agencies must make decisions based on the potential costs and benefits of long- term maintenance (see below, Implications for Policies and Management). Preservation and maintenance ensure the security of objects and their fitness for use by future users. This is the second most common service provided, though collections within an institution may receive different levels of this expen- sive service. Environmental controls on temperature, humidity, exposure to sunlight and other factors can be critical in preventing the deterioration that would render objects useless for research or other activities (Stauderman and Tompkins, 2016). Some collections require highly specialized preservation (e.g., cryo- preservation, alcohol immersion). For renew- able collections of living specimens, maintaining and producing healthy and viable organisms are major costs. Beyond these two basic services, agencies vary widely in providing the other four services, depending on each agency’s mission and the support available. Documenting additions to collections involves the transfer of data and metadata (e.g., collecting location and date, collector’s name) from project collections and researchers or from other institutional collections. These data and metadata may contain errors and ambiguities that persist unless they are corrected at this stage or at a later stage through data cura- tion. Metadata from renewable project collections can include instructions for maintaining living organisms, as well as data on collectors and collecting localities. Providing access to users can include loan and visitor programs as well as digital access through web portals with information, digital images, and even trait data from the objects in the collection. This is a key service for increasing the benefits generated by institutional collections and their extramural users. These services make communities of users aware of the objects in collections and the rules governing their access and use. NSF is supporting Advancing Digitization of Bio- logical Collections,10 a 10- year funding initiative for non- Federal collections, to raise the visibility, discoverability, and use of collections through digitization. Collection managers are responsible for promoting access and use, while also safeguard- ing objects to ensure their long- term availability and fitness for use. Information about and access to some collections are not made publicly available. These include collections of virulent pathogens, crime investigations, and others that could pose a threat to public health and national security. Data curation has developed rapidly in a few types of collections over the past decades, especially those with large- scale digitization initiatives. Taken together, digitization and data curation have moved these collections into the realm of “big data.” Data curation activities usually include: • Detecting and correcting errors; • Developing and implementing community- driven data standards and ontologies; • Developing automated translation of analog data to digital format; • Automating data quality control; • Developing and using standardized specimen/sample identifiers; and • Linking digital collection records (using standardized identifiers) to publications citing samples/specimens and data derived from samples/specimens in public databases. For example, DNA sequences in GenBank include references to the voucher specimens from which the sequences were derived; not all collections include references to related GenBank records. Some collections have active intramural research programs that document the characteristics of objects in the collec- tion. Examples include the USDA’s National Plant Germplasm System11 (NPGS; see Box 2) and Agricultural Research Ser- vice’s (ARS) Culture Collection12 (Box 3), and the CDC’s NHANES Biospecimen Program (see Box 4). Data curation would include linking public data to the digital records of objects in the collection. The costs and benefits related to extramural research activities that characterize objects in collections are discussed in a following discussion on value added by users. 12 Economic Analyses of Federal Scientific Collections Some Federal collections are used to increase public understanding through education and outreach. Objects from the Smithsonian’s institutional scientific collections are displayed in the exhibits of two public museums that attract more than 10 million visitors per year (National Museum of Natural History, 7+ million visitors per year; National Zoological Park, 3+ million), as well as being featured in magazine articles and television programs. NSF’s support for scientific collec- tions is awarded partly on the basis of their broader impact, including increasing public awareness of science. Cost Categories and Accounting The types of expenditures related to delivering these services fall into a small number of cost categories. Table 2 shows which cost categories are relevant to each of the services offered by collections. These associations between cost categories and particular services do not vary significantly among types of collections, geographic location, or agencies, though the costs within categories will vary. Viewing overall operating costs in the context of the services provided enables collection managers to make evidence- based management decisions, especially when the benefits arising from particular services are considered (see Implications for Policies and Management, below). Baker et al. (2014) presented cost data in terms of a similar breakdown of services provided by a non- Federal collection. The IWGSC (2009) report included the results of a survey with more than 150 responses from 14 agencies represent- ing about 300 Federal collections. The responding agencies indicated that • 27% of collections have a budget line- item devoted to maintenance and management; and • 41% of collections have no funds specifically allocated for collection care and management. Presentations and discussions during this study confirmed that the availability of dedicated funding for the operation of institutional collections varies widely among agencies and even among collections in the same agency. Relatively few in- stitutional collections have distinct budgets that support the cost of providing the services described above. In these few cases, the operating costs of the institutional collection are well documented and provide a solid basis for evaluating costs relative to the benefits generated by the collection. Table 3, column A presents three examples. The first two examples involve transfers of funds to a collection specifically for support of operating costs. One collection is managed by a Federal TABLE 2. Cost categories associated with services provided by scientific collections Personnel, Training, and Staff Travel Facility Space and Modification Equipment Acquisition and Development Utilities Materials and Consumables Shipping and Receiving IT, Web and Communications Services Maintenance and Security Contracts Contracts for Exhibit/ Material Design and Fabrication 1. Accessioning X X X X X X 2. Preserving and maintaining X X X X X X 3. Documenting additions X X X 4. Providing access to users X X X X X X X 5. Data curation X X X 6. Increasing public understanding through education and outreach X X X X Costs Related to Federal Collections 13 agency through an Interagency Agreement and the other is managed by a non- Federal contractor (see Table 3A for exam- ples). These collections created systems for documenting and recovering all operating costs (see Method for Reconstructing Collection Budgets, below). More commonly, a single agency budget will combine the operating costs of one or more institutional collections along with unrelated costs, such as: collecting activities; management of project collections; research activities including analy- ses of objects; and preparation of publications. Other agencies have considered collection- related budgets as too small to segregate so they have included them in much larger organizational units, sometimes unrelated to research. In these latter cases, special efforts are needed to identify the costs associated with the institutional collection (see examples, Table 3B). Method for Reconstructing Collection Budgets Agencies that do not have separate budgets for their collections or use contractors for collection management can docu- ment their operating costs in another way. NSF- ICF uses the following method to develop the budget for its Interagency Agreement with NSF. The International Cooperative Administrative Support Services is a similar system used by the U.S. State Department to divide the cost of administrative services provided by the State Department (e.g., office space, IT sup- port, utilities, motor pool) among the Federal agencies housed in each embassy. Table 2 shows the cost categories associated with each of six services that can be offered by an institutional collection. This framework allows collection managers to reconstruct their operating budgets using the following procedure: • Itemize the full- time equivalents, square footage, number of computers and other measures of resource utilization in each cost category of the services they provide; • Calculate personnel costs directly using the compensation of the staff members involved in each service; • Use the budget of the parent organization to determine total support for space, utilities, IT and security services, and other cost categories used by the collection; and • Prorate the portion of space, utilities, security and other services used by the collection, using the appropriate utiliza- tion measure (e.g., number of computers for IT services, square footage for facility space and security). TABLE 3. Examples of institutional collections with and without dedicated budgets A. Institutional Collections with Dedicated Budgets B. Institutional Collections without Dedicated Budgets Two of the three NHANES institutional collections are stored and managed by non- CDC repositories under a contract with CDC. The budgets submitted by the contractors and paid by CDC document all operational costs other than personnel costs of CDC staff that oversee the contracts. These operating costs include storing and maintaining samples and distributing samples to users whose proposals have been approved after CDC staff review. The third NHANES institutional collection includes samples directly accessioned into the CDC Biorepository after being collected from health survey participants. Management of the NHANES collection (storage, maintenance, and shipment of samples to qualified users) and several other institutional collections is provided by the CDC Biorepository. It can be difficult to know the operating costs of any particular collection in a centralized facility. The National Science Foundation Ice Core Facility (NSF- ICF)13 belongs to NSF but it is housed at the Denver Federal Center, where it is managed by the U.S. Geological Survey (USGS) and supported by an NSF- USGS Interagency Agreement. The agreement’s budget represents the operating budget of the collection, including personnel costs of USGS staff. The Food and Drug Administration (FDA) of the Department of Health and Human Services (HHS) has collections of food- borne pathogens and other food safety collections14 that serve the agency’s regulatory mission. Since the services provided by the collections serve regulatory activities that have separate budget line- items (e.g., seafood safety, marketplace inspections, food and feed safety, outbreak response), support for collections is distributed among several budget lines. ARS has non- renewable institutional collections (e.g., preserved insects, fungi) used for agricultural research. They include objects relevant to several USDA commodity programs that contribute funds for research and collection operations. Combining support for research and collections obscures the true operating costs of the collections.15 14 Economic Analyses of Federal Scientific Collections Once the services/cost categories have been established and their associated costs prorated from higher organizational totals, the operational costs of a collection can be summed across services and cost categories. Most institutional collections have stable operating budgets except for construction of new facilities, major equipment upgrades, or relocation to new facilities. Other cost variations are smaller, involving occasional changes in staff positions, space utilization, or the addition of new services. Cost Recovery This study considered user fees as a mechanism for reducing costs, not for generating benefits or returns on investments. Accordingly, we address the issue of cost recovery in this section on costs, rather than in the following section, which fo- cuses on benefits. We found the full range of agency philosophies concerning cost recovery. At one end of the spectrum, many agencies do not charge users any access fees. There is general concern among these agencies that charging users for access will reduce user interest, especially among potential users with limited resources (e.g., students, small institutions). Based on one year of records from three Canadian biobanks, Albert, Bartlett, Johnston, Schacter and Watson (2014) argued that cost recovery is probably limited to 5–25% of total operating costs. USDA/ARS maintains germplasm of agriculturally important plants, animals, microbes, and insects. ARS has no statutory authority to charge user fees for access, with one exception.16 The 1990 Farm Bill established the USDA National Genetic Resources Program (NGRP), which includes renewable national germplasm collections of agriculturally important plants, animals, microbes, and insects. The Bill stipulated that the NGRP’s germplasm be made available to users free- of- charge. NSF- ICF is also proactive in promoting use and does not seek cost recovery through user fees. Other agencies recover some or all operating costs. NSF’s Living Stock Centers Program17 does not require grantees to recover full operating costs; they are encouraged to become more sustainable through user fees. The National Cancer Institute (Odeh, Miranda and Rao, et al. 2015) and the University of British Columbia18 have developed software tools that help biobanks determine cost recovery fees based on data from cost categories. The NHANES Biospecimen Program charges a set fee per sample to cover some operational and transactional costs: collecting, storing, and processing samples; preparation of data files; and personnel costs associated with the process of reviewing proposals for access to samples. User fees are not intended to recover full operating costs. At the other end of the spectrum, the National Institute of Standards and Technology (NIST) is required by statute (15 USC 275c) to recover costs related to its Standard Reference Materials (SRM) Program. NIST maintains clear documen- tation of the costs associated with developing, marketing, producing, and distributing SRMs. Estimating the number of units they expect to sell each year allows the program to set prices that will recover the required operational costs (Research Triangle Institute, 2000). 15 Benefits Generated by Federal Collections M any, perhaps all, IWGSC agencies have experience in justifying their requests for collection support by describing in various ways the benefits their collections generate for the agency and taxpayers. Their efforts to describe these benefits can use monetary terms (e.g., commercial revenue generated, cost reductions, productivity improve- ments). Rates of return on investment and benefit to cost ratios can be calculated in these cases, assuming that operating costs are also well known. Other benefits are specific but qualitative, and these are often described as impacts. IWGSC (2009) included a series of side- bar examples of these benefits, each identified with areas of impact such as public health (Horowitz et al., 2010; DiEuliis et al., 2016), environmental quality (Lawrey, 1993), or public safety and national security. Other previous attempts to place a value on scientific collections have used qualitative terms (Suarez and Tsutsui 2004) or the cost of creating the collections (Bradley et al., 2014). In reviewing methods used for evaluating benefits generated by scientific collections, the study’s priority was finding evidence- based methods, regardless of whether the evidence was monetary, qualitative, or descriptive. The evidence used by methods reviewed during the study include: survey responses from users; market value data; historical records of collec- tion use; expenditures by users on objects from collections; and potential savings to society through emergency mitigation. The following sections describe five methodologies available to Federal agencies. Each one views the impacts of collec- tions from slightly different perspectives and comes with particular assumptions. Each one has strengths and weaknesses; some are more time- and labor- intensive than others. Agencies may find that employing several approaches in combination is useful. Technology/Knowledge Transfer There is a substantial body of research into the monetary impact of federally funded research in science and technology. Many studies have attempted to trace products in the marketplace and the jobs and wealth they generate back to their ori- gins in government grants and research labs (see review by Ammon, Salter, and Martin, 2001). These studies must address the delays inherent in the R&D process leading from discovery through application, proof of concept, patenting, product development and testing, licensing, manufacture and eventual marketing. They must also somehow apportion the eventual market value of new products among all the contributing links in this value chain. For example, new crop varieties have been developed from accessions obtained from plant germplasm collections (see Box 2) but the monetary value of the new crops was also due to crop breeding that may have involved crops already in production, improvements in farming tech- nique, and other inputs (Rubenstein, Heisey, Shoemaker, Sullivan, and Frisvold, 2015). Partitioning the net value of the new crop among these inputs would require replicate treatments that isolate and control for each contributing factor (e.g., Evenson and Gollin, 1997; Güereña, Lehmann, Thies, Enders, Karanja and Neufeldt, 2015). Since the funds invested in the original research were unavailable for other uses during this delay, appropriate discount rates must be applied to adjust for the actual cost of the research (including opportunity costs) prior to calculating returns on investment. This same approach can be applied to scientific collections, which are often the basis of research leading to discover- ies, product development, and new products and processes. The U.S. Department of Agriculture, like many other Federal agencies in this study, publishes annual reports of technology transfer19 for ARS and for USDA as a whole. These reports present numbers of patents and licenses resulting from USDA research and the royalties received. Due to the challenges and assumptions associated with this type of economic analysis, these reports do not attempt to document the portion of the marketplace value that can be credited to USDA research or collections. 16 Economic Analyses of Federal Scientific Collections Several USDA collections are the starting points of industrial research and development. NPGS acquires, maintains, develops, conducts research and distributes plant varieties in support of breeding new crop varieties (see Box 2). Evenson and Gollin (1997) discuss the challenges associated with estimating returns on investment and benefit to cost ratios of plant germplasm collections. The ARS Culture Collection provides microbial samples that have been critical in generating new knowledge disseminated in academic publication and patents for new commercial products (see Box 3). Collections of living microbial strains (often referred to as “culture collections”) are renewable collections that are used for basic research in microbiology, and for applied research in a variety of commercial areas (e.g., agribusiness, pharmaceu- ticals). Furman and Stern (2011) discuss the economic valuation of Biological Resource Centers such as the ARS Culture Collection. Agencies involved in safeguarding public health face several challenges in demonstrating the benefits generated by the use of institutional collections. There is little doubt that they contribute to the prevention, detection, and cure of diseases, but the precise pathways between the use of collections and tangible, measurable benefits can be difficult to follow (see Box 4). Economic analyses of technology/knowledge transfer that include benefit to cost estimates involve considerable effort and expense. They are normally done by contractors who specialize in economic studies. As an alternative, agencies can Box 2. National Plant Germplasm System NPGS is a distributed network of 25 plant genebanks and support labs cooperatively operated by USDA/ARS, State Agricultural Experiment Stations, and Land- Grant Universities. The repositories receive, preserve, maintain, and distribute germplasm samples and associated information to support agricultural production by making germplasm available to users around the world, including researchers, plant breeders, growers, and other qualified users. NPGS is a renewable collection so preservation and maintenance involve propagating and testing plants to ensure their health and viability. NPGS genebanks collectively manage approximately 600,000 separate germplasm accessions. Each year, NPGS distributes 250,000–300,000 samples to users and charges no user fees. The Germplasm Resources Information Network20 (GRIN) is USDA’s searchable data system that manages the inventories of plant and animal accessions USDA’s genebanks, as well as information about the plant, animal and microbial accessions held by NPGS and other USDA germplasm collections. NPGS documents the origins of the accessions it receives and their physiological and other traits. Some accessions arrive at NPGS with considerable data and metadata, while others have little and are studied and characterized through intramural research. These new data and metadata are added to GRIN. User requests for germplasm samples vary widely, including requests for highly characterized accessions that are already cultivated as crops. These may be used in cultivation or in plant breeding aimed at developing new and more productive cultivars. Other users request wild plant relatives of crops that are poorly characterized and have no history as crop plants. These may be used for basic research and to identify, isolate and incorporate useful traits into new plant varieties. The benefits generated by NPGS take many forms: increased agricultural output; reduced losses to pests and environmental stress; increased knowledge that may contribute to food security; and increased potential to respond to food insecurity. Some of these can be measured quantitatively (Bretting, 2018), but sources other than NPGS also played important roles in generating these measurable benefits. Photo courtesy of USDA ARS Benefits Generated by Federal Collections 17 use qualitative terms to document the use of their collections by commercial entities, or non- commercial users who are working on commercial development projects. This can be accomplished by adding data gathering from users of a collec- tion to other data curation tasks. Success Stories Several of the sidebar examples in IWGSC (2009) are “success stories”—case studies in which collections helped to solve a problem or prevent losses that represented millions to billions of dollars. For example, USDA’s collections of agricul- tural pests help prevent catastrophic crop losses or trade wars over disagreements about imports with dangerous insects (see Box 8). Vaccines developed using samples in collections can curtail epidemics and save lives and avoid productivity losses, documented in “cost- of- illness” economic studies (Byford, Torgerson, and Raftery, 2000). Other examples are used Box 3. ARS Culture Collection The ARS Culture Collection (also known as the Northern Regional Research Lab Collection [NRRL]) is one of the largest public collections of bacteria and fungi in the world. It is housed within the Mycotoxin Prevention and Applied Microbiology Research Unit at the National Center for Agricultural Utilization Research in Peoria, Illinois. NRRL’s intramural research focuses on advancing agricultural production, food safety, public health, and economic development. Data about strains in the collection provided by depositors, users, and intramural staff are added to NRRL’s public database, which improves the community’s ability to find samples for further study. NRRL includes two collections. • The “Open Collection” contains 90,000 microbial strains that are owned by USDA and are made available to academic and commercial researchers without charge. These isolates represent a broad sample of biological diversity collected over more than a century. On average, 4,000 microbial cultures are distributed by the NRRL each year at no cost to users. If obtained from private culture centers, users would be charged approximately $1 million for these samples. In 2018, NRRL isolates were provided to government, academic, and industry scientists across the U.S. and 42 other countries. • The “Patent Collection” contains 7,600 microbial strains that have been deposited, typically in association with a patent application, under the NRRL’s International Depositary Authority. The NRRL is one of only two International Depositary Authorities for bacteria and fungi in the United States. Deposition of isolates in NRRL fulfills requirements of U.S. patent applications and all other countries that have signed the Budapest Treaty on the International Recognition of the Deposit of Microorganisms for the Purposes of Patent Procedures. These isolates are made available to the scientific community upon issuance of an associated patent or at the request of the depositor. NRRL distributes 400 isolates from the Patent Collection per year, on average. A portion of the operating costs of the Patent Collection is recovered in two ways. Depositors are charged a one- time fee when their isolates are accessioned and requesters are charged an access fee. These fees were authorized in the 1985 Farm Bill and are updated via U.S. Patent and Trade Office communications to the World Intellectual Property Organization. NRRL contributes directly and indirectly to technology development and business enterprises. Direct benefits can be traced through patents that cite NRRL samples. NRRL samples contribute indirectly to technology development through the new knowledge presented in research publications that cite them. NRRL isolates have been cited in more than 65,000 scientific publications, as well as 7,500 patents. A formal economic analysis of the monetary value of these direct and indirect contributions has not been attempted. Photo courtesy of USDA ARS 18 Economic Analyses of Federal Scientific Collections routinely by agencies to highlight the potential value of collections that can provide high- impact solutions to applied prob- lems. However, the benefits generated by success stories are often in areas other than the one for which they were collected. That is, the benefits are not the products of the day- to- day work of the collection or even the mission of the agency that owns the collection. For example, IWGSC (2009) included a sidebar about geologic rock cores that were collected for oil and gas discovery but are also proving valuable for mapping and predicting earthquakes. These success stories can be effec- tive for public relations and for raising awareness about collections, but they are not often viewed as compelling evidence of returns on investment or estimators of benefit to cost ratios. Economists sometimes draw an analogy between success stories and winning lottery tickets. Both involve a small in- vestment (for a lottery ticket or a few objects in a collection), a very large payoff, and very low probabilities of success. They also point out some important differences. Lotteries have a known delay between purchasing a ticket and when the winner is chosen, but the waiting time until the next collection- based success story is unknowable. In addition, the value of a winning lottery ticket is known when the winner is drawn, but there is no way to predict the value of a solution that might be found in a collection. Finally, a lottery has at least one guaranteed winner each time the winning number is picked, but there’s no guarantee that the solution to a critical problem is waiting somewhere in a collection. The random, unpredictable nature of success stories may limit their value in demonstrating returns on investment. The following example of success stories, and the subsequent example of option values, may suggest ways that agencies can highlight benefits that are valu- able but unpredictable. The National Park Service (NPS) protects and preserves an extraordinary range of habitats and life forms, from high alpine mammals to marine microorganisms. A well- known success story involves a bacterial species, Thermus aquaticus, collected and described from a hot spring in Yellowstone National Park. It was found to produce an enzyme that catalyzed Box 4. NHANES Value Chain The National Health and Nutrition Examination Survey (NHANES) is a unique public health program that obtains blood and urine samples at the same time physical exam information and responses to standardized interviews are collected. These samples and data have been collected continuously since 1999 from a statistical sample of the U.S. population. The NHANES Biospecimen Program was developed to address future medical, environmental, and public health issues challenging the Nation by maintaining a collection of serum, plasma, urine, and DNA specimens. Data produced from research using NHANES biospecimens are added to the NHANES database and made available to the public on the NHANES website. The NHANES Biospecimen Program makes samples available to any qualified researcher, though most users are other CDC Centers, Federal agencies, and academic institutions. Researchers use NHANES samples and data to establish the distributions of values for new health markers and exposure to environmental toxins in a statistically significant sample. These data are released through the NHANES website where they can be accessed for translational research that benefits public health and society. The $40 million in annual Federal support for NHANES samples and data has been critical in generating these benefits, but the same can be said for other sources of support for new health markers. Calculating the monetary and societal benefits and assigning them to the different sources of support would take considerable time and effort and has not been attempted by CDC. Photo courtesy of HHS CDC Benefits Generated by Federal Collections 19 the growth of the polymerase chain reaction (PCR) into a global biotechnology enterprise worth billions of dollars annually. Box 5 describes this success story. Option Value Bishop (1982) and Fisher and Haneman (1986) described “option value” as the benefit of having something available in case it is needed in the future, even though the probability of future use and its future value are unknown. This concept of options is used in the sale and purchase of stocks, wines, and works of literature that might become the basis of commercial films (to name a few examples). Option value can be thought of as an insurance policy that is only worth something in the occurance of an unforeseen event. Option values are therefore related to the values claimed in success stories, described above, with one important difference. As used here, option values are directly related to agency missions and the everyday uses of collections. The missions of many Federal collections include preventing or mitigating threats to public health and safety, such as risks to the nation’s food supply. For example, NPGS is a resource for developing new and better crops (see Box 2). FDA’s Foodborne Pathogen Collections are critical for ensuring the safety of food in the marketplace (see Box 6). The U.S. has witnessed many such threats to our food security: declines in agricultural output of specific crop variet- ies, foodborne disease outbreaks, and crop failures due to the introduction of insects, mold, fungi, and other agricultural pests. The same is true for epidemics, airline crashes due to bird strikes, earthquakes and other threats to public safety described in IWGSC (2009). The economic impacts of many of such events have been estimated. Agencies could compile focused knowledge bases of past events that illustrate the scope of potential losses that face the country. These past losses illustrate the option value of maintaining collections that could prevent or mitigate these losses. Box 5. Yellowstone National Park Provides Basis for Biotechnology Breakthrough The development of DNA sequencing was hampered by its minute quantities found in biological tissues. Cetus Corp. developed the Polymerase Chain Reaction (PCR), which could produce billions of DNA copies by dividing double strands by heating and assembling new complementary copies by cooling. This thermal cycling required an enzyme that could assemble complementary DNA strands while functioning at high temperatures. Kary Mullis, a Cetus researcher, was awarded the 1993 Nobel Prize in Chemistry for inventing the PCR method. Independently, microbiologists had been exploring the microbes that live in thermal hot springs. Thomas Brock, a microbiologist at Indiana University, discovered the bacterium Thermus aquaticus in Yellowstone National Park, naming it formally in 1969 (Brock and Freeze, 1969). Brock deposited representative cultures of T. aquaticus in the American Type Culture Collection (ATCC; Brock, 1997), at that time a non- Federal collection in Rockville, MD with significant NSF and NIH support. Taq polymerase is the synthetic enzyme developed from Brock’s cultures in ATCC from Yellowstone National Park. Its heat stability and efficiency enabled the success of PCR as a research tool and business enterprise. Thermal cyclers (generally known as “PCR machines”) appeared on the market in the mid- 1980s (see image21). They were soon widespread in genetics labs; a recent report put PCR- related sales in 2017 at $7.41 billion.22 Photo courtesy of DOI NPS 1985 PCR machine, SI 20 Economic Analyses of Federal Scientific Collections Value Added by Users Success stories and option values are based on rare and unpredictable uses of collections. In contrast, this approach to doc- umenting benefits is based on normal, everyday activities in collections. “Providing access to users” is one of the services listed in Table 1. Since this service increases the number of researchers who use a collection, it can be viewed as the service which has the greatest impact on the potential benefits generated by a collection. Collections often explain their value in terms of growth (accessions per year), or activity (numbers of visitors who use the collections, or numbers of specimens/ samples distributed to users), but these are input measures, not indicators of outputs or benefits. Some collections try to report the publications and/or datasets generated by users, though collecting this information from users is often difficult. Published articles and datasets are important outputs, but prior to their use by others it is difficult to assess their beneficial value beyond the professional standing of the authors. Collections will often highlight important discoveries made by users of the collection, similar in some ways to success stories described above. The Core Research Center (CRC) of the U.S. Geological Survey26 has been proactive in promoting community use of the collection and increasing the discoverability of objects in the collection through data curation (see Box 7). Their policies Box 6. FDA Foodborne Bacteria Collections The Food and Drug Administration maintains about 10 institutional collections. Among them are some of the world’s largest and most diverse collections of pathogens associated with human and veterinary illnesses found in the food and feed supply. They are housed in several facilities of FDA’s Center for Food Safety and Applied Nutrition (CFSAN) in the Washington, DC and Chicago areas. The largest of these contains approximately 40,000 strains of the bacterial genus Salmonella.23 A second collection of foodborne bacteria focuses on environmental pathogens found in seafood. It includes 5,000 well- characterized strains of the genus Vibrio housed in FDA’s Gulf Coast Seafood Laboratory24 on Dauphin Island, AL. The CFSAN strains whose whole genomes have been submitted to Genome Trakr25 are maintained as vouchers to characterize them and to ensure the reproducibility of sequencing results. FDA’s intramural research relies on their collections to develop and improve methods for detecting pathogenic bacteria and discriminating among strains, ranging from the species level down to the agents responsible for specific disease outbreaks. The collections provide strains that are assembled into test sets that are distributed for a variety of applications such as: proficiency testing of labs, (including those contributing whole genome DNA sequences to Genome Trakr); providing positive and negative controls on cross- contamination of DNA sequencing runs; validating the sensitivity, specificity and reliability of new methods for detecting and identifying foodborne pathogens; and rapid field testing for the presence of specific pathogens. CFSAN has genetically engineered some of these control strains to make them fluorescent and easily detectable. Strains in the collection are also subjected to high levels of sanitary treatments to detect the emergence of highly resistant strains. FDA responds to 100–150 qualified extramural users per year by providing 500–750 isolates to academic researchers (approximately 50% of users); industrial labs (30%); and State and other Federal agencies (20%). The FDA has not attempted to trace monetary or other impact of the use of its collections. These impacts are generated through new and improved capabilities of public health agencies, hospitals, universities, and private companies that prevent and respond to disease outbreaks caused by foodborne pathogens. However, the use of FDA collections by these sectors is known and can be considered in the context of the foodborne disease outbreaks each year and their cost to the Nation. Photo courtesy of HHS/FDA Benefits Generated by Federal Collections 21 and practices are comparable to those of companies that re- invest profits in their R&D efforts in order to increase future productivity. CRC’s “virtuous cycle” is based on promoting use of the collection, capturing and incorporating the outputs from this use into the collection, and thereby increasing future use. CDC’s NHANES Biospecimen Program also collects the results of laboratory testing done by intra- and extramural users and connects these data to the NHANES database. A list of publications resulting from studies using NHANES institutional collection samples are provided on the NHANES Biospec- imen website. When the results of analyses and sample preparations done by users are integrated back into a collection and its public databases through data curation, future users are more likely to find the samples, specimens, and data they need. The collection’s value to those users has increased. Accordingly, this study views the expenses borne by users for analytical procedures and sample preparations as “co- investments” in the collection and a form of return on investment. Patterns of user demand are the basis for understanding and documenting co- investments in collections. For example, user demand for objects in a collection might be high soon after they are added to the collection but decline soon after. Al- ternatively, user demand might be unrelated to the length of time since objects were accessioned. To determine the degree to which collections retain the user interest that drives co- investment, the study obtained historical data on user demand over time from several Federal collections. USGS’s CRC has accepted donated cores from 1974 to the present, with most coming from around the 1980s. The core samples requested from 2016 to 2018 showed this same age distribution (see Figure 2). The Smithsonian’s National Museum of Natural History (NMNH) has more than 125 million specimens and samples of plants, animals, fossils, rocks, minerals, and human artifacts. NMNH, which began computerizing its loan records in the 1960s, provided information on more than 120,000 loan requests from nine scientific departments. Figure 3 shows loan data from the Invertebrate Zoology Department, which had the most complete data. Box 7. USGS Core Research Center (CRC) CRC is an institutional collection27 that contains approximately 10,000 rock cores and more than 50,000 borehole cuttings, 95% of which were donated by private companies whose intramural use of the cores did not justify the costs of storage and maintenance. USGS evaluates the quality and rarity of donations before accepting them into the Denver, CO based repository. The collection includes cores from 35 States and cuttings from 27 States, with coverage concentrated in the Rocky Mountain region. CRC receives more than a thousand research visits per year and provided users with 10,000 samples from 2016–18. USGS does not charge academic researchers any user fees when they request and receive samples, thereby minimizing barriers to use. However, all users must provide the following within a proscribed period after receiving samples from CRC. Non- compliance can result in loss of future access to the collection. Users are required to provide: • All data derived from analyses of CRC samples; and • A duplicate of thin sections made from CRC core samples. Thin sections that users provide become part of the collection and are available to future users. Photos of thin sections and cores and all associated data records are hosted on the CRC website and they are downloadable from the collection’s catalog. Finally, CRC estimates the dollar value of the analyses and thin sections reported by users, based on market values. Photos courtesy of DOI USGS 22 Economic Analyses of Federal Scientific Collections FIGURE 2 . CRC sample requests from 2016 to 2018 Horizontal axis represents accession years of cores. FIGURE 3 . Loan data for natural history museum specimens. (A) Distribution of requested samples by year they were accessioned into collection. (B) Distribution of requested samples showing year of request versus year the requested sample was collected. Benefits Generated by Federal Collections 23 Figures 2 and, clearly, 3 show that users request access to samples and specimens in proportion to their representation in the collection. There is no evidence that their value to users diminishes over time. This suggests that the value added through co- investment by users over time could be substantial. Some agencies with collections are proactive in documenting their collections and promoting their extramural use. In order to document patterns of use and co- investment by users, agencies will need to have systems for capturing data on: • Research visitors and requests for loans and sample distributions, and • Publications and datasets released on public data repositories. To incorporate the value generated by users into the collection (making them co- investments that add value), the collection must also provide data curation services. This would involve: • Receiving and curating sample preparations done by users and returned to the collection; • Adding metadata about the returned sample preparations to the digital records of the original sample or specimen; • Capturing the metadata, digital identifiers, and web addresses of publications and datasets released on public data repositories; and • Incorporating these metadata, identifiers, and addresses into the digital records of the original sample or specimen. Counter- Factual Scenarios Counter- factual scenarios are well- established devices for economic analyses. They shift the frame of reference normally used to evaluate the value of scientific collections. Rather than exploring the value of collections to users, counter- factual scenarios explore the costs to users of not having access to the collections. Two examples are presented here. In a study of USDA’s Animal and Plant Health Inspection Services (APHIS; see Box 8 and Lichtenberg, Olson and Lawley, 2009), the agency was the principal user of the collection. The absence of the collection would have had a direct and clear effect on the agency’s ability to fulfill one of its core mission responsibilities. This allowed the study to use pro- grammatic data to estimate the financial impact of not having access to the collections. Companies were the focus of a study of the benefits generated by NIST’s SRM Program (see Box 9 and Martin, Gallaher and O’Connor, 2000). The study relied on user surveys and interviews to estimate the financial impact on the companies if they did not have SRMs. NIST does not consider the SRM Program a scientific collection, though it has many of the char- acteristics of renewable collections as described in Box 1, and it provides some of the services described in Table 1. NIST’s use of counter- factual scenarios may therefore be instructive for Federal scientific collections. Economic analyses that employ counter- factual scenarios are data- rich but labor- intensive. These studies are normally done by contractors rather than agency staff to avoid any appearance of conflicts of interest and to increase credibility. The APHIS study (Box 8) relied on programmatic data because the absence of collections (the counter- factual premise) would result in clear consequences (rejection of certain imports at ports of entry). The NIST study (Box 9) relied on structured interviews of users because there were many possible consequences to users if SRMs did not exist. The Paperwork Reduction Act limits the burden placed on non- Federal survey respondents by restricting Federal surveys to nine requests for information. OMB can issue waivers based on formal requests and some agencies have been granted waivers for cause. Complying with the Government Performance and Results Act by conducting economic analy- ses is one possible basis for requesting OMB waivers. Comparison among Methods Federal scientific collections have many stakeholders: agency researchers and administrators; Congress; OSTP and OMB; the non- Federal research community; and U.S. taxpayers. These stakeholders have different reasons for wanting to know if taxpayers are getting a good return on investments in Federal collections. Different stakeholders have different views of what constitutes “value,” “benefits,” and “returns on investment.” They may be looking for cost savings, or ways of in- creasing cost- effectiveness, or seeking ways to make management and policies more evidence- based. No single evaluation 24 Economic Analyses of Federal Scientific Collections method can give clear and simple benefit- to- cost ratios that will address these questions. The methods described here will shed new light on these matters in different ways, and the following table of strengths and weaknesses may help agencies select appropriate methods. Box 8. Counter- Factual Scenario for USDA’s Animal Plant Health Inspection Services Customs and Border Protection (CBP) staff inspect incoming agricultural shipments at U.S. ports of entry for potential agricultural pests. When they encounter evidence of insects, fungi or other potential pests, they collect, preserve and send them to USDA/APHIS area identifiers, located at larger ports and plant inspection stations around the United States. APHIS area identifiers provide the first layer of authoritative identification for commonly intercepted and unambiguously identifiable pests. When the area identifiers cannot identify them with certainty, they send samples to APHIS National Identification Services (NIS) for identification. NIS has a staff of National Taxonomists, responsible for final, authoritative identification of intercepted pests and pathogens using morphological and molecular techniques. In addition, NIS contracts with ARS staff and both use collections in ARS Systematic Laboratories and the Smithsonian Institution as definitive resources in making identifications. The costs of on- site CBP inspections, ARS contracts, and specific other APHIS activities (e.g., quarantine of imported live plants) are recovered through fees paid by importers. The long- term costs of the USDA collections that are housed near experts at ARS facilities28 and in the Smithsonian’s National Museum of Natural History29 are paid by taxpayers. At the end of this process, all shipments can be classified as: a) No potential pests found: safe for entry; b) Potential pest found and identified as benign: safe for entry; c) Potential pest found and identified as harmful but treatable: safe for entry after treatment; d) Potential pest found and identified as harmful and untreatable, not safe for entry; or e) Potential pest found and taxonomic uncertainty prevents definitive identification: not safe for entry. Lichtenberg, Olson, and Lawley (2009) developed a counter- factual scenario based on the premise that there are no reference collections or identification guides based on them. Any shipment in which potential pests were found (cases b to e, above) would not be considered safe for entry. The absence of collections would result in the rejection of all shipments in categories b and c. APHIS inspection and USDA shipment records showed that $180 million in imports fell into these categories over one year (mid- 2006 to mid- 2007). All inspection- related APHIS and ARS research and collection costs totaled $27 million during this period, resulting in a benefit to cost ratio of 4.87. USDA APHIS Image Gallery Photo by Jim Young, USDA Benefits Generated by Federal Collections 25 TABLE 4. Principal advantages and disadvantages of five methods for documenting benefits from scientific collections described in Sections 4A- E Method Principal Advantages Principal Disadvantages Technology/Knowledge Transfer • Based on tangible outcomes, often monetary • Usually connected to normal collections- based work • Can be expressed in quantitative terms (e.g., benefit- cost ratios) • Difficult to connect use of collection to ultimate outcome (delays, other contributors to process) • Sometimes serendipitous Success Stories • Can be dramatic, high value • Easily understood • Based on rare events that can’t be predicted • Can be serendipitous and unrelated to normal collections- based work Option Value • Can be dramatic, high value • Connects to historical events, easily understood • Based on probability of future use, not past performance Value Added by Users • Based on normal collection activities • Highlights patterns of collection use • Can be expressed in quantitative terms (e.g., rates of return) • Requires cooperation of users • Requires data curation • Uses narrow definition of “value” (i.e., value to users, not others) Counterfactual Scenarios • Highlights unique role of collections • Based on customer feedback and/or performance data • Can be expressed in quantitative terms (e.g., rates of return) • Customer surveys can be expensive, labor- intensive • Limitations on Federal surveys (Paperwork Reduction Act) • Distrust of survey results Box 9. Counter- factual scenario for NIST’s Standard Reference Materials Program NIST produces and sells more than 1,300 types of highly characterized and standardized SRMs to industry, academia, and government—including companies that develop, manufacture, and or use analytical instruments that are critical to assuring quality, verifying accuracy of measurements, and ensuring compliance with Federal regulations. The SRM Program shares similarities with renewable collections as defined in this report. The approach taken by NIST using economic impact analyses for the SRM Program highlights an opportunity for institutional collections. An economic analysis was commissioned by NIST to study the value of SRMs developed to measure the sulfur content of fossil fuels (Research Triangle Institute, 2000). The study interviewed representatives of nine companies from industries (e.g., coal processing, oil refining, steel production) that purchased the sulfur SRMs, asking them to report how they would have met regulatory standards if the NIST SRMs were not available. The survey asked for yearly estimates (starting with the year the company began purchasing SRMs) of: the costs and delays from finding an alternative; lost productivity and business; increased transaction costs; regulatory penalties; and other losses they would have suffered. Their estimated costs from 1988- 2003, adjusted for inflation and discounted because of delays, amounted to $412 million. The adjusted cost of operating the program (producing and marketing the SRMs, shipping, billing, overhead and other administrative expenses) was $3.7M. This represented a benefit- to- cost ratio of 112. Photo courtesy of DOC NIST 27 Implications for Policies and Management T he methods described here for documenting and estimating costs and benefits can equip agencies to make two evidence- based decisions: • What kinds of objects should they accession each year, and how many of them? • Which services should they provide, and how much of them? Some agencies operate under authorizing legislation (including the organic acts that created them), in which require- ments to retain Federal ownership of certain objects is specified or implied. This may limit an agency’s ability to make evidence- based decisions on what kinds and how many objects they accession. The following discussion explores the tensions and trade- offs caused by unfunded mandates, and how these mandates may come into conflict with the value of understanding operating costs as called for in America COMPETES. In containing costs and generating benefits, the managers of Federal institutional collections and their agency leader- ship face three structural constraints on policies and management: A. All agencies with collections face decisions on the intake and the removal of objects from their collections in order to maximize potential use and future benefits (within budget constraints), while minimizing the risk of not having important objects when need for them arises. Agencies lose the ability to make informed decisions when legislation or policies mandate that an agency must obtain and keep certain objects; B. Attracting resources (facilities, staff, and funding for operating costs) enable collections to generate benefits that advance the agency’s mission, but budget processes and environments are often zero- sum or declining. Unfunded mandates to obtain and keep certain objects limit an agency’s abilities to make informed decisions about resource allocation; and C. Expanding the range of collection- related services provided by a collection can generate benefits that advance the agency’s mission, but offering too many services (some of which may be mandated by law) may limit the quality and impact of each service provided. Constraint A Decisions about which objects to obtain, retain, and discard are difficult because future use and impact are unknown. Knowledge of future operational costs per object and agency priorities for areas of future benefits provide valuable guid- ance for these decisions. When agencies are required by legislation to maintain and preserve whole categories of objects, they lose control over future costs and the ability to pursue particular benefits. NIH has developed policies concerning the management of their project collections; USGS has done the same for its working collections. These policies30 have accompanying implementation guides31 that include criteria and decision trees for making decisions about which objects should be transferred from NIH’s project collections to institutional collections or from the Department of the Interior’s (DOI) working collections to museum property. Factors such as the uniqueness of the objects, their relevance to agency mission, the cost of long- term maintenance, and the degree to which additions complement the rest of a collection are reasonable criteria. These policies serve as examples of the informed decisions that agencies can make when not constrained by legislative and regulatory mandates. The missions of DOI and USDA include the management of different categories of Federal land (e.g., National Parks and Forests). Several Federal laws mandate that certain types of objects (e.g., archaeological artifacts, vertebrate fossils) collected from designated Federal lands must be maintained and conserved.32 28 Economic Analyses of Federal Scientific Collections Constraint B Federal appropriations are the primary source of support for the services provided by Federal collections, so funding levels limit the amount of services the collections can provide. The growth, maintenance, and preservation of collections gener- ate most of their costs and without adequate support, these basic services can limit the ability to offer other services that generate tangible benefits, especially user access, data curation, and education and outreach. Legislative, regulatory, and policy mandates have significantly increased the number of objects that agencies must ac- cession, maintain, and preserve, thereby limiting the resources available for support of objects that serve other parts of an agency’s mission. For example, DOI’s collections contain tens of millions of scientific objects, many of them added to DOI collections in compliance with legal and policy mandates. Financial and management arrangements can be difficult when ownership and stewardship are assigned to different institutions. Many DOI collections are housed and managed by non- Federal repositories, often those of the researchers who made the collections. DOI retains ownership of these collections but non- Federal institutions are their stewards. DOI’s appropriations have not been adequate for supporting collection services.33 In addition, these non- Federal institutions are not eligible to receive NSF funding for projects that would improve collections, or portions of collections, that are owned by the Federal Government. Constraint C Beyond the basic services of accessioning, documenting, maintaining, and preserving, collections face important and difficult decisions about which services to offer. If a collection or agency tries to offer other services, they may not be able to devote the resources needed to generate benefits, and they are at risk of being accused of “mission creep.” Even basic services such as preservation may suffer. Once policies about accessioning and services are set, it is sometimes necessary to reduce collection growth and services if increases in funding and other resources do not keep pace. Such reductions are likely to reduce the benefits generated, so they are very difficult to explain to collection stakeholders. Decisions about which services a collection should offer are at the center of the relationship between costs and bene- fits. That is, the services provided by a collection drive its operational costs, the types and amounts of benefits the collec- tion can be expected to generate, and the appropriate methods used to document and estimate those benefits. Collections that perform only the basic service—accessioning material—have lower costs that are easier to document, but benefits will be more difficult to generate if no other services are provided. Beyond the basic service of accessioning, each additional service increases costs, but they may also increase the types and amounts of benefits a collection can generate: • Preserving and maintaining a collection extends the time that objects will survive and can produce reliable analytical results; • Documenting additions to a collection will facilitate use by intramural researchers; • Providing access to users will expand the user base to the extramural research community, including other countries and scientific disciplines; • Data curation will add value to the collection and permit the collection to document the benefits of this co- investment; and • Increasing public understanding through education and outreach through collections can create societal benefits be- yond research and development. 29 Recommendations T his report was based on examples and experiences obtained from Federal scientific collections, but the following recommendations may be applicable to scientific collections in general. A. The framework of services, costs, and benefits described here provides collections and organizations which own col- lections with an approach for greater evidence- based policy formulation and management decision- making. Federal institutional collections should consider testing and adopting them as tools for improving operations, as well as for documenting and explaining the value of their collections to taxpayers. B. IWGSC member agencies should consider testing one or more of the methods presented here for documenting the benefits generated by collections. Several would require new data collecting efforts and added expense. CDC NHANES (Box 4) and USGS/CRC (Box 7) are collecting and providing access to value added data provided by users; USDA/ APHIS (Box 8) is using agency data for analysis in a counter- factual scenario. C. In choosing among the methods presented here for documenting benefits, officials should consider their mission and the types of benefits their collections can generate. For example, i. Collections that contribute more directly to economic development might favor technology/knowledge transfer and/or counter- factual scenarios. The former can identify the collection uses related to successful innovations or outcomes, and the latter can document the costs of not having the collections; ii. Collections that contribute to societal benefits by preparing for environmental shocks (e.g., disease outbreaks, major crop failures) might find success stories and option value more useful. Collection managers can use the former method to highlight recent events in which their collections came into use, while the latter method can describe the costs of similar shocks in the past. iii. Collections that primarily contribute new knowledge in the form of public data and academic publications might prefer value added by users and counter- factual scenarios. As described above, collections can gather data on pat- terns of collection use and co- investment by users, and can ask selected users (through surveys) what they would have done if they did not have access to the collection. D. Groups such as the IWGSC, the International Society for Biological and Environmental Repositories (ISBER), the So- ciety for the Preservation of Natural History Collections (SPNHC) and others should continue in their roles as forums for information exchange and sharing of best practices as they apply the methods described here. As collections and organizations which own collections begin to generate reports and other documents concerning costs and benefits, the IWGSC Clearinghouse34 can continue to serve as a useful platform for information exchange. 31 Appendix 1. Abbreviations and Acronyms ARS Agricultural Research Service (part of USDA) APHIS Animal and Plant Health Inspection Service (part of USDA) ATCC American Type Culture Collection BLM Bureau of Land Management (part of DOI) CBP Customs and Border Protection (part of DHS) CDC Centers for Disease Control and Prevention (part of HHS) CFSAN Center for Food Safety and Applied Nutrition (part of FDA) CRC Core Research Center (part of USGS) DHS Department of Homeland Security DOC Department of Commerce DOD Department of Defense DOE Department of Energy DOI Department of the Interior DOJ Department of Justice DOS Department of State DOT Department of Transportation EPA Environmental Protection Agency ERS Economic Research Service (part of USDA) FBI Federal Bureau of Investigation (part of DOJ) FDA Food and Drug Administration (part of HHS) FS Forest Service (part of USDA) GRIN Germplasm Resources Information Network (part of ARS) HHS Department of Health and Human Services IWGSC Interagency Working Group on Scientific Collections (part of NSTC) NASA National Aeronautics and Space Administration NDU National Defense University (part of DOD) NGRP National Genetic Resources Program (part of ARS) NHANES National Health and Nutrition Examination Surveys (part of CDC) NIH National Institutes of Health (part of HHS) NIS National Identification Services (part of APHIS) NIST National Institute of Standards and Technology (part of DOC) NLM National Library of Medicine (part of NIH) NMFS National Marine Fisheries Service (part of NOAA) NMNH National Museum of Natural History (part of SI) NOAA National Oceanic and Atmospheric Administration (part of DOC) NPGS National Plant Germplasm System (part of ARS) NPS National Park Service (part of DOI) NRRL Northern Regional Research Lab Collection (part of ARS) NSF National Science Foundation 32 Economic Analyses of Federal Scientific Collections NSF- ICF National Science Foundation Ice Core Facility NSTC National Science and Technology Council (part of OSTP) OMB Office of Management and Budget OSTP Office of Science and Technology Policy PCR Polymerase Chain Reaction PPQ Plant Protection and Quarantine Division (part of APHIS) SI Smithsonian Institution SRM Standard Reference Materials Program (part of NIST) STPI Science and Technology Policy Institute USAID Agency for International Development USDA Department of Agriculture USFSC Registry of U.S. Federal Scientific Collections USGS U.S. Geological Survey (part of DOI) VA Department of Veterans Affairs 33 Appendix 2. Collections Cited Name Collection Discipline URL Registry record Pages Department of Defense DoD Serum Repository Health Biomedical Sciences https://health.mil/Military-Health-Topics /Combat-Support/Armed-Forces-Health -Surveillance-Branch/Data-Management -and-Technical-Support/Department-of -Defense-Serum-Repository — 8 Department of Health and Human Services, Centers for Disease Control National Health and Nutrition Examination Surveys (NHANES) Health Biomedical Sciences https://www.cdc.gov/nchs/nhanes /biospecimens/biospecimens.htm https://registry.gbif.org/collection/a4f7e9a3 -c9df-443e-b874-6a7d0585453e 8, 11, 13, 18, 20, 29 Department of Health and Human Services, Food and Drug Administration Foodborne pathogens Health Biomedical Sciences https://www.fda.gov/ https://registry.gbif.org/institution/f00c1 f94-8fbf-4fc0-ac39-abeb3ec48723 13, 19, 28 CFSAN Foodborne Bacteria collection Health Biomedical Sciences — https://registry.gbif.org/collection/85b3 c137-6d2a-4a4d-a5c5-ae570c184d46 28 Gulf Coast Seafood Lab Health Biomedical Sciences — https://registry.gbif.org/collection/ebbebdb2 -6b55-4f33-9ce1-41add4cc48c7 28 Department of the Interior, National Park Service Yellowstone National Park Archaeology; Anthropology; Biological Sciences https://www.nps.gov/yell/index.htm https://registry.gbif.org/institution/d4e852 68-a913-4943-bfa2-eeb498c0ab1d 18–19, 26 Department of the Interior, U.S. Geological Survey Core Research Center Geological & Earth Sciences https://www.usgs.gov/core-science -systems/nggdp/core-research-center https://registry.gbif.org/collection/ced54b13 -6914-402c-bf26-29a7ac2c18a5 20–22, 29 National Institute of Standards and Technology Standard Reference Materials Material Sciences; Agricultural Sciences & Natural Resources; Health Biomedical Sciences https://www.nist.gov/srm https://registry.gbif.org/institution/fbcb0b2b -2d4c-4f07-9e4d-bee5737fce74 14, 23, 25 National Science Foundation NSF Ice Core Facility (NSF-ICF) Geological & Earth Sciences; Atmospheric Sciences https://icecores.org/ https://registry.gbif.org/institution/7a717 903-d4c7-4a83-a0cb-aaaca212228e 13 Living Stock Centers Program https://nsf.gov/funding/pgm_summ .jsp?pims_id=505541&org=DBI&from =home — 19 Non-Federal, Private American Type Culture Collection (ATCC) Biological Sciences https://www.atcc.org/ https://registry.gbif.org/institution/dc1823 b1-3b46-47a0-bb92-ebe6f2a4a2dd 26 https://health.mil/Military-Health-Topics/Combat-Support/Armed-Forces-Health-Surveillance-Branch/Data-Management-and-Technical-Support/Department-of-Defense-Serum-Repository https://www.cdc.gov/nchs/nhanes/biospecimens/biospecimens.htm https://registry.gbif.org/collection/a4f7e9a3-c9df-443e-b874-6a7d0585453e https://www.fda.gov/ https://registry.gbif.org/institution/f00c1f94-8fbf-4fc0-ac39-abeb3ec48723 https://registry.gbif.org/collection/85b3c137-6d2a-4a4d-a5c5-ae570c184d46 https://registry.gbif.org/collection/ebbebdb2-6b55-4f33-9ce1-41add4cc48c7 https://www.nps.gov/yell/index.htm https://registry.gbif.org/institution/d4e85268-a913-4943-bfa2-eeb498c0ab1d https://www.usgs.gov/core-science-systems/nggdp/core-research-center https://registry.gbif.org/collection/ced54b13-6914-402c-bf26-29a7ac2c18a5 https://www.nist.gov/srm https://registry.gbif.org/institution/fbcb0b2b-2d4c-4f07-9e4d-bee5737fce74 https://icecores.org/ https://registry.gbif.org/institution/7a717903-d4c7-4a83-a0cb-aaaca212228e https://nsf.gov/funding/pgm_summ.jsp?pims_id=505541&org=DBI&from=home https://www.atcc.org/ https://registry.gbif.org/institution/dc1823b1-3b46-47a0-bb92-ebe6f2a4a2dd 34 Economic Analyses of Federal Scientific Collections Name Collection Discipline URL Registry record Pages Smithsonian Institution, National Museum of Natural History Department of Invertebrate Zoology Biological Sciences; Ocean & Marine Sciences https://naturalhistory.si.edu/research /invertebrate-zoology https://registry.gbif.org/collection/0174f5b3 -da29-4967-b8dc-ce75ed53e35d 21–22 U.S. Department of Agriculture, Agricultural Research Service National Plant Germplasm System (NPGS) Agricultural Sciences & Natural Resources https://www.ars-grin.gov/ https://registry.gbif.org/institution/e45 f5702-3f7a-4eaa-8cbe-bc11cee53412 11, 13, 14, 16, 18, 19, 22 ARS Culture Collection Agricultural Sciences & Natural Resources; Health Biomedical Sciences; Biological Sciences https://nrrl.ncaur.usda.gov/ https://registry.gbif.org/collection/2f212b5e -8619-412f-baf6-ee5f0d4b5c67 11, 16, 17 Multiple ARS Systematic collections Biological Sciences — ARS: https://registry.gbif.org/institution /search?q=usda/ars%20systematic 13, 24, 34 https://naturalhistory.si.edu/research/invertebrate-zoology https://registry.gbif.org/collection/0174f5b3-da29-4967-b8dc-ce75ed53e35d https://www.ars-grin.gov/ https://registry.gbif.org/institution/e45f5702-3f7a-4eaa-8cbe-bc11cee53412 https://nrrl.ncaur.usda.gov/ https://registry.gbif.org/collection/2f212b5e-8619-412f-baf6-ee5f0d4b5c67 https://registry.gbif.org/institution/search?q=usda/ars%20systematic 35 Notes 1. Abbreviations and acronyms used throughout this document are specified in Appendix 1. 2. See IWGSC Clearinghouse: https://iwgsc.nal.usda.gov 3. 42 USC 6624; Public Law 111–358—January 4, 2011, https://www.congress.gov/111/plaws/publ358/PLAW-111publ358.pdf 4. See also IWGSC, 2013b. 5. U.S. Federal Scientific Collections are registered in the Global Registry of Scientific Collections, https://www.gbif.org/grscicoll, which is managed by the Global Biodiversity Information Facility, http://gbif.org. 6. IWGSC agency collections policies: https://iwgsc.nal.usda.gov/agency-documents 7. CDC NHANES collection: https://registry.gbif.org/collection/a4f7e9a3-c9df-443e-b874-6a7d0585453e and https://www.cdc.gov /nchs/nhanes/biospecimens/biospecimens.htm 8. DoD Serum Repository record; see also Perdue et al., 2015 9. When a research project ends, agencies can decide that objects in a project collection are no longer needed for mission-related research. If these objects are not considered appropriate for institutional collections, they can be offered to other agencies or non-Federal institutions. If transferred to a non-Federal institution, these objects may become part of another project collection or they may be accessioned into an institutional collection. For example, the National Marine Fisheries Service (NMFS) often transfers objects from completed project collections to university-based institutional collections. NSF has supported the integration of Fed- eral project collections that have been transferred from agencies (e.g., the National Oceanic and Atmospheric Administration, the Forest Service) and accessioned by non-Federal institutions. 10. NSF Advancing Digitization of Biological Collections Program: https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503559 11. USDA National Plant Germplasm System collections: https://registry.gbif.org/grscicoll/institution/e45f5702-3f7a-4eaa-8cbe-bc11 cee53412 12. ARS Culture Collection record: https://registry.gbif.org/grscicoll/collection/2f212b5e-8619-412f-baf6-ee5f0d4b5c67 13. National Ice Core Facility record: https://registry.gbif.org/institution/7a717903-d4c7-4a83-a0cb-aaaca212228e; also see https://ice cores.org/ and https://www.usgs.gov/mission-areas/core-science-systems/about/national-science-foundation-ice-core-facility 14. FDA food safety collections: https://registry.gbif.org/grscicoll/institution/f00c1f94-8fbf-4fc0-ac39-abeb3ec48723 15. Documenting additions to renewable collections (Table 1, Service 3) involves characterizing objects, (e.g., NPGS, Box 2; ARS Cul- ture Collection, Box 3). These could be considered research or a collection service unique to this type of collection. 16. The Farm Bill authorizes the ARS Culture Collection in Peoria, IL to charge user fees for access to its Patent Collection (see Box 3). 17. NSF support for Living Stock Centers is provided through the Collections in Support of Biological Research (CSBR) Program: https://nsf.gov/funding/pgm_summ.jsp?pims_id=505541&org=DBI&from=home 18. University of British Columbia’s Biobank Resource Center: https://biobanking.org/webs/biobankcosting 19. USDA and ARS: Annual Reports on Technology Transfer: http://ars.usda.gov/office-of-technology-transfer/tt-reports/ 20. Germplasm Resources Information Network: https://www.ars-grin.gov/ 21. “Mr. Cycle” was an early PCR thermal cycler; see https://americanhistory.si.edu/collections/search/object/nmah_1000862 22. https://www.globenewswire.com/news-release/2018/03/27/1453732/0/en/Global-Polymerase-Chain-Reaction-Market-Will-Reach -USD-10-62-Billion-by-2023-Zion-Market-Research.html] 23. CFSAN Foodborne Bacteria collection record: https://registry.gbif.org/grscicoll/collection/85b3c137-6d2a-4a4d-a5c5-ae570c184 d46 24. Gulf Coast Seafood Lab record: https://registry.gbif.org/grscicoll/collection/ebbebdb2-6b55-4f33-9ce1-41add4cc48c7 25. Genome Trakr is a global network of 43 U.S. and 20 non-U.S. institutions that is assembling a publicly available database of whole genome sequences of foodborne pathogens. The network includes more than 40 U.S. institutions (Federal and State public health agencies, universities, and hospitals) and 20 non-U.S. institutions (see https://www.fda.gov/food/whole-genome-sequencing-wgs -program/genometrakr-network). 26. USGS Core Research Center collection record: https://registry.gbif.org/grscicoll/collection/ced54b13-6914-402c-bf26-29a7ac2c 18a5 27. IWGSC considers CRC an institutional collection. DOI does not use the terms “institutional collections” and “project collections” as defined in IWGSC (2013a). DOI classifies its collections as “museum property” and “working collection” (defined in DOI Depart- mental Manual Part 411, https://www.doi.gov/sites/doi.gov/files/uploads/411dm1_museum_property_policy.pdf). USGS considers CRC a working collection. https://iwgsc.nal.usda.gov https://www.congress.gov/111/plaws/publ358/PLAW-111publ358.pdf https://www.gbif.org/grscicoll http://gbif.org https://iwgsc.nal.usda.gov/agency-documents https://registry.gbif.org/collection/a4f7e9a3-c9df-443e-b874-6a7d0585453e https://www.cdc.gov/nchs/nhanes/biospecimens/biospecimens.htm https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503559 https://registry.gbif.org/grscicoll/institution/e45f5702-3f7a-4eaa-8cbe-bc11cee53412 https://registry.gbif.org/grscicoll/collection/2f212b5e-8619-412f-baf6-ee5f0d4b5c67 https://registry.gbif.org/institution/7a717903-d4c7-4a83-a0cb-aaaca212228e https://icecores.org/ https://icecores.org/ https://www.usgs.gov/mission-areas/core-science-systems/about/national-science-foundation-ice-core-facility https://registry.gbif.org/grscicoll/institution/f00c1f94-8fbf-4fc0-ac39-abeb3ec48723 https://nsf.gov/funding/pgm_summ.jsp?pims_id=505541&org=DBI&from=home https://biobanking.org/webs/biobankcosting http://ars.usda.gov/office-of-technology-transfer/tt-reports/ https://www.ars-grin.gov/ https://americanhistory.si.edu/collections/search/object/nmah_1000862 https://www.globenewswire.com/news-release/2018/03/27/1453732/0/en/Global-Polymerase-Chain-Reaction-Market-Will-Reach-USD-10-62-Billion-by-2023-Zion-Market-Research.html https://registry.gbif.org/grscicoll/collection/85b3c137-6d2a-4a4d-a5c5-ae570c184d46 https://registry.gbif.org/grscicoll/collection/ebbebdb2-6b55-4f33-9ce1-41add4cc48c7 https://www.fda.gov/food/whole-genome-sequencing-wgs-program/genometrakr-network https://registry.gbif.org/grscicoll/collection/ced54b13-6914-402c-bf26-29a7ac2c18a5 https://www.doi.gov/sites/doi.gov/files/uploads/411dm1_museum_property_policy.pdf https://www.cdc.gov/nchs/nhanes/biospecimens/biospecimens.htm https://registry.gbif.org/grscicoll/institution/e45f5702-3f7a-4eaa-8cbe-bc11cee53412 https://registry.gbif.org/grscicoll/collection/85b3c137-6d2a-4a4d-a5c5-ae570c184d46 https://www.fda.gov/food/whole-genome-sequencing-wgs-program/genometrakr-network https://registry.gbif.org/grscicoll/collection/ced54b13-6914-402c-bf26-29a7ac2c18a5 36 Economic Analyses of Federal Scientific Collections 28. Beltsville fungal collection, https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center /mycology-and-nematology-genetic-diversity-and-biology-laboratory/docs/us-national-fungus-collections-bpi/us-national -fungus-collections-databases/ 29. USDA Systematic Entomology Laboratory, https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural -research-center/systematic-entomology-laboratory/; NMNH insect collection, https://naturalhistory.si.edu/research/entomology 30. NIH Collections Policy https://policymanual.nih.gov/1189; USGS Policy on Scientific Working Collections https://www.usgs.gov /products/scientific-collections/usgs-policy-scientific-working-collections 31. Companion Guide: Guidance for Implementation of the NIH Policy for the Management of and Access to Scientific Collections https://osp.od.nih.gov/wp-content/uploads/2016/08/Companion%20Guide.pdf; USGS implementation guide to collections policy https://www.usgs.gov/products/scientific-collections/guide-planning-and-managing-scientific-working-collections-us 32. Examples include the 1906 Antiquities Act, the 1916 NPS Organic Act, and most recently, the 2009 Paleontological Resources Preservation Act (Subtitle D of the Omnibus Land Management Act of 2009: https://www.congress.gov/111/plaws/publ11/PLAW -111publ11.pdf). Legislation concerning DOI’s museum collections policies: https://www.doi.gov/museum/policy 33. See DOI Office of the Inspector General 2009 and 2016, and annual DOI reports on museum property management: https://www .doi.gov/museum/annual-reports 34. https://iwgsc.nal.usda.gov https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center/mycology-and-nematology-genetic-diversity-and-biology-laboratory/docs/us-national-fungus-collections-bpi/us-national-fungus-collections-databases/ https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center/systematic-entomology-laboratory/ https://naturalhistory.si.edu/research/entomology https://policymanual.nih.gov/1189 https://www.usgs.gov/products/scientific-collections/usgs-policy-scientific-working-collections https://osp.od.nih.gov/wp-content/uploads/2016/08/Companion%20Guide.pdf https://www.usgs.gov/products/scientific-collections/guide-planning-and-managing-scientific-working-collections-us https://www.congress.gov/111/plaws/publ11/PLAW-111publ11.pdf https://www.doi.gov/museum/policy https://www.doi.gov/museum/annual-reports https://iwgsc.nal.usda.gov https://www.doi.gov/museum/annual-reports https://www.congress.gov/111/plaws/publ11/PLAW-111publ11.pdf https://www.usgs.gov/products/scientific-collections/usgs-policy-scientific-working-collections https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center/systematic-entomology-laboratory/ https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center/mycology-and-nematology-genetic-diversity-and-biology-laboratory/docs/us-national-fungus-collections-bpi/us-national-fungus-collections-databases/ https://www.ars.usda.gov/northeast-area/beltsville-md-barc/beltsville-agricultural-research-center/mycology-and-nematology-genetic-diversity-and-biology-laboratory/docs/us-national-fungus-collections-bpi/us-national-fungus-collections-databases/ 37 References Albert, M., J. Bartlett, R. N. Johnston, B. Schacter, and P. Watson. 2014. Biobank Bootstrapping: Is Biobank Sustainability Possible through Cost Recovery? Biopreservation and Biobanking 12(6). https://doi.org/10.1089/bio.2014.0051 American Alliance of Museums. 2017. “Museums as Economic Engines: A National Report.” Commissioned by the American Alliance of Museums and conducted by Oxford Economics. Ammon, J., A. J. Salter, and B. R. Martin. 2001. The Economic Benefits of Publicly Funded Basic Research: A Critical Review. Research Policy 30(3):509–532. Available online February 28, 2001. https://doi.org/10.1016/S0048- 7333(00)00091- 3 Baker, R. J., L. C. Bradley, H. J. Garner, and R. D. Bradley. 2014. “‘Door to Drawer’ Costs of Curation, Installation, Documentation, Da- tabasing, and Long- Term Care of Mammal Voucher Specimens in Natural History Collections.” Occasional Papers of the Museum of Texas Tech University, Lubbock, Texas. Bishop, R. C. 1982. Option Value: An Exposition and Extension. Land Economics 58(1):1–15. Bradley, R. D., L. C. Bradley, H. J. Garner, and R. J. Baker. 2014. “Assessing the Value of Natural History Collections and Addressing Issues Regarding Long- Term Growth and Care.” BioScience 64:1150–1158. https://doi.org/10.1093/biosci/biu166 Bretting, P. K. 2018. 2017 Frank Meyer Medal for Plant Genetic Resources Lecture: Stewards of Our Agricultural Future. Crop Science 58:2233–2240. https://doi.org/10.2135/cropsci2018.05.0334 Brock, T. D. 1997. The Value of Basic Research: Discovery of Thermus aquaticus and Other Extreme Thermophiles. Genetics 146(4): 1207–1210. Brock, T. D., and H. Freeze. 1969. Thermus aquaticus gen. n. and sp. n., a Nonsporulating Extreme Thermophile. Journal of Bacteriology 98(1):289–297. Byford, S., D. J. Torgerson, and J. Raftery. 2000. Cost of Illness Studies. BMJ 2000; 320:1335. https://doi.org/10.1136/bmj.320.7245.1335 DiEuliis, D., K. R. Johnson, S. S. Morse, and D. E. Schindel. 2016, Opinion: Specimen Collections Should Have a Much Bigger Role in Infectious Disease Research and Response. Proceedings of the National Academy of Sciences 113(1):4- 7. https://doi.org/10.1073 /pnas.1522680112 Evenson, R. E., and D. Gollin. 1997. Genetic Resources, International Organizations, and Improvement in Rice Varieties. Economic De- velopment and Cultural Change 45(3):471–500. Fisher, A. C., and W. M. Hanemann. 1986. Option Value and the Extinction of Species. Advances in Applied Micro- Economics 4:169–190. Furman, J. L., and S. Stern. 2011. “Climbing atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research.” American Economic Review 101:1933–1963. Graves, P. 2003. “Valuing Public Goods.” Challenge 46:100- 112. Güereña D. T., J. Lehmann, J. E. Thies, A. Enders, N. Karanja, and H. Neufeldt. 2015. Partitioning the Contributions of Biochar Proper- ties to Enhanced Biological Nitrogen Fixation in Common Bean (Phaseolus vulgaris). Biology and Fertility of Soils 51:479–491. https:// doi.org/10.1007/s00374- 014- 0990- z Halewood M. 2013. What Kind of Goods Are Plant Genetic Resources for Food and Agriculture? Towards the Identification and De- velopment of a New Global Commons. International Journal of the Commons 7(2):278–312. https://www.jstor.org/stable/26523131 Horowitz, D. B., E. C. Peters, I. Sunila, and J. C. Wolf. 2010. Treasures in Archived Histopathology Collections: Preserving the Past for Future Understanding. Histologic XLIII(1):1–8. Interagency Working Group on Scientific Collections. 2009. “Scientific Collections: Mission- Critical Infrastructure for Federal Science Agencies.” Report of the IWGSC. https://iwgsc.nal.usda.gov/sites/default/files/IWGSC_GreenReport_FINAL_2009.pdf Interagency Working Group on Scientific Collections. 2013a. “Recommendations for Departmental Collections Policies.” Report of the IWGSC. https://iwgsc.nal.usda.gov/sites/default/files/IWGSC_Recommend_Collxns_Policies_2013_01_28.pdf Interagency Working Group on Scientific Collections. 2013b. “Best Practices for Budgeting for Scientific Collections.” Report of the IWGSC. https://repository.si.edu/bitstream/handle/10088/99408/IWGSC%20Collections%20Budget%20Recommendations%20FINAL -2013.pdf?sequence=1&isAllowed=y Kothamasi, D., M. Spurlock, and E. Kiers. 2011. Agricultural Microbial Resources: Private Property or Global Commons? Nature Biotech- nology 29:1091–1093. https://doi.org/10.1038/nbt.2056 Lawrey, J. D. 1993. Lichens as Monitors of Pollutant Elements at Permanent Sites in Maryland and Virginia. The Bryologist 96:339–341. Lichtenberg, E., L. J. Olson, and C. Lawley. 2009. The Value of Systematics in Screening Imports for Invasive Pests. Unpublished man- uscript, June 17, 2009. Link, A. N., and J. T. Scott. 2012. “The Theory and Practice of Public- Sector R&D Economic Impact Analysis.” National Institute of Standards and Technology Planning Report 11- 1, January 2012. https://doi.org/10.1089/bio.2014.0051 https://doi.org/10.1016/S0048-7333(00)00091-3 https://doi.org/10.1093/biosci/biu166 https://doi.org/10.2135/cropsci2018.05.0334 https://doi.org/10.1136/bmj.320.7245.1335 https://doi.org/10.1073/pnas.1522680112 https://doi.org/10.1007/s00374-014-0990-z https://doi.org/10.1007/s00374-014-0990-z https://www.jstor.org/stable/26523131 https://iwgsc.nal.usda.gov/sites/default/files/IWGSC_GreenReport_FINAL_2009.pdf https://iwgsc.nal.usda.gov/sites/default/files/IWGSC_Recommend_Collxns_Policies_2013_01_28.pdf https://repository.si.edu/bitstream/handle/10088/99408/IWGSC%20Collections%20Budget%20Recommendations%20FINAL-2013.pdf?sequence=1&isAllowed=y https://doi.org/10.1038/nbt.2056 https://doi.org/10.1073/pnas.1522680112 https://repository.si.edu/bitstream/handle/10088/99408/IWGSC%20Collections%20Budget%20Recommendations%20FINAL-2013.pdf?sequence=1&isAllowed=y 38 Economic Analyses of Federal Scientific Collections Martin, S. A., M. P. Gallaher, and A. C. O’Connor. 2000. “Economic Impact of Standard Reference Materials for Sulfur in Fossil Fuels.” National Institute of Standards and Technology Planning Report 00- 1, February 2000. Odeh, H., L. Miranda, A. Rao, J. Vaught, H. Greenman, J. McLean, D. Reed, S. Memon, B. Fombonne, P. Guan, and H. M. Moore. 2015. The Biobank Economic Modeling Tool (BEMT): Online Financial Planning to Facilitate Biobank Sustainability. Biopreservation and Biobanking 13(6):421–429. https://doi.org/10.1089/bio.2015.0089 Office of Science and Technology Policy. 2010. “Policy on Scientific Collections.” Memorandum to the Heads of Executive Departments and Agencies. OSTP, October 6, 2010. https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO_Scientific_Collxns_Policy_2010 _10(1).pdf Office of Science and Technology Policy. 2014. “Improving the Management of and Access to Scientific Collections.” Memorandum to the Heads of Executive Departments and Agencies. OSTP, March 20, 2014. https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO _Scientific_Collxns_FINAL_2014_03(1).pdf Perdue, C. L., A. A. Eick- Cost, and M. V. Rubertone. 2015. A Brief Description of the Operation of the DoD Serum Repository. Military Medicine 180(10):10–12. Reichman, J. H., P. F. Uhlir, and T. Dedeurwaerdere. 2015. Governing Digitally Integrated Genetic Resources, Data, and Literature: Global Intellectual Property Strategies for a Redesigned Microbial Research Commons. New York: Cambridge University Press. Research Triangle Institute. 2000. “Economic Impact of Standard Reference Materials for Sulfur in Fossil Fuels.” National Institute of Standards and Technology Planning Report 00- 1. Robbins, J. 2006. The Search for Private Profit in the Nation’s Public Parks. New York Times, November 28, 2006. https://www.nytimes .com/2006/11/28/science/28yell.html Rubenstein, K. D., P. Heisey, R. Shoemaker, J. Sullivan, and G. Frisvold. 2005. “Crop Genetic Resources: An Economic Appraisal.” A Report of the Economic Research Service. Economic Information Bulletin Number 2, May 2005. U.S. Department of Agriculture. https://www.ers.usda.gov/webdocs/publications/44121/17452_eib2_1_.pdf?v=0 Schuhmann, P. W. 2014. “Non- Market Valuation: Methods and Data.” Slide set, University of North Carolina, Wilmington. https://www .sesync.org/sites/default/files/education/economics- 4.pdf Smale, M., and B. Koo. 2003. Introduction: Taxonomy of Genebank Value. Brief 7. In What is a Genebank Worth? ed. Smale, M., and B. Koo, Briefs 7–12, Biotechnology and Genetic Resource Policies. International Food Policy Research Institute, December 2003. Stauderman, S., and W. G. Tompkins, eds. 2016. Proceedings of the Smithsonian Institution: Summit on the Museum Preservation Environment. Washington, DC: Smithsonian Institution Scholarly Press. https://doi.org/10.5479/si.9781935623878 Suarez, A. V., and N. D. Tsutsui. 2004. The Value of Museum Collections for Research and Society. BioScience 54:66–74. Tassey, G. 2003. “Methods for Assessing the Economic Impacts of Government R&D.” National Institute of Standards and Technology Planning Report 03- 1, September 2003. U.S. Department of the Interior, Office of Inspector General. 2009. “Museum Collections: Accountability and Preservation.” Report No. C- IN- MOA- 0010- 2008, December 2009. https://www.doioig.gov/sites/doioig.gov/files/2010- I- 0005.pdf U.S. Department of the Interior, Office of Inspector General. 2016. Verification Review – Recommendations for the Report “Department of the Interior’s Accountability and Preservation of Museum Collections” (Audit No. C- IN- MOA- 0010- 2008),” Report No. 2016- CR- 018. https://www.doioig.gov/sites/doioig.gov/files/2016CR018Public_0.pdf https://doi.org/10.1089/bio.2015.0089 https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO_Scientific_Collxns_Policy_2010_10(1).pdf https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO_Scientific_Collxns_FINAL_2014_03(1).pdf https://www.nytimes.com/2006/11/28/science/28yell.html https://www.ers.usda.gov/webdocs/publications/44121/17452_eib2_1_.pdf?v=0 https://www.sesync.org/sites/default/files/education/economics-4.pdf https://doi.org/10.5479/si.9781935623878 https://www.doioig.gov/sites/doioig.gov/files/2010-I-0005.pdf https://www.doioig.gov/sites/doioig.gov/files/2016CR018Public_0.pdf https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO_Scientific_Collxns_FINAL_2014_03(1).pdf https://iwgsc.nal.usda.gov/sites/default/files/OSTP_MEMO_Scientific_Collxns_Policy_2010_10(1).pdf https://www.nytimes.com/2006/11/28/science/28yell.html https://www.sesync.org/sites/default/files/education/economics-4.pdf Economic Analyses of Federal Scientific Collections Methods for Documenting Costs and Benefits David E. Schindel and the Economic Study Group of the Interagency Working Group on Scientific Collections Open access PDF available from Smithsonian Institution Scholarly Press at https://doi.org/10.5479/si.13241612 Smithsonian Scholarly Press Schindel_final high res C1.pdf Schindel-IWGSC_FINAL_web.pdf Schindel_C4_web.pdf Blank Page