![[PREVIOUS]](http://www.istl.org/graphics/backward.gif)
![[CONTENTS]](http://www.istl.org/graphics/contents.gif)
![[NEXT]](http://www.istl.org/graphics/forward.gif)
| Issues in Science and Technology Librarianship | Winter 2000 |
|---|
| DOI:10.5062/F4DF6P6J |
| URLs in this document have been updated. Links enclosed in {curly brackets} have been changed. If a replacement link was located, the new URL was added and the link is active; if a new site could not be identified, the broken link was removed. |
Much has been written on issues pertaining to licensing and archiving of digital information. Until recently, there has not been enough information to evaluate how these digital products, particularly journals, are being used. Furthermore, meaningful data are often difficult to obtain as some publishers and vendors supply little or no data or only information they feel supports the purchase of their products. As it becomes increasingly difficult to afford all digital content, librarians must be able to measure digital use of e-journals and books in order to make the best purchasing decisions for their institutions. Librarians must develop their own solutions as well as solutions in collaboration with publishers so that better evaluation of digital content use can occur.
Understanding what is "of value" becomes even more difficult and complex as an increasing number of our resources become digitized. Different digital formats, interfaces, pricing structures, and access restrictions complicate our ability to evaluate journal resources using consistent measures. The very nature of information is changing; by virtue of desktop access, ability to manipulate information, and value-added features, we need to ask if the same content is as valuable today as it was five years ago. Considering the changing nature of our library collections, it is probably time to redefine the role of performance as well as the nature of what and how we measure it.
Here is a scenario that is similar to recent discussions at the Bernard Becker Medical Library at Washington University School of Medicine in St. Louis:
As the journal renewal process for the year 2000 takes place, a group of librarians gather to review a report listing journal usage statistics, the cost per use, and the subscription rate for all print journals housed in the library. A line is drawn at a certain use level based upon the available funding. Once the total is hit, all titles below that line are considered for cancellation. A title falling beneath that line is found to also have an electronic counterpart. However, there are no electronic statistics available for that particular title from the publisher. Thus, it is impossible to determine accurately the actual overall use of the journal. The librarians wonder what the level of electronic use was and whether that use would move that particular title above the cancellation line. The librarians also wonder about the title just above the cancellation line that does not have an electronic counterpart. Does the library retain it over the slightly lesser used title that possibly has electronic uses? For example, when evaluating Current Opinions in Psychiatry, librarians found that the journal was used only 16 times in print format. However, the corresponding statistics for Current Opinions in Psychiatry in Journals@Ovid revealed that it was accessed nearly 300 times. Based on this additional information the title was retained.
Additionally, another title with low use is provided as part of a bundled set of journals. Cancellation of this title, no matter how poorly used in any format, nullifies the library's contract resulting in increased prices for the other titles. Nevertheless, knowing the amount of electronic usage could help the librarians decide when it might be time to renegotiate or attempt to modify the contract.
Finally, the library group notices that the overall use for important print journals is falling off. The nature and historical importance of these journals are such that there is no doubt that they are not losing their scholarly presence. The answer is fairly obvious: they are being accessed online. With this increased online access, many researchers are accessing these journals from their desktops rather than visiting the library.
Generally most journal evaluation reports inform librarians how their print collections are utilized. They do not, however, supply any information on how often or what resources are being used on the client desktops. Moreover, the typical journal evaluation report does not identify an expanding web-based user base. It also does not help determine if the library is reaching previously underserved populations nor does it suggest whether users will forgo information found only in the library for the convenience of getting similar information in their offices.
So what should be our performance measures and market research questions?
This paper will focus on two reporting mechanisms that offer useful information for evaluative purposes.
We can begin with one set of measures available for some Highwire Press titles and determine what they tell us about performance. The statistics provided for these limited titles1 are among the most extensive on the market. An example of the December 1999 use-data for the Journal of Biological Chemistry (JBC) is included in Figure 1. As shown, the statistics include:
The Highwire statistics provide some basic data as well as some meaningful novel data. What is particularly useful is the number of unique IP addresses that access this site. We know for instance, that during the month of December there were over 550 unique IP addresses that accessed JBC over 1200 times. From this data we can conjecture that a wide range of researchers frequently use this resource. Thus, this type of reporting not only provides a sense of the volume, but also the breadth of use for this particular title.
Other statistics for JBC give us the opportunity to learn more about our users' habits. While this information may not correlate directly to purchase and cancellation decisions, it helps us ascertain how researchers are using these value-added resources. In addition, this information helps us identify areas in which training might be appropriate. For instance, during the month of December, 1,169 searches were performed in JBC. This suggests that users are doing more than going to the site to obtain a specific citation. They may be using this journal's special online feature to search for new information and use it as an information-gathering tool. It also gives us a window into the format preferences of our users. Many users tell us they prefer PDF, but for JBC in most months in 1999, most JBC users chose HTML format over PDF.
|
Journal of Biological Chemistry December 1999
| |
| Subscriber ID: | xxxxx |
| Contact: | Xxxxxxxx Xxxx Associate Director for Technical Services |
| Phone: | 314-362-6802 |
| Email: | xxxxx@msnotes.wustl.edu |
| Total-usage statistics | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Access Events to: | Home Page | Current issue TOC | All TOCS | Searches | Abstracts | Full text HTML | PDFs | User Names | IP addresses in use |
| Total Usage | 1,262 | 389 | 604 | 1,169 | 436 | 1,608 | 1,420 | ||
| Unique Events | 1 | 57 | 338 | 1,014 | 897 | 1 | 558 | ||
| Article Usage by Section | |||||||
|---|---|---|---|---|---|---|---|
| Section: | Total Articles Online | Abstracts | Full text HTML | PDFs | |||
| Unique Articles Accessed | Total Accesses | Unique Articles Accessed | Total Accesses | Unique Articles Accessed | Total Accesses | ||
| Mini-Reviews | 442 | 0 | 0 | 46 | 103 | 38 | 116 |
| Communications | 1,042 | 13 | 24 | 50 | 70 | 50 | 88 |
| Carbohydrates, Lipids, and Other Natural Products | 1,024 | 14 | 17 | 30 | 36 | 23 | 27 |
| Cell Biology and Metabolism | 9,009 | 160 | 210 | 516 | 741 | 493 | 755 |
| Enzymology | 2,276 | 16 | 19 | 57 | 78 | 48 | 66 |
| Membranes and Bioenergetics | 1,776 | 9 | 12 | 50 | 60 | 34 | 59 |
| Nucleic Acids, Protein Synthesis, and Molecular Genetics | 5,397 | 49 | 63 | 152 | 198 | 102 | 142 |
| Protein Chemistry and Structure | 3,791 | 18 | 20 | 106 | 154 | 105 | 160 |
| Additions and Corrections | 235 | 0 | 0 | 5 | 6 | 2 | 3 |
| Other Articles | 44,935 | 59 | 71 | 0 | 0 | 0 | 0 |
| Totals | 69,937 | 338 | 436 | 1,012 | 1,446 | 895 | 1,416 |
| Top 10 Articles in December 1999 | |||||
|---|---|---|---|---|---|
| Full text HTML | Abstracts | Total Accesses | Age of Article in days from 12/31/1999 | Article | |
| 11 | 19 | 0 | 30 | 15 | Section: Mini-Reviews Moshe Oren Regulation of the p53 Tumor Suppressor Protein Dec 17, 1999 274: 36031-36034 [Full Text] |
| 5 | 17 | 0 | 22 | 8 | Section: Mini-Reviews Mikhail Bogdanov, William Dowhan Lipid-assisted Protein Folding Dec 24, 1999 274: 36827-36830 [Full Text] |
| 2 | 12 | 0 | 14 | 1 | Section: Mini-Reviews Michael D. Kaytor, Stephen T. Warren Aberrant Protein Deposition and Neurological Disease Dec 31, 1999 274: 37507-37510 [Full Text] |
| 3 | 11 | 0 | 14 | 22 | Section: Communications Masatoshi Yamasaki, Noriyo Hashiguchi, Chiharu Fujiwara, Tsuneo Imanaka, Toshiro Tsukamoto, Takashi Osumi Formation of Peroxisomes from Peroxisomal Ghosts in a Peroxisome-deficient Mammalian Cell Mutant upon Complementation by Protein Microinjection Dec 10, 1999 274: 35293-35296 [Abstract] [Full Text] |
| 2 | 9 | 2 | 13 | 22 | Section: Communications Inaki Azpiazu, Humberto Cruzblanca, Ping Li, Maurine Linder, Min Zhuo, N. Gautam A G Protein gamma Subunit-specific Peptide Inhibits Muscarinic Receptor Signaling Dec 10, 1999 274: 35305-35308 [Abstract] [Full Text] |
| 7 | 6 | 0 | 13 | 862 | Section: Cell Biology and Metabolism Dayang Wu, Herschel D. Wallen, Naohiro Inohara, Gabriel Nunez Interaction and Regulation of the Caenorhabditis elegans Death Protease CED-3 by CED-4 and CED-9 Aug 22, 1997 272: 21449-21454 [Abstract] [Full Text] |
| 7 | 6 | 0 | 13 | 15 | Section: Cell Biology and Metabolism Yuping Yuan, Suhasini Kulkarni, Philippe Ulsemer, Susan L. Cranmer, Cindy L. Yap, Warwick S. Nesbitt, Ian Harper, Nayna Mistry, Sacha M. Dopheide, Sascha C. Hughan, David Williamson, Corinne de la Salle, Hatem H. Salem, Francois Lanza, Shaun P. Jackson The von Willebrand Factor-Glycoprotein Ib/V/IX Interaction Induces Actin Polymerization and Cytoskeletal Reorganization in Rolling Platelets and Glycoprotein Ib/V/IX-transfected Cells Dec 17, 1999 274: 36241-36251 [Abstract] [Full Text] |
| 2 | 8 | 3 | 13 | 1 | Section: Nucleic Acids, Protein Synthesis, and Molecular Genetics Jennifer R. Saam, Jeffrey I. Gordon Inducible Gene Knockouts in the Small Intestinal and Colonic Epithelium Dec 31, 1999 274: 38071-38082 [Abstract] [Full Text] |
| 5 | 7 | 0 | 12 | 22 | Section: Mini-Reviews Kostas Tokatlidis, Gottfried Schatz Biogenesis of Mitochondrial Inner Membrane Proteins Dec 10, 1999 274: 35285-35288 [Full Text] |
| 3 | 5 | 4 | 12 | 22 | Section: Communications Onno Kranenburg, Ingrid Verlaan, Wouter H. Moolenaar Dynamin Is Required for the Activation of Mitogen-activated Protein (MAP) Kinase by MAP Kinase Kinase Dec 10, 1999 274: 35301-35304 [Abstract] [Full Text] |
| 47 | 100 | 9 | 156 | 99 (avg age) | Totals for Top 10 Articles in December 1999 |
What does this information tell us about our performance? First and foremost it tells us that this particular resource is heavily used. In addition, it identifies subject areas most frequently viewed, providing clues into what areas might be good targets for training. It also tells us that researchers frequently use the search feature. This information may support the development and implementation of classes devoted to searching full-text information. The report further suggests that our users will download an HTML document rather than a PDF document. This information may help us make decisions about other product formats. Most importantly, the report indicates that this online journal is a valuable resource for our users since it is accessed over 1,300 times a month. In comparison, the print version of JBC is used on average only 475 times a month.
The next example that helps define performance comes from the Becker Library. Before Ovid Technologies, Inc. included a statistical package with its system, members of library staff and Medical School Computing Services (MSCNS) wrote a statistical package that could use Ovid's log files. We are able to extract data from the log files into a Sequel Server database and write reports in Microsoft Access to answer usage questions. Using this software, we are able to identify:
We are also able to tailor a report for a particular need. For instance, when there was a billing problem concerning the usage of the Internal Medicine residents, we were able to generate a usage report for that group alone. Additionally, we can examine an individual's account to determine how many hours an individual has used Ovid for a given period of time. A portion of one report generated from library data is illustrated in Figure 2.
Percentage of EUCLID Use by Departments and Divisions FY99
| Department Name | Percentage |
|---|---|
| MEDICAL HOUSE STAFF-I&R | 9.52% |
| PATHOLOGY-INSTRUCTION&RESEARCH | 4.87% |
| GEN SURGERY ADMIN CLINC. PRACT | 4.14% |
| CELL BIOLOGY AND PHYSIOLOGY I&R | 3.67% |
| BIOLOGY | 2.93% |
| PSYCHIATRY | 2.85% |
| INT MED-CARDIOLOGY | 2.79% |
| OCCUPATIONAL THERAPY | 2.61% |
| MOLECULAR BIOLOGY & PHARMACOLOGY | 2.58% |
| OPHTHALMOLOGY | 2.56% |
| SURGICAL PATHOLOGY | 2.40% |
| NEUROLOGY | 2.33% |
| INT MED-PULMONARY DISEASES | 2.24% |
| PEDIATRICS-Administration | 2.20% |
| BIOCHEMISTRY & MOLECULAR BIOPHYS | 1.94% |
| INT MED-GASTROENTEROLOGY | 1.91% |
| INT MED-LABORATORY MEDICINE | 1.83% |
| PHILOSOPHY | 1.83% |
| ANATOMY & NEUROBIOLOGY | 1.70% |
| INT MED-RENAL DISEASE | 1.55% |
| INT MED-METABOLISM | 1.51% |
| ORTHOPEDIC SURGERY DEPT. | 1.46% |
| INT MED-INFECTIOUS DISEASES | 1.45% |
| BARNARD CANCER CENTER | 1.32% |
| INT MED-DERMATOLOGY DIV-ADMIN | 1.31% |
| OTOLARYNGOLOGY-INSTRUCTION&RESEA | 1.18% |
| DIVISION OF ONCOLOGY | 1.07% |
This information has been valuable for marketing services, identifying groups or departments for training, deciding how many licenses to buy for each database and journal title, and determining which databases and journals to retain or cancel. What is surprising is that for every item canceled from Ovid, we tend to replace it with another. We have not decreased our business with this information vendor; rather we have used this information to obtain the best and most economical products for our particular user population.
Contrast these two examples with the usage data obtained from other commercial vendors. One vendor sends statistics that list, for each day, the number of queries, the number of sessions, average length of session and the number of "turnaways" (i.e., the number of times users have exceeded licenses). Another vendor sends a counting of the types of pages accessed (e.g., "list of journals," "articles," "issues," "help," etc.), the top 20 titles accessed each month and the journal titles accessed but not subscribed to by the library. In addition the form and type of article downloaded are also summarized. This type of limited information can give the library a sense of the preferred format, the length of an average session, or what the most popular resources are. Moreover, this information can provide us with a sense of the volume of use, though often not the breadth. Unfortunately, this information is often incomplete, frequently generating more questions concerning usage patterns than it answers.
Currently, the Association of Research Libraries is establishing a project on determining usage measures for electronic information resources; see {http://www.arl.org/stats/program/newmeas.html}. In addition, The University of Michigan University Library and The Program for Research on the Information Economy are sponsoring a research conference on the economics and use of digital library collections; see {http://www.si.umich.edu/PEAK-2000/}. Internationally, the UK Serials Group's 2000 conference will be holding workshops on monitoring e-journal usage and as well as a session on performance measurement for academic libraries. On a local level, the University of Colorado library system is expanding its pilot study to take a more rigorous and refined look at the use of online products for its four campuses.
A more organized and standardized approach to obtaining meaningful and useful data should not lead to mass cancellations of subscriptions. On the contrary, informed decisionmaking should lead to an increased trust between libraries and publishers and expand and enhance a library's ability to respond to information needs. This will only help maintain library budgets that in turn will help to maintain or increase purchasing power. Finally, will this improve our performance as providers of information? The answer is obvious. By placing ourselves firmly at the center of gathering data and determining the electronic use characteristics of our products and researchers, we position ourselves as the true knowledge managers for the 21st Century.
We welcome your comments about this article.
