College and Research Libraries Some Benefits of the Online Catalog Malcolm Getz Library users benefit from automated services by being able to complete library tasks more quickly and by increasing the likelihood of success. This essay reports measures of the gains in search time and likelihood of success in known-item searches with the Acorn automated catalog at Vanderbilt. User time and search success are valued in dollar terms to make the measures meaningful for system-design choices. he Heard Library at Vanderbilt Universitv introduced an on- line catalo'g called Acorn in Sep- tember 1985. 1 We treat the event as a natural experiment and make ''before'' and ''after'' comparisons of li- brary use in order to judge how the online system affects library service. We ascribe dollar values that facilitate comparison of costs and benefits in future work. A system designer faces choices that would permit a system to function more quickly but at an increased cost. By valu- ing the time saved, we may inform such a decision. Likewise, a designer may have opportunities to increase the likelihood of success but would result in a more expen- sive system. By valuing the improvement in search success, we help a designer face such a choice. We view this report on the effect of Acorn for users as a first step. During 1986, the circulation and acquisitions functions were implemented, and future investigation will address the benefits of these and other developments in auto- mated services. This is an exploratory effort to assess the most important immediate effects of an electronic catalog. Other benefits remain unmeasured; indeed, there will be other long-run consequences of automated sys- tems that ultimately will p:rove to be quite important. At this point, we work with what we can measure. The study relies primarily on Paul Kan- tor's techniques for measuring access and availability of library materials both before and after the implementation of the elec- tronic catalog. 2 We began with a survey of persons entering the library to determine what tasks they perform there. Then we explored how several library-user tasks are changed by Acorn. In particular, we asked how much time users save when finding. materials with the electronic sys- tem and how much more successful they are when using the electronic catalog. The Heard Library serves Vanderbilt University, a private university with 5,500 undergraduates and 3,500 graduate and professional-school students. The univer- sity, including the Medical School, has about 1,300 full-time faculty. The Heard Library holds about 1.6 million volumes in seven divisions including the Central/Sci- ence Division. This study relies primarily on evidence from the Central Library's collection of about 600,000 volumes in the humanities and social sciences. Conclu- Malcolm Getz is Associate Provost for Information Services and Technology at Vanderbilt University, Nash- ville, Tennessee 37203. Margaret Ann Trotter led the survey efforts underlying this study, and her work is much appreciated. Flo Wilson provided useful comments and supplementary data about Acorn. The staff of the Heard Library assisted in gathering the necessary data. I wish to thank the following for their helpful comments: Patricia Battin, Yale M. Braunstein, Jane Burke, Richard Burkhauser, Richard M. Dougherty, Cliff Huang, Paul B. Kantor, Michael J. McGill, Bruce A. Miller, Timothy Richards, and two anonymous referees. 224 sions are generalized to apply to all but the Law, Medical, and Music divisions. In 1984-85 the library used a key-sort card circulation system and card catalogs in each division. Acorn came online in September 1985. VVe surveyed card- catalog use in January, April, and July 1985 and Acorn use in December 1985 and February and March 1986. In these survey Some Benefits of the Online Catalog 225 periods, Acorn included 500,000 titles, about two-thirds of Heard's collections. VVe expect the total 1.6 million-volume collection to represent about 800,000 titles. LIBRARY USE Library users were asked to complete the survey form included as figure 1. The survey was distributed to each person en- Please tell us what you do in the library today. Check all that apply. ACTIVITIES Finding Time in ___ _ 0 Se~ch catalog by author/title 0 Se~ch catalog by subject 0 Se~ch index to periodicals 0 Ask libr~ian or staff 0 Browse shelves Other 0 Photocopy 0 Microform printer 0 Interlibrary loan 0 Check books out 0 Return books Time out ___ _ Reading 0 Your own materials 0 Reference materials 0 Reserve materials 0 Other library books 0 Journals 0 Microform materials 0 Government documents Other ______________________________ __ PURPOSE Class Relate(! 0 Assigned reading 0 Work on term paper 0 Work on speech Research 0 Qualifying exams 0 Thesis or dissertation Other ____________ __ 0 Sponsored rese~ch project 0 Prep~g a publication STATUS 0 Faculty 0 Grad./prof. student 0 Undergraduate 0 Staff 0 Other professional 0 Personal School/dept.--------------------------- Ye~cametoVU _________________________ _ Other ____________________________ _ COMMENTS FIGURE 1 The Jean and Alexander He~d Library-User Survey 226 College & Research Libraries tering the library during a complete day for several arbitrarily chosen days in the spring and summer of 1985 in the Cen- tral/Science and Education Library divi- sions. Respondents completed 1,690 usa- ble forms. This figure represents about a 60% response rate among persons enter- ing the libraries during the survey inter- val. Faculty, graduate and undergraduate students, and others responded as shown in the first column of table 1. As a check for validity, we compared the proportion of respondents who are faculty (6.8%); undergraduates (45.5%); graduate students (36.3%); and others (11.5%) 3 against the same proportions for total at- tendance in the library: faculty (4.8%); un- dergraduates (49.7%); graduate students (40.4%); and others (5.2%). The Heard Li- brary has an access-control system that checks a bar code at entry and tabulates users by type. (Total attendance for 1984-85 is shown in the first column of ta- ble 8.) Faculty were slightly more likely to complete the survey form at each visit than the undergraduate and graduate stu- dents.4 In generalizing our findings, we weight the result for each of the four groups by the known aggregate annual at- tendance; this slight bias in rate of reply should not distort our overall findings. We asked users about the purpose of their visits and about any finding, read- ing, and other activities they undertook during their visits. Table 1 reports the total number of completed forms showing any finding activity, 41.9% overall; any read- May 1987 ing activity, 72.0%; and any other activity, 43.1 %. In contrast, table 2 reports the number of tasks marked on the survey forms, that is, a tally of all the boxes marked under the finding, reading, and ''other'' categories respectively. Table 3 shows a breakdown of the find- ing tasks by type. Overalt about 18% of the visits include one or more author/title searches; about 12% engage in each of the following: subject search in the catalog, search in a periodical index, consult a li- brarian, and browse. Author/title search- ing is much more important among fac- ulty; periodical indexes are much more important for graduate students. Subject searching and consulting a library staff member are most important among un- dergraduates engaged in finding. Because an online catalog is intended to improve methods of finding materials in the li- brary, we are most interested in exploring how Acorn affects these tasks. Although finding tasks are more fre- quent than reading tasks for faculty, read- ing is much more frequent among the other groups. Table 4 reports the pattern of reading tasks. Although all groups make important use of their own materi- als, reading one's own material is most important for undergraduates. We can identify the share that particular reading tasks hold in relation to the total number of reading tasks involving library materi- als, that is, exclusive of the "own materi- als" tasks. Considering reading tasks that involve library materials as the denomina- TABLE 1 LffiRARY USE: COUNT OF RESPONDENTS* Activities Purpose Number of Oass Patrons Finding Reading Other Related Research Faculty 114 73 68 43 18 85 64.0% 59.6% 37.7% 15.8% 74.6% Graduate/professional 613 314 439 336 321 322 51.2% 71.6% 54.8% 52.4% 52.5% Undergraduate 769 227 594 257 589 93 29.5% 77.2% 33.4% 76.6% 12.1% Other 194 94 115 92 45 91 48.5% 59.3% 47.4% 23.2% 46.9% Total 1,690 708 1,216 728 973 591 41.9% 72.0% 43.1% 57.6% 35.0% *Percentages indicate the proportion of respondents who engage in any task under a given activity. Percentages total more than 100% because many users do more than one activity per visit. ,. +> 1.,.1 Some Benefits of the Online Catalog 227 TABLE2 LffiRARY USE: TALLY OF ALL TASKS MARKED UNDER A GIVEN ACTIVITY* Activities Purpose Number of Class Patrons Finding Reading Other Related Research Faculty 114 89 97 52 27 99 78.1% 85 .1% 45 .6% 23.7% 86.8% Graduate/professional 613 517 772 476 435 386 84.3% 125.9% 77.7% 71.0% 63.0% Undergraduate 769 383 831 340 784 103 49.8% 108.1% 44 .2% 102.0% 13.4% Other 194 134 150 100 50 112 69.1% 77.3% 51.5% 25.8% 57.7% Total 1,690 1,123 1,850 968 1,296 700 66.4% 109.5% 57.3% 76.7% 41.4% *The tally of tasks exceeds the number of respondents shown on table 1 because respondents may engage in more than one task in a given activity . TABLE 3 FINDING TASKS Search Search Number of Catalog by Author Catalog by Patrons or Title Subject Facul~ 114 43 5 % o visits 37.7% 4.4% % of finding tasks 48 .3% 5.6% Graduate/professional 613 137 98 %of visits 22.3% 16% % of finding tasks 26.5% 19.0% Undergraduate 769 80 83 %of visits 10.4% 10.8% % of finding tasks 20.9% 21.7% Other 194 37 26 %of visits 19.1% 13.4% % of finding tasks 27.6% 19.4% Total 1,690 297 212 %of visits 17.6% 12.5% % of finding tasks 26.4% 18.9% tor, we see that just over a quarter of all visits involve consultation of reference materials, that reserve materials are prin- cipally consulted by undergraduates, and that about a fifth of all groups consult the books in the stacks. Only about a sixth of the undergraduate library collection read- ing tasks involve journals. The pattern of reading tasks should not be immediately affected by the introduction of the elec- tronic catalog, but as other electronic ser- vices develop there may be an effect. Other user tasks include charging books, requesting interlibrary loans, and making copies. The rates of occurrence of these tasks are reported in table 5. Photo- copying is most important to students; in- Search Ask Double Single Periodical Librarian Count Count Index or Staff Browse Total Aggregate 8 16 17 89 73 7% 14% 14.9% 78.1% 64% 9.0% 18.0% 19.1% 100% 118 74 90 517 314 19.2% 12.1% 14.7% 84.3% 51.2% 22.8% 14.3% 17.4% 100% 64 91 65 383 227 8.3% 11.8% 8.4% 49.8% 29.5% 16.7% 23.8% 17.0% 100% 18 27 26 134 94 9.3% 13.9% 13.4% 69.1% 48.5% 13.4% 20.1% 19.4% 100% 208 208 198 1,123 708 12.3% 12.3% 11.7% 66.4% 41 .9% 18.5% 18.5% 17.6% 100% terlibrary loans are most important to fac- ulty. Book charge occurs on only about one in seven library visits overall. As with reading activities, the electronic catalog is not expected to have a direct effect on these tasks, and we report this summary simply to provide a full context for think- ing about what our users do in the library. The survey establishes a baseline for considering how electronic services affect library use. We can consider how the pat- terns of use change over time. More espe- cially for this essay, we can consider those particular tasks that are affected by the on- line catalog and undertake investigations to explore exactly how the tasks are af- fected. By seeing t~e effects in the context 228 College & Research Libraries May 1987 TABLE 4 READING TASKS Your Other Double Single Number of Own Reference Reserve Library Microform Government Count Count Patrons Materials Materials Materials Books Journals Materials Documents Total Aggregate Facultl 114 19 20 4 14 31 4 5 97 68 % o visits 16.7% 17.5% 3.5% 12.3% 27.2% 3.5% 4.4% 82.5% 59.6% %of library materials 25.6% 5.1% 17.9% 39.7% 5.1% 6.4% 80.4% Graduate/ 613 175 169 70 105 187 38 28 772 439 Professional %of visits 28.5% 27.6% 11.4% 17.1% 30.5% 6.2% 4.6% 125.9% 71.6% %of library materials 28.3% 11.7% 17.6% 31.3% 6.4% 4.7% 77.3% Undergraduate 769 447 115 88 79 65 21 16 831 594 %of visits 58.1% 15% 11.4% 10.3% 8.5% 2.7% 2.1% 108.1% 77.2% %of library materials 29.9% 22.9% 20.6% 16.9% 5.5% 4.2% 46.2% Other 194 39 31 7 25 39 6 3 150 115 %of visits 20.1% 16% 3.6% 12.9% 20.1% 3.1% 1.5% 77.3% 59.3% %of library materials 27.9% 6.3% 22.5% 35.1% 5.4% 2.7% 74% Total 1,690 680 335 169 223 322 69 52 1,850 1,216 %of visits 40.2% 19.8% 10% 13.2% 19.1% 4.1% 3.1% 109.5% 72% %of tasks with library materials 28.6% 14.4% 19.1% 27.5% 5.9% 4.4% 63.2% TABLE 5 OTHER TASKS Number of Microform Patrons Photocopy Printer Faculty 114 18 0 15.8% 0% Graduate/professional 613 185 14 30.2% 2.3% Undergraduate 769 174 11 22.6% 1.4% Other 194 40 1 20.6% .5% Total 1,690 417 26 24.7% 1.5% of the overall use· of the library, we can de- termine the importance of particular changes relative to total library use. CHANGES IN RATES OF CIRCULATION AND LIBRARY USE Aggregate patterns of library use do not appear to have been affected immediately by Acorn. Table 6 reports annual totals for circulation, reshelving, photocopying, reference questions, and number of pho- tocopies made in Central Library for each Inter- Check Library Books Return Loan Out Books Other Total 10 13 5 6 52 8.8% 11.4% 4.4% 5.3% 45.6% 26 120 82 49 476 4.2% 19.6% 13.4% 8.0% 77.7% 6 83 50 16 340 .8% 10.8% 6.5% 2.1% 44.2% 2 25 24 18 110 1.0% 12.9% 12.4% 9.3% 51.5% 44 241 161 89 978 2.6% 14.3% 9.5% 5.3 57.9% of seven years. Figures for the first six years show the annual changes during the period before Acorn, and those for the last year, 1985-86, show the outcomes for the first year with Acorn. The figures in table 6 are portrayed graphically in figure 2. The last column of table 6 indicates that enroll- ment changed little during this period: changes in library use are not caused by changes in enrollment. A sharp departure might have been ex- pected with the introduction of Acorn. ~ l Some Benefits of the Online Catalog 229 TABLE6 CENTRAL LffiRARY USE Reference Number of Photocopy Circulation Reserve Reshelving Questions Photocopies Income Enrollment* 85/86 99,759 20,454 221,013 16,380 960,812 $41,428 9,069 84/85 106,006 22,296 271,746 17,777 793,188 34,662+ 9,117 83/84 116,346 23,068 293,199 19,162 467,573 36,353 9,035 82/83 112,204 21,747 276,481 19,941 414,296 32,125 8,782 81/82 107,868 21,960 216,936 18,600 428,323 34,576 8,911 80/81 105,800 23,980 188,773 18,674 417,135 32,107 8,874 79/80 105,749 29,434 171,489 17,806 371,458 29,361 9,125 *Enrollment figures are for the fall semester. +Price per copy decreased to 5¢ from 10¢ . 1,000,000 900,000 800,000 .... 700 , 000 600,000 500 , 000 400 , 000 Photocopies 300 , 000 200 , 000 Reshe1ving 100,000 ~ Circulation Reserve 79/80 80/81 81/82 82/83 83/84 84/85 85/86 86/87 FIGURE2 Central Library Use 1979 to 1986 230 College & Research Libraries However, the results for 1985-86 are char- acteristic of those for the preceding six years. Acorn has not lead to a sudden shift in library use. Perhaps with more years of experience, we will be able to identify some change in trends associated with the arrival of Acorn. We do note the very sharp increase in photocopying in 1984-85, an increase as- sociated with a decrease in the price per copy from $.10 to $.05. The further in- crease in 1985-86 may be associated with the introduction of a magnetic debit card for photocopying, replacing coins. The fall in circulation in these last two years might be associated with the increase in photo- copying, suggesting that photocopying and book circulation are partial substitutes-convenient and inexpensive photocopying allows a user to copy a few pages rather than check out a book. Records for attendance at the library are incomplete-there does not appear to be any significant trend in attendance. We conclude that the introduction of the electronic catalog has had no effect on the aggregate rate of library use at Vanderbilt, at least in its first year. We turn then to a consideration of how the electronic ser- vice affects the time required to find an item in the library. ACCESS TIME A principal benefit of an electronic cata- log is the decrease in time required to find materials. We compared the finding time for known-item searches using Paul Kan- tor's "Measure of Access Effort by Simu- lation." Each member of a group of Van- derbilt students was given six author/title citations chosen randomly from the Cen- tral Library's shelflist. Each student took a stop watch and timed the four steps of finding each item seriatnn: (1) at the cata- log, (2) to the stacks, (3) return to circula- tion point and complete any needed form, and (4) checkout. Experienced library users may find a dif- ferent degree of advantage in the elec- tronic system than inexperienced users, and so we measure access time with two different groups of subjects: (1) a group of relatively inexperienced library users and (2) a group of student library employees May 1987 from technical services who had been ex- tensively trained in finding materials. Each group completed the simulation in 1985 before the electronic catalog became available; similar groups completed the simulation in 1986 after the catalog had been in place for several months. Citations were randomly assigned to the searchers, using a different list each year. The average time for completing each of the four steps in finding a given item is re- ported in table 7. Section I reports the time for the inexperienced student library us- ers, line A showing the time at the manual catalog and line B, time at the electronic catalog. Section II reports similar times for the exp~rienced users. At this point, we were interested in comparing the time required to find a cita- tion in the catalog because that is the task that should be affected by the introduction of the electronic system. We report the times for the other three steps for com- pleteness, and because we expect future research to make use of these. For exam- ple, providing better location cues in the catalog might reduce the time'' to stacks.'' The introduction of bar codes and auto- mated circulation should reduce the time "to return" and "checkout." First, let's ask why we should consider experienced users separately from the in- experienced. Section III of Table 7 reports t-ratios comparing the mean time of expe- rienced and inexperienced users in using the card catalog (E) and in using the elec- tronic catalog (F). We quickly see that the 1 minute, 24 seconds required at the card catalog by an inexperienced user is signifi- cantly more than the 58 seconds required by an experienced user in a statistical sense. Similarly, the 1 minute, 16 seconds required at the electronic catalog by an in- experienced user is significantly more than the 39 seconds required by an experi- enced user. Experienced users do appear to perform differently at the catalog than the inexperienced. In comparing the per- formance of different cataloging systems, then, we will want to look separately at ex- perienced and inexperienced users. In comparing the time required at the catalog for the card system and for Acorn, first consider the experienced users. The Some Benefits of the Online Catalog 231 TABLE 7 MEAN TIME TO FIND AND CHECK OUT GIVEN ITEMS, ONE AT A TIME MINUTES: SECONDS (STANDARD DEVIATIONS ARE IN PARENTHESES) I. Inexperienced Users A. Card catalog, 1985 B. Acorn, 1986 T -statistics for difference in means (degrees of freedom) F-statistic in variances (degrees of freedom) II. Experienced Users C. Card catalog, 1985 D. Acorn, 1986 T -statistics for difference in means (degrees of freedom) F-Statistic for difference in variances (degrees of freedom) III. T-statistics for comparison of inexperienced and experienced users E. Card catalog, 1985 (comparing A and C) (degrees of freedom) F. Acorn, 1986 (comparing Band D) (degrees of freedom) *Statistically significant at the O.Ollevel tStatistically significant at the 0.05 level student library workers who had been trained in using the catalog required an average of 58 seconds to locate an item in the card catalog and only 39 seconds in the electronic system. The mean difference of 19 seconds per item is statistically signifi- cant at the 1% level. We conclude, then, that the introduction of Acorn has re- duced the time required for experienced users to find a known item in the catalog. In comparing the times for the inexperi- enced users, we have a more complex story. The mean time at the card catalog of 1 minute, 24 seconds looks similar to the 1 minute, 16 seconds at Acorn. We cannot conclude, however, that the electronic system made no difference. The t-test to compare the sample means ·depends on the assumption that the vari- ances of the distributions are the same. Here, the variances of the samples of "at catalog'' times are significantly different at the 1% level. The F-statistic with 41 and At Catalog To Stacks Return Checkout Total n 1:24 ( :51) 1:16 1:33 0.58 (100) 3.75* (41,57) 0.58 ( :31) 0:39 ( :23) 3.4* (94) 1.12 (59,35) 3.34+ (118) 2.71* ( 76) 3:21 (1:45) 2:34 (1:46) 2:19 (1:06) 2:25 (1:07) 3.59* (106) 0.75 ( 70) 2:39 ( :56) 2:40 ( :29) 2:28 ( :37) 2:36 ( :45) 1.17 (102) 0.40 ( 70) 0:44 8:08 58 ( :40) 0:45 7:15 42 ( :42) 0.32 6:17 60 ( :30) 0:35 6:15 36 ( :25) 1.71+ (102) 1.29 ( 70) 57 degrees of freedom is 3.75. We con- clude, then, that the samples are drawn from different distributions, the Acorn sample having a higher variance. With the electronic system we are able to look at the search behavior in more detail. We examined the log of the Acorn ses- sions of some of our searchers and learned that one individual experienced difficulty using the electronic system. If the two ex- traordinarily long search times, one of 5 minutes and one of 8 minutes, are ex- cluded, the variance is reduced and the mean search time by the inexperienced us- ers would be cut by 16 seconds, making the mean times significantly different. This individual experienced misses on Acorn because of entering authors' first name first, entering titles beginning with the word ''the,'' and continuing to type a title beyond the allowed length and so overwriting the first part of the entry. For these reasons, the individual took an ex- 232 College & Research Libraries traordinary amount of time to complete a search. The Acorn log reported 23 searches leading to hits, and 4 searches leading to misses. The mean time from the first enter key being touched, initiating a search, until the desired citation is found was 18.94 seconds for the 23 successes. The time for the successes ranged from 0.2 seconds to 46.1 seconds. The longer times occurred when the searcher used a strat- egy involving several steps, for example, an author's last name as the starting point. Experienced users learn more direct search strategies. For the four misses, the mean time from the first enter key until the last response is 3 minutes 2 seconds. All the failed searches were by the same searcher looking for one item. Of course, similar miscues will confuse one in a card catalog, and we have not at- tempted to observe these among our sub- jects in the card searches. We suspect that improvements in Acorn software may mitigate the problem of search errors of the type found here. We should then be able to find measurable time savings among inexperienced users in locating known items in the catalog. One could consider more effort at training inexperi- enced users, but training efforts among in- experienced users have not had a high level of acceptance at Vanderbilt. We con- clude that most inexperienced users have found significant reductions in time at the catalog, but that some have found a frus- trating increase in search times. Our goal in this study is to measure the time savings experienced by using Acorn. We have differentiated relatively experi- enced from relatively inexperienced us- ers. We have not sought to measure the whole learning curve of library users as they approach Acorn for the first time. Such an investigation would be a worth- while subject for future work. In the re- mainder of this analysis, we exclude the subject who, despite a claim to the con- trary, showed no previous experience with Acorn whatever. By this assumption, we are able to assume a 19-second time saving for each known-item search, whether by experienced or inexperienced users. To establish how many users have different levels of experience and so to as- May1987 sign an individual time saving to each group is beyond the scope of the present effort. THE VALUE OF SEARCH TIME SAVED Let's suppose that, after an initial intro- duction, our users save an average of 19 seconds on each author/title search per- formed in the library. This represents are- duction of about one-third for experienced users, and about one-fifth for inexperi- enced users. On average, each person searching for known items sought 2.2 items per visit, a seek rate we observed in monitoring success rates below. If 19 sec- onds are saved on each search, an average of 41.8 seconds is saved per author/title searcher per visit to the library. We can estimate the value of the search time saved by users. Such an estimate of value may be useful in making design choices when we face choices about new automated services. For example, the in- troduction of keyword and Boolean search capabilities may reduce search time, on average. Such capabilities, however, will come at the cost of significantly larger computer systems. We may wish to judge value of time saved in improved searching against the added system cost. The notion of valuing user time relies on an assumption about library user behav- ior, namely, that users seek to minimize the amount of time required to achieve a particular outcome and would be willing to pay to achieve an outcome more quickly. Users may take such time savings and do yet more searching, or they may use the time in other ways. In many hu- man endeavors, people behave as though they place a monetary value on time and so we assume they value time saved in completing library tasks as well. 5 If users value time at zero, there would be little point in organizing a library to make it convenient. If users place a very high value on time, we would expect libraries to be willing to go to great lengths to in- crease convenience. We will assume that the value users place on their time bears some relationship to their wage rate. To address the value of time saved, we estimate the total number of author/title { .. / .. ... J j ' searches performed in the library per year and so estimate the total number of hours saved. Then we make an assumption about the value library users place on their time and so arrive at an estimate of the to- tal value of time saved. As noted above, the Heard Library has a bar code-activated turnstile system that records each entrance to the library by ma- jor category of user. Column A of table 8 reports the aggregate total attendance at the four divisions of the Heard Library with turnstile counts (excludes Law, Med- ical, and Music) for 12 months in the 1984-85 year. We use the irlformation about finding tasks from our survey of library use re- ported above. The author/title search rate given in table 3 is reported again in column B of table 8. Recall, for example, 37.7% of faculty visitors engaged in author/title searching. Multiplying the av- erage number of incidents of author/title searching per visit times the number of visits yields an estimate of the number of author/title searchers for a year for each group of users as reported in column C of table-B. Assuming that each author/title searcher saves 41.8 seconds, on average, for each author/title search session under- taken with Acorn, we find the number of hours of author/title searching saved as shown in column D. We estimate, then, that about 1,294 hours of user time is saved annually in author/title searching as Some Benefits of the Online Catalog 233 a consequence of Acorn. This time savings might be compared to the total amount of time users spend in the library. Users av- erage about 95 minutes in the Central Li- brary in each visit, thus there are about 1 million hours of use annually. The author/ title search time saved seems quite small relative to this total. A more appropriate assessment, how- ever, is in terms of the value of user's time. We use the count of faculty by rank from Vanderbilt's Registry and reports in The Chronicle of Higher Education about Van- derbilt's average faculty salaries by rank to compute an average hourly rate of about $18. From published job listings that indi- cate educational requirements and start- ing salaries, we determine that the aver- age hourly rate paid to entry level Vanderbilt employees with bachelor's de- grees was about $6.90 per hour. Similarly, we learn that the average rate paid for staff persons with less than a college degree was about $4.60 in 1986. The value stu- dents place on their time is likely to be at least the wage rate they could have earned were they not students, and so these wage rates seem to be a relevant starting point in determining the value of time saved. Other factors might be considered in valuing search time as well. In evaluating transportation systems, investigators have routinely found that time that is high in nuisance value, that is, time spent walk- ing, waiting, and transferring between conveyances, is valued by commuters by TABLES VALUE OF TIME SAVED IN AUTHOR/mLE SEARCHES, CENTRAL LffiRARY B. D. F. G . A. Author/ c. Hours E. Value Value ~~~te Title AIT Saved Value of Search of Time Searching Searches 41.8 Seconds of Time Time Saved (41.8sec.) UserOass Visits per Visit Annually per Searcher per Hour Saved per Searcher Faculth 31,509 0.377 11,879 137.9 $25 $3,448 29.0¢ Grad. prof. 265,250 0.223 59,151 686.8 7.50 5,151 8.7¢ Undergrad 26,568 0.104 33,963 394.3 5.00 1,972 5.8¢ Other 33,977 0.191 6,4QO 75.4 7.50 ,566 8.7¢ Total 657,304 0.176 115,686 1,294.4 $8.63 $11/137 10.0¢ Sources: A. Information from entrace turnstiles, 12 months 1984-85 B. Sample survey of 1,690 users of the Heard Library, 1985 C. column A times column B , D. Assumes 41.8 seconds saved for each author/title searcher, 2.2 searches per searcher. See table 7. E. Value of time reflects wage and salaries of each group, roughly. F. column D times column E G . column E times 41.8/3600. 234 College & Research Libraries about three times more than time spent moving. That is to say, a person will typi- cally be willing to spend three times more to avoid a minute of walk, wait, and trans- fer time than he or she will be willing to spend to avoid a minute of moving time. If we view time spent searching as nuisance time, time akin to walk, wait, and transfer time, then we may suspect that persons will be willing to spend significantly more than their wage rates to avoid a minute of searching time. This line of reasoning seems especially plausible for faculty. We know of faculty who never used the card catalog, but instead went directly to the stacks and depended on the shelf order to find even known items. Five percent of students at Vanderbilt who responded to a student service's poll indicated that they did not use any catalog when using the li- brary. Such persons may find the time saved with the online system sufficient to induce them to use the catalog routinely. Finally, Vanderbilt students appear to have life prospects that are significantly better than average and so a value of time above local wage rates seems appropriate. Investigation into how library users value their time would be a desirable avenue for research but is beyond the scope of the present essay. In order to make clear that the values for time are rough estimates and to account for the likely nuisance factor in catalog use, we choose round numbers for the value of time for each class of user: $25 per hour for faculty, $7.50 per hour for grad- uate/professional students and the ''other'' group, and $5 per hour for under- graduate students. Weighting these rates by the annual searching of each group yields an average value of time for library users of $8.63 per hour as shown in column E of the table. Applying these rates, we can value the searching time saved as shown in column F of table 8. The total value of time saved appears to be about $11,000 annually, with about 77% of it being attributable to faculty and graduate students. The uncer- tainty underlying the valuation of time might lead one to prefer expressing the value as a range. Given the 41.8 seconds saved per searcher, the total value of time May1987 saved might plausibly range from $8,000 to $18,000. Another way of applying the value of time is to determine the value of the 41.8 seconds saved by each author/title searcher. These rates are shown in column G of table 8, with 29.0 cents worth of time saved for each faculty searcher, 8. 7 cents saved for each graduate student searcher, and 5.8 cents saved for each un- dergraduate searcher. AVAILABILITY Another major consequence of an auto- mated catalog is the increased probability of success in finding specific items. To in- vestigate this issue, we applied Paul Kan- tor's "Measure of Availability" tech- nique. Users of the catalog were stopped and asked whether they were looking for a specific item. Those who responded posi- tively were asked to complete a form re- porting the author and title, the call num- ber if found, and to mark if they found the item on the shelf. The form was collected at the exit to the library, and library staff members immediately followed up those cases marked ''not found'' to determine the cause of failure. With the card catalog, in spring 1985, we surveyed 637 known item searches at the Central Library and learned that 367 items were found on the shelf, an overall suc- cess rate of 57.6%, as shown in table 9. With Acorn in operation, we sur- veyed 325 known-item searches in fall 1985 and learned that 208 of the items were found on the shelf, an overall suc- cess rate of 64%, as shown on line 2 of the table. This difference is significant at the 5% level with a one-tailed test. However, it may be inappropriate to in- fer that the improvement is due to the electronic catalog. By exploring the causes of failure, we gain more insight. Kantor defines a series of conditional success rates, that, when multiplied together, yield the overall success rate. (Kantor takes account of illegible responses and non-existent titles by apportioning such outcomes among the analyzed failures. The figures reported here and in table 9 have been adjusted in this manner.) Consider the first reason for failure, { namely that the Central Library had not acquired the item. Of the 637 known items searched, 112 or,17.6% were not acquired, including 68 that were never ordered, 5 that were on order but unavailable, and 32 that were owned by other divisions. The remaining 525 were owned by Central and so we have a conditional success rate of 525/637, or 82.4%, meaning that if all books that were acquired were available on the shelf and found by searchers, the overall success rate would be 82.4%. In the fall, this rate was 85.5%, a significant im- provement. Seventeen items among the 525 ac- quired by the library were described in the catalog but not found there by the search- ers, implying a conditional success rate at the catalog of (525-17)/525, or 96.7%. In the fall, this rate was 95.2%, an inconse- quential variation in performance. Of the 508 items acquired and identified by searchers in the catalog, 51 were not on the shelf because they were in circulation, implying a conditional success rate II at the shelf" of (508-51)/508, or 89.9%. In the fall, this rate was 88.6%, an insignificant differ- ence. Of the 457 items owned, found in the catalog, and not checked out, 65 were on reserve, at the bindery, or otherwise miss- ing, implying a conditional success rate of (457-65)/457 or 85.7%. This rate was 90.6% in the fall, a significant improvement. Of the 392 items owned, found in the catalog, not checked out, and not missing, 25 were on the shelf but not found by us- ers, implying a conditional success rate in the stacks of 93.6%. In the fall, this rate was 98.1, a statistically significant differ- ence. In looking at the comparison of spring and fall success rates as an indication of the effect of the online system, one should ignore those that involve circulation, ab- sence, and stack failure because, in fall 1985, the system did little to influence these matters. Instead, one should focus on line D, the success rate at the catalog and on line C, those items owned in other divisions. Acorn did not lead to improvement in success at finding known items (line D), perhaps because only two-thirds of Cen- Some Benefits of the Online Catalog 235 tral Library's holdings were included in the electronic database at that time. As the electronic catalog database approaches full coverage, perhaps this conditional success rate will improve. Improvements in the searching software may increase this conditional success rate as well. Acorn does score a success as a union catalog for the library, as indicated by line C of Table 9, that is, with respect to items owned by other divisions. The card cata- log was not a complete union catalog, and so items from across all the divisions could not be found there. 6 Users and library staff might consult the public OCLC terminals to determine that the Heard Library owned the item, but could not determine which division owned it. As a union cata- log, Acorn provides holdings information for all divisions (for items in the electronic database). Therefore, the items counted as not owned by Central in spring 1985 are appropriately considered failures-they accounted for 12.7% of the failures. In the fall, however, the items counted as owned by other divisions should count as catalog successes because the catalog provides complete location information. The consequence of having a union cata- log can be seen by comparing row I with row II at the bottom of table 9. Assuming that the other divisions have similar con- ditional success rates in the other determi- nants of availability, the overall success rate for known item searches is given by row I when only Central is considered. Row II gives the overall conditional suc- cess rate when the holdings of the whole system are considered. We compute the system success rate by moving the 34 "in catalog at other divisions'' from the failure side to the success side, and applying the other conditional success rates to the re- vised catalog success rate. In the spring, the aggregate success rate for Central alone as 57.5. This rate would have been 61.2% if the card catalog were a union cat- alog. In the fall, the aggregate success rate for Central alone was 64.1, but given the Acorn union catalog, the system success rate was 70.1%. The union catalog feature of Acorn is an unequivocal gain in the availability of known items. The investment in Acorn 236 College & Research Libraries May1987 TABLE9 MEASURES OF AVAILABILITY Spring 1985 Fall1985 l.n 2. Found on shelf at Central 3. Not found on shelf at Central llie~ble titles (distributed) as' not found") A. Never ordered B. Ordered, not available C. In catalog at other divisions D. In catalog, but missed E. Checked out F. On reserve, at bindery or otherwise missing G. On shelf, but missed oy patron I. Overall success at Central* II. Overall success at System* *Varies slightly due to rounding. Count 637 367 253 17 7~} 34 17 51 65 25 +Statistically significant at the .05 level, one-tail. tStatistically significant at the .Ollevel, one-tail. made practical the union catalog. The ap- parent overall success rate for a user of Central Library, then, moved from 57.5% in spring 1985, with a local card catalog, to 70.1% in fall 1985, with the Acorn union catalog. Of course, not all of the gain from 57.5 to 70.1 can be attributed to Acorn alone be- cause some of the gain is in areas of library performance not directly affected by Acorn. The union catalog effect was 3.7 percentage points (61.2-57.5) in the Spring, and 6.0 percentage points (70.1- 64.1) in the Fall. The average union catalog effect at Vanderbilt, then, is a 4.9 percentage point gain in overall success rate, an important gain that can be fully as- cribed to Acorn. In 1986, we expect the acquisition sys- tem of Acorn to be implemented. Acorn will then provide information about items on order. The circulation function is being implemented in 1986 as well. Information about circulation status, binding, andre- serve will then be displayed simulta- neously with the bibliographic informa- tion. The apparent success rate for known-item searches will change mark- edly. When a searcher is given current in- formation about status without prompt- ing, the search may be viewed as a catalog success: the searcher will get detailed cues Conditional Success Rate Count Conditional Success Rate Z-statistic to Test Difference in Proportions 325 57.6 42.4 325 208 114 3 64.0 35.1 87.81 82.4 26} 25 14 30 93.5} ss .s)2.732:1: } 2.063+ 96.7 89.9 85.7 93.6 57.5 61.2 95.2 -1.65 88.6 -0.55 22 90.6 1.82t 4 98.1 2.46:f: 64.1 2.075t 70.1 2.695:1: to finding the desired item, cues not avail- able from a card catalog. The apparent success rate, namely th-.~ of knowing in- stantly where the item is, could reach 85% with no other change in library operation. Of course, Acorn may provide better tracking of circulation, reserve, binding, ordering, and processing so that library operations improve and availability itself increases. A number of years may be re- quired before such gains can be realized and measured. VALUING IMPROVEMENTS IN AVAILABILITY Let's assume an improvement in the availability of specific items sought by us- ers. How can we value improved availabil- ity and so contribute to an assessment of the benefit of the electronic system? We define the demand for an item of in- formation as the quantity of information a searcher will seek at a given cost. Here we might think in terms of successful known- item searches. Generally, the first few items sought per year will have quite high value. By value we mean what one would be willing to pay to use the item if one had to pay to use it. If one could have only 5 books in a year, one would choose only the most valuable. If one will seek 50 books in a year, one will include progres- sively less valuable books in the group sought. If one will look at 300 books in a year, one will include books of very much less value. In sum, a person with prior in- formation about the content of books will rank the value of books to be explored and choose the most valuable first. Hence, we expect the marginal or incremental value of items sought to decline as more books are sought. We depict the quantity of items sought relative to the additional value of seeking one more item as the downward sloping line, labelled MB in figure 3. MB stands for marginal benefit, the extra benefit of one more successful search. Alternatively, one can consider the per- son with no prior information about the books. We can think of the search process, then, as stochastic. Each additional search provides one more draw from the unex- plored pool of available items. Drawing at random, a first draw will have some given probability of yielding an item whose value exceeds a given threshold of useful- ness. The second draw will have a some- what smaller probability of yielding a book more valuable than the first. The third draw will have a smaller probability of yielding a book more valuable than ei- ther of the first two. The tenth draw will have a very much smaller probability of yielding a book more valuable than any of the first nine. The probability that the next item drawn will prove more valuable than any of the preceding items found declines the more items one has already drawn. Thus, the marginal benefit of looking at another item declines as more items are examined. The height of the line indicates the value a searcher places on finding one more item, given the quantity already found. 7 Now let's consider the cost to a searcher of conducting additional searches. If the only cost of searching were the 8 minutes of time required to find an item (as indi- cated in table 7), then we would expect searchers to keep searching until the value of an additional ''hit'' were just equal to the value of the 8 minutes it takes to get it. For a faculty member, 8 minutes is worth about $3.33; for an undergraduate, 8 min- utes is worth about $.67. Of course, if a Some Benefits of the Online Catalog 237 searcher seeks several items at once, the extra time to find one more item may be less than the 8 minutes required to find one item at a time. Nevertheless, the time cost of an added search places a lower bound on the value of finding one more item. Searchers will keep looking for more items until the cost of another search just equals the expected value of the last item found. Searchers will not look for items they expect to be less valuable than the cost of finding it. If the library contained only items that were currently in print, an alternative to availability in the library would be pur- chase of the item. Purchase involves pay- ing the price of the item plus the attendant transactions costs associated with pur- chase, the inconvenience of ordering, paying, and waiting. For in-print materi- als in the library, the purchase price of the items plus the value of the transactions cost sets an upper bound for the value of a successful search in the library. At the Heard Library in 1984-85, the average price of a monograph was $27.16; the av- erage price of a periodical was $99.10; and the average price of other serials was $54. Weighting these values by their size in ac- quisitions yields an average purchase price p~r volume of $48.11. Adding 25% (our rough guess) for the cost and delay associated with purchase yields a value of about $60 per item as an upper limit on the value of in-print items. Most of the items in stock at the library are no longer in print and so may have a value greater than the purchase price. The cost of replicating information that is no longer available could be extremely high. Or one might think of the delay costs of in- terlibrary loan if the item is available somewhere else. On the other hand, ma- terial is most valuable when it is new, es- pecially so in the sciences and social sci- ences. Even in the humanities, the average rate at which material is consulted falls as it ages indicating that average val- ues fall. 8 As with merchandise over 25 years old and with houses, for those few items whose value increases sharply with age, there are many that continue to be viewed as junk. We are prepared to assert, then, that on average the purchase price of 238 College & Research Libraries materials poses an upper bound on the value of an extra item successfully found in our library. We can think of the cost of extra searches as being represented by another line in figure 3, the line marked MC for marginal cost of a search. This line depicts the extra cost in time and otherwise to a user of searching for one more item, given that some number has already been sought. This line might be horizontal if the cost to a user of one more search does not depend on the number of searches under- taken. For us, however, searching is ate- dious business and the more searching that has to be done, the greater the nui- sance. Therefore, we display the marginal $/Search Vo V' 0 May 1987 cost of searching as upward sloping. Given the marginal benefit relationship, MB, and the marginal cost relationship, MC, we can identify the quantity of searching that a user will find sufficient, namely Q 0 • Although more searching will yield more benefit (the marginal benefit is still positive), the extra benefit of another search is less than the cost of the search and therefore, not worthwhile. At this level of searching, the value of the last search is the height of MB at that point, namely, Yo. We can think of the introduction of the online system as shifting the marginal cost relationship downward to MC': the cost of an additional search is now less at each MC' -:::::::: ·-- . ·--·-· Qo O' MB Items Sought MB : Marginal benefit in dollar terms from one more unit of search, given that a particular amount of search has already been com- pleted . MC: marginal cost, primarily in user time, of additional units of search. MC' : the lower schedule of searching cost resulting from improvements due to automation. V 0 : The value of the last search before automation . Q 0 : the quantity of search performed before automation. V': the value of the last unit of search given automation . Q ': the quantity of search performed given automation . The area of the trapezoid under MB from Q 0 to Q' : the value of the increased search performed as a consequence of the fall in cost (user time) to complete a search given automation. FIGURE3 Marginal Benefits and Costs of Search J level of searching. With the new marginal cost of searching, the user will change the quantity of searching to Q' and the mar- ginal value of one more search will be V'. The value of the additional searches will be the area under the marginal benefit line from Qo to Q'; in per. unit terms roughly the average of Vo and V'. For small changes in costs, Vo and V' will be close to each other in value and so the value of one more search will be close to the cost of an added search. Our discussion of the time saved in author/title searching by using an online system seems to indicate that the effect on the total cost of searching is rela- tively small. Therefore, we conclude that the value of another successful search is in the $0.67 (for undergraduates) to $3.33 (for faculty) range, and choose $1 as a round number. A round number best por- trays the imprecision of the estimate. We can estimate the increased number of successful author/title searches due to the union catalog provided by Acorn. We start with the estimated total number of author/title searchers reported in Table 8. We find the average overall success rate as a local catalog from Table 9, namely, 60.8%, and multiply to find an estimated number of successes as a local catalog as shown on line C of figure 4. Then, we find the average overall success rate as a union catalog, namely 65.7, and apply it as well to the total number of search attempts. We estimate, then, that there will be 12,344 more successful author/title searches per year as a consequence of having the union catalog. Valuing each success at $1, we . find an estimated increased value of suc- cesses of $12,344. The value of increased success at author/title searching due to the Some Benefits of the Online Catalog 239 union catalog might plausibly range from $10,000 to $20,000. SUMMARY Acorn, the Heard Library's online cata- log, has had a statistically significant effect in reducing the amount of time required for a search and in increasing the probabil- ity of success in finding a known item. It is possible to value these consequences of automation in dollar terms, at least to an approximation. We find about $11,000 of annual benefit due to the time saved in author/title searching and about $12,000 due to increased success at author/title searching, benefits that result from the fact that the electronic system is a union catalog. These figures do not include pos- sible gains at the law, medical, and music libraries because we do not have atten- dance information from these libraries. The estimated dollar value of these bene- fits might be added to the estimated dollar value of other benefits, for example, those in subject searching and in reference. Ex- tensions and improvements in the auto- mated service, especially dial-up access, seem likely to increase significantly the benefits of library automation in the years ahead. This style of investigation may have some influence on future investments in automated services. For example, there may be gains in both time savings and , search success from giving more explicit cues to locations than simply the call num- ber. There may be gains from identifying the most common errors made by inexpe- rienced users and modifying the system so that a user can correct and adapt more quickly. Delivering the electronic catalog A. Total annual author/title search (from table 8) (115,686 x 2.2 items per searcher) 254,509 60.8 154,741 65.7 167,085 12,344 $12,344 B. Average success rate I for Central Library (from table 9) (57.5 + 64.1)/2 C. Number of successful author/title searches annually (A times B) D. Average success rate II for system union catalog (64.2 + 70.1)/2 E. Number of successful author/title searches annually (A times D) F. Increase in successes annually (E = C) due to union catalog G. Value of increased successes at $1 per success FIGURE4 Value of Added Availability 240 College & Research Libraries to desktops around the university will fur- ther enhance the benefit of the service by reducing search time. We look forward to future investigations documenting the benefits of these advances. This essay is a report of work in prog- ress, work I hope others will join. The re- finement of measurements, the applicabil- ity to other libraries, the extension to other benefits, each stands as a research May 1987 agenda. Indeed, many assumptions made in this essay should be viewed as hypothe- ses and be subjected to appropriate tests. The methods may also serve to measure the benefits of collection and service poli- cies as well as automation programs. The payoff from such investigations should be a deeper, more detailed understanding of what constitutes quality in an information service. REFERENCES AND NOTES 1. The Heard Library licensed NOTIS software from Northwestern University, installed it on an mM , 4361 computer dedicated to the library's use, and used it to provide online catalog services. The 1 name Acorn, a symbol from the university's shield, refers to the whole local system. 2. Paul Kantor, Objective Perfonnance Measures for Academic and Research Libraries (Washington, D.C.: Association of Research Libraries, 1984). 3. The "other" category includes primarily alumni and university staff; however, all respondents who did not indicate their status are included in this category and constitute about 35% of the group. 4. Faculty, alumni, and staff tend to visit the library only once a day. Graduate students and under- graduates who use the library in part as a study hall between classes tend to come and go several times during the day. They tended to be willing to fill out only one form for the entire day rather than a form for each visit. This may account for the lower response rate among students. 5. Jon P. Nelson, "Accessibility and the Value of Time in Commuting," Southern Economic Journal 43:1321-29 Gan. 1977). 6. Central's card catalog holds cards for Central, Science, Management, and the Observatory but not for the other units of the Heard Library. A microfiche union serials list, updated quarterly, includes serials from more divisions but is not part of the catalog. 7. These ideas fit reasonably well with a discussion of the book purchase decision in Joseph Newhouse and Arthur Alexander, An Economic Analysis of Public Library Services (Lexington, Mass.: ) Lexington Bks., 1972), p.8-13; S. A. Lippman and J. J. McCall, "The Economics of Job Search: A Survey," Economic Inquiry 14:115-89, 347-68 (1976). 8. Philip Morse, Library Effectiveness (Cambridge, Mass.: MIT Pr., 1968); and Richard W. Trueswell, "Growing Libraries: Who Needs Them? A Statistical Basis for the No-Growth Collection," in Fare- well to Alexandria, Solutions to Space, Growth, and Perfonnance Problems of Libraries, ed. Daniel Gore (Westport, Conn.: Greenwood, 1976), p.72-104; Herman H. Fussier and Julian L. Simon, Patterns in the Use of Books in Large Research Libraries (Chicago: Univ. of Chicago Library, 1961.)