Mill.indd Undergraduate Information Resource Choices David H. Mill This study provides a thorough overview of information consumption at a modest-sized liberal arts college. It is based on a citation analysis of an extensive sample of bibliographies drawn from papers written for 64 intermediate and advanced courses. The papers, representing 17 academic departments, were written during the 2004–05 academic year. The citations were analyzed by type of resource, and for books and journals, local ownership and age information was noted.The format (electronic or print) of locally available journals was also recorded. Both divisional and overall results are presented, and statistically significant results are noted. he abundance of information available through the Internet makes it readily possible for undergraduate students to write research papers without using a single library-supplied resource. The evi- dence suggests that students are indeed relying less and less on the traditional re- sources supplied by the academic library. Nationally, library circulation statistics have been in decline for a number of years.1 Studies of college student informa- tion habits, underscored by the shared anecdotal evidence of academic librar- ians, suggest that undergraduate students are inclined to turn to Web resources over traditional library resources, books, and journals.2 Researchers and librarians have cautioned that this trend will have a del- eterious effect on scholarship. An implicit conclusion is that academic libraries are becoming increasingly irrelevant, regard- less of whether they are building print or electronic collections, as students turn to more convenient and familiar informa- tion sources. While a growing body of research—much of it survey-based—may make this conclusion seem inevitable, in fact, few studies actually have taken an in-depth look at the undergraduate choice of information resources as revealed by student research paper bibliographies. This study was undertaken to gain a comprehensive understanding of the un- dergraduate citation behavior of students enrolled in intermediate and advanced courses at the researcher’s home institu- tion, a modest-sized liberal arts college. The overarching goal of the project was to gather information so that the library could best employ its resources and adapt its services to meet the needs of its pri- mary patrons, the college’s students. The study was designed to assess the types of resources cited, their proportional representation, and their ages. An equally important goal was to determine how well the library’s collections were meet- ing the information needs of the students by checking library ownership for the David H. Mill is Information Technology Librarian in the Myrin Library at Ursinus College; e-mail: dmill@ursinus.edu. 342 mailto:dmill@ursinus.edu Undergraduate Information Resource Choices 343 materials cited, as well as measuring the extent that students were relying on ma- terials obtained through its Interlibrary Loan service. There were a number of other factors that were important in the study. In the case of journals, the researcher hoped that the use of the library’s various electronic collections, such as JSTOR, Project Muse, and ScienceDirect, could be gauged. To provide feedback to the teaching faculty, it was also important to determine the extent to which students were using scholarly journal titles. Beyond measuring traditional library sources, another sig- nificant goal of the study was to measure the pervasiveness of open Web resources in student bibliographies. This goal was of particular interest for the purposes of designing and targeting future informa- tion literacy initiatives. It was also im- portant to determine whether different pa erns of resource use among academic divisions might drive adjustments in col- lection development and management policies. Finally, the researcher hoped that the study would establish a baseline for future studies, both at the researcher’s institution and at similar institutions elsewhere. Literature Review Citation analysis studies have long been employed by libraries as a means of evaluating the information habits of their users, o en with the intent of determin- ing how well local collections support these populations. Most frequently, the focus has been on the journal citation behavior of graduate students or faculty, though some studies have taken a more inclusive look at information resources. With the primary intent of assessing a periodical collection, Peritz & Sor ex- amined psychology masters theses from four Israeli universities to determine the types of materials cited, ages, and range of subject fields.3 By analyzing graduate theses in psychology, Thomas, at Califor- nia State University, Long Beach, gathered data in support of journal cancellation decisions.4 To evaluate the journal collec- tion at St. Mary’s University, Sylvia and Lesher studied citations from psychology and counseling graduate student theses and dissertations.5 To help develop a new serials management policy, Klas- sen employed the Institute of Scientific Information’s citation indexes to analyze Wesleyan faculty citation behavior.6 Smith studied the bibliographies of graduate student theses and dissertations from the University of Georgia to compare the usefulness of local collections by broad academic discipline.7 Considerably fewer citation studies have examined the bibliographies of un- dergraduate papers. St. Clair & Magrill collected undergraduate bibliographies from four institutions and analyzed them by material type and age.8 As a means of assessing the Western Illinois University library collection, Joswick compared bib- liographies from freshman composition papers with local holdings.9 Hovde stud- ied the effectiveness of library instruction by analyzing the quality of undergradu- ate bibliographies from freshman English composition papers at Northern Illinois University.10 More recent investigations have also considered the impact of the World Wide Web on undergraduate citation behav- ior. In a series of three seminal studies, Davis & Cohen and Davis looked at the bibliographies from one multisection, introductory economics course at Cornell University, assessing the type and quality of resources, in addition to measuring the pervasiveness of open Web citations.11 Grimes and Boening, a er interviewing students and faculty and analyzing the bibliographies from two English compo- sition classes, concluded that community college students used “unevaluated or inappropriate” Web resources.12 Jenkins, analyzing an informal sample of 116 bibliographies drawn from a range of academic disciplines at the College of Mount St. Joseph, determined that while open Web citations were common, they were overshadowed by traditional library http:resources.12 http:citations.11 http:University.10 344 College & Research Libraries July 2008 resources, articles, and books.13 Leiding analyzed a random sample of under- graduate honors theses at James Madison University wri en during the period of 1993–2002 with the primary intent of as- sessing local collections and found that under 8 percent of the citations from post-Web bibliographies were composed of open Web resources.14 Tomaiuolo, who gauged student use of Web resources indirectly through a survey of English faculty, concluded that college students “continue to use open Web resources extensively.”15 The present study is most closely re- lated to those of Davis, Davis & Cohen, Jenkins, and Leiding, though it differs in one important respect. While each of these studies has provided valuable insight into undergraduate citation behavior, none has looked at an undergraduate student population as a whole. In fact, to the author’s knowledge, no published study has used a collegewide, random sample of undergraduate papers to gain an overall understanding of the student use of information resources. Background Ursinus College, located 28 miles north- west of Philadelphia, is an independent, coeducational liberal arts institution with an enrollment of approximately 1,500 students at the time of the study. The col- lege offers 28 majors and has a strong aca- demic tradition with some 14 prestigious honor societies, including a chapter of Phi Beta Kappa. Undergraduate research is strongly encouraged. The campus is fully networked, and all entering students receive a laptop computer. The Myrin Library, the only library on campus with a circulating collection, houses approxi- mately 250,000 volumes. Methodology At the beginning of each of the two target semesters for the 2004–05 academic year, all syllabi were checked to identify those courses with writing assignments requir- ing the use of outside (i.e., non–course-sup- plied) resources. In addition to consulting the syllabi, the researcher also contacted departmental chairs as well as individual faculty members to ask about possible paper assignments. Since the focus of the study was on intermediate and advanced courses, the census deliberately excluded first-year English composition classes. Foreign language courses were also not considered in the study. A er identifying the relevant courses, the researcher asked the 43 faculty members teaching these courses if a copy of the cover page and bibliography page for every student paper could be obtained. All agreed to provide this information, though 2 stated that they first wanted to secure their students’ permission. The total for both semesters was 73 discrete courses having a combined enrollment of 1,293 students. The researcher collected a total of 941 bibliographies, 73 percent of the absolute maximum possible, assuming that every single enrolled student produced a paper. (This assumption is not completely accu- rate, as some courses had group projects and others gave students the choice of either writing a paper or doing another type of project. Thus, the true percent- age is likely to be somewhat higher than 73 percent.) The divisional rate of return was 85 percent for the sciences, 82 percent for the humanities, and 63 percent for the social sciences. The researcher drew a random sam- ple—stratified by academic division— consisting of approximately 25 percent of the total available bibliographies. The total number of bibliographies in the sample, 236, was composed of 63 from the humanities, 64 from the sciences, and 109 from the social sciences. The sample included 64 courses and 72 classes from 17 academic departments. (See table 1 for divisional and departmental breakdown.) Of the 236 bibliographies, 5 percent were from first-year students, 37 percent from sophomores, 28 percent from juniors, and 30 percent from seniors. While some faculty submi ed bibliog- raphies in electronic format, the majority http:resources.14 http:books.13 Undergraduate Information Resource Choices 345 provided photocopies of the originals, which were then scanned and converted to rtf (rich text format) using the program ABBYY PDF Transformer 1.0. Over the course of the year following the collec- tion of the data, the researcher analyzed the bibliographies by resource type, age, and library ownership. During the analysis, the citations from each bibliography were broken down into the following categories: journals, TABLE 1 Course and Class Breakdown by Academic Division and Department Humanities Courses Classes Art 2 2 English 8 9 History 8 8 Honors 1 1 Totals 19 20 Sciences Courses Classes Biology 8 9 Chemistry 3 3 Exercise & Sport Science 1 1 Honors 1 1 Math & Computer Science 1 2 Neuroscience 3 3 Totals 17 20 Social Sciences Courses Classes Anthropology 2 2 Business and Economics 1 2 Education 1 2 Environmental Studies 1 1 Honors 1 1 International Relations 1 1 Media and Communication Studies 1 1 Politics 6 7 Psychology 11 12 Sociology 3 3 Totals 28 32 Grand Totals 64 72 books, open Web sites, newspapers, and other. For journals and books, the oldest item, newest item, and average age were recorded for each bibliography. Book cita- tions were checked against local library holdings. In a given bibliography, an ed- ited work or anthology was counted only once, regardless of the number of times cited. The researcher also checked book and journal citations to see if any had been obtained via Interlibrary Loan requests. Open Web sites, for the purposes of this study, were defined as sites with user-generated content (for instance, Wikipedia) as well as those with publicly available Web pages. The researcher excluded online journals, magazines, newspapers, and books from this category. All cited Web sites were also tested to see if they were reachable. The researcher checked cited journal titles against the library’s holdings and, if not held, against the full-text titles available through InfoTrac Expanded Academic Index ASAP, the library’s most compre- hensive multidisciplinary database. Journals were coded as being avail- able in print format only, electronic format only, or in both print and electronic formats. If the format was electronic, a note was made of the possible supplying vendor(s). To assess the quality of the cited journals, the researcher divided the titles into two categories, scholarly and nonscholarly. Scholarly journals were defined as those that publish original research studies (Studies in English Literature, Perceptual and Motor Skills, Science, and so forth). A journal did not have to be refereed to be included in the scholarly category. All other titles (for instance, general news magazines such as Time and trade journals such as Variety) were classified as nonscholarly. The researcher used the so ware program MINITAB (Release 14) to produce both descriptive and infer- 346 College & Research Libraries July 2008 TABLE 2 Pooled Citations by Resource Type and Academic Division Humanities Sciences Social Sciences All Divisions Journals 114 24.5% 421 66.2% 618 46.7% 1,153 47.6% Books 283 60.7% 110 17.3% 333 25.2% 726 29.9% Web Sites 49 10.5% 95 14.9% 262 19.8% 406 16.7% Other 19 4.1% 9 1.4% 53 4.0% 81 3.3% Newspapers 1 0.2% 1 0.2% 57 4.3% 59 2.4% Totals 466 100% 636 100% 1,323 100% 2,425 100% ential statistics. All testing of population means, unless otherwise noted, was done using two-tailed, two-sample t-tests. Results Table 2 presents a gross analysis of the pooled citations by resource type both by and across academic divisions. While journals, books, and Web sites were the predominant resources, they were present at substantially different rates. It is note- worthy that traditional resources—books, journals, and newspapers—accounted for nearly 80 percent of the total references. The average number of items in a bibliography across divisions was 10.3 (median 8.5). The shortest bibliography had a single citation and the longest had 55 items. On average, the social sciences bibliographies were the longest with 12.1 items (median 10) and the humanities bibliographies the shortest with 7.4 items (median 6). This difference was statistical- ly significant (p<.01), as was the difference (p<.05) between the humanities and the sciences bibliographies, which contained an average of 9.9 items (median 8). The difference in bibliography length between the sciences and social sciences did not reach statistical significance, however. Looking at the pooled citations by academic division (see table 2), the domi- nant resource types—journals, books, Web sites—and their respective ranks re- mained the same as in the cross-divisional comparison, except in the case of the humanities, where books predominated. Also, for the social sciences, newspapers were slightly more represented than the “Other ” category, while this order was reversed for the humanities and social sciences. The tallies for traditional resources—books, journals, and newspa- pers—were similar for the humanities and sciences, 85.4 percent and 83.7 percent respectively, while somewhat lower for the social sciences, 76.2 percent. Table 3 presents the proportions of an “average” bibliography by resource type both by academic division and across divisions. Note that the percentage values differ somewhat from those in the analy- TABLE 3 Proportions of Average Bibliography by Resource Type and Academic Division Humanities N = 63 Sciences N = 64 Social Sciences N = 109 All Divisions N = 236 Journals 26.1% 62.2% 46.7% 45.4% Books 59.1% 21.0% 26.3% 33.7% Web Sites 12.2% 15.7% 20.1% 16.8% Other 2.6% 0.9% 3.4% 2.5% Newspapers 0.1% 0.1% 3.5% 1.7% Undergraduate Information Resource Choices 347 sis of pooled citations since these values are sensitive to the varying lengths of the bibliographies. The differences among the resource types and academic divisions will be considered in turn below. Books Of the 236 bibliographies analyzed, 78.8 percent cited one or more monographs. The divisional differences ranged from 93.7 percent for the humanities to 72.5 percent for the social sciences with the sciences falling in between at 75 percent. While a majority of all the bibliographies cited at least one book, the mean propor- tion of monographs per bibliography was only 33.7 percent for all divisions. This value shows considerable variation among the academic divisions with the humanities having a much higher aver- age proportion at 59.1 percent than either the sciences at 21.0 percent or the social sciences at 26.3 percent. This difference is highly significant statistically (p<.001). The difference between the mean percent- age of monographs for the sciences and the social sciences was not significant. However, if the comparison is limited to only those bibliographies which do cite monographs, the social sciences bibliog- raphies did include a statistically (p<.05) higher proportion of books than the sci- ences (28% vs. 36%). Table 4 provides monograph age infor- mation—mean oldest, mean average, and mean newest—for each academic division and across divisions. The mean of the average age of the monographs in each bibliography across academic divisions was 13.1 years (median 9). At 19.8 years (median 14) , this value for the humanities bibliographies was significantly greater than for the sciences at 8.9 years (median 5) or the social sciences at 10.9 years (me- dian 9) (p<.01 and p<.05, respectively). There was no statistically significant dif- ference in mean average age between the sciences and social sciences. Across divisions, the average oldest book cited was 24.2 years (median 16). The average age of the oldest monograph for the humanities was 39 years (median 30.5); significantly older than that of the sciences at 12.7 years (median 7) or the social sciences at 20.3 years (median 16) (p<.001). Likewise, the difference between the average age of the oldest cited book for the sciences and the social sciences was also statistically significant (p<.05). The average age of the most recent monograph cited in each bibliography across academic divisions was 5.7 years (median 3). While this value varied by academic division from a low of 4.7 years (median 2) for the social sciences to a high of 7 years (median 4) for the humanities with the sciences falling in between at 5.5 years (median 2), these divisional differ- ences were not statistically significant. The library held 62 percent of all the monographs cited in the bibliographies. (See table 5.) The divisional differences for local holdings ranged from a high of 65.8 percent for the social sciences to a low of 51.8 percent for the sciences, with humanities coming between at 62 per- cent. Students obtained a small number of books (10 or 1.4% of the total cited) through the library’s Interlibrary Loan service. TABLE 4 Ages of Books and Journals in Years Mean Newest Mean Oldest Mean Average Books Journals Books Journals Books Journals Humanities 7.0 16.0 39.0 36.0 19.8 26.5 Sciences 5.5 3.0 12.7 18.0 8.9 8.2 Social Sciences 4.9 3.5 20.3 16.3 10.9 9.2 All Divisions 5.7 5.9 24.2 20.9 13.1 12.3 348 College & Research Libraries July 2008 TABLE 5 Local Availability of Cited Materials Books Journals Humanities 62.0% 86.8% Sciences 51.8% 62.0% Social Sciences 65.8% 58.7% All Divisions 62.0% 62.7% Journals Of the 236 bibliographies in the sample, 188 or 79.7 percent cited one or more journal articles. The divisional differences ranged from 95.3 percent for the sciences to 60.3 percent for the humanities, with the social sciences falling between at 81.7 percent. The mean proportion of journal citations per bibliography (see table 3) was 45.4 percent across all academic di- visions. The level for the sciences at 62.2 percent was significantly greater than for the humanities at 26.1 percent or the social sciences at 46.7 percent (p<.01). Likewise, the difference between the social sciences and the humanities was also statistically significant (p<.01). Table 4 summarizes the journal age information both for each academic divi- sion and across divisions. Divisionwide, the mean of the average age of the jour- nals cited in each bibliography was 12.3 years (median 7). As was the case with monographs, the mean average age of the journals cited in the humanities bibli- ographies, at 26.5 years (median 21), was significantly older than for the sciences at 8.2 years (median 5) or the social sciences at 9.2 years (median 7) (p<.01). The differ- ence in this value between the sciences and social sciences was not statistically significant. For all divisions, the average age of the oldest article cited was 20.9 years (median 13). For the humanities, this value was 36 years (median 36.5); significantly older than that of the sciences at 18.0 (median 11) or the social sciences at 16.3 (median 11) (p<.01). The difference between the average oldest journal cited in the sciences and the social sciences was not statisti- cally significant. Divisionwide, the average age of the most recent article cited was 5.9 years (me- dian 1). In terms of divisional differences, the results mirrored that of the average oldest cited article. For the humanities, the average age of the newest article cited was 16 years (median 10), significantly older than for the sciences at 3 years (median 0) or the social sciences at 3.5 years (median 1) (p<.01). There was no statistically sig- nificant difference between these values for the sciences and social sciences. Across divisions, the average bibliog- raphy that cited journals was composed of 81 percent scholarly citations, with the humanities having the highest proportion at 90.8 percent, the sciences next with 85.8 percent, and the social sciences lowest at 73.7 percent. Both the humanities and sciences bibliographies had a statistically higher average level of scholarly journal content than did the social sciences (p<.01 and p<.05, respectively). The difference between the levels for the humanities and the sciences, however, did not reach the level of statistical significance. Note that these results apply only to those bibliog- raphies which did cite journals. In total, 62.7 percent of all the journal articles cited were available either through the library’s local periodical holdings or through Expanded Academic Index ASAP. Across divisions, 24.5 percent of these were held in the print journal collection; 56.4 percent were from the electronic journal collection; and 19.1 percent were available in both print and electronic formats. Thus, 75.5 percent—the sum of the la er two categories—of all cited journal articles held by the library were accessible in electronic format. These results strongly suggest that students heavily prefer the library’s electronic journals over print or microfilm. Students obtained 5.4 percent of all cited journal articles through the library’s Interlibrary Loan Department. The divisional differences in local journal availability ranged from a high of Undergraduate Information Resource Choices 349 TABLE 6 Average Proportion of Open Web Sites per Bibliography All Bibliographies Bibliographies with Open Web Sites Humanities 12.2% N = 63 47.9% N = 16 Sciences 15.7% N = 64 25.2% N = 40 Social Sciences 20.1% N = 109 37.2% N = 59 All Divisions 16.8% N = 236 34.5% N = 115 86.8 percent for the humanities to a low of 58.7 percent for the social sciences, with the sciences falling between at 62 percent. The humanities were the highest poten- tial users of electronic journals with 90.9 percent of the locally available journals found in electronic format. The sciences and social sciences had lower but similar rates at 74.7 percent and 71.9 percent respectively. The researcher had difficul- ties determining whether students had actually used electronic or print journals, as 72 percent of the bibliographies citing journals did not include any retrieval statements. This value was highest for the sciences at 85 percent and lowest for the humanities at 59 percent, with the social sciences between at 67 percent. Open Web Sites Across all divisions, just under half of the bibliographies, 48.7 percent, cited one or more open Web sites. For the sciences, this number was 62.5 percent; for the social sciences, 54.1 percent; and for the humanities, 25.4 percent. Table 6 registers the mean propor- tion of open Web site citations per bibliography at 16.8 percent, with the social sciences having the high- est proportion at 20.1 percent, humanities the lowest at 12.2 percent, and the sciences falling between at 15.7 percent. At face value, these divisional results would appear to be very dif- ferent; however, because of the great variability in the data, the differences were not statistically significant. The results are markedly different, however, when com- paring only those bibliographies—just under half of the total—that did cite open Web sites. In this case, the aver- age proportion of open Web site cita- tions per bibliography, divisionwide, was 34.5 percent, with the humanities highest at 47.9 percent, the sciences lowest at 25.2 percent, and the social sciences coming between at 37.2 percent. In terms of statis- tical significance, both the humanities and social sciences had a higher proportion of open Web sites in their bibliographies than did the sciences (p<.05). The dif- ference between the proportions for the humanities and the social sciences, how- ever, did not reach the level of statistical significance. The researcher hypothesized that bibliographies that contained open Web site references would consequently include fewer citations to both journals and books. To test this hypothesis, the bibliographies for each academic division were divided into two groups, Open-Web and No-Open-Web, and the mean num- ber of books, journals, and total citations calculated for each. For the humanities and sciences, there were no significant differences between the Open-Web and No-Open-Web bibliographies for any of these values. Table 7 compares the aver- age number of books, journals, and total citations for social sciences bibliographies TABLE 7 Social Sciences: Open-Web vs. No-Open- Web Bibliographies Total Citations Books Journals No-Open-Web (N = 50) 10.90 3.10 7.36 Open-Web (N = 59) 13.19 3.02 4.24 350 College & Research Libraries July 2008 TABLE 8 Open Web Sites by Domain .com .edu .gov & .us .net .org other 31.77% 13.79% 10.84% 2.22% 29.06% 12.32% with and without open Web references. The differences in average bibliography length and number of books cited were not statistically significant. The differ- ence in the average number of journals per bibliography, 4.24 vs. 7.36, did reach statistical significance (one-tailed t-test, p<.05), suggesting that students who cited open Web sites did so at the expense of journal articles. Because of imperfect character transla- tion in the conversion of the scanned pdf files to text files, the accuracy of the URLs provided in the bibliographies was not checked; however, the researcher a empt- ed to reach all of the Web sites within a year following the collection of the data. Some 92.4 percent of all the open Web site citations were reachable either at the URL provided, within the root Web site, or at another URL obtained by searching the citation through Google (www.google. com). Table 8 provides a breakdown of the open Web site citations by domain. Newspapers Newspapers, divisionwide, accounted for only 2.4 percent of the total citations, the smallest percentage of any of the resource types. Only 9.3 percent of the bibliographies cited one or more, and only 2.5 percent of the bibliographies had greater than 25 percent of their total con- tent derived from newspapers. Divisional differences were pronounced, with only a single newspaper cita- tion appearing in both the humanities and the sciences bibliogra- phies. In contrast, 18.3 percent of the social sciences bibliographies cited one or more newspaper articles with 4.3 percent of the total citations coming from newspapers. Overall, the mean proportion of newspa- per citations per bibliography was 1.69 percent, with the social sciences having the highest average at 3.54 percent. The humanities and sciences had much lower rates at 0.08 percent and 0.14 percent, respectively. Because of the very small numbers involved, no a empt was made to determine statistical significance. Other Across academic divisions, 16.5 percent of the total bibliographies included citations from the Other category, though only 1.7 percent derived more than 25 percent of their content from this group of assorted resource types. In the humanities, only 11 percent of the bibliographies included any Other publication types. This figure was lowest for the sciences at 9.4 percent and highest for the social sciences at 23.9 percent. The Other category accounted for 3.3 percent of the total citations, lower than any resource type except for News- papers. Divisional values were lowest for the sciences at 1.4 percent, with the social sciences and humanities having very similar rates at 4 percent and 4.1 percent respectively. The mean proportion of the Other cita- tions per bibliography was 2.51 percent with the social sciences having the highest TABLE 9 Resource Types in Other Category by Division Interviews Misc. Movies Speeches Unknown Humanities 9 47.4% 4 21.1% 6 31.6% 0 0.0% 0 0.0% Sciences 1 11.1% 3 33.3% 0 0.0% 0 0.0% 5 55.6% Social Sciences 9 17.0% 21 39.6% 1 1.9% 6 11.3% 16 30.2% All Divisions 19 23.5% 28 34.6% 7 8.6% 6 7.4% 21 26.0% www.google Undergraduate Information Resource Choices 351 rate at 3.42 percent, sciences the lowest at 0.89 percent, and the humanities falling in between at 2.59 percent. These differences, however, were not statistically significant even when the comparison was limited to just those bibliographies that did cite materials from the Other category. The largest identifiable resource type in the Other category was “Interviews,” which accounted for 23.5 percent of the citations. “Movies/videos” amounted to 8.6 percent and Speeches 7.4 percent. “Miscellaneous” publication types, which included abstracts, annual reports, ar- chival materials, articles in press, class handouts/notes, class performances, dissertations, le ers, national constitu- tions, newswires, and unpublished data, accounted for 34.6 percent. The remainder of the Other category, 26 percent of the citations, was composed of publication types that could not be identified because of incomplete citations. Table 9 presents the divisional breakdowns. Discussion While this study was undertaken for a variety of reasons, the researcher hoped to help answer a single fundamental ques- tion: are traditional library collected and managed resources, regardless of format, the dominant category of citations in un- dergraduate student papers, or has the user-generated content available through open Web sites usurped that role? The re- sults are unequivocal. Whether measured in terms of pooled citations or average proportions per bibliography, students at Ursinus College enrolled in intermediate TABLE 10 Sample Composition by Academic Division St. Clair & Magrill Leiding* Jenkins Present Study Humanities 61.7% 18.2% 27.2% 19.2% Sciences 8.9% 25.3% 42.4% 26.2% Social Sciences 29.5% 56.5% 30.4% 54.6% * For the sake of comparison, Business was combined with Social Sciences in the Leiding breakdown. and advanced courses cited traditional li- brary resources—books and journals—at a much higher rate (77.5% vs. 16.7% and 79.1% vs. 16.8%, respectively) than open Web resources. Unlike books or journals, open Web site citations appeared in a minority of the bibliographies analyzed, 48.7 per- cent, accounting for just 16.7 percent of the total citations. The corresponding number reported by previous studies that sampled multiple undergraduate courses varies considerably from a low of 9.6 percent (combined average for the years 1997–2002) to a high of 24 percent in 2001 (Jenkins).16 Davis, studying a single, multiple-section course, reported values of 9 percent, 21 percent, 22 percent, and 13 percent for the years 1996, 1999, 2000, and 2001, respectively.17 Despite these differing results, it is noteworthy that books and journals overshadowed open Web sites in every case. Since just over half of the bibliog- raphies did not include any open Web citations whatsoever, this average propor- tion of 16.8 percent creates a somewhat misleading impression. Looking at just those bibliographies that did contain open Web sites—slightly under half of the total—a quite different picture emerges, with the average proportion increasing to 34.5 percent. Thus, students who did cite open Web sites relied on them, on average, for over one-third of their total citations, a result that demands serious a ention, particularly for those involved in information literacy initiatives. Even so, only for the social sciences, however, did the use of open Web sites appear to impact negatively on the use of traditional resources, in this case, journals. The reason for the sharp divide in the pop- ulation between open- Web and no-open-Web bibliographies is un- clear and invites further study. Of the 61 discrete http:respectively.17 http:Jenkins).16 352 College & Research Libraries July 2008 courses in the sample, not including Hon- ors projects, only 16 (26%) had bibliogra- phies without any open Web citations, sug- gesting that the majority of faculty do not impose an outright ban on their use. This would be in agreement with the results of a faculty survey by Tomaiuolo.18 A survey of individual faculty policies regarding the use of open Web sites would be useful in learning more about this phenomenon. Table 10 illustrates the inherent dif- ficulty of comparing the results of this study with the findings of past studies. The differences in divisional composition, a reflection of local institutional se ings, vary considerably from study to study. For example, the oldest study in the grouping, St. Clair and Magrill, was heav- ily biased towards the humanities with nearly 62 percent of the bibliographies coming from this division.19 Likewise, the divisional composition of Jenkins’ sample varied markedly from that of the present study.20 While the composition of the pres- ent study and the Leiding samples were, in fact, very similar in terms of divisional breakdown, the sample populations were presumably different. Leiding’s study was limited to honors theses, while this study included a broad sampling of interme- diate and advanced courses, as well as honors theses.21 Despite these differences in sample makeup, it is notable that both Jenkins and Leiding arrived at results remarkably similar to those of the present study in terms of the prevalence of books and jour- nals. Jenkins found 76 percent of content coming from books and articles; Leiding measured 76.3 percent in post-Web bibli- ographies; and the present study arrived at a figure of 77.5 percent.22 Overall, students cited journals far more frequently than books, 47.6 percent vs. 29.9 percent. While the proportions vary, these results also are in general agreement with those of Leiding and Jenkins.23 As discussed below, this preference may be the result of traditional disciplinary prac- tice. It may also reflect the convenience of accessing e-journal content. Divisional differences in the use of books and journals were obvious, with the sciences showing a strong preference for journals, and the humanities an almost equally strong preference for books. The social sciences cited more journals than books, though the difference was not as pronounced. Again, these results were in general agreement with both Leiding and St. Clair & Magrill, reinforcing long-held views that the humanities rely more on books while the sciences and social sci- ences use journals more heavily.24 Unfortunately, because of the absence of retrieval statements in the majority of the bibliographies, it was not possible to learn unequivocally the source of cited articles. However, given that over 75 percent of the cited journals held by the library were available in electronic format, it is likely that students were primarily making use of the library’s collection of electronic journals. Of the studies noted in table 10, only St. Clair & Magrill, a pre-Web study, included information on the age of cited materi- als, recording data on oldest and newest items. However, since their sample, as noted earlier, included a much higher percentage of humanities bibliographies than the present one, and because the study was done in a university se ing, comparisons must be viewed as tentative. Across academic divisions, the average oldest book cited in the present study was 24.2 years (median 16) while St. Clair & Magrill reported figures considerably older at 30.4–32.4 years (medians 20–22).25 They also found somewhat older results for the average newest monograph at 6.1–8.1 years (median 1.1–3.1) compared to 5.7 years (median 3) from this study.26 Divisionwide, the average age of the most recent article cited was 5.9 years (median 1) which is in keeping with the findings of St. Clair & Magrill at 5.3–7.3 years (median 1–3).27 At 20.9 years (medi- an 13), the average age of the oldest article cited was substantially older, however, than the values of approximately 11–13 years (medians 5–7) reported by St. Clair http:study.26 http:20�22).25 http:heavily.24 http:Jenkins.23 http:percent.22 http:theses.21 http:study.20 http:division.19 http:Tomaiuolo.18 Undergraduate Information Resource Choices 353 & Magrill.28 Again, though comparisons must be done with caution, it is notewor- thy that students in a post-Web environ- ment were actually citing older journal articles. To explore this phenomenon in greater detail, the researcher assessed the ages of those journals that were available through the library in electronic format or were cited as open-access electronic journals. The mean oldest age was 18.9 years (median 10), and the mean newest was 7.8 years (median 3); again, both older than the values reported by St. Clair & Magrill. This finding suggests that students are using electronic journals to reach back further in time than in the pre-Web era. The convenience of access- ing journal backfiles online as opposed to using print or microform is likely the deciding factor. The results highlight pronounced divisional differences in the ages of cited materials. Whether gauged by the average mean age, the average oldest, or the average newest, the humanities bibliographies cited significantly older journal articles than either the sciences or social sciences. The outcome was similar for monographs, with the humanities citing significantly older books whether measured by the average mean age or the average oldest age than the other two academic divisions. In terms of average mean age and average oldest age, the social sciences bibliographies also cited older books than did the sciences. These findings suggest that age should not be the overriding factor in making weeding decisions with both humanities and social sciences materials. Science and humanities bibliographies contained the highest average proportion of “scholarly” journal content, as defined in this study. While still high at 74 percent, the average for the social sciences was sta- tistically lower than the other disciplines. It should be noted that the definition used for “scholarly” titles may have been inher- ently biased against many of the journals commonly cited in the social sciences, the International Relations and Politics Departments in particular. While pub- lications such as Congressional Quarterly Almanac, Economist, and Nation may not meet the strict definition of “scholarly,” it certainly could be argued that they are appropriate resources for undergraduate students. In terms of local holdings, both books and journals were available at nearly identical rates, 62 percent vs. 62.7 percent. Leiding’s corresponding values were similar at 65.4 percent and 58.2 percent, respectively.2 9 Since the researcher ’s library, by policy, does not acquire text- books, ideally, they should be taken out of the comparison for a more meaningful assessment of the usefulness of the col- lection. Similarly, it would also be useful to know which cited articles are made available to students as course-provided readings. Newspapers accounted for a very small proportion of the total citations, the low- est of any of the resource types. This is a somewhat surprising finding given that the library has had a long-standing sub- scription to Lexis Nexis Academic Universe, a database that includes over 350 full-text newspapers. Citations in the “other ” category, which encompassed interviews, movies/videos, speeches, miscellaneous, and unidentified resource types, while more numerous than newspaper articles, were still quite low. Interlibrary Loan Use While some 9.3 percent of the 236 bibli- ographies included at least one article or book obtained from Interlibrary Loan, the total number of these citations accounted for only 3.8 percent of the combined book and journal totals. Twenty-two of the 208 students in the study—just under 11 per- cent—cited materials obtained through Interlibrary Loan. The majority of these students, 19, cited journal articles from ILL, while only 6 cited books. Given that the Interlibrary Loan Department sup- plied some 1,495 journal articles and 644 monographs to the student body during the 2004–05 academic year, the researcher http:respectively.29 http:Magrill.28 354 College & Research Libraries July 2008 found it surprising that only 62 journal articles and 10 books from ILL were cited in the sample bibliographies. Possibly, students do not necessarily cite many of the materials that they receive through Interlibrary Loan; or, because they are slow in requesting these materials in the first place, they do not have them in time to make use of them. Another likely explanation is that students who make heaviest use of ILL are a subset of the student population. Conclusion Ursinus College Students, writing re- search papers for intermediate and advanced courses during the 2004–05 academic year, used traditional library sources—books and journals—for a clear majority of their citations. With the excep- tion of the humanities courses, journals were cited more frequently than books, and the majority of the journal titles cited were “scholarly,” as defined by this study. Given that some 75 percent of the cited journals held by the library were avail- able in electronic format, students likely used electronic journals than much more heavily than print. However, because only a minority of citations included a retrieval statement, it was impossible, in most cases, to determine the actual source. Despite the likely preference for electronic journals, students did not limit their citations to newer publications, but cited relatively old articles as well. A majority of the books and journals cited, 62 percent and 62.7 percent, respectively, were available locally in the library’s col- lections. Nearly 11 percent of the students cited at least one item obtained through Interlibrary Loan, though these citations accounted for a relatively small propor- tion of the total. Newspapers were not a significant source of citations for the humanities or the sciences, though they did comprise over 4 percent of the social sciences citations. While students also cited in- terviews, movies, speeches, and other miscellaneous materials, they accounted for only a small percentage of the total citations. In keeping with the findings of other undergraduate citation studies, open Web sources placed a distant third to book and journal citations, with just over half of the sample bibliographies containing no open Web citations whatsoever. In 2002, Davis, reporting on the prevalence of open Web resources in student bibliographies from a multisection economics course, conclud- ed that a “possible crisis of undergraduate education [was] at hand.”30 These results suggest, at least upon first glance, that this threat may have been overstated. Nonetheless, the fact that almost half of the sample bibliographies derived, on av- erage, just over one-third of their content from open Web resources indicates that there may indeed be legitimate cause for concern. Assessing the quality of these Web sites—a task that was beyond the scope of this project—would doubtless provide useful information for informa- tion literacy initiatives. Notes 1. Martha Kyrillidou and Mark Young, ARL Statistics 2004–05 (Washington, D.C.: Association of Research Libraries, 2006). Available online from www.arl.org/bm~doc/arlstat05.pdf. [Accessed 14 December 2006]. 2. Steve Jones and Mary Madden, The Internet Goes to College (Washington, D.C.: Pew In- ternet & American Life Project, September 15, 2002), available online from www.pewinternet. org/pdfs/PIP_College_Report.pdf [Accessed 9 December 2006]; OCLC Online Computer Center, Inc., “OCLC White Paper on the Information Habits of College Students: How Academic Librar- ians Can Influence Students’ Web-Based Information Choices” (June 2002), available online from www2.oclc.org/oclc/pdf/printondemand/informationhabits.pdf [Accessed 9 December 2006]. 3. Bluma C. Peritz and Dina Sor, “The Use of Libraries by Graduate Students in Psychology as Indicated by Citations,” Collection Management 12, no. 3/4 (1990): 11–23. 4. Joy Thomas, “Graduate Student Use of Journals: A Bibliometric Study of Psychology www.pewinternet www.arl.org/bm~doc/arlstat05.pdf Undergraduate Information Resource Choices 355 Theses,” Behavioral & Social Sciences Librarian 12, no. 1 (1993): 1–7. 5. Margaret Sylvia and Marcella Lesher, “What Journals Do Psychology Graduate Students Need? A Citation Analysis of Thesis References,” College & Research Libraries 56 (July 1995): 313–18. 6. Timothy Klassen, “Measuring Serials Usage Using Faculty Cited Journals Data,” Bo om Line 14, no. 1 (2001): 37–43. 7. Erin T. Smith, “Assessing Collection Usefulness: An Investigation of Library Ownership of the Resources Graduate Students Use,” College & Research Libraries 64 (Sept. 2003): 344–55. 8. Gloriana St. Clair and Rose Mary Magrill, “Undergraduate Use of Four Library Collections: Format and Age of Materials,” Collection Building 11, no. 4 (1992): 2–15. 9. Kathleen E. Joswick, “Library Materials Use by College Freshmen: A Citation Analysis of Composition Papers,” College & Undergraduate Libraries 1, no. 1 (1994): 43–66. 10. Karen Hovde, “Check the Citation: Library Instruction and Student Paper Bibliographies,” Research Strategies 17 (2000): 3–9. 11. Philip M. Davis and Suzanne A. Cohen, “The Effect of the Web on Undergraduate Citation Behavior 1996-1999,” Journal of the American Society for Information Science and Technology 52, no. 4 (2001): 309–14;.Philip M. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update,” College & Research Libraries 63 (January 2002): 53–60; Philip M. Davis, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholarship in a Networked Age,” portal: Libraries and the Academy 3, no. 1 (2003): 41–61. 12. Deborah J. Grimes and Carl H. Boening, “Worries with the Web: A Look at Student Use of Web Resources,” College & Research Libraries 62 (Jan. 2001): 11–22. 13. Paul O. Jenkins, “They’re Not Just Using Web Sites: A Citation Study of 116 Student Papers,” College & Research Libraries News 63 (Mar. 2002): 164. 14. Reba Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies to Evaluate Library Collections,” College & Research Libraries 66 Sept.r 2005): 417–29. 15. Nicholas G. Tomaiuolo, “Faculty Views of Open Web Resource Use by College Students,” Journal of Academic Librarianship 31 (Nov. 2005): 559–66. 16. Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies,” 421; Jenkins, “They’re Not Just Using Web Sites,” 164. 17. Davis, “Effect of the Web on Undergraduate Citation Behavior,” portal (2003): 47. 18. Tomaiuolo, “Faculty Views of Open Web Resource Use by College Students,” 563. 19. St. Clair and Magrill, “Undergraduate Use of Four Library Collections,” 3. 20. Jenkins, “They’re Not Just Using Web Sites,” 164. 21. Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies,” 417. 22. Jenkins, “They’re Not Just Using Web Sites,” 164; Leiding, “Using Citation Checking of Undergraduate Honors Thesis,” 423. 23. Leiding, “Using Citation Checking of Undergraduate Honors Thesis,” 421; Jenkins, “They’re Not Just Using Web Sites,” 164. 24. Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies,” 426; St. Clair and Magrill, “Undergraduate Use of Four Library Collections,” 6. 25. St. Clair and Magrill, “Undergraduate Use of Four Library Collections,” 8. 26. Ibid. 27. Ibid. 28. Ibid. 29. Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies,” 424. 30. Davis, “The Effect of the Web on Undergraduate Citation Behavior,” College & Research Libraries (Jan. 2002): 59.