525 The Scholarly Impact of Books Acquired via Approval Plan Selection, Librarian Orders, and Patron-Driven Acquisitions as Measured by Citation Counts David C. Tyler, Brianna D. Hitt, Francis A. Nterful, and McKenna R. Mettling* Patron-driven acquisition has been an important, if contentious, topic for decades, with numerous programs having been piloted, adopted, and reported on, largely favorably, in the library literature. Still, questions and doubts persist for academic libraries, especially where the composition of vendor plans and packages and the judgment of patrons are concerned. Past literature has approached the assessment of patron-driven acquisition by analyzing circulation/usage, comparing peer-library holdings, seeking patrons’ or librarians’ judgments of utility and suitability, looking for evidence of collection imbal- ances, and testing for overlap in patrons’ and librarians’ purchases. To contribute to this literature, this study addresses scholarly impact and examines whose selections—ap- proval plans’, librarians’, or patrons’—have been most heavily cited. For the social sci- ences, the sciences, and the humanities, the authors gathered topic-matched random samples of books acquired via approval plans and librarian orders during the first five years of operation of their institutions’ interlibrary loan purchase-on-demand patron- driven acquisition program and compared their citation counts to the counts of books acquired via the program. Google Scholar was employed to tally citations. Introduction Academic librarians with collection development responsibilities and library administrators who have followed the library literature on acquisitions and collection development since the 1999 publication of Perdue and Van Fleet’s study of the successful program at Bucknell University1 could hardly be unaware of the explosion of interest in patron-driven acquisition (henceforth, PDA)—or, alternately, demand-driven (DDA), user-driven, and/or evidence-based acquisition. In 2010 and in 2012, respectively, the ACRL Research Planning and Review Committee identi- fied collection growth driven by patron demand and patron-driven e-book acquisition as top ten trends in academic libraries.2 As well, several authors contemporaneously named PDA * David C. Tyler is Collections Analyst & Strategist in the University Libraries at the University of Nebraska- Lincoln; email: dtyler2@unl.edu. Brianna D. Hitt is a PhD student, and Francis A. Nterful and McKenna R. Met- tling are MS graduates, all in the Department of Statistics at the University of Nebraska-Lincoln. ©2019 David C. Tyler, Brianna D. Hitt, Francis A. Nterful, and McKenna R. Mettling, Attribution-NonCommercial (http:// creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:dtyler2@unl.edu http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/ 526 College & Research Libraries May 2019 one of the most widely discussed acquisition models in the library literature.3 In fact, Walker declared US research libraries to be on a PDA “tipping point” in 2011.4 The PDA approach would certainly seem to have been quickly and widely adopted. In a national survey reported on in 2010, Lenares and Delquie found just 32 respondents with a current program, 42 with plans to implement a program within a year, and 90 institutions indicating that they would like to implement a program within the next three years.5 The following year, Osorio presented a survey of librarians in Illinois that found that a mere 12.6 percent and 23.1 percent of respondents had a PDA or DDA program for print books or e- books, respectively.6 In a 2013 article adapted from a 2012 report to The Andrew W. Mellon Foundation, Esposito, Walker, and Ehling estimated that there were just 400 to 600 programs in existence.7 More recently published survey results, however, have shown that between 85 and 95 percent of respondents report that their libraries have PDA/DDA programs in place.8 The field’s high level of interest and the increasing number of programs piloted and ad- opted by academic libraries have produced a veritable avalanche of literature. Caminita, in a chapter published in 2014, reported that searches of Google Scholar and of EBSCO’s Library, Information Science and Technology Abstracts (LISTA) employing a selection of terms associated with college libraries and PDA found more than 2,000 and more than 400 items, respectively, had been indexed since 2009.9 A search of Google Scholar employing Caminita’s terms performed by the lead author of this study during the fall of 2017 for items indexed since the beginning of 2013 found nearly 12,000 items.10 Thus, it would seem that the field’s interest in PDA has been and continues to be quite keen, and adoption of the model has been widespread. In fact, several in the field have argued that this “new” patron-driven approach has or will soon become standard library practice,11 although perhaps it should be noted that several authors, both PDA critics and supporters, have taken pains to declare that patron-driven collecting is actually nothing new and is part of what libraries have always done.12 One can only speculate, but perhaps this reflects a desire on the part of the model’s critics to call into question the boosterish enthusiasm of some of PDA’s promoters and a desire on the part of the model’s promoters to reassure that a seemingly new and potentially risky acquisition practice is not actually new and is, therefore, not risky. Regardless of whether one enthusiastically embraces or unenthusiastically laments the development and adoption of PDA, the authors of this study would caution that it is early yet to draw firm conclusions concerning the model and its effects. Thus far, the published literature has cast PDA as a new and exciting disruptive technology/practice that moves aca- demic libraries away from a wasteful and moribund “just-in-case” model and toward a lean and responsive “just-in-time” model;13 as a subversion of library hegemonies, hierarchies, and authority/power relations;14 as a potential revolution in library work;15 as a major stra- tegic change;16 as a “powerful shift in how libraries acquire materials”;17 and as a continua- tion of or a return to past practice in a new guise …something both new and not.18 From the midst of this promotional whirl, several authors have thoughtfully called into question the PDA model and argued that it has several failings that will produce numerous unfavorable outcomes. Early studies, though, regardless of the metrics employed or the issues addressed and assessed, have largely produced findings and conclusions favorable to the PDA model.19 Many of these findings have not, though, been entirely generalizable or particularly robust. As Walker has noted, the published research on, as well as the criticism of, PDA “has consisted, in the main, of opinion pieces debating the merits of PDA and case studies of libraries’ expe- The Scholarly Impact of Books Acquired 527 riences implementing PDA.”20 Likewise, Dewland and See have noted that a shortcoming of the model and its literature has been the absence of consistent, objective, and standardized metrics for success.21 Remedying this entirely would be well beyond the current authors’ abilities or the scope of a single article, yet the authors do hope that they may here introduce an approach that will address one of the more frequently raised criticisms of PDA: its potential for acquiring books of low scholarly merit. Unfortunately, “scholarly merit” is something of an amorphous and multifaceted measure with a very high potential for subjectivity. Unsurprisingly, the library literature thus far has tended to approach the idea obliquely and fractionally rather than di- rectly and comprehensively, most commonly through assessments of PDA-purchased books’ appropriateness for academic library collections. One facet of scholarly merit that has yet to be addressed is scholarly impact. So far as the authors have been able to ascertain, no one has yet examined whether the books supplied/ordered via the various selection/acquisition modes—approval plan selection, librarian firm order, or PDA—tend to be more heavily cited. To begin to remedy this, the authors propose to examine the citation counts of the books of the three disciplines—the social sciences, the sciences, and the humanities—acquired via the three main acquisition methods (henceforth, “Order Type[s]”) employed by the University of Nebraska-Lincoln Libraries (UNL Libraries). The books in question were added to the circulat- ing collection during the first five years (2003/04 to 2007/08) of the UNL Libraries’ interlibrary loan-driven purchase-on-demand (ILL-PoD) program for print books. For those unfamiliar with the terminology, Carrico, Leonard, and Gallagher have defined ILL-PoD as follows: “ILL-PoD is a patron-driven purchase model whereby patrons request books not owned by their local library, and based on various criteria…, the book is purchased rather than bor- rowed from another library.”22 For the criteria employed by the UNL Libraries, excepting the list of barred publishers, please refer to appendix A. The citation counts for the books were collected via Google Scholar from January to April of 2016, roughly 8 to 13 years after they had been added to the collection and 12 years, on average, after they had been published. In this manner, the authors hope to discover which Order Type, if any, attracted the most citations, which the authors hope will serve as an at least partial measure of the Order Types’ scholarly impact and merit. Review of Literature As was noted above, a great deal has been written about the PDA model and about PDA/ DDA programs. Much of the published research literature that has assessed and evaluated the outcomes of PDA programs has supported the approach,23 but some of the library literature has been cautionary, if not outright critical. The criticisms of PDA pertinent to this study can be distilled to two issues: first, the difference in the objectives of the traditional, collection- building, “just-in-case” model and the more patron-oriented, “just-in-time,” PDA model; and second, the different motivations and aims of traditionalist academic librarians, of vendors and publishers, and of library patrons. When developing collections in accord with the tra- ditional model, academic librarians act as experts, agents, and gatekeepers, in which roles they identify, collect, manage, and preserve the best and most significant scholarship of their day and of the past, as identified by themselves and other authorities, such as the teaching faculty and reviewers of academic books. The intent behind the model is to build a collection to meet institutions’ educational missions, to preserve the scholarly record, and to meet not 528 College & Research Libraries May 2019 just the needs of the day’s patrons but the needs of future scholars working 10, 20, 50, and even 100 years hence.24 By way of contrast, the PDA model was derived at least in part from an approach to manufacturing and warehousing developed by Toyota that was designed to meet customers’ needs as they arose, rather than anticipating them.25 As Costello explains in the preface to her recent book on the subject, “Demand-driven acquisitions (DDA) describes any acquisitions process that is driven by the desires of patrons or their usage of materials rather than predic- tive processes like package plans and librarian selection.”26 Thus, as one of its most vigorous proponents, Rick Anderson of the University of Utah, has suggested, the PDA model is not intended for building collections in the traditional sense at all.27 For PDA’s critics, the fairly recent, widespread, and enthusiastic adoption of PDA/DDA approaches by academic libraries marks an abdication of one of academic librarians’ main professional responsibilities,28 the “ultimate source of pride of many collection specialists,”29 through a turning away from collection-building as an end in itself, what Anderson has termed the “monument to Western Civilization” collections mindset.30 For PDA’s advocates, it instead marks a positive turn toward the adoption of what several authors have termed a facilitated-collection or collections-as-service mindset centered on present patrons’ expressed, rather than anticipated, needs.31 Not incidentally, it also marks something of a reduction in academic librarians’ control over the composition of their libraries’ collections and a corre- sponding empowerment of patrons.32 As such, it is not difficult to suppose why traditionalist academic librarians who understand collection building to be one of their core duties find PDA worrisome and upsetting, especially when some of its more radical champions have sug- gested it ought to become the primary, if not the sole, method for adding books to academic libraries’ collections.33 In addition to being concerned with the lessening of collection building as academic librarians’ raison d’être, PDA’s critics have also called into question vendors’/publishers’ and patrons’ motives, expertise, and knowledge. With respect to vendors/publishers, they have warned that book jobbers and publishers are primarily in the business of selling books, so academic libraries ought to be wary of their PDA plans and e-book DDA packages lest their library catalogs come to resemble online shopping malls crowded with inappropriate and low-quality materials.34 Similarly, with respect to PDA patrons, several authors have warned against, or have reported hearing colleagues opine against, allowing patrons too much freedom to add books to academic libraries’ collections. Critics have cautioned that patrons tend to be too narrowly focused on their immediate needs, that patrons request/access books whose utility and scholarly worth is unknown to them, that patrons will unbalance collections in their pursuit of idiosyncratic or trivial popular topics, and so forth.35 The criticisms of vendors strike the authors of this study as somewhat problematic. It has been a generally accepted professional truism that good collection development is a product of good partnerships and the sharing of expertise between librarians on the one hand and vendors and publishers on the other, and one can certainly see this attitude expressed in the library literature from both sides.36 Yet, in the critics’ support, there has been some recent, empirically supported counter-literature that has found that approval plan selections pro- duce fewer circulations than do librarians and/or patrons’ purchases37 and offer worse value in terms of cost-per-circulation,38 although it should also be noted that a number of studies have found that approval plan selections slightly outcirculated books ordered by individual The Scholarly Impact of Books Acquired 529 selectors.39 Approval plans also have been found to do a poor job of providing libraries with books from small presses and from small societies, with highly specialized books, and with books whose subject matter or approach fall outside traditional disciplinary boundaries.40 There also does seem to be some evidence that DDA e-books packages warrant scrutiny. Two recent studies have both reported their authors having discovered nonacademic materials in their e-book packages despite both pairs of authors having worked closely with vendors to exclude such materials.41 An internal study of e-book subscription packages conducted at the UNL Libraries similarly found questionable materials in the vendors’ packages, as well as books from suspect publishers. Thus, cautions against giving over collection development too much to vendors/publishers would certainly seem worth considering. The library literature’s response to criticisms of patrons as selectors has been varied, and the authors would be hard-pressed to do justice here to the voluminous scholarship on this subject or to summarize it succinctly. For a lengthy overview of the approaches taken in the literature, the authors recommend Costello’s recent Evaluating Demand-Driven Acquisitions.42 For the authors’ current purposes, it should suffice to note that the attitudes apparent behind much of the criticisms of patrons have been chided for exhibiting a snobbish paternalism43 and that much of the early research on PDA programs would seem to support the model’s ability to deliver collection-appropriate books of acceptable scholarly merit. As Tyler et al. have noted: The primary worry about inappropriate books has largely been answered in the existing body of literature. Authors have reported small numbers or percentages of their programs’ requested or purchased items as being inappropriate to their libraries, with subject, Dewey call number or Library of Congress (LC) subclass, genre, readership-/content-level, publisher, or material type variously employed as criteria. However, librarians who have directly reviewed the requested or pur- chased items or who have reviewed requests by subject criteria, publisher, material type, readership-level, reviewers’ recommendations, or against peer institutions’ holdings, have found PDA items to largely have been worthy and appropriate purchases for their collections.44 Still, although critics’ dire predictions against patrons as selectors have yet to find much solid support in the research literature, they should not be dismissed entirely. There does seem to be some scattered evidence in the critics’ favor. For example, with respect to the prob- lem of “hot topics,” there is, of course, the infamous “banana books” incident, recounted by Polanka at the Charleston Conference, wherein a library’s PDA budget was quickly spent in response to a class assignment.45 Additionally, concerning patrons’ much-feared fondness for less-scholarly materials, Goedeken and Lawson have discovered that their patrons appear to have an affection for books in Wiley’s Dummies series.46 Of interest to critics questioning patrons’ motivations for accessing/requesting books would be the finding of Reynolds et al. that patrons’ second-most-frequent reason given for using the “Suggest a Purchase” service at Texas A&M was for recreational reading.47 A few critics, rather than call into question all patrons, have limited themselves to sug- gesting that perhaps only undergraduates ought to be barred from adding books to academic libraries collections and that only faculty and, perhaps, graduate students should be allowed. With respect to the issue of undergraduates’ potential for being problematic selectors, one 530 College & Research Libraries May 2019 recent study of a PDA program for undergraduates did find that the books undergraduates requested tended not to be widely held by peer institutions,48 whereas studies of programs that were less restricted by patrons’ status have found PDA/DDA books to be held widely by peer institutions or held widely in general.49 The same study on undergraduates as selectors also reported that selections made by undergraduates appeared not to obtain as great a circulation advantage as has been reported elsewhere for less restricted programs.50 In yet another study, Purdue University’s librarians found that books requested by undergraduates tended to be subsequently checked out most frequently by other undergraduates.51 These studies raise the strong possibility that undergraduates’ PDA/DDA purchases may be qualitatively different from purchases made by faculty, graduate students, and other types of patrons, which has understandably made placing the patrons on an equal footing to selectors an issue of concern for some PDA critics.52 In light of the uncertainties recounted above, and owing to the fact that no one has yet looked into the scholarly impact of PDA books, it would seem worthwhile to examine whether books provided by vendors or publishers via approval plans or e-book packages, purchased by librarians via firm ordering, or acquired via PDA/DDA programs produce a greater schol- arly impact. As well, given the uncertainty surrounding the various types of academic library patrons as purchasers of scholarly works, it would also seem worthwhile to examine whether PDA books acquired by the various types of patrons are of commensurate interest to scholars. Background, Data Collection, and Method of Analysis Toward the end of the 2002/03 school year, the manager of the UNL Libraries’ ILL department brought Perdue and Van Fleet’s article on Bucknell’s successful PDA program to the attention the Collection Development Committee and suggested that it might consider a pilot project involving ILL PoD. As a result, since the start of the 2003/04 school year, the UNL Libraries has operated an ILL PoD PDA program for recently published nonreference academic books. After five years’ operation, in response to a call for papers for a special issue of the journal Collection Management, data were drawn for the PDA-acquired books and for all print books added to the circulating collection during the interval via traditional acquisition modes. The resulting data set produced several articles concerning the circulation performance of this set of books, the cost-per-circulation, whether the books were being monopolized, and the selection patterns of librarians and patrons at the level of Library of Congress (LC) subclass. For the current study, the authors gathered records for PDA-acquired books in the data set with LC subclasses associated with the social sciences, the sciences, and the humanities. From the contemporaneous approval plans’ selections and librarian firm-ordered titles, the authors then drew equal numbers of books for each LC subclass via simple random selection without replacement. Subclasses for which not all three Order Types had purchases were not included in the draw. Thus, the authors produced three “matched” samples for each discipline’s purchases (for example: if PDA patrons had purchased 10 books in subclass QA—Mathematics, then 10 books for that subclass would have been randomly drawn from the approval plans’ and from the librarians’ books, resulting in a pool of 30 books for purposes of analysis and comparison). Very small numbers of books were purchased by the PDA patrons in some LC subclasses, so the authors, to facilitate analysis, found it necessary to merge several of the LC subclasses into topical groups (in other words, “Topic”). For the composition of the Topic groups of each discipline, please see table 1. The Scholarly Impact of Books Acquired 531 TABLE 1 Composition of the Samples by Discipline and Topic (Grouped Library of Congress Subclasses) Topics with ILL PDA Purchases Books per Order Type Social Sciences: Anthropology & Recreation (GF–GN, GT–GV) 29 Business & Economics (HB–HG) 106 Education (LB, LC, LD, LJ) 47 Geography & Environmental Sciences (G–GE) 9 Law (K, KB–KBU, KF–KFZ, KJ–KKZ, KL–KWX, KZ) 43 Military & Naval Sciences (U, UG) 5 Political Science (JA–JF, JK, JN–JQ, JV, JZ) 45 Psychology (BF) 46 Social Sciences (General) & Statistics (H–HA) 9 Sociology & Related Fields (HM–HX) 152 Books per Order Type 491 Total Social Sciences Books 1,473 Sciences: Agriculture (SB, SF) 10 Arts & Crafts (TT) 1 Building Construction (TH) 1 Chemical Technology & Manufacturing (TP, TS) 18 Dentistry (RK) Life Sciences (QH–QR) 55 1 Engineering & Technology (General) (T–TD, TN) 23 Engineering (Mechanical, Electrical, & Automotive [TJ–TL]) 17 Life Sciences (QH–QR) 55 Medicine (Clinical & Internal) (RC–RD, RF–RJ) 81 Medicine (General, Public Health, & Pathology) (R–RB) 40 Physical Sciences (QB–QE) 34 Science (General) & Mathematics (Q–QA) 59 Therapeutics & Pharmacy (RM–RS) 3 Books per Order Type 343 Total Sciences Books 1,029 Humanities: Architecture (NA) 37 English Language & Literature (PE, PR, PS) 82 Fine Arts (N, NB–ND, NK–NX) 70 General Works, Biography, Library & Information Science (AZ, CT, Z–ZA) 21 History (CB–CC, CN, D–DA, DC–DG, DJK–DK, DR–DU, E, F) 166 Music (ML–MT) 22 Non-English Languages & Literatures (PA, PJ, PL, PQ, PT) 18 Philology, Linguistics, & Literature (General) (P, PN) 65 532 College & Research Libraries May 2019 In addition to being curious in general about the performance of the selection modes that comprise the three Order Types, the authors’ attention, as was noted above, was also drawn to the performance of the library’s patrons by the literature critical of PDA/DDA programs for failing to exclude certain sorts of patrons. Therefore, the authors further grouped the books in the UNL Libraries’ PDA data sets into books ordered by one of five “Patron Types” by status. As can be seen from table 2 below, PDA at UNL has mostly been a means for graduate stu- dents and for members of the faculty to make additions to the circulating collection, which is consistent with results reported elsewhere in the library literature for PDA programs for print materials,53 although not necessarily for e-book DDA programs.54 Unfortunately, the UNL Li- braries’ ILL department’s data collection form allowed patrons to self-report their status, which reporting was not mandatory, so it would appear from the table that roughly 11.4 percent of PDA patrons did not sufficiently identify themselves and had to be classed as “Unknown.” To gather data on the apparent scholarly interest in/impact of these books, as indicated by cumulative citation over the study’s interval, the lead author searched Google Scholar from January through April of 2016 for the books’ index entries and citation counts. In a number of instances, there were multiple index entries for books due to variations in how they had been cited, so each book was searched for with three different combinations of search terms drawn from the author, title, and publisher fields of their catalog records. Upon discovering the books’ index entries, the books’ citations were recorded and, if necessary, summed. In a few instances, books in the data set proved to be earlier or later editions or to be books from numbered series. In such instances, only citation counts for the appropriate editions or series TABLE 1 Composition of the Samples by Discipline and Topic (Grouped Library of Congress Subclasses) Topics with ILL PDA Purchases Books per Order Type Philosophy (B–BD, BH–BJ) 44 Photography (TR) 19 Religions (BL–BP, BR–BX) 71 Books per Order Type 615 Total Humanities Books 1,845 Note: No PDA books were purchased in LC subclasses not listed, such as GR—Folklore, so they were not included in the study. TABLE 2 Percentage of PDA Books Purchased by Patron Type Patron Types Disciplines Faculty Graduates Undergraduates Staff Unknown Social Sciences 21.0% 53.4% 9.4% 6.1% 10.2% Sciences 19.8% 50.4% 7.3% 8.7% 13.7% Humanities 31.5% 44.2% 8.8% 4.4% 11.1% Totals 25.2% 48.8% 8.6% 6.0% 11.4% Note: Percentages rounded. The Scholarly Impact of Books Acquired 533 numbers were included in the tallies. In a very few instances, index entries for the books were not discoverable. In those instances, the lead author conducted a fourth search that employed the books’ first authors’ last names and a few words from the books’ titles as a phrase. The lead author then manually tallied the number of times the books appeared in the first 10 pages of search results. The author found, when forced to employ this approach, that none of these books appeared beyond the fifth or sixth page of search results. The collected data produced highly right-skewed distributions of citation counts (see appendix B). As an initial, informal testing of the waters, the authors attempted within each discipline to test for differences in performance by Order Type, by Order Type within Topic groups, by Patron Type within the PDA subset, and by Patron Type within Topic groups using Poisson regression, but this initial test run produced results that were unlikely, with nearly every comparison proving to be significant, and the Poisson models were poor fits. Subsequent model fitting suggested that the negative binomial distribution was a much better fit for the data, so the authors reanalyzed the performance of the Order Types and Patron Types using negative binomial regression analysis with p value adjustments for multiple comparisons. Research Questions As should be evident from the foregoing, the foci of this study are the citation count perfor- mances of the three Order Types and, within the PDA data sets, the performances of the five Patron Types. For each of the three disciplines, the authors first asked a pair of questions: If the three Order Types had been given an opportunity to select/purchase equal numbers of books on the same Topics for the UNL Libraries, would there be differences in the books’ performance as generators of citation activity, and, if so, which Order Type’s books would have been cited more? As a follow-up research question, the authors asked: Would there be differences in Order Types’ citation count performances within Topic groups? If there were no significant differences between Order Types’ performances in general, the authors reasoned, perhaps there would be performance differences concealed within particular subject areas. Alternately, if there were differences between the Order Types’ performances in general, these differences might be present across all Topics or might be limited to outsized differences within just a few. For the study’s final research question, the authors repeated Research Question #1 but turned their attention to the PDA data sets and to the five Patron Types: Within the PDA data sets of each discipline, did any of the Patron Types outperform any of the others? Results Research Question #1: For each discipline, were there significant differences in performance by Order Type? If so, which Order Types were cited more? For this first question, the authors tested within disciplines the three Order Types’ citation counts for statistically significant differences and then performed post hoc tests with adjust- ments for multiple comparisons to determine which Order Types, if any, outperformed one or more of the others. As the F values for the three tests of fixed effects in table 3 show, there would fairly definitely seem to be at least one significant difference in performance present in each discipline. To determine the nature and direction of the differences between Order Types, the authors performed the aforementioned post hoc comparisons, whose results are to be found in the “Differences of Order Type Least Squares Means” portions of the table. 534 College & Research Libraries May 2019 TABLE 3 Citation Count Performance within Disciplines by Order Type: Approval Plans, Librarian Orders, and PDA Patrons Social Sciences: Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 1,470 33.26 < .0001 Order Type Least Squares Means Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Appr. 5.0816 0.07150 1,470 71.07 < .0001 161.03 11.5145 Libs. 5.8916 0.07145 1,470 82.45 < .0001 362.00 25.8663 PDA 5.6199 0.07147 1,470 78.64 < .0001 275.86 19.7149 Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p Appr. Libs. –0.8100 0.1011 1,470 –8.01 < .0001 < .0001 Appr. PDA –0.5383 0.1011 1,470 –5.32 < .0001 < .0001 Libs. PDA 0.2717 0.1011 1,470 2.69 0.0073 0.0198 Sciences: Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 1,026 68.61 < .0001 Order Type Least Squares Means Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Appr. 4.6697 0.08742 1,026 53.42 < .0001 106.67 9.3254 Libs. 6.0116 0.08731 1,026 68.86 < .0001 408.13 35.6321 PDA 5.8110 0.08732 1,026 66.55 < .0001 333.95 29.1529 Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p Appr. Libs. –1.3418 0.1236 1,026 –10.86 < .0001 < .0001 Appr. PDA –1.1412 0.1236 1,026 –9.24 < .0001 < .0001 Libs. PDA 0.2006 0.1235 1,026 1.62 0.1046 0.2357 Humanities: Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 1,842 31.27 < .0001 Order Type Least Squares Means Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Appr. 3.8868 0.06229 1,842 62.40 < .0001 48.7561 3.0371 Libs. 4.1691 0.06223 1,842 67.00 < .0001 64.6585 4.0234 PDA 4.5788 0.06216 1,842 73.66 < .0001 97.3967 6.0539 The Scholarly Impact of Books Acquired 535 For the social sciences, as one can see from the reported t values and adjusted p values presented in the top third of the table, the librarians significantly outperformed both the ap- proval plans and the patrons, and the patrons significantly outperformed the approval plans. (Note: In reading this and the tables to follow, positive t values indicate that the Order Type or Patron Type in the left-hand column outperformed the one in the right-hand column; a negative value indicates the reverse.) How great were the differences in citation-count perfor- mance? To answer this question, one will need to subtract the mean values found in the table. For example, the difference in citation counts between the approval plans and the librarians in the social sciences was 161.03–362.00 = –200.97. That is to say that the book jobbers’ selec- tors underperformed the UNL social sciences librarians by roughly 201 citations per book. Similarly, the librarians outperformed the PDA patrons by roughly 86 citations per books, and the patrons outperformed the approval plan selectors by roughly 115 citations per book. For the sciences, the post hoc comparisons show that there were two statistically signifi- cant performance differences to be found among the Order Types. Both the librarians and the patrons handily outperformed the approval plans. The librarians outperformed the approval plans by approximately 301.46 citations per book, and the patrons outperformed the approval plans by roughly 227.28 citations per book. The librarians did outperform the patrons by approximately 74 citations per book. Were one to multiply this difference by the number of books in question, one would find that the librarians’ books attracted more than 25,000 more citations than did the patrons’ books; that certainly seems like a large amount, but, when one consults the table, one can see that this difference was not statistically significant. In fact, it was not significant even prior to the adjustment for multiple comparisons (p = 0.1046). Still, it is difficult not to see it as substantial. Finally, for the humanities, as was the case with the social sciences, all three post hoc comparisons reveal statistically significant differences. The librarians again outperformed the approval plans’ selectors. Unlike with the social sciences, the PDA patrons outperformed both of the other Order Types. If one consults the mean values displayed and/or consults appendix B, one will notice that the parameters and counts for the humanities tended to be quite a bit smaller than those of the social sciences and of the sciences, although the shape of the cita- tion counts’ distributions tended to be quite similar. Despite these differences between the disciplines, statistically significant differences were still very much present in the humanities. The humanities librarians attracted almost 10,000 more citations than did the approval plans, and the patrons garnered roughly 20,000 more than the former and almost 30,000 more than the latter. TABLE 3 Citation Count Performance within Disciplines by Order Type: Approval Plans, Librarian Orders, and PDA Patrons Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p Appr. Libs. –0.2823 0.08805 1,842 –3.21 0.0014 0.0039 Appr. PDA –0.6920 0.08800 1,842 –7.86 <.0001 <.0001 Libs. PDA –0.4097 0.08795 1,842 –4.66 <.0001 <.0001 Note: Statistically significant effects/differences appear in bold. 536 College & Research Libraries May 2019 Research Question #2: For each discipline, were there significant differences in performance between Order Types within Topics? If so, which Order Types were cited more? As a follow-up to the first research question, the authors parsed the data sets a bit more finely and again tested the performance of the Order Types, this time within the Topic categories presented in table 1. The authors tested the Order Types’ performances within the 10 Topic categories of the social sciences; the 13 categories of the sciences, although only the results of 9 will be presented; and the 11 Topic categories of the humanities. The four missing sciences categories—Arts & Crafts, Building Construction, Dentistry, and Therapeutics & Pharmacy—were included in the modeling and testing of the data, but they comprised so very few books that the authors felt that it was not worthwhile—and would perhaps be misleading—to report their results. As was the case above, the authors looked for the presence of statistically significant differ- ences in the three disciplines’ data sets and then followed with a battery of post hoc comparisons to determine where and in what direction the significant differences in performance within Topics, if any, occurred. Because of the sizable number of comparisons being made, the authors unfortunately found it necessary to split the table employed above into three parts, one for each discipline. In the upper portion of each discipline’s subtable, the authors once again will report the results of the tests of fixed effects, this time looking for Order Type and Topic effects and, most important, for Order Type*Topic interaction effects. In the remainder of each subtable, the authors will again report on the post hoc comparisons performed, as in table 3 above. As one can see from a quick review of tables 4a-c below, there were statistically significant Order Type*Topic interaction effects for all three disciplines, with the social sciences and the humanities producing roughly equal F values; but, with the sciences, despite its enormous F value displayed in table 3 above, producing a value slightly less than one half of either. With Topic taken into account, perhaps the sciences exhibited fewer or less extreme differences than the other disciplines. To obtain a clearer picture of what occurred in the data sets, one will need to turn to the post hoc comparisons. First, for the social sciences, the post hoc testing determined that performance differences between Order Types were not present across all Topic groups. As the t values and adjusted p values in table 4a show, there were statistically significant differences in performance in just 8 of the 30 comparisons made, after the adjustment for multiple comparisons. For example, in the Anthropology & Recreation Topic group, there was no statistically significant difference in performance between the approval plans and the librarians, despite the latter having at- tracted roughly 48 more citations per book. The PDA patrons, however, outperformed both in a statistically significant manner, garnering roughly 219 more citations per book than did the approval plans and roughly 171 more per book than did the librarians. The close reader of the tables may notice that some of the differences in means appear to be quite large, yet the adjusted p values show the differences not to be significant at all, such as was the case with the Topic Social Sciences (General) & Statistics. In such cases, an explanation will likely be found by referring back to the composition of the samples presented in table 1 and referring to the standard error of the means column (“S.E. Mean”) in the table under review. If one were to do so with this example case, one would see that the number of books being tested was small and that the standard errors, partially as a result, were quite large. Thus, one could speculate that had more books been purchased in the LC subclasses H and HA, then this Topic would The Scholarly Impact of Books Acquired 537 TABLE 4A Social Sciences: Comparison of Citation Counts by Order Type and Topic: Approval Plans, Librarian Orders, and PDA Patrons Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 1,443 4.04 0.0178 Topic 9 1,443 7.95 < .0001 Order Type*Topic 18 1,443 3.85 < .0001 Differences of Order Type Least Squares Means with Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer) Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p. Anthropology & Recreation Appr. 59.931 17.0728 Appr. Libs. –1.47 0.1422 0.3064 Libs. 108.24 30.7864 Appr. PDA –3.82 0.0001 0.0004 PDA 279 79.2585 Libs. PDA –2.36 0.0186 0.0488 Business & Economics Appr. 153.86 22.876 Appr. Libs. –0.97 0.3337 0.5979 Libs. 188.55 28.0264 Appr. PDA –2.99 0.0028 0.0079 PDA 288.71 42.8978 Libs. PDA –2.03 0.0428 0.1061 Education Appr. 100.74 22.5114 Appr. Libs. –4.99 < .0001 < .0001 Libs. 487.66 108.78 Appr. PDA –4.63 < .0001 < .0001 PDA 434.34 96.8954 Libs. PDA 0.37 0.7137 0.9284 Geography & Enviro. Science Appr. 157.89 80.5612 Appr. Libs. 0.22 0.8229 0.9727 Libs. 134.33 68.5585 Appr. PDA –0.73 0.4666 0.7468 PDA 267 136.16 Libs. PDA –0.95 0.3412 0.6073 Law Appr. 121.3 28.3275 Appr. Libs. –1.64 0.1010 0.2287 Libs. 208.53 48.6629 Appr. PDA 0.30 0.7633 0.9512 PDA 109.81 25.6493 Libs. PDA 1.94 0.0523 0.1273 Military & Naval Science Appr. 190.6 130.45 Appr. Libs. 0.84 0.4032 0.6806 Libs. 84.8 58.1182 Appr. PDA 1.02 0.3096 0.5666 PDA 71.2 48.8207 Libs. PDA 0.18 0.8569 0.9822 Political Science Appr. 214.44 48.9161 Appr. Libs. –1.30 0.1923 0.3928 Libs. 326.62 74.4786 Appr. PDA –0.21 0.8357 0.9765 PDA 229.29 52.2985 Libs. PDA 1.10 0.2728 0.5160 Psychology Appr. 193.22 43.5969 Appr. Libs. –0.53 0.5928 0.8542 Libs. 229.17 51.701 Appr. PDA 0.75 0.4512 0.7315 PDA 151.91 34.2874 Libs. PDA 1.29 0.1978 0.4019 Social Sciences & Statistics Appr. 800.56 408.03 Appr. Libs. 0.04 0.9649 0.9989 Libs. 775.56 395.29 Appr. PDA –1.07 0.2858 0.5342 PDA 1728.44 880.84 Libs. PDA –1.11 0.2664 0.5069 Sociology Appr. 151.01 18.7498 Appr. Libs. –7.71 < .0001 < .0001 Libs. 584.71 75.5247 Appr. PDA –2.56 0.0105 0.0282 PDA 236.82 29.3901 Libs. PDA 5.15 < .0001 < .0001 Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, some statistics could not be displayed. Full tables are available upon request. 538 College & Research Libraries May 2019 TABLE 4B Sciences: Comparison of Citation Counts by Order Type and Topic: Approval Plans, Librarian Orders, and PDA Patrons Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 990 7.39 0.0007 Topic 12 990 12.56 <.0001 Order Type*Topic 24 990 1.61 0.0322 Differences of Order Type Least Squares Means with Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer) Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p. Agriculture Appr. 50.5000 24.4422 Appr. Libs. 0.26 0.7985 0.9647 Libs. 42.4000 20.5383 Appr. PDA –1.21 0.2261 0.4467 PDA 115.60 55.8174 Libs. PDA –1.47 0.1428 0.3076 Chemical Tech. & Manufacturing Appr. 82.3333 29.6536 Appr. Libs. –1.25 0.2101 0.4217 Libs. 155.89 56.0767 Appr. PDA –2.31 0.0209 0.0544 PDA 267.22 96.0706 Libs. PDA –1.06 0.2895 0.5394 Engineering & Tech. (Gen.) Appr. 191.78 61.0151 Appr. Libs. –0.79 0.4271 0.7065 Libs. 274.17 87.1982 Appr. PDA –2.11 0.0347 0.0874 PDA 496.48 157.84 Libs. PDA –1.32 0.1870 0.3841 Engineering (Mech., Elect., & Auto.) Appr. 292.47 108.19 Appr. Libs. –1.23 0.2206 0.4382 Libs. 555.29 205.34 Appr. PDA –0.95 0.3444 0.6113 PDA 479.71 177.40 Libs. PDA 0.28 0.7797 0.9578 Life Sciences Appr. 68.2364 14.0671 Appr. Libs. –4.32 <.0001 <.0001 Libs. 240.33 49.4328 Appr. PDA –3.45 0.0006 0.0017 PDA 186.60 38.3915 Libs. PDA 0.87 0.3846 0.6595 Medicine (Clinical & Int.) Appr. 68.1358 11.5746 Appr. Libs. –3.75 0.0002 0.0006 Libs. 167.54 28.4084 Appr. PDA –3.28 0.0011 0.0031 PDA 149.56 25.3623 Libs. PDA 0.47 0.6359 0.8837 Medicine (Gen.) Appr. 109.07 26.3363 Appr. Libs. –2.20 0.0277 0.0709 Libs. 231.47 55.8319 Appr. PDA –2.53 0.0116 0.0311 PDA 258.55 62.3563 Libs. PDA –0.32 0.7458 0.9437 Physical Sciences Appr. 119.15 31.1984 Appr. Libs. –4.52 <.0001 <.0001 Libs. 634.09 165.79 Appr. PDA –2.79 0.0054 0.0149 PDA 334.44 87.4711 Libs. PDA 1.73 0.0840 0.1946 Science (Gen.) & Mathematics Appr. 122.44 24.3370 Appr. Libs. –7.69 <.0001 <.0001 Libs. 1062.08 210.78 Appr. PDA –6.48 <.0001 <.0001 PDA 754.78 149.80 Libs. PDA 1.22 0.2239 0.4434 Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, some statistics could not be displayed. Full tables are available upon request. The Scholarly Impact of Books Acquired 539 also have produced a batch of statistically significant results. Unfortunately, as things stand, this can only be a speculation. With respect to the sciences, if one were to peruse the mean values reported for each Topic, one would see that the librarians’ books attracted more citations in eight of the nine Topic categories reported on than did the approval plans’citations, with the sole exception being Agriculture. Likewise, the patrons’ mean citations were higher than the approval plans’ citations in all nine Topics. Both results would certainly be in line with the more general results reported on in table 3. If one were to review the t and adjusted p values for the approval plans’ post hoc comparisons, one would see that the approval plans were statistically significantly outperformed in 9 of the 18 comparisons involving them, and they were outperformed by nearly statistically significant amounts in an additional three comparisons—versus librarians in Medicine (General) and versus patrons in Chemical Technology & Manufacturing and Engineer- ing & Technology (General). In reviewing the Topic means for librarians and for patrons, one can see that the performance results were more mixed. Librarians outperformed patrons by substantial amounts in four Topics, patrons substantially outperformed librarians in three topics, and the performance of both was roughly equal for Medicine (Clinical & Internal) and just slightly favorable for patrons in Medicine (General, Public Health, & Pathology). Although some of the performance differences appear substantial to the eye, however, none of the post hoc comparisons for librarians and patrons produced differences that were statistically signifi- cant. In fact, just one, the comparison for Physical Sciences, was even close to being significant. TABLE 4C Humanities: Comparison of Citation Counts by Order Type and Topic: Approval Plans, Librarian Orders, and PDA Patrons Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Order Type 2 1,812 15.52 < .0001 Topic 10 1,812 31.13 < .0001 Order Type*Topic 20 1,812 3.58 < .0001 Differences of Order Type Least Squares Means with Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer) Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p. Architecture Appr. 26.8649 6.3963 Appr. Libs. 1.36 0.1724 0.3596 Libs. 16.9459 4.0556 Appr. PDA –0.06 0.9528 0.9981 PDA 27.4054 6.5239 Libs. PDA –1.42 0.1546 0.3287 English Language & Literature Appr. 28.7195 4.5906 Appr. Libs. –2.14 0.0322 0.0816 Libs. 46.5854 7.4225 Appr. PDA –4.70 <.0001 <.0001 PDA 82.8293 13.1675 Libs. PDA –2.56 0.0106 0.0287 Fine Arts Appr. 23.5857 4.0877 Appr. Libs. 0.55 0.5792 0.8441 Libs. 20.5857 3.5730 Appr. PDA –1.97 0.0492 0.1205 PDA 38.1714 6.5901 Libs. PDA –2.52 0.0117 0.0315 General Works, Biography, & LIS Appr. 67.5714 21.2405 Appr. Libs. 0.07 0.9461 0.9975 Libs. 65.5714 20.6141 Appr. PDA 0.86 0.3875 0.6629 PDA 46.0 14.4839 Libs. PDA 0.80 0.4257 0.7051 540 College & Research Libraries May 2019 Finally, let us turn to the humanities. A review of the reported means shows the per- formance of the approval plans to be quite mixed vis-à-vis the librarians, with the approval plans having higher means in six Topics and with the librarians having higher means in five. The approval plans’ performance against the PDA patrons was not as good, with the latter having higher means in 11 of the 12 Topic categories. Against the PDA patrons, the librarians’ performance was similarly poor, with the patrons having higher mean values for nine Topics. A review of the post hoc comparisons supports this sense of the Order Types’ performances within Topics. Of the 33 post hoc comparisons, 15 proved to be statistically significant, with the patrons outperforming the approval plans or the librarians in 12 of them. Additionally, the patrons outperformed the approval plans by a nearly significant amount in the Topic Fine Arts and the librarians in the Topic Non-English Languages & Literatures. Of the three remaining comparisons whose results were statistically significant, the approval plans outperformed both the librarians and the patrons in Photography, and the librarians outperformed the approval plans in Philology, Linguistics, & Literature (General). The librarians also nearly outperformed the approval plans by a statistically significant amount in English Language & Literature. Thus, the results reported above for the general samples presented in table 3, which showed the patrons strongly outperforming both the approval plans and the librarians and which showed TABLE 4C Humanities: Comparison of Citation Counts by Order Type and Topic: Approval Plans, Librarian Orders, and PDA Patrons History Appr. 60.1988 6.7334 Appr. Libs. 0.91 0.3634 0.6346 Libs. 52.1325 5.8348 Appr. PDA –3.00 0.0027 0.0076 PDA 96.7711 10.8078 Libs. PDA –3.91 <.0001 0.0003 Music Appr. 33.5 10.3256 Appr. Libs. –1.56 0.1182 0.2620 Libs. 66.1364 20.3130 Appr. PDA –2.93 0.0034 0.0096 PDA 119.73 36.7128 Libs. PDA –1.37 0.1716 0.3583 Non-English Languages & Literatures Appr. 22.0 7.5246 Appr. Libs. 1.12 0.2633 0.5025 Libs. 12.7778 4.4043 Appr. PDA –0.88 0.3764 0.6501 PDA 33.7222 11.4905 Libs. PDA –2.00 0.0454 0.1119 Philology, Linguistics & Literature (Gen.) Appr. 76.8923 13.7325 Appr. Libs. –4.11 <.0001 0.0001 Libs. 216.89 38.6572 Appr. PDA –3.00 0.0027 0.0077 PDA 163.97 29.2351 Libs. PDA 1.11 0.2673 0.5083 Philosophy Appr. 79.75 17.3093 Appr. Libs. –0.77 0.4389 0.7189 Libs. 101.14 21.9370 Appr. PDA –3.85 0.0001 0.0004 PDA 259.45 56.1951 Libs. PDA –3.07 0.0021 0.0061 Photography Appr. 43.0526 14.2566 Appr. Libs. 4.97 <.0001 <.0001 Libs. 3.9474 1.3774 Appr. PDA 2.54 0.0111 0.0300 PDA 13.0 4.3600 Libs. PDA –2.46 0.0139 0.0370 Religions Appr. 43.8732 7.5148 Appr. Libs. –0.53 0.5991 0.8587 Libs. 49.831 8.5297 Appr. PDA –3.25 0.0012 0.0034 PDA 96.1831 16.4256 Libs. PDA –2.72 0.0066 0.0181 Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, some statistics could not be displayed. Full tables are available upon request. The Scholarly Impact of Books Acquired 541 the performance difference between the approval plans and the librarians to have been slightly less pronounced, would seem to have been well supported here. Research Question #3: For each discipline, were there significant differences in performance by Patron Types? If so, which types were cited more? The study’s third research question is, essentially, the application of Research Question #1 to the citation counts of just the PDA-acquired books, with the five Patron Type categories as- suming the place of the three Order Types. As such, tables 5a–c below should each be read in much the same manner as were the disciplines’ sections of table 3 above. In the upper portion of each discipline’s section of the table, one will again find the tests of fixed effects, which indicate whether or not at least one statistically significant difference was present in the data sets being analyzed. As one can see via a quick review of the three pertinent sections of the tables, there was strong evidence of at least one statistically significant difference in the social sciences data set, considerably weaker evidence of at least one difference in the sciences data set, and strong evidence for at least one difference in the humanities data set. TABLE 5A Social Sciences: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, Undisclosed-status Patrons (Unknown), and Undergraduates Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Patron Type 4 486 12.62 < .0001 Order Type Least Squares Means Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Faculty 5.0817 0.1386 486 36.67 < .0001 161.05 22.3204 Graduates 5.6126 0.08684 486 64.63 < .0001 273.86 23.7826 Staff 6.5879 0.2565 486 25.68 < .0001 726.23 186.27 Unknown 6.0071 0.1987 486 30.23 < .0001 406.32 80.7487 Undergrads 4.6898 0.2075 486 22.60 < .0001 108.83 22.5863 Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p Faculty Graduates –0.5309 0.1636 486 –3.25 0.0013 0.0109 Faculty Staff –1.5062 0.2915 486 –5.17 < .0001 < .0001 Faculty Unknown –0.9254 0.2423 486 –3.82 0.0002 0.0014 Faculty Undergrads 0.3920 0.2496 486 1.57 0.1169 0.5171 Graduates Staff –0.9753 0.2708 486 –3.60 0.0003 0.0032 Graduates Unknown –0.3945 0.2169 486 –1.82 0.0695 0.3636 Graduates Undergrads 0.9229 0.2250 486 4.10 < .0001 0.0005 Staff Unknown 0.5807 0.3245 486 1.79 0.0741 0.3870 Staff Undergrads 1.8981 0.3299 486 5.75 < .0001 < .0001 Unknown Undergrads 1.3174 0.2873 486 4.58 < .0001 < .0001 Note: Statistically significant effects/differences appear in bold. 542 College & Research Libraries May 2019 For the social sciences, as the t and adjusted p values in the table show, there were several statistically significant differences in performance between Patron Types, and every Patron Type was involved in at least one such pairing. The surprise in the social sciences PDA data set for the authors was the performance of Staff, which outperformed all of the other Patron Types and outperformed three of the other four by statistically significant amounts. Its out- performance of the Unknown patrons was also nearly significant prior to the adjustment of p values. The next-best-performing Patron Type among the categories whose members were identifiable was Graduates, who significantly outperformed Faculty and Undergraduates. Surprisingly, Faculty did not significantly outperform Undergraduates, although it should be noted that the Faculty mean was higher. Before proceeding to the next discipline, the authors thought it would be worthwhile to note the magnitude of some of the differences in performance in the social sciences. The top-performing Patron Type, Staff, outperformed Faculty by roughly 565 citations per book, Graduates by roughly 452 citations per book, and Undergraduates by roughly 617 citations per book. Graduates, by way of comparison, slightly outperformed Faculty and Undergraduates by roughly 113 and 165 citations per book, respectively, and Faculty outperformed Undergraduates by just 52 citations per book. The performance of social sciences’ Staff was quite remarkable. TABLE 5B Sciences: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, Undisclosed-status Patrons (Unknown), and Undergraduates Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Patron Type 4 338 2.66 0.0327 Order Type Least Squares Means Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Faculty 5.9401 0.1833 338 32.40 <.0001 379.97 69.6523 Graduates 5.9496 0.1149 338 51.77 <.0001 383.61 44.0860 Staff 5.2818 0.2761 338 19.13 <.0001 196.73 54.3238 Unknown 5.3368 0.2206 338 24.19 <.0001 207.85 45.8511 Undergrads 5.5868 0.3024 338 18.48 <.0001 266.88 80.7034 Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p Faculty Graduates –0.00952 0.2164 338 –0.04 0.9649 1.0000 Faculty Staff 0.6582 0.3314 338 1.99 0.0478 0.2750 Faculty Unknown 0.6033 0.2868 338 2.10 0.0362 0.2210 Faculty Undergrads 0.3533 0.3536 338 1.00 0.3185 0.8557 Graduates Staff 0.6678 0.2991 338 2.23 0.0262 0.1702 Graduates Unknown 0.6128 0.2487 338 2.46 0.0143 0.1015 Graduates Undergrads 0.3628 0.3235 338 1.12 0.2628 0.7951 Staff Unknown –0.05497 0.3534 338 –0.16 0.8765 0.9999 Staff Undergrads –0.3049 0.4095 338 –0.74 0.4570 0.9458 Unknown Undergrads –0.2500 0.3743 338 –0.67 0.5047 0.9631 Note: Statistically significant effects/differences appear in bold. The Scholarly Impact of Books Acquired 543 In the PDA data set for the sciences, the Patron Type performances were more in keeping with what one would expect from having read the cautionary PDA literature. Faculty and Graduates had roughly equal means and handily outperformed the other three Patron Types by more than one hundred citations per book. Surprisingly, however, none of the post hoc comparisons proved to be statistically significant after the adjustment of p values. Faculty and Graduates did significantly outperform the Staff and Unknown types prior to adjustment, but not the Undergraduates. A closer inspection of the data set revealed that Undergraduates books had the greatest performance variability, so this may in part account for the lack of a detectable difference. The bulk of Undergraduates books performed no better than did Staff and Unknown books, but the best-performing of these books performed as well as the best- performing Graduates books and nearly as well as the best-performing Faculty books. From this, one would expect that, were Undergraduates acquisitions to be increased, eventually a statistically significant difference in performance would develop. The performances of the Patron Types in the humanities’ PDA data set were more prob- lematic than they were in the social sciences or sciences. The tests of fixed effects showed the presence of at least one likely real difference in the humanities’ PDA data set; but, after the adjustment for multiple comparisons, the only significant differences that remained were TABLE 5C Humanities: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, Undisclosed-status Patrons (Unknown), and Undergraduates Type III Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F Patron Type 4 610 5.89 0.0001 Order Type Least Squares Means Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean Faculty 4.2792 0.1083 610 39.52 <.0001 72.1804 7.8161 Graduates 4.6096 0.09137 610 50.45 <.0001 100.44 9.1776 Staff 3.8774 0.2907 610 13.34 <.0001 48.2963 14.0397 Unknown 5.1714 0.1826 610 28.33 <.0001 176.16 32.1623 Undergrads 4.5852 0.2051 610 22.36 <.0001 98.0185 20.1018 Differences of Order Type Least Squares Means with Adjustment for Multiple Comparisons (Tukey-Kramer) Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p Faculty Graduates –0.3304 0.1417 610 –2.33 0.0200 0.1362 Faculty Staff 0.4018 0.3102 610 1.30 0.1957 0.6942 Faculty Unknown –0.8922 0.2123 610 –4.20 <.0001 0.0003 Faculty Undergrads –0.3060 0.2319 610 –1.32 0.1875 0.6792 Graduates Staff 0.7322 0.3047 610 2.40 0.0166 0.1159 Graduates Unknown –0.5618 0.2042 610 –2.75 0.0061 0.0479 Graduates Undergrads 0.02442 0.2245 610 0.11 0.9134 1.0000 Staff Unknown –1.2940 0.3433 610 –3.77 0.0002 0.0017 Staff Undergrads –0.7078 0.3558 610 –1.99 0.0471 0.2723 Unknown Undergrads –0.5862 0.2746 610 –2.14 0.0332 0.2065 Note: Statistically significant effects/differences appear in bold. 544 College & Research Libraries May 2019 those where the Unknown type outperformed the Faculty, Graduates, and Staff Patron Types, which information is, clearly, not useful. If one were to review the preadjustment p values, one would see that Graduates nearly significantly outperformed Faculty and Staff and that Undergraduates nearly outperformed Staff. Though Graduates and Undergraduates had nearly equal means, the latter’s performance did not as closely approach significance, likely because of the small number of books purchased by Undergraduates and because of the com- paratively high variability in their citation performance, as was the case with the sciences’ Undergraduates above. Analysis For the social sciences, the results displayed in table 3 suggest that some of PDA’s naysayers’ and critics’ concerns about potential differences in the quality of vendors’, librarians’, and patrons’ book selections may be justified and that their arguments in favor of librarian title- by-title selection were warranted. Librarians’ selections during the interval had more of an impact, as measured by citation counts, than did book vendors’ and patrons’ selections by statistically significant amounts. Patrons’ selections, of course, also had more impact than did book vendors’ selections for the approval plans, also by a statistically significant amount. The librarians’ sample of 491 books garnered 42,293 more citations than did those of the patrons, for a difference of roughly 86 citations per book. When looking at these numbers, it would be difficult to argue that librarian selection did not add value or that librarian selection and patron selection were equivalent where scholarly impact is concerned. That said, these results would suggest to the authors that the more important story was the dismal performance of the approval plans’ book selectors. Over the study interval, the sample of titles that the book vendors sent to the UNL Libraries garnered almost 100,000 fewer citations than did the titles in the librarians’ sample and slightly more than 56,000 fewer cita- tions than did the patrons’ purchases. However correct one may count PDA’s critics concern- ing patrons’ failings as book selectors for the ages, the citation counts would suggest that the critics would be substantially more justified in concluding that book vendors deserve close, constant, and careful scrutiny. As was noted above, to produce a more nuanced picture of the three selectors’/purchas- ers’ performance, the authors elected also to compare the performance of the Order Types within Topics; but, where the approval plans were concerned, there was very little nuance to be found. In table 4a, the librarians’ purchases had higher mean citation counts in 7 of the 10 Topic groups, and most of these differences showed the librarians’ books to have collected two, three, or four times as many citations. Likewise, the patrons outperformed the book vendors in 6 of the 10 Topic groups, with an additional two near ties. The sole Topic group where the approval plan selectors outperformed the other Order Types was “Military & Naval Sciences,” and even there the differences in performance were not statistically significant. There may be some room for nuance where Librarians and PDA Patrons are concerned. The librarians had higher mean counts in half of the Topic groups, the patrons had higher mean counts in 4 of the 10 Topic groups, and the two had roughly equal performances in one group. Of the performance differences, each Order Type had just one that was statistically significantly different. Thus, of the three Order Types, the authors would suggest that firm- ordering librarians and ILL-PoD-requesting patrons were much more nearly comparable in their performances than were the book vendors and the librarians or the book vendors and the The Scholarly Impact of Books Acquired 545 patrons, although, again, the librarians were clearly the superior selectors when the samples were looked at as a whole. For both PDA’s critics and its advocates, the results presented by table 5a must have produced the most bafflement, in that they both confirmed and countered expectations. Of the five Patron Types, for example, Undergraduates performed the worst, just as PDA’s critics have contended they would. However, of the two Patron Types that both critics and advo- cates esteem, only one, Graduates, outperformed Undergraduates by a statistically significant amount. The Faculty decidedly did not. A review of the results for a Patron Types within Topics analysis performed by the authors but not reported on did suggest that Faculty books pretty consistently produced higher mean citation counts than did Undergraduates books in Topic groups where purchasing overlapped, but, again, the differences in Faculty and Under- graduates performance were largely statistically indistinguishable. Librarians critical of PDA have suggested to the lead author that this unexpected finding should be read as the result of undergraduates and faculty having purchased books with different qualities that both attract low citation counts. On the one hand, the former, in accordance with the anti-undergraduate and anti-PDA argument, obviously must have purchased books with low scholarly merit. On the other hand, the latter, in accordance with librarians’ generally pro-faculty attitude, must have purchased books of high scholarly merit, but these obviously were so advanced, abstruse, and/or esoteric as to attract less widespread attention. This may well be the case, but, without empirical support, this argument seems to the authors to be tainted by motivated reasoning. And what should PDA’s critics and advocates make of the results for the university staff? This is a group of library patrons that has largely been ignored in the PDA/DDA literature, yet here they clearly outperformed the other three groups with identifiable patrons. The so- cial sciences’ Staff purchases attracted roughly 5,200 more total citations than did Faculty’s, despite their having made 30 purchases to Faculty’s 103. In essence, the Staff books attracted 4.5 citations for every citation that the Faculty books attracted. Perhaps one ought to expand Walter’s argument against letting undergraduates add to the collection and make a point of including university staff in PDA/DDA purchasing. How ought one to interpret this surprising result? Unfortunately, because of policies intended to protect patron privacy, the authors were unable to delve into just what was be- hind the outsized performance of the university staff, but we are able to offer some plausible speculation based upon our knowledge of local culture that may suggest that the staff’s performance may be read as a local effect. First, some university administrators at UNL may be listed in the patron database as members of the university staff rather than as members of the faculty. Second, it is not entirely uncommon at UNL for recently graduated graduate students, especially students in education and in a handful of the social sciences, to take various sorts of staff positions on campus. Once in these positions, some of them spend a few years conducting research and building their résumés prior to applying for faculty and administrative positions at UNL or elsewhere. Thus, it might be that some sizable percentage of the Staff performance could more accurately be characterized as having been produced by administrators with active research agendas and by lingering, highly motivated graduate students intent on improving their chances in a tough career market. Thus, if one’s institution has a culture similar to that of UNL, one may see a similar performance from one’s staff; if not, then likely not. Regardless, this study’s findings for Research Question #3 suggest that, in the social sciences at least, some of PDA’s critics’ assumptions concerning the intersection 546 College & Research Libraries May 2019 between academe’s natural hierarchy and the quality of what academic PDA patrons purchase may be inaccurate and misleading. Where the sciences were concerned, the statistics presented in appendix B and the results of the first batch of statistical tests reported on in table 3 suggest that PDA’s critics’ opinions in favor of librarian title-by-title selection were, again, justified. Their concerns with PDA, however, do not appear to have been supported by the data analyzed here. Far and away, the concerns that were most clearly and substantially supported by the UNL Libraries’ sciences data were, once again, critics’ concerns over the motivations and/or performances of vendors. Both the librarians’ and the patrons’ science books had higher citation counts by statistically significant amounts than did the book jobbers’ approval plan selections. Librarians’ books gar- nered just over 103,000 more citations than did the plans’ books, and Patrons’ books garnered almost 78,000 more. Librarians’ books did outperform Patrons’ by more than 25,000 citations, but this difference, again, proved not to be statistically significant. Thus, in reviewing the Order Types in general, it would also be difficult to argue that librarian selection in the sciences does not add value, and it would certainly be appropriate to see the results of this study as supporting the contention of several of the more caution- ary voices in the PDA debate that librarian selection should not be abandoned in favor of universal PDA just yet.55 The results of this study do not, so far, support the contention of reactionary voices in the debate that have suggested PDA/DDA ought not to be employed in academic libraries. The citation counts of the PDA Patrons books were nearly as good as were Librarians, so there would seem to be little justification for excluding academic library patrons in the sciences from collection development or for closely monitoring and mediating their collecting, especially if the distribution of one’s library’s PDA requests tends to resemble the UNL Libraries’ skew in favor of Faculty and Graduates. With minimal and reasonable restrictions in place (again, see appendix A), the UNL Libraries’ patrons per- formed quite well. The same, of course, cannot be said for the book vendors’ contributions to the collection via approval plans. The results of the table 3 tests for the sciences, as they did for the social sciences, suggest that critics of vendors may be more right than they know. Approval plan selectors for the UNL Libraries have not only proven themselves to be poor at selecting books that meet local needs,56 but they have, so far, proven themselves to be poor at identifying and selecting books that meet the interests of researchers generally, as well. Again, to produce a more nuanced picture of the three Order Types’ performance, with Research Question #2 the authors also compared the sciences’ books’ performance by Order Type within Topic groups. Of the 27 head-to-head, within-Topic comparisons reported on, the 9 involving Librarians versus PDA Patrons showed no statistically significant differences in performance, even prior to the adjustments made to their p values to account for multiple comparisons. In fact, the preadjustment p values showed only a single difference in perfor- mance being nearly significant. Thus, these results support the conclusion that there is little to choose from where Librarian and PDA Patron selection in the sciences are concerned. The same, once again, cannot be said for the approval plan selections. Half of the 18 post hoc comparisons showed Approval Plans as having been outperformed by either the Librar- ians or the Patrons by statistically significant amounts, even after adjustment for multiple comparisons. An additional three comparisons—against Patrons in Chemical Technology & Manufacturing and in Engineering & Technology (General) and against Librarians in Medicine The Scholarly Impact of Books Acquired 547 (General, Public Health, & Pathology)—were also nearly significant. Thus, Approval Plans’ per- formance proved to be pretty consistently poor across the several Topics. With respect to the criticisms and cautions raised and aired concerning which sorts of patrons ought or ought not to be allowed to add books to academic libraries’ collections, the results for the final research question would appear to offer some support for the contention that members of the faculty and graduate students can be relied upon as worthy selectors, but that undergraduate students and others might not be as reliable. In the set of tests reported on in table 5b, Faculty and Graduates citation counts were statistically indistinguishable, and there was also little to distinguish one from the other in additional Patron Types within-Topics post hoc tests performed later by the authors but not reported on. In the general tests of per- formance, both well outperformed Undergraduates, Staff, and the Unknown Patron Types, although none of the post hoc head-to-head comparisons’ results proved to be statistically significant. Still, Faculty and Graduates would seem a safer and more consistent bet in the sciences. The humanities data—including the statistics in appendix B, the general tests of the Order Types presented in table 3, and the Order Types within-Topics tests presented in table 4c—offer very strong support for the PDA approach. For advocates of librarian title-by-title selection, this data set also offers some welcome, if not entirely unproblematic, support as well. Consistent with the results presented for the social sciences and for the sciences, the humani- ties’ results for the general testing of the three Order Types support both PDA and librarian title-by-title selection approaches to some extent and call into question the value of approval plans for meeting researchers’ needs. The humanities PDA books attracted tens of thousands more citations than did the librarians’ firm-ordered books and the approval plans’ selections, and they, unsurprisingly, strongly and statistically significantly outperformed both in table 3. As well, in the 22 Order Types within-Topics comparisons reported on in table 4c, the PDA books pretty consistently outperformed the other two Order Types, with 18 of the 22 PDA mean citation counts being higher than either the Librarians’ or Approval Plans’ mean counts and with 12 of the 22 post hoc comparisons in table 4c favoring PDA by statistically signifi- cant amounts. Thus, it would seem worthwhile to allow humanities patrons a fair amount of freedom to add books to an academic library collection. Librarians, as was the case with the other disciplines, also outperformed Approval Plans by a statistically significant amount in table 3, although the humanities’ difference in perfor- mance was not as outsized as were those of the social sciences or the sciences. In table 4c, which presented the Order Types within-Topics comparisons, however, the Librarians versus Approval Plans relationship was shown to be much less clear. The within-Topics comparisons actually would appear to have been something of a draw. The two Order Types had higher means in roughly equal numbers of Topic categories, and the majority of the post hoc com- parisons strongly support failing to reject the null hypothesis. In essence, table 4c suggests that the liaison librarians for languages and literatures were able to outperform the approval plans, but none of the other humanities librarians were able to distinguish themselves in any meaningful way. Finally, for those critics who have taken a position against allowing undergraduates to add books to academic library collections via PDA, the UNL Libraries’ humanities data would offer little to no support. None of the known Patron Types performed better or worse than any of the others by statistically significant amounts, although there was some slight performance 548 College & Research Libraries May 2019 variation between types of patrons. Thus, the results reported here would support letting humanities students and faculty, and perhaps even university staff, to order what they like within the guidelines and constraints of the program. Limitations to the Study The first limitation to this study of which to be cognizant is one that affects much of the library literature generally and, as Walker noted, the PDA literature particularly: this was a single study conducted with data from a single site. This, of course, limits its generalizability, as it is impossible to determine whether or not it was merely characteristics of the setting that produced the effects presented. Thus, the results reported and conclusions drawn may not be universal, and studies of this sort should be conducted at other sites to reduce or eliminate the impact of local factors. The issue of the study’s generalizability naturally leads to the second important limita- tion to the study: the issue of its robustness. Recently, the robustness of scientific findings in a myriad of fields has been called into question,57 and this has led to something of a crisis in validity and to a renewed interest in replication. Recently, researchers have had difficulty reproducing the results of highly cited papers in clinical research,58 the findings of classic studies in psychological science,59 the conclusions medical studies and drug trials,60 findings concerning structural brain-behavior correlations,61 and so forth. Reproducibility is a defin- ing feature of science, the final arbiter of findings’ validity, and the cornerstone of cumulative science,62 and this should be no less the case in library science. The current study, as far as the authors are aware, has been the first of its kind. Without additional studies of this sort, it will be difficult, if not impossible, to determine how much the results of this study actually explain rather than merely describe, to determine whether the authors have observed system- atic regularities, idiosyncrasies in the data, or mere products of chance.63 Thus, it would be incorrect to conclude that the issues raised by PDA’s critics and advocates have been firmly and entirely settled here. The third important limitation to this study centers upon the type of PDA program stud- ied. As has been noted, previous research on PDA has shown that Order Type has been a good predictor of circulation.64 At the UNL Libraries, ILL PoD purchasing has been associated with greater circulation than librarians’ orders and approval plans’ selections.65 In essence, that an academic library patron has a need, perceived a book to have the potential to meet that need, was willing to go through the UNL Libraries’ ILL process to request it, and was then willing to wait for a few days to receive the requested item has predicted that the requested item will circulate significantly more in the short term. Thus, ILL-PoD–derived findings may not hold for other sorts of PDA programs that have lower barriers to patron satisfaction, such as catalog-integrated DDA programs for e-books that allow one to merely click on a link to ac- cess a book.66 The ILL-PoD findings also may not hold for programs that merely encourage patrons to recommend books for purchase, such as mediated “suggest-a-book” programs that direct patron input to selector librarians for vetting.67 A fourth limitation to the study involves the research questions and the metric employed to answer them. As was noted above, this study only addresses scholarly merit via scholarly impact as measured by citation counts. As such, it offers the conclusion that Order Type may be a useful predictor of scholarly attention, at least in the short term. This study has nothing to say about the quality of the scholarship in the books under examination, about how last- The Scholarly Impact of Books Acquired 549 ing their influence will be in the long run, about how well received they were by scholars, or about any of the other, more qualitative conclusions one could come to about the books and their impact. A fifth limitation to the study that the authors would like to note involves the samples employed. For the first research question, depending upon the discipline, the study had be- tween 300 and 500 books per Order Type to analyze. These samples should be sufficiently large to draw and support some meaningful conclusions. As the study progressed to its second question, however, the authors had to “slice” the samples more finely. For the third question, the authors employed just the books that comprised the three PDA samples. Thus, the results for these latter questions should be treated a bit more cautiously, and the conclusions drawn should be treated as being less certain. Finally, the reader should keep in mind that this study did not control for the disciplinary affiliations of the librarians and PDA patrons. The study’s samples were drawn and divided solely on the basis of Order Type and Topic. As a result, one cannot conclude that librarians or patrons with various disciplinary affiliations were or were not particularly good at select- ing books of high interest in their areas of specialization. For example, the samples for the “Education” Topic contained books purchased by librarians with liaison assignments to de- partments in the College of Education and Human Sciences, but the samples also contained books purchased by librarians with assignments to other departments and programs, such as anthropology, sociology, ethnic studies, and so forth. In most Topics, pertinent disciplin- ary affiliations did account for the majority of purchases in most Topics, but one should still not conclude too much from this study about the value of narrow and specific disciplinary knowledge. Conclusions The conclusions to be drawn from this study would seem to be several. For critics who have warned against a too eager, uncritical, and comprehensive adoption of PDA as a collections panacea and for advocates of the value of librarians as selectors, this study has provided some much-needed empirical support. Librarians performed very well in the social sciences and in the sciences, as well as or better than did the PDA patrons and much better than did the approval plans’ selectors. In the humanities, of course, the librarians were substantially outperformed by the patrons, but they did still significantly outperform the approval plans in their turn. Thus, it would seem far too soon to worry over whether selection is dead, as a panel discussion at a recent Charleston Conference provocatively inquired. The results of this study would support the myriad panelists’ arguments that librarian selection is still valuable and vital.68 For PDA critics who have loudly opined against allowing patrons in general or against undergraduates in particular to add titles to academic libraries’ collections, this study pro- vides some much-needed evidential discouragement. The PDA patrons performed well in all three disciplines, and the cases for the faculty and against the undergraduates were nowhere near as clear or strong as PDA’s critics would apparently have them. Graduate students were consistently good performers across all disciplines and within many Topics, and Faculty performed very well in the sciences, but the results of this study certainly do not support a blanket disparagement or lionization of any one type of patron. Certainly, concerned academic librarians could more closely monitor undergraduate students’ requests if doing so felt neces- 550 College & Research Libraries May 2019 sary, but if their institutions’ undergraduate students perform as well and make up as small a percentage of PDA purchases as do UNL’s, then the authors would be inclined to suggest that such heightened gatekeeping would largely be a waste of time. Which of the final two conclusions to this study will prove the most pertinent will depend upon the nature of the reader’s library and the principles guiding its collecting, as well as where one’s library falls along what Nardini, in an early article on the history of the adoption of approval plans, posited as the depth and breadth versus precision divide.69 Some number of this journal’s readers are employed by large research libraries with budgets and mandates oriented more toward comprehensiveness. Many more, no doubt, are employed by academic libraries with constrained budgets whose collection-building focus is directed more toward meeting patrons’ current needs. For the large research library whose intent is to collect more comprehensively and to build, as Anderson would have it, a cultural monument, this study should provide some reassur- ance that operating a PDA program will not precipitate the arrival of a “bibliopocalypse” that destroys the library, regardless of what some PDA critics appear to have been prophesying. The UNL PDA books in this study attracted good amounts of scholarly attention during the study interval, and nothing in this study would suggest that PDA polluted or debased the UNL Libraries’ collection with low-quality dross, beneath the interest of scholars. Past surveys of PDA programs have found that early adopters were devoting 1 to 5 percent of their book budgets to PDA.70 Given the results of this study and of past studies of PDA books’ circulation and collection suitability, it would be difficult to imagine that a large research library, however monumental its collection, could not profitably devote at least such a small percentage of its book budget to meeting its current patrons’ expressed needs. For the more typical sort of academic library, the library hoping to direct limited col- lection dollars toward impactful titles, the results of this study suggest that the PDA critics’ cautions against vendors’ and publishers’ plans and packages could profitably be heeded. Nardini, in a recently published chapter, positioned approval plans as an important part of academic libraries’ selection infrastructure, one that provides libraries with numerous ben- efits and advantages, especially in the areas of efficiency and collection balance.71 From the results of this study, however, it would be difficult not to conclude that, as prices continue to increase and as collection dollars grow ever more dear, approval plans’ advantages may be outweighed by their failings. The results of this and past studies at the UNL Libraries incline the authors to wonder whether, in the librarian-vendor relationship, the shadowy outlines of a classic principal-agent problem may not be discerned, the sort of problem where one party believes the other to be acting to advance its interests when the second party is in fact acting to advance its own.72 Contrary to the rosy picture advanced by Nardini in the aforementioned chapter, much of the early criticisms of the approval plan model warned of just such potential conflicts of interest.73 Approval plans and e-book packages may well deliver the efficiencies and savings promised, but at UNL the approval plans appear to have been doing so with books that comparatively few researchers want. The not infrequent response in the field to this dilemma is to suggest that vendors’ products require monitoring and modification to function perfectly,74 but employing this approach at UNL actually produced worse perfor- mance on several occasions. If academic libraries with similarly constrained book budgets find themselves experiencing similarly unsatisfactory results with vendor plans or packages, they may wish to investigate the benefits of divesting. The Scholarly Impact of Books Acquired 551 To close: as was noted above, the library literature on PDA has so far been largely posi- tive and supportive of the model, and a number of studies have substantiated the claims of its advocates, especially where circulation/usage and collection suitability have been concerned. The counter literature skeptical of or cautioning against PDA, to this point, has warned of the dangers and shortcomings of the model almost entirely speculatively. The authors hope that this study has contributed to a continued, evidence-driven, and empirically supported conversation on the PDA model and furthered the examination of its benefits and pitfalls for academic libraries. Effective and efficient materials selection is a difficult process, and academic librarians engaged in collection building often find themselves striving to satisfy multiple and sometimes contradictory masters. Concerning the struggle to purchase books that meet current patrons’ needs and the simultaneous struggle to purchase books of high and sustained scholarly interest, this study suggests that PDA’s advocates may be right: Giv- ing the patrons what they want today may very well be supplying the collection with what it needs for tomorrow. 552 College & Research Libraries May 2019 APPENDIX A. UNL University Libraries’ ILL Purchasing Project Criteria (2010–2011) All books requested via ILLiad, published between 2007 and 2010 and not owned by UNL Libraries must be considered for purchase (books in IRIS that are counted as missing, lost, or on search cannot be purchased). Selected books will be judged by the following criteria. Books must be available from Barnes and Noble online and must arrive in a timely manner. We cannot order any book that takes longer than 2–3 days to ship. We also cannot order books that are marked as “Pre-orders.” The price limit on purchasing project books is $175; we may not order books that cost more than this amount. Books purchased have academic merit. The following type of books may not be purchased: • Textbooks (any book designated by OCLC or Barnes & Noble as a textbook) • Foreign language books • Fiction, poetry, plays • Music scores • Lab manuals, workbooks, field guides • Solutions manuals • Popular interest (nonfiction, best sellers, self-help, and the like) • Popular biographies (biographies can be decided on a case-by-case basis) • Journal volumes/serials • Computer books • Anthologies • Older editions of a book that still fall within the 3-year period Most books not marked as textbooks published by university presses are appropriate for the ILL purchasing program. Another way to judge a book that may be published by an unknown publisher but is not disqualified by any of the above criteria is to look in the OCLC holdings to see what other libraries own this item. If it is owned by many GWLA and ARL libraries, it should be purchased. If it is owned by mostly public libraries, it should be obtained via interlibrary loan. If it is a new book not owned by many libraries or you think it would be a good addition to the collection, decide which subject area the book falls under and consult the library liaison. Ask if they feel the book is a worthwhile purchase. If they feel that it is, you may buy it. If the liaison is not available or the book falls under the prohibited categories but still seems like a good addition to the library collection, ask [the Chair of Technical Services] for permission to purchase it. The Scholarly Impact of Books Acquired 553 APPENDIX B. Sample Distributions and Parameters Social Sciences: Distributions and Parameters of Citation Counts by Order Type: 1) Approval Plans, 2) Librarians, and 3) PDA Patrons 1) Pe rc en t Sum = 79,068, Range = 0–6,050, Mean = 161.035, Median = 43, St. Dev. = 498.968, Skew = 8.491, Skew S.E. = .110 2) Pe rc en t Sum = 177,741, Range = 0–28,479, Mean = 361.998, Median = 47, St. Dev. = 1,728.174, Skew = 10.791, Skew S.E. = .110 3) Pe rc en t Sum = 135,448, Range = 0–11,758, Mean = 275.862, Median = 71, St. Dev. = 975.865, Skew = 8.224, Skew S.E. = .110 Times Cited Curve Normal Note: Values rounded to the nearest 1/1,000th where appropriate. 554 College & Research Libraries May 2019 Sciences: Distributions and Parameters of Citation Counts by Order Type: 1) Approval Plans, 2) Librarians, and 3) PDA Patrons 1) Pe rc en t Sum = 36,588, Range = 0–3,381, Mean = 106.671, Median = 36, St. Dev. = 241.988, Skew = 8.290, Skew S.E. = .132 2) Pe rc en t Sum = 139,987, Range = 0–23,877, Mean = 408.125, Median = 41, St. Dev. = 1,840.102, Skew = 9.860, Skew S.E. = .132 3) Pe rc en t Sum = 114,545, Range = 0-12,252, Mean = 333.950, Median = 77, St. Dev. = 1,044.363, Skew = 8.937, Skew S.E. = .132 Times Cited Curve Normal Note: Values rounded to the nearest 1/1,000th where appropriate. The Scholarly Impact of Books Acquired 555 Humanities: Distributions and Parameters of Citation Counts by Order Type: 1) Approval Plans, 2) Librarians, and 3) PDA Patrons 1) Pe rc en t Sum = 29,985, Range = 0–1,800, Mean = 48.756, Median = 17, St. Dev. = 113.845, Skew = 8.029, Skew S.E. = .099 2) Pe rc en t Sum = 39,765, Range = 0–3,995, Mean = 64.659, Median = 14, St. Dev. = 216.044, Skew = 11.726, Skew S.E. = .099 3) Pe rc en t Sum = 59,899, Range = 0–5,042, Mean = 97.397, Median = 26, St. Dev. = 314.550, Skew = 9.480, Skew S.E. = .099 Times Cited Curve Normal Note: Values rounded to the nearest 1/1,000th where appropriate. 556 College & Research Libraries May 2019 Notes 1. Jennifer Perdue and James A. Van Fleet, “Borrow or Buy? Cost-effective Delivery of Monographs,” Journal of Interlibrary Loan, Document Delivery & Information Supply 9, no. 4 (1999): 19–28. 2. Association of College and Research Libraries, Research, Planning and Review Committee, “2010 Top Ten Trends in Academic Libraries: A Review of the Current Literature,” College & Research Libraries News 71, no. 6 (2010): 286–92; Association of College and Research Libraries, Research, Planning and Review Committee, “2012 Top Ten Trends in Academic Libraries: A Review of the Trends and Issues Affecting Academic Libraries in Higher Education,” College & Research Libraries News 73, no. 6 (2012): 311–20. 3. Cristina Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” in Customer-based Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014), 1–12; Karin J. Fulton, “The Rise of Patron-driven Acquisitions: A Literature Review,” Georgia Library Quarterly 51, no. 3 (2014): Article 10, available online at http://digitalcommons.kennesaw.edu/glq/vol51/iss3/10 [accessed 28 November 2017]; Denise A. Garofalo, “Tips from the Trenches,” Journal of Electronic Resources Librarianship 23, no. 3 (2011): 274–76; Rick Lugg, “Collecting for the Moment: Patron-driven Acquisitions as a Disruptive Technology,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 7–22; Norm Medeiros, “King of the Hill,” OCLC Systems & Services 28, no. 2 (2012), 64–66. 4. Kizer Walker, “Patron-Driven Acquisition in US Academic Research Libraries: At the Tipping Point in 2011?” Bibliothek Forschung und Praxis 36, no. 1 (2012): 126–30. 5. Deborah Lenares and Emilie Delquié, “Give the People What They Want: Patron Driven Acquisition: Re- sults and Reflections on a Survey Completed by Publishers Communication Group” (presentation, STM Spring Conference, Le Meridien Cambridge-MIT, Cambridge, MA, April 21–29, 2010). 6. Nestor L. Osorio, “A User’s Participatory Selecting Model: Librarians Point of Views” (presentation, Illi- nois Library Association Annual Meeting, Rosemont, IL, October 19, 2011), available online at http://hdl.handle. net/10760/16507 [accessed 26 January 2018]. 7. Joseph Esposito, Kizer Walker, and Terry Ehling, “PDA and the University Press: A Report Prepared for The Andrew W. Mellon Foundation,” Journal of Scholarly Publishing 44, no. 3 (2013): s1–s62. 8. Steven Carrico, Michelle Leonard, and Erin Gallagher, Implementing and Assessing Use-Driven Acquisitions (Lanham, MD: Rowman & Littlefield, 2016); Tony Horava and Michael Levine-Clark, “Current Trends in Col- lection Development Practices and Policies,” Collection Building 35, no. 4 (2016): 97–102. 9. Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” 1–12. 10. Google, Google Scholar, n.d., s.v. “college, libraries, patron, driven, acquisitions,” available online at http:// scholar.google.com/ [accessed 31 October 2017]. 11. Dee Ann K. Allison, The Patron-driven Library: A Practical Guide for Managing Collections and Services in the Digital Age (Oxford: Chandos Publishing, 2013); Rick Anderson, “Collections 2021: The Future of the Library Collection Is Not a Collection,” Serials 24, no. 3 (2011): 211–15; Faye A. Chadwell, “What’s Next for Collection Management and Managers? User-centered Collection Management,” Collection Management 34, no. 2 (2009): 69–78; Joseph Esposito, Kizer Walker, and Terry Ehling. “The New Supply Chain and Its Implications for Books in Libraries,” Educause Review 47, no. 5 (2012): 58–59; Esposito, Walker, and Ehling, “PDA and the University Press,” s1–s62; Fulton, “The Rise of Patron-driven Acquisitions,” Article 10; Edward A. Goedeken and Karen Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” College & Re- search Libraries 76, no. 2 (2015): 205–21; Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30. 12. John Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions: A Summary,” in Customer- based Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014): 159–76; Barbara Fister, “Problematizing Patron-driven Acquisitions,” Library Journal: Peer to Peer Review (blog, November 11, 2010); Barbara Fister, “Puzzled by Patron-driven Acquisitions,” Library Babel Fish (blog, November 11, 2010), available online at https://www.insidehighered.com/blogs/library-babel-fish/puzzled-patron-driven-acquisitions [accessed 12 April 2019]; Mary Gilbertson, Elizabeth Chadbourn McKee, and Lutishoor Salisbury, “Just in Case or Just in Time? Outcomes of a 15-month Patron-driven Acquisition of e-Books at the University of Arkansas Libraries,” Library Collections, Acquisitions & Technical Services 38, no. 1/2 (2014): 10–20; William Miller, “Patron-driven Acquisitions (PDA): The New Wave in Book Acquisitions Is Coming,” Library Issues: Briefings for Faculty and Administrators 31, no. 5 (2011): 1–4; Scott A. Smith, “PDA: Driving Off the Cliff or, New Wine in Old Bottles,” Against the Grain 23, no. 3 (2011): 40; Suzanne M. Ward, Guide to Implementing and Managing Patron-Driven Acquisitions (Chicago: American Library Association, 2012). 13. Esposito et al., “The New Supply Chain and Its Implications for Books in Libraries,” 58–59; Esposito et https://digitalcommons.kennesaw.edu/glq/vol51/iss3/10/ http://hdl.handle.net/10760/16507 http://hdl.handle.net/10760/16507 http://scholar.google.com/ http://scholar.google.com/ https://www.insidehighered.com/blogs/library-babel-fish/puzzled-patron-driven-acquisitions The Scholarly Impact of Books Acquired 557 al., “PDA and the University Press,” s1–s62; Goedeken and Lawson, “The Past, Present, and Future of Demand- driven Acquisitions in Academic Libraries,” 205–21; Lugg, “Collecting for the Moment,” 7–22. 14. Kathleen A. Lehman, “Collection Development and Management,” Library Resources & Technical Services 58, no. 3 (2014): 169–77; Miller, “Patron-driven Acquisitions (PDA),” 1–4; Jane Schmidt, “Demand-driven Acquisi- tions: The Hegemony of the Canon Interrupted,” in Creating Sustainable Community: The Proceedings of the ACRL 2015 Conference (Portland, Oregon, March 25–28, 2015), 168-73, available online at www.ala.org/acrl/sites/ala.org. acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf [accessed15 November 2017]; Elena S. Smith, “Power and Practice in Academic Library Materials Selection Paradigms,” SLIS Student Research Journal 1, no. 2 (2011): Article 6, available online at http://scholarworks.sjsu.edu/slissrj/vol1/iss2/6/ [accessed 1 May 2012]; William H. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” Library Resources & Technical Services 56, no. 3 (2012): 199–213. 15. Medeiros, “King of the Hill,” 64–66. 16. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30. 17. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, xiii. 18. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Esposito et al., “PDA and the University Press,” s1–s62; Gilbertson et al., “Just in Case or Just in Time?” 10–20; Smith, “PDA: Driving Off the Cliff or, New Wine in Old Bottles,” 40; Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30. 19. David C. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials: A Chronological Review and Summary of Findings,” Library Philosophy and Practice (2011): Article 635, available online at http://digitalcommons.unl.edu/libphilprac/635/ [accessed 26 January 2018]; David C. Tyler, Joyce C. Melvin, MaryLou Epp, and Anita M. Kreps, “Patron-driven Acquisition and Monopolistic Use: Are Patrons at Academic Libraries Using Library Funds to Effectively Build Private Collections?” Library Philosophy and Practice (2014) 11, Article 1,149, available online at http://digitalcommons.unl.edu/libphilprac/1149/ [accessed 26 January 2018]. 20. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 129. 21. Jason C. Dewland and Andrew See, “Notes on Operations: Patron Driven Acquisitions: Determining the Metrics for Success,” Library Resources & Technical Services 59, no. 1 (2015): 13–23. 22. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, 95. 23. Laura Costello, Evaluating Demand-driven Acquisitions (Cambridge, UK: Chandos Publishing, 2016); Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635; Tyler et al. “Patron-driven Acquisition and Monopolistic Use,” Article 1,149. 24. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213; Aaron Wood, “Mainstream Patron-Driven Acquisition: Topicality Over the Scholarly Record… and the Cello Suites,” Against the Grain 25, no. 5 (2013): 22. 25. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21. 26. Costello, Evaluating Demand-driven Acquisitions, ix. 27. Joseph Esposito, “A Dialogue on Patron-Driven Acquisitions,” The Scholarly Kitchen (blog, January 3, 2012), available online at https://scholarlykitchen.sspnet.org/2012/01/03/a-dialogue-on-patron-driven-acquisitions/ [ac- cessed 15 November 2017]. 28. Sheila Corrall, “Community Engagement in Collection Development: Social Responsibility or Professional Abdication?” (presentation, Community Engagement & Social Responsibility: ALISE’17 Annual Conference, Atlanta, GA, January 17–20, 2017). 29. James A. Cogswell, “The Organization of Collection Management Functions in Academic Research Li- braries,” Journal of Academic Librarianship 13, no. 5 (1987): 270. 30. Rick Anderson, “What Patron-Driven Acquisition (PDA) Does and Doesn’t Mean: An FAQ,” The Scholarly Kitchen (blog, May 31, 2011), available online at https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven- acquisition-pda-does-and-doesnt-mean-an-faq/ [accessed 15 November 2017]; Anderson, “Collections 2021,” 211–15. 31. Anderson, “Collections 2021,” 211–15; Lorcan Dempsey, “Library Collections in the Life of the User: Two Directions,” Liber Quarterly 26, no. 4 (2017): 338–59; Julie Linden, Sarah Tudesco, and Daniel Dollar, “Collections as a Service: A Research Library’s Perspective,” College & Research Libraries 79, no. 1 (2018): 86–99; Doug Way, “Transforming Monograph Collections with a Model of Collections as a Service,” portal: Libraries and the Acad- emy 17, no. 2 (2017): 283–94. 32. Miller, “Patron-driven Acquisitions (PDA),” 1–4; Schmidt, “Demand-driven Acquisitions: The Hegemony http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf http://scholarworks.sjsu.edu/slissrj/vol1/iss2/6/ http://digitalcommons.unl.edu/libphilprac/635/ http://digitalcommons.unl.edu/libphilprac/1149/ https://scholarlykitchen.sspnet.org/2012/01/03/a-dialogue-on-patron-driven-acquisitions/ https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-acquisition-pda-does-and-doesnt-mean-an-faq/ https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-acquisition-pda-does-and-doesnt-mean-an-faq/ 558 College & Research Libraries May 2019 of the Canon Interrupted,” 168–73; Smith, “Power and Practice in Academic Library Materials Selection Para- digms,” Article 6; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213. 33. Douglas Jones, “On-demand Information Delivery: Integration of Patron-driven Acquisition into a Com- prehensive Information Delivery System,” Journal of Library Administration 51, no. 7/8 (2011): 764–76; Michael Levine-Clark, “Developing a Multiformat Demand-driven Acquisition Model,” Collection Management 35, no. 3/4 (2010): 201–07; Michael Levine-Clark, “Developing a Model for Long-term Management of Demand-driven Acquisitions,” Against the Grain 23, no. 3 (2011): 24–26; Michael Levine-Clark, “Building a Demand-Driven Col- lection: The University of Denver Experience,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 45–60; Peter Spitzform, “Patron-driven Acquisition: Collecting as if Money and Space Mean Something,” Against the Grain 23, no. 3 (2014): 20–24; Peter Spitzform and Pongracz Sennyey, “A Vision for the Future of Academic Library Collections,” International Journal of the Book 4, no. 4 (2007): 185–89. 34. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Fister, “Problema- tizing Patron-driven Acquisitions” (blog); Fister, “Puzzled by Patron-driven Acquisitions” (blog); Jean-Mark Sens and Anthony J. Fonseca, “A Skeptic’s View of Patron-driven Acquisitions: Is It Time to Ask the Tough Questions?” Technical Services Quarterly 30, no. 4 (2013): 359–71. 35. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Chadwell, “What’s Next for Collection Management and Managers?” 69–78; Candice Dahl, “Primed for Patron-driven Acquisition: A Look at the Big Picture,” Journal of Electronic Resources Librarianship 24, no. 2 (2012): 119–26; Lindsey Reno, “Out of the Shadows: A Public Face for Acquisitions in Academic Libraries,” Against the Grain 24, no. 6 (2012): 22–26; F.K. Rottmann, “To Buy or to Borrow: Studies of the Impact of Interlibrary Loan on Collection Development in the Academic Library,” Journal of Interlibrary Loan & Information Supply 1, no. 3 (1991), 17–27; Sens and Fonseca, “A Skeptic’s View of Patron-driven Acquisitions,” 359–71; Smith, “PDA: Driving off the Cliff or, New Wine in Old Bottles,” 40; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213; Ward, Guide to Implementing and Managing Patron-Driven Acquisitions; Aaron Wood, “Mainstream Patron-Driven Acquisition,” 22. 36. Audrey Fenner, “The Approval Plan: Selection Aid, Selection Substitute,” The Acquisitions Librarian 16, no. 31/32 (2004): 227–40; Sherry E. Gelbwasser, “Vendor-Library Relationships in ARL Libraries,” The Acquisitions Librarian 8, no. 16 (1996): 109–17; Alison Griffin and Sarah Forzetting, “Behind the Scenes: Approval Profiles from a Vendor’s Perspective,” in Library Collection Development for Professional Programs: Trends and Best Practices, ed. Sarah Holder (Hershey, PA: IGI Global, 2012), 16–31; Bob Nardini, “Approval Plans and Patron Selection: Two Infrastructures,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 23–43; Robert F. Nardini, “Approval Plans: Politics and Performance,” College & Research Libraries 54, no. 5 (1993): 417–26. 37. Wenli Gao, Irene Ke, and Cherie Turner, “Programming Plus Subject Expertise: A Combined Approach for Approval Profile Modification,” Collection Management (forthcoming); Irene Ke, Wenli Gao, and Jackie Bronicki, “Does Title-By-Title Selection Make a Difference? A Usage Analysis on Print Monograph Purchasing,” Collection Management 42, no. 1 (2017): 34–47; David C. Tyler, Christina Falci, Joyce C. Melvin, MaryLou Epp, and Anita M. Kreps, “Patron-driven Acquisition and Circulation at an Academic Library: Interaction Effects and Circulation Performance of Print Books Acquired via Librarians’ Orders, Approval Plans, and Patrons’ Interlibrary Loan Requests,” Collection Management 38, no. 1 (2013): 3–32; David C. Tyler, Yang Xu, Joyce C. Melvin, Marylou Epp, and Anita M. Kreps, “Just How Right Are the Customers? An Analysis of the Relative Performance of Patron- Initiated Interlibrary Loan Monograph Purchases,” Collection Management 35, no. 3/4 (2010): 162–79. 38. David C. Tyler, Joyce C. Melvin, Yang Xu, Marylou Epp, and Anita M. Kreps, “Effective Selectors? Inter- library Loan Patrons as Monograph Purchasers: A Comparative Examination of Price and Circulation-Related Performance,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 21, no. 1/2 (2011): 57–90. 39. Denise Brush, “Circulation Analysis of an Engineering Monograph Approval Plan,” Collection Building 26, no. 2 (2007): 59–62; Erin L. Ellis, Nikhat J. Ghouse, Monica Claassen-Wilson, John M. Stratton, and Susanne K. Clement, “Comparing Approval and Librarian-Selected Monographs: An Analysis of Use,” in Library Data: Empowering Practice and Persuasion, ed. Darby Orcutt (Santa Barbara, CA: Libraries Unlimited, 2010), 53–68; Marcie Kingsley, “Circulation Statistics for Measuring Approval Plan Effectiveness,” Against the Grain 8, no. 4 (1996): 1, 16–17; James Cory Tucker, “Collection Assessment of Monograph Purchases at the University of Nevada, Las Vegas Libraries,” Collection Management 34, no. 3 (2009): 157–81; Michele Wolff, “Evaluating the Art Approval Plan: A Case Study,” Art Documentation 18, no. 1 (1999): 5–9. 40. John S. Brantley, “Approval Plans, Discipline Change, and the Importance of Human Mediated Book Selection,” Library Collections, Acquisitions, and Technical Services 34, no. 1 (2010): 11–24; Mary Eldredge, “Major The Scholarly Impact of Books Acquired 559 Issues in Approval Plans: The Case for Active Management,” The Acquisitions Librarian 8, no. 16 (1996): 51–59. 41. Kady Ferris and Tina Herman Buck, “An Ethos of Access: How a Small Academic Library Transformed Its Collection-building Processes,” Collection Management 39, no. 2/3 (2014): 127–44; Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21. 42. Costello, Evaluating Demand-driven Acquisitions. 43. Esposito, Walker, and Ehling, “PDA and the University Press,” s1–s62. 44. Tyler et al., “Patron-driven Acquisition and Monopolistic Use,” 11. 45. Jason Price and John McDonald, “Beguiled by Bananas: A Retrospective Study of the Usage & Breadth of Patron vs. Librarian Acquired eBook Collections,” in Proceedings of the Charleston Library Conference (Charleston, South Carolina, 2009), doi:10.5703/1288284314741; Gabrielle Wiersma and Yem Fong, “Patron-driven e-Book Solu- tions: Moving beyond the Banana Books Incident,” in Proceedings of the Charleston Library Conference (Charleston, South Carolina, 2011), doi:10.5703/1288284314973. 46. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21. 47. Leslie J. Reynolds, Carmelita Pickett, Wyoma Vanduinkerken, Jane Smith, Jeanne Harrell, and Sandra Tucker, “User-driven Acquisitions: Allowing Patron Requests to Drive Collection Development in an Academic Library,” Collection Management 35, no. 3/4 (2010): 244–54. 48. Jeffrey H. Waller, “Undergrads as Selectors: Assessing Patron-driven Acquisition at a Liberal Arts Col- lege,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 23, no. 3 (2013): 127–48. 49. Gilbertson et al., “Just in Case or Just in Time?” 10–20; Doug Way, “The Assessment of Patron-initiated Collection Development via Interlibrary Loan at a Comprehensive University,” Journal of Interlibrary Loan, Docu- ment Delivery & Electronic Reserve 19, no. 4 (2009): 299–308. 50. Waller, “Undergrads as Selectors,” 127–48. 51.. Judith M. Nixon and E. Stewart Saunders, “A Study of Circulation Statistics of Books on Demand: A Decade of Patron-Driven Collection Development, Part 3,” Collection Management 35, no. 3/4 (2010): 151–61. 52. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213. 53. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635. 54. Emily K. Chan and Susan L. Kendall, “Identifying Users of Demand-driven E-book Programs: Applica- tions for Collection Development,” in Imagine, Innovate, Inspire : Proceedings of the ACRL 2013 Conference (Chicago: Association of College and Research Libraries, 2013), 301–08. 55. Rick Anderson, “Is Selection Dead? The Rise of Collection Management and the Twilight of Selection,” Against the Grain 23, no. 2 (2011): 40–44. 56. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just How Right Are the Customers?” 162–79. 57. John P.A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (2005): e124, available online at http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 [accessed 23 April 2018]. 58. John P.A. Ioannidis, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Re- search,” JAMA 294, no. 2 (2005): 218–28. 59. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 349, no. 6251 (2015): aac4716; Harold Pashler and Eric-Jan Wagenmakers, “Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?” Perspectives on Psychological Science 7, no. 6 (2012): 528–30. 60. C. Glenn Begley and Lee M. Ellis, “Drug Development: Raise Standards for Preclinical Cancer Re- search.” Nature 483, no. 7391 (2012): 531–33; Florian Prinz, Thomas Schlange, and Khusru Asadullah, “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery 10, no. 9 (2011): 712. 61. Wouter Boekel, Eric-Jan Wagenmakers, Luam Belay, Josine Verhagen, Scott Brown, and Birte U. Forstmann, “A Purely Confirmatory Replication Study of Structural Brain-Behavior Correlations,” Cortex 66 (2015): 115–33. 62. Jennifer Crocker and M. Lynne Cooper, “Addressing Scientific Fraud,” Science 334, no. 6060 (2011): 1182; Jeremy Freese, “Replication Standards for Quantitative Social Science: Why Not Sociology?” Sociological Methods & Research 36, no. 2 (2007): 153–72; Gary King, “Replication, Replication,” PS: Political Science & Politics 28, no. 3 (1995): 444–52; Barbara R. Jasny, Gilbert Chin, Lisa Chong, and Sacha Vignieri, “Again, and Again, and Again…,” Science 334, no. 6060 (2011): 1225; Matthew C. Makel and Jonathan A. Plucker, “Facts Are More Important Than Novelty: Replication in the Education Sciences,” Educational Researcher 43, no. 6 (2014): 304–16; Stefan Schmidt, “Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences,” Review https://doi.org/10.5703/1288284314741 https://doi.org/10.5703/1288284314973 http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 560 College & Research Libraries May 2019 of General Psychology 13, no. 2 (2009): 90–100. 63. Nassim Nicholas Taleb, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (New York: Random House, 2005). 64. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635. 65. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just How Right Are the Customers?” 162–79. 66. Costello, Evaluating Demand-driven Acquisitions. 67. Rebecca Schroeder, “When Patrons Call the Shots: Patron-driven Acquisition at Brigham Young Univer- sity,” Collection Building 31, no. 1 (2012): 11–14. 68. Anderson, “Is Selection Dead?” 40–44. 69. Nardini, “Approval Plans: Politics and Performance,” 420. 70. Lenares and Delquié, “Give the People What They Want.” 71. Nardini, “Approval Plans and Patron Selection,” 23–43. 72. Michael C. Jensen and William H. Meckling, “Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure,” Journal of Financial Economics 3, no. 4 (1976): 305–60. 73. Association of Research Libraries, Approval Plans in ARL Libraries, SPEC Kit, no. 83 (Washington, DC: ARL, 1982), 79–84; Margaret Dobbyn, “Approval Plan Purchasing in Perspective,” in Background Readings in Building Library Collections, 2nd ed., eds. Phyllis Van Orden and Edith B. Phillips (Metuchen, NJ: Scarecrow Press, 1979): 194; Jane Maddox, “Approval Plans—Viable?” Journal of Academic Librarianship 1 (1976): 22; Rose Mary Magrill and Mona East, “Collection Development in Large University Libraries,” in Advances in Librarianship, vol. 8, ed. Michael H. Harris (San Diego, CA: Academic Press, 1978): 6; Norman D. Stevens, “The Resources and Technical Services Division of ALA,” Library Resources & Technical Services 13 (1969): 334. 74. Leslie Czechowski, “Edging toward Perfection: Analysis of a New Approval Plan in a Health Sciences Library,” Library Collections, Acquisitions, & Technical Services 32 (2008): 107–11; Eldredge, “Major Issues in Ap- proval Plans,” 51–59.