Resource Discovery: Comparative Survey Results on Two Catalog Interfaces Heather Hessel and Janet Fransen RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 21 ABSTRACT Like many libraries, the University of Minnesota Libraries-Twin Cities now offers a next-generation catalog alongside a traditional online public access catalog (OPAC). One year after the launch of its new platform as the default catalog, usage data for the OPAC remained relatively high, and anecdotal comments raised questions. In response, the libraries conducted surveys that covered topics such as perceptions of success, known-item searching, preferred search environments, and desirable resource types. Results show distinct differences in the behavior of faculty, graduate student, and undergraduate survey respondents, and between library staff and non-library staff respondents. Both quantitative and qualitative data inform the analysis and conclusions. INTRODUCTION The growing level of searching expertise at large research institutions and the increasingly complex array of available discovery tools present unique challenges to librarians as they try to provide authoritative and clear searching options to their communities. Many libraries have introduced next-generation catalogs to satisfy the needs and expectations of a new generation of library searchers. These catalogs incorporate some of the features that make the current web environment appealing: relevancy ranking, recommendations, tagging, and intuitive user interfaces. Traditional OPACs are generally viewed as more complex systems, catering to advanced users and requiring explicit training in order to extract useful data. Some librarians and users also see them as more effective tools for conducting research than next-generation catalogs. Academic libraries are frequently caught in the middle of conflicting requirements and expectations for discovery from diverse sets of searchers. In 2002, the University of Minnesota-Twin Cities Libraries migrated from the NOTIS library system to the ALEPH500™ system and launched a new web interface based on the ALEPH online catalog, originally branded as MNCAT. In 2006, the libraries contracted with the Ex Libris Group as one of three development partners in the creation of a new next-generation search environment called Primo. During the development process, the libraries conducted multiple usability studies that provided data to inform the direction of the product. Participants in the usability studies generally characterized the Primo interface as “clear” and “efficient.”1 A year later the University Heather Hessel (heatherhessel@yahoo.com) was Interim Director of Enterprise Technology and Systems, Janet Fransen (fransen@umn.edu) is the librarian for Aerospace Engineering, Electrical Engineering, Computer Science, and History of Science & Technology, University of Minnesota, Minneapolis, MN. mailto:heatherhessel@yahoo.com mailto:fransen@umn.edu INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 22 Libraries branded Primo as MNCAT Plus, rebranded the ALEPH OPAC as MNCAT Classic, and introduced MNCAT Plus to the Twin Cities user community as a beta service. In August 2008, MNCAT Plus was configured as the default search for the Twin Cities catalog on the libraries’ main website, with the libraries continuing to keep a separate link active to the ALEPH OPAC. A new organizational body called the Primo Management Group was created in December 2008 to coordinate support, feedback, and enhancements of the local Primo installation. This committee’s charge includes evaluating user input and satisfaction, coordinating communication to users and staff, and prioritizing enhancements to the software and the normalization process. When the Primo Management Group began planning its first user satisfaction survey, the group noted that a significant number of library users seemed to prefer MNCAT Classic. Therefore, two surveys were developed in response to the group’s charge. These two surveys were identical in scope and questions, except that one survey referenced MNCAT Classic and was targeted to MNCAT Classic searchers (appendix A), while the other survey referenced MNCAT Plus and was targeted to MNCAT Plus searchers (appendix B). These surveys were designed to produce statistics that could be used as internal benchmarks to gauge library progress in areas of user experience, as well as to assist with ongoing and future planning with regard to discovery tools and features. RESEARCH QUESTIONS In addition to evaluating user satisfaction and requesting user input, the Primo Management Group also chose to question users about searching behaviors in order to set the direction of future interface work. Questions directed toward searching behaviors were informed by the findings from a 2009 University of Minnesota Libraries report on making resources discoverable.2 The group surveyed respondents about types of items they expect to find in their searches, their interest in online resources, and the entry point for their discovery experience. The Primo Management Group crafted the surveys to get answers to the following research questions:  How often do users view their searching activity as successful?  How often do users know the title of the item that they are looking for, as opposed to finding any resource relevant to their topic?  What search environments do users choose when looking for a book? A journal? Anything relevant to a topic?  How interested are users in finding items that are not physically located at the University of Minnesota?  Are there other types of resources that users would find helpful to discover in a catalog search? RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 23 Although it can be tempting to think of the people using the catalog interfaces as a homogeneous group of “users,” large academic libraries serve many types of users. As Wakimoto states in “Scope of the Library Catalog in Times of Transition,” On the one hand, we have ‘Net-generation users who are accustomed to the simplicity of the Google interface, are content to enter a string of keywords, and want only the results that are available online. On the other hand, we have sophisticated, experienced catalog users who understand the purpose of uniform titles and Library of Congress classifications and take full advantage of advanced search functions. We need to accommodate both of these user groups effectively.3 The Primo Management Group planned to use the demographic information to look for differences among user communities; therefore the surveys requested demographic information such as role (e.g., student) and college of affiliation (e.g., School of Dentistry). In designing the surveys, the group took into account the limitations of this type of survey as well as the availability of other sources of information. For example, the Primo Management Group chose not to include questions about specific interface features because such questions could be answered by analyzing data from system logs. The group was also interested in finding out about users’ strategies for discovering information, but members felt that this information was better obtained through focus groups or usability studies rather than through a survey instrument. RESEARCH METHOD The Primo Management Group positioned links to the user surveys in several online locations, with the libraries’ home page providing one primary entry point. Clicking on the link from the home page presented users with an intermediate page, where they were given a choice of which survey to complete: one based on MNCAT Plus, and the other on MNCAT Classic. If desired, users could choose to complete a separate survey for each of the two systems. Links were also provided from within the MNCAT Plus and MNCAT Classic environments, and these links directed users to the relevant version of the survey without the intermediary page. In addition to the survey links in the online environment, announcements were made to staff about the surveys, and librarians were encouraged to publicize the surveys to their constituents around campus. The survey period lasted from October 1 through November 25, 2009. At the time of the surveys, the University of Minnesota Libraries was running Primo version 2 and ALEPH version 19. Because participants were self-selected, the survey results represent a biased sample, are more extreme than the norm, and are not generalizable to the whole university population. Participants were not likely to click the survey link or respond to e-mailed requests unless they had sufficient incentive, such as strong feelings about one interface or the other. Thirty percent of respondents provided an e-mail address to indicate that they would be willing to be contacted for focus groups or further surveys, indicating a high level of interest in the public-facing interfaces the libraries employ. In considering a process for repeating this project, more attention would be paid to methodology to address validity concerns. FINDINGS AND ANALYSIS INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 24 Findings relevant to each research question are discussed here. Six hundred twenty-nine surveys contained at least one response—476 for MNCAT Plus and 153 for MNCAT Classic. Responses by Demographics As shown in table 1, graduate students were the primary respondents for both MNCAT Plus and MNCAT Classic, followed by undergraduates and faculty members. Library staff made up 13 percent of MNCAT Classic respondents and 4 percent of MNCAT Plus respondents, although the actual number of library staff responding was nearly identical (twenty-one for MNCAT Plus, twenty for MNCAT Classic). Library staff members were disproportionately represented in these survey responses and the group analyzed the results to identify categories in which library staff members differed from overall trends in the responses. Questions about affiliation appeared at the end of the surveys, which may account for the high number of respondents in the “Unspecified” category. MNCAT Classic Respondents Frequency MNCAT Plus Respondents Frequency Graduate student 50 33% Graduate student 176 37% Undergraduate student 31 20% Undergraduate student 110 23% Library staff 20 13% Faculty 40 8% Faculty 21 14% Staff (non-library) 28 6% Staff (non-library) 10 7% Library staff 21 4% Community member 2 1% Community member 11 2% (Unspecified) 19 12% (Unspecified) 90 19% Total 153 100% Total 476 100% Table 1. Respondents by User Population A comparison of the student survey responses shows that graduate students were overrepresented, while undergraduates were underrepresented, at close to a reverse ratio. Of the total number of graduate and undergraduate students, 62 percent of the respondents were graduate students, even though they accounted for only 32 percent in the larger population. Conversely, undergraduates represented only 38 percent of the student respondents, even though they accounted for 68 percent of the graduate and undergraduate total. Regrettably, the surveys did not include options for identifying oneself as a non-degree-seeking or professional student, so the analysis of students compared with overall population in this section includes only graduate students and undergraduates. Differences were also apparent in the representation of all four categories of students within a particular college unit. At least two college units were underrepresented in the survey responses: RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 25 Carlson School of Management and the College of Continuing Education. One college unit was overrepresented in the survey results; 59 percent of the overall student respondents to the MNCAT Classic survey, and 47 percent of the MNCAT Plus students indicated that they were housed in the College of Liberal Arts (CLA), and yet CLA students only represent 32 percent of the total number of students on campus. Table 2 shows the breakdown of percentages by college or unit and the corresponding breakdown by survey respondent, highlighting where significant discrepancies are evident. Twin Cities Overall Percentage of Students MNCAT Classic Student Survey Respondents +/- MNCAT Plus Student Survey Respondents +/- Carlson School of Management 9% 0% -9% 2% -7% Center for Allied Health 0% 2% +1% 1% 0% Col of Educ/Human Development 10% 9% -1% 14% +3% Col of Food, Agr & Nat Res Sci 5% 4% 0% 7% +2% Coll of Continuing Education 8% 1% -7% 1% -7% College of Biological Sciences 4% 6% +2% 5% 0% College of Design 3% 3% 0% 3% 0% College of Liberal Arts 32% 59% +27% 47% +15% College of Pharmacy 1% 1% 0% 0% -1% College of Veterinary Medicine 1% 1% 0% 1% 0% Graduate School 0% 0% 0% 0% 0% Humphrey Inst of Publ Affairs 1% 1% 0% 1% 0% Institute of Technology (now College of Science & Engineering) 14% 9% -5% 10% -4% Law School 2% 1% -1% 1% 0% Medical School 4% 2% -3% 5% 0% School of Dentistry 1% 1% 0% 0% -1% School of Nursing 1% 0% -1% 0% -1% School of Public Health 2% 1% -1% 3% +1% Table 2. Student Responses by Affiliation INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 26 Faculty and staff together totaled only eighty-nine respondents on the MNCAT Plus survey and fifty-one respondents on the MNCAT Classic survey. In keeping with graduate and undergraduate student trends, the College of Liberal Arts (CLA) was clearly over-represented in terms of faculty responses. The CLA faculty group represents about 17 percent of the faculty at the University of Minnesota. Yet over half the faculty respondents on the MNCAT Plus survey were from CLA; over 80 percent of the MNCAT Classic faculty respondents identified themselves as affiliated with CLA. Faculty groups that were underrepresented include the Medical School and the Institute of Technology. Perceptions of Success A critical area of inquiry for the surveys was user satisfaction and perceptions of success: “Do users perceive their searching activity as successful?” Asked in both surveys, the question’s responses allowed the Primo Management Group to compare respondents’ perceived success between the two interfaces. Results show a marked difference: While 86 percent of the MNCAT Classic respondents reported that they are “usually” or “very often” successful at finding what they are looking for, only 62 percent of the MNCAT Plus respondents reported the same perception of success. Respondents reported very similar rates of success regardless of school, type of affiliation, or student status. Figure 1. Perceptions of Success: MNCAT Plus and MNCAT Classic These results should be interpreted cautiously. Because MNCAT Plus is the libraries’ default catalog interface, MNCAT Classic users are a self-selecting group whose members make a conscious decision to bookmark or click the extra link to use the MNCAT Classic interface. One cannot assume that MNCAT users in general also would have an 86 percent perception of success were they to use MNCAT Classic; familiarity with the tool could play a part in MNCAT Classic users’ success. 14% 24% 44% 18% 4% 11% 32% 54% 0% 10% 20% 30% 40% 50% 60% Rarely Sometimes Usually Very often MNCAT Classic MNCAT Plus RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 27 Another possible factor in the reported difference in user success is the higher proportion of known-item searching—finding a book by title—occurring in MNCAT Classic. A user’s criteria for success differ when searching for a known item versus conducting a general topical search. It is easier for a searcher to determine that they have been successful in a situation where they are looking for a specific item. Some features of MNCAT Classic, such as the start-of-title and other browse indexes, are well suited to known-item searching and had no direct equivalent in MNCAT Plus, which defaults to relevance-ranked results. (Primo version 3 has implemented new features to enhance known-item searching.) Comments received from users suggest that several factors played a role. One MNCAT Classic respondent praised the “precision of the search...not just lots of random hits” and noted that MNCAT Classic supports a “[m]ore focused search since I usually already know the title or author.” In contrast, a MNCAT Plus respondent commented that the next-generation interface was “great for browsing topics when you do not have a specific title in mind.” This comment is consonant with the results from other usability testing done on next-generation catalogs. In "Next Generation Catalogs: What Do They Do and Why Should We Care?", Emanuel describes observed differences between topical and known-item searching: “During the testing, users were generally happy with the results when they searched for a broad term, but they were not happy with results for more specific searches because often they had to further limit to find what they wanted in the first screen of results.”4 A common characteristic of next-generation catalogs is that they return a large result set that can then be limited using facets. Training and experience may also explain some of the differences in success. MNCAT Plus also enables functionality associated with the Functional Requirements for Bibliographic Records (FRBR), which is intended to group items with the same core intellectual content in a way that is more intuitive to searchers. However, this feature is unfamiliar to traditional catalog searchers and requires an extra step to discover very specific known-items in Primo. One MNCAT Plus user expressed dissatisfaction and added, “I'm not sure if it's my lack of training/practice or that the system is not user-friendly.” In focus group analyses conducted in 2008, OCLC found that “when participants conducted general searches on a topic (i.e., searches for unknown items) that they expressed dissatisfaction when items unrelated to what they were looking for were returned in the results list. End users may not understand how to best craft an appropriate search strategy for topic searches.”5 How Often do Users Know the Title of the Item that They are Looking For? Users come to the library with different goals in mind. In “Chang's Browsing,” available in Theories of Information Behavior, Chang identified five general browsing themes,6 adapted to discovery by Carter.7 For the purposes of the survey, the Primo Management Group grouped those themes into two goals: finding an item when the title is known, and finding anything on a given topic. The Primo Management Group had heard concerns from faculty and staff that they have more difficulty finding an item when they know the title when using MNCAT Plus than they did with MNCAT Classic. The group was interested in knowing how often users search for known items. To explore this topic and its impact on perceptions of success, the surveys included two questions on known-item and topical searching. The survey results shown in table 3 indicate that a significantly higher proportion of MNCAT Classic respondents (30 percent plus 43 percent = 73 percent) than MNCAT Plus respondents (24 INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 28 percent plus 29 percent = 53 percent) were “very often” or “usually” searching for known items. It may be that users in search of known items have learned to go to MNCAT Classic rather than MNCAT Plus. Rarely Sometimes Usually Very often Total I already know the title of the item I am looking for MNCAT Classic 7% (11) 19% (29) 30% (46) 43% (66) 152 MNCAT Plus 15% (69) 33% (151) 24% (111) 29% (132) 463 I am looking for any resource relevant to my topic MNCAT Classic 14% (21) 32% (47) 20% (29) 34% (51) 148 MNCAT Plus 14% (62) 29% (133) 29% (133) 28% (127) 455 Table 3. Responses to “I already know the title of the item I am looking for” When the Primo Management Group considered how often researchers in different user roles searched for known items versus anything on a topic, clear patterns emerged as shown in figure 2. In the MNCAT Plus survey, only 34 percent of undergraduate MNCAT Plus searchers “usually” or “very often” search for a particular item, versus 74 percent of faculty. Conversely, 75 percent of undergraduate respondents “usually” or “very often” search for any resource relevant to a topic, versus 37 percent of faculty. Graduate student respondents showed interest in both kinds of use. If successful browsing by topic is best achieved using post-search filtering, it may help to explain differences between undergraduate students and faculty. The analysis of usability testing done on other next generation catalogs described in “Next Generation Catalogs: What Do They Do and Why Should We Care?” states that “users that did not have extensive searching skills were more likely to appreciate the search first, limit later approach, while faculty members were faster to get frustrated with this technique.”8 Results for all MNCAT Classic respondents showed a preference for known item searching, but undergraduate students still indicated that they search more for anything on the topic and less for known items than faculty respondents. No significant differences were identified by discipline. RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 29 Figure 2. Searching for a Known Item vs. Any Relevant Resource Some qualitative comments from survey takers suggest that respondents view the library interface as a place to go to find something already known to exist, e.g., “I never want to search by topic. Library catalogs are for looking up specific items.” However, with respect to discovering resources for a subject in general, both MNCAT Classic and MNCAT Plus respondents showed that they would also like to find items relevant to their topic (figure 2). There was no significant difference between MNCAT Classic and MNCAT Plus respondents on this question; in both environments, only 14 percent of the users said that they would “rarely” be interested in general results relevant to their topic. Perceptions of Success by Specific Characteristics For MNCAT Plus, the majority of respondents “somewhat agree” or “strongly agree” that items available online or in a particular collection are easy to find. One-third of the MNCAT Plus respondents had never tried to find an item in a particular format. Over 40 percent had never tried to find an item with a particular ISBN/ISSN. Interface features may be a factor here: ISBN/ISSN searching is not a choice in the MNCAT Plus drop down menu, so users may not know that they can do such a search. A higher percentage of MNCAT Classic respondents “strongly agree” that it is easy to find items by collection, available online, or in a particular format, than MNCAT Plus respondents. Figure 3 shows results based on particular characteristics. INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 30 Figure 3. Perception of Success by Characteristic Although the surveys were primarily intended to gather reactions from end users, some interesting data emerged about usage by library staff. As demonstrated in figure 4, library staff respondents were much more likely to have performed the specific types of searches listed in this section than users generally, and reported a much higher rate of perceived success with MNCAT Classic. Figure 4. Perception of Success by Characteristic: Library Staff RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 31 Searching by Location: Local Collections and Other Resources In a large research institution with several physical library locations and many distinct collections, users need the ability to quickly narrow a search to a particular collection. But even the largest institution cannot collect everything a researcher might need. The Primo Management Group wondered not only whether users felt successful when they looked for an item in a particular collection but also wanted to explore whether users want to see items not owned by the institution as part of their search results. Finding items among the many library locations was not a problem for either MNCAT Plus or MNCAT Classic respondents: 72 percent either somewhat or strongly agreed that it is easy to find items in a particular collection using MNCAT. Furthermore, survey respondents of both interfaces agreed that they are interested in items no matter where the items are, which underlines the value of a service such as WorldCat; 73 percent of MNCAT Plus respondents and 78 percent of MNCAT Classic respondents expressed a preference for seeing items held by other libraries, knowing they could request items using an interlibrary loan service if necessary. Preferred Search Environments Three of the survey questions asked users about their preferred search environments for different searching needs:  When looking for a particular book  When looking for a particular journal article  When searching without a particular title in mind Each survey presented respondents with a list of choices and space to specify other sources not listed. Respondents were encouraged to mark as many sources as they regularly use. When searching for a specific book, users of the two catalog environments identified a number of other sources. The top five sources in each survey are listed in table 4. When I am looking for a specific book, I usually search (check all that apply): MNCAT Classic Respondents (Frequency) MNCAT Plus Respondents (Frequency) 1. MNCAT Classic (116) 1. MNCAT Plus (217) 2. WorldCat (50) 2. Google (165) 3. Amazon (50) 3. MNCAT Classic (163) 4. Google (49) 4. Amazon (160) 5. Google Books (31) 5. Google Books (108) Table 4. Search Environment for Books INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 32 Qualitative comments indicated that users like being able to connect to Amazon and Google Books in order to look at tables of contents and reviews. They also specifically mentioned Barnes and Noble, as well as other local libraries. These results show that MNCAT Plus respondents were more likely to also use MNCAT Classic than vice-versa. The data do not suggest why this would be the case, but familiarity with the older interface may play a role. MNCAT Classic respondents were more likely than MNCAT Plus users to return to their search environment when searching for a particular book (82 percent versus 53 percent). One MNCAT Plus respondent commented “I didn't know I could still get to MNCAT Classic.” When searching for a specific journal article, users of both systems chose “Other databases (JSTOR, PubMed, etc.)” above all the other choices. Even more respondents would likely have marked this choice if not for confusion over the term “Other databases.” Most of the comments mentioned specific databases, even when the respondent had not selected the “Other databases” choice. One user commented, “Most of these choices would be illogical. You don't list article indexes, that's where I go first.” Table 5 lists the five responses marked most often for each survey. When I am looking for a specific journal article, I usually search (check all that apply): MNCAT Classic Respondents (Frequency) MNCAT Plus Respondents (Frequency) 1. Other databases (JSTOR, PubMed, etc.) (92) 1. Other databases (JSTOR, PubMed, etc.) (232) 2. MNCAT Classic (53) 2. Google Scholar (131) 3. Google Scholar (40) 3. E-Journals List (130) 4. E-Journals List (34) 4. MNCAT Plus (110) 5. Google (29) 5. MNCAT Plus article search (101) Table 5. Search Environment for Articles. Qualitative comments from respondents indicated that interfaces would be more useful if they helped users find online journal articles. This raised some questions with regard to MNCAT Plus, which includes a tab labeled “Articles” for conducting federated article searches. However, MNCAT Plus respondents noted that they used the Plus “Articles” search almost as much as they did MNCAT Plus. Other Plus comments included: I tried to use this for journal articles but it only has some in the database I guess and when I did my search it only found books and no articles. I don't understand it. I tried this new one and it came up with wierd [sic] stuff in terms of articles. My professor said to give up and use the regular indexes because I wasn't getting what I needed to do the paper. It wasted my time. This desire for federated search coupled with the expressions of dissatisfaction with the existing federated search platform is consistent with the mixed opinions expressed in other studies, such as Sam Houston State University’s assessment of use of and satisfaction with the WebFeat RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 33 federated search tool. That study found “[f]ederated search use was highest among lower-level undergraduates, and both use and satisfaction declined as student classification rose.”9 The new search tools that contain preindexed articles, such as Primo Central, Summon, WorldCat Local, and EBSCO Discovery Service, may address the frustrations that more experienced searchers express regarding federated search technology. When researching a topic without a specific title in mind, “Google” and “Other databases” were nearly equal and ranked first for MNCAT Plus respondents, while “Other databases” ranked first for MNCAT Classic respondents. Table 6 lists the five responses marked most option for each survey. When I am researching a topic without a specific title in mind, I usually search (check all that apply): MNCAT Classic Respondents (Frequency) MNCAT Plus Respondents (Frequency) 1. Other databases (JSTOR, PubMed, etc.) (84) 1. Google (197) 2. MNCAT Classic (76) 2. Other databases (JSTOR, PubMed, etc.) (192) 3. Google (63) 3. Google Scholar (155) 4. Google Scholar (47) 4. MNCAT Plus (145) 5. WorldCat (32) 5. MNCAT Classic (101) Table 6. Search Environment for Topics Significant differences based on school affiliation were evident in the area of preferred search environments for topical research. For example, Institute of Technology respondents reported using Google much more often when researching without a specific title in mind than respondents in other areas. Evidence from the health sciences is limited in that only seven percent of respondents in total identified themselves as being from this area. However, these limited results show that health sciences respondents relied more on library databases than on Google. Respondents in the liberal arts relied more on MNCAT, in either version, than did respondents in the other fields. Desired Resource Types One feature of the Primo discovery interface is its ability to aggregate records from more than one source. University Libraries maintains several internal data sources that are not included in the catalog, and the possibility of including some of these in the MNCAT Plus catalog has been considered many times since Primo’s release. The Primo Management Group was interested to hear from users whether they would find three types of internal sources useful: research reports and preprints, online media, and archival finding aids. The group also asked users to mark “Online journal articles” if they would find article results helpful. The question did not specify whether journal articles would appear integrated with other search results in a MNCAT “Books” search or INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 34 in a separate search such as that already provided through a metasearch on the MNCAT Plus Articles tab. The surveys asked users what kinds of resources would make MNCAT more useful. The results for both MNCAT Plus and MNCAT Classic were similar and response counts for both surveys were ordered as shown in table 7. Respondents could mark more than one of the choices. I would find MNCAT more useful if it helped me find: MNCAT Classic Frequency MNCAT Plus Frequency Online journal articles 65 255 U of M research materials (e.g., research reports, preprints) 34 149 Online media (e.g., digital images, streaming audio/visual) 27 134 Archival finding aids 27 90 Table 7. Desired Resource Types The Primo Management Group noted that more MNCAT Plus respondents chose “Online Journal Articles” more frequently than the other categories even though the MNCAT Plus interface includes an “Articles” tab for federated searching. It is unclear whether the respondents were not seeing the “Articles” tab in MNCAT Plus because they would like to see search results integrated, or if they were using the “Articles” tab and were not satisfied with the results. Comments from respondents generally supported the inclusion of a wider range of resources in MNCAT. However, several respondents also expressed concerns about the trade-offs that might be involved in providing wider coverage. One user liked the idea of having the databases “all … in one place,” but added that “it would have to just give you the stuff that you need.” Several users cited the varying quality of the material discovered through library sources. One user supported the inclusion of articles “if it included GOOD articles and not the ones I got.” A MNCAT Classic respondent gave the variable quality of the material he or she had found through a database search as a reason for leaving the coverage of MNCAT as it is: “I use the best sources depending on my needs.” Another MNCAT Classic user expressed doubt that coverage of all disciplines was feasible. In commenting on the content of MNCAT, respondents also mentioned specific types of material that they wanted to see (e.g. archives of various countries), as well as difficulties with particular classes of material (“the confusing world of government documents”). One MNCAT Plus user related his or her interest in public domain items to a specific item of functionality that would enhance their discovery, namely a date sort. In general, the interest in University of Minnesota research material was fairly high. However, faculty members ranked University of Minnesota research materials last in terms of preference: Only twelve faculty respondents chose the option, out of sixty-one total faculty respondents. RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 35 CONCLUSIONS The data from two surveys, conducted concurrently in 2009 on a traditional OPAC (MNCAT Classic) and next-generation catalog (MNCAT Plus), point to differences in the use and perceptions of both systems. There appeared to be fairly strong “brand loyalty” with MNCAT Classic, given that this interface is no longer the default search for the libraries. Surveys for both systems suggest a perception of success that is lower than desirable and that there is room to improve the quality of the discovery experience. It is unclear from the data if the reported perceptions of success were the result of the systems not finding what the user wants, or if the systems did not contain what the user wanted to find. MNCAT Classic respondents were more likely to use WorldCat to find a specific book than MNCAT Plus respondents. MNCAT Plus respondents indicated a use of MNCAT Classic, but not vice versa. Both sets of surveys described use of Amazon and Google for discovery. MNCAT Plus respondents reported lower rates of success at finding known items than MNCAT Classic respondents. MNCAT Classic respondents were far more likely to have a specific title in mind that they wanted to obtain; half of the MNCAT Plus respondents reported having a specific title in mind. The team that examined the survey responses found that the data suggested several key attributes that should be present in the libraries discovery environment. Further discussion of the results and suggested attributes was conducted with library staff members in open sessions. Results also informed local work on improving discovery interfaces. The results suggested:  The environment should support multiple discovery tasks, including known-item searching and topical research.  Support for discovery activity should be provided to all primary constituent groups, noting the significant survey response by graduate student searchers.  Users want to discover materials that are not owned by the libraries, in addition to local holdings.  A discovery environment should make it easy for users to find and access resources in vendor-provided resources, such as JSTOR and PubMed. While the results of the 2009 surveys provided a valuable description of usage, the survey team recognized that methodological choices limit the usefulness in applying results to a larger population. The team also recognized that there were a number of questions yet unanswered. Some of these outstanding questions present opportunities for future research and suggest that a variety of formats might be useful, including surveys, focus groups, and targeted interviews.  To what extent do users expect to find integrated search results among different kinds of content, such as articles, databases, indexes, and even large scale data sets?  What general search strategies do users use to navigate the complex discovery environment that is available to them, and where are the failure points?  How much of the current environment requires training and how much is truly intuitive to users? INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 36  How can the University Libraries identify and serve users who did not complete the surveys?  How useful would users find targeted results based on a particular characteristic such as role, student status, or discipline? Since the surveys were conducted, the University Libraries upgraded to Primo version 3, which included features to address some of the concerns respondents identified in the surveys, such as known-item searching. Primo version 3 allows users to conduct a left-justified title search (“Title begins with…”), as well as sort by fields such as title and author. Once the new version has been in place long enough for users to develop some comfort with the interface, the Primo Management Group intends to resolve methodological issues and repeat its surveys, measuring users’ reactions against the baseline data set in the 2009 surveys. ACKNOWLEDGEMENTS We would like to thank the other members of the Primo Management Group, who helped to design and implement the surveys, as well as analyze and communicate the results: Chew Chiat Naun (chair), Susan Gangl, Connie Hendrick, Lois Hendrickson, Kristen Mastel, R. Arvid Nelsen, and Jeff Peterson. We also want to acknowledge the helpful feedback and guidance of the group’s sponsor, John Butler. REFERENCES 1 Tamar Sadeh, “User Experience in the Library: A Case Study.” New Library World 109, no. 1/2 (2008): 7–24. 2 Cody Hanson et al., Discoverability Phase 1 Final Report (Minneapolis: University of Minnesota, 2009), http://purl.umn.edu/48258/ (accessed Dec. 20, 2010). 3 Jina Choi Wakimoto, “Scope of the Library Catalog in Times of Transition.” Cataloging & Classification Quarterly 47, no. 5 (2009): 409–26. 4 Jenny Emanuel, “Next Generation Catalogs: What Do They Do and Why Should We Care?” Reference & User Services Quarterly 49, no. 2 (Winter, 2009): 117–20. 5 Karen Calhoun, Diane Cellentani, and OCLC, Online Catalogs : What Users and Librarians Want: An OCLC Report (Dublin, Ohio: OCLC, 2009). 6 Shan-ju Chang, “Chang's Browsing,” In Theories of Information Behavior, ed. Karen E. Fisher, Sandra Erdelez and Lynne McKechnie, 69-74 (Medford, N.J.: Information Today, 2005). 7 Judith Carter, “Discovery: What do You Mean by that?” Information Technology & Libraries 28, no. 4 (December 2009): 161–63. 8 Jenny Emanuel, “Next Generation Catalogs: What Do They Do and Why Should We Care?” Reference & User Services Quarterly 49, no. 2 (Winter, 2009): 117–20. 9 Abe Korah and Erin Dorris Cassidy. “Students and Federated Searching: A Survey of Use and Satisfaction,” Reference & User Services Quarterly 49, no. 4 (Summer 2010): 325–32. https://purl.umn.edu/48258 RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 37 APPENDIX A. MNCAT Classic Survey The library catalog is intended to help you find an item when you know its title, as well as suggest items that are relevant to a given topic. We’d like to know how often you use MNCAT Classic for these different purposes. 1. When I visit MNCAT Classic… Very often Usually Sometimes Rarely I already know the title of the item I am looking for     I am looking for any resource relevant to my topic     Many people use tools other than the library catalog to find books, articles, and other resources. For the different situations below, please tell us what other tools you find helpful. 2. When I am looking for a specific book, I usually search (check all that apply):  Amazon  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat  Google Books  MNCAT Plus article search  Google Scholar  Libraries OneSearch Other (please specify) _______________________________________________________ 3. When I am looking for a specific journal article, I usually search (check all that apply):  Amazon  Google Books  MNCAT Plus article search  Citation Linker  Google Scholar  Libraries OneSearch  E-Journals List  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat Other (please specify) ___________________________________________________ INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 38 4. When I am researching a topic without a specific title in mind, I usually search (check all that apply):  Amazon  Google Scholar  Libraries OneSearch  E-Journals List  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat  Google Books  MNCAT Plus article search Other (please specify) ___________________________________________________ Now we’d like to know what you think of MNCAT Classic and what new features (if any) you’d like to see. 5. When I use MNCAT Classic Very often Usually Sometimes Rarely I succeed in finding what I’m looking for     6. It is easy to find the following kinds of items in MNCAT Classic Strongly agree Somewhat agree Somewhat disagree Strongly disagree I haven’t looked for this with MNCAT Classic An item that is available online      An item within a particular collection (e.g., Wilson Library, University Archives, etc.)      An item in a particular physical format (e.g., DVD, map, etc.)      An item with a specific ISBN or ISSN      RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 39 7. I would find MNCAT Classic more useful if it helped me find (check all that apply):  Online journal articles  Online media (e.g., digital images, streaming audio/visual)  Archival finding aids  U of M research material (e.g., research reports, preprints) Other (please specify) ___________________________________________________ 8. The WorldCat catalog allows you to search the contents of many library collections in addition to the University of Minnesota. Which of the following best describes your level of interest in this type of catalog?  Yes, I am interested in what other libraries have regardless of where they are, knowing I could request it through interlibrary loan if I want it  Yes, I am interested, but only if I can get the items from a nearby library  No, I am interested only in what is available at the University of Minnesota Libraries Please share anything you particularly like or dislike about MNCAT Classic. 9. What I like most about MNCAT Classic is: ___________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ 10. What I like least about MNCAT Classic is: ___________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ We want to understand how different groups of people use MNCAT Classic, as well as other tools, for finding information. Please answer the following questions to give us an idea of who you are. 11. How are you affiliated with the University of Minnesota?  Faculty  Graduate student  Undergraduate student  Staff (non-library) INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 40  Library staff  Community member 12. With which University of Minnesota college or school are you most closely affiliated?  Allied Health Programs  Food, Agricultural and Natural Resource Sciences  Pharmacy  Biological Sciences  Law School  Public Affairs  Continuing Education  Liberal Arts  Public Health  Dentistry  Libraries  Technology (engineering, physical sciences & mathematics)  Design  Management  Veterinary Medicine  Education & Human Development  Medical School  None of these  Extension  Nursing 13. We are interested in learning more about how you find the materials you need. If you would be willing to be contacted for further surveys or focus groups, please provide your e-mail address: _______________________________________________ RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 41 APPENDIX B. MNCAT Plus Survey The library catalog is intended to help you find an item when you know its title, as well as suggest items that are relevant to a given topic. We’d like to know how often you use MNCAT Plus for these different purposes. 1. When I visit MNCAT Plus… Very often Usually Sometimes Rarely I already know the title of the item I am looking for     I am looking for any resource relevant to my topic     Many people use tools other than the library catalog to find books, articles, and other resources. For the different situations below, please tell us what other tools you find helpful. 2. When I am looking for a specific book, I usually search (check all that apply):  Amazon  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat  Google Books  MNCAT Plus article search  Google Scholar  Libraries OneSearch Other (please specify) _______________________________________________________ 3. When I am looking for a specific journal article, I usually search (check all that apply):  Amazon  Google Books  MNCAT Plus article search  Citation Linker  Google Scholar  Libraries OneSearch  E-Journals List  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat Other (please specify) ___________________________________________________ INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 42 4. When I am researching a topic without a specific title in mind, I usually search (check all that apply):  Amazon  Google Scholar  Libraries OneSearch  E-Journals List  MNCAT Classic  Other databases (JSTOR, PubMed, etc.)  Google  MNCAT Plus  WorldCat  Google Books  MNCAT Plus article search Other (please specify) ___________________________________________________ Now we’d like to know what you think of MNCAT Plus and what new features (if any) you’d like to see. 5. When I use MNCAT Plus Very often Usually Sometimes Rarely I succeed in finding what I’m looking for     6. It is easy to find the following kinds of items in MNCAT Plus Strongly agree Somewhat agree Somewhat disagree Strongly disagree I haven’t looked for this with MNCAT Plus An item that is available online      An item within a particular collection (e.g., Wilson Library, University Archives, etc.)      An item in a particular physical format (e.g., DVD, map, etc.)      An item with a specific ISBN or ISSN      RESOURCE DISCOVERY: COMPARATIVE SURVEY RESULTS | HESSEL AND FRANSEN 43 7. I would find MNCAT Plus more useful if it helped me find (check all that apply):  Online journal articles  Online media (e.g., digital images, streaming audio/visual)  Archival finding aids  U of M research material (e.g., research reports, preprints) Other (please specify) ___________________________________________________ 8. The WorldCat catalog allows you to search the contents of many library collections in addition to the University of Minnesota. Which of the following best describes your level of interest in this type of catalog?  Yes, I am interested in what other libraries have regardless of where they are, knowing I could request it through interlibrary loan if I want it  Yes, I am interested, but only if I can get the items from a nearby library  No, I am interested only in what is available at the University of Minnesota Libraries Please share anything you particularly like or dislike about MNCAT Plus. 9. What I like most about MNCAT Plus is: ___________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ 10. What I like least about MNCAT Plus is: ___________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ We want to understand how different groups of people use MNCAT Plus, as well as other tools, for finding information. Please answer the following questions to give us an idea of who you are. 11. How are you affiliated with the University of Minnesota?  Faculty  Graduate student  Undergraduate student  Staff (non-library) INFORMATION TECHNOLOGY AND LIBRARIES | JUNE 2012 44  Library staff  Community member 12. With which University of Minnesota college or school are you most closely affiliated?  Allied Health Programs  Food, Agricultural and Natural Resource Sciences  Pharmacy  Biological Sciences  Law School  Public Affairs  Continuing Education  Liberal Arts  Public Health  Dentistry  Libraries  Technology (engineering, physical sciences & mathematics)  Design  Management  Veterinary Medicine  Education & Human Development  Medical School  None of these  Extension  Nursing 13. We are interested in learning more about how you find the materials you need. If you would be willing to be contacted for further surveys or focus groups, please provide your e-mail address: _______________________________________________