Fagan 140 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 Visual search interfaces have been shown by researchers to assist users with information search and retrieval. Recently, several major library vendors have added visual search interfaces or functions to their products. For pub- lic service librarians, perhaps the most critical area of interest is the extent to which visual search interfaces and text-based search interfaces support research. This study presents the results of eight full-scale usability tests of both the EBSCOhost Basic Search and Visual Search in the context of a large liberal arts university. L ike the Web, online library research database inter- faces continue to evolve. Even with the smaller scope of library research databases, users can still suffer from information overload and may have difficulty in processing large results sets. Web search-engine research has shown that the number of searchers viewing only the first results page has increased from 29 percent in 1997 to 73 percent in 2002 for United States-based Web search- engines users.1 Additionally, the mean number of results viewed per query in 2001 was 2.5 documents.2 This may indicate either increasing relevance in search results or an increase in simplistic Web interactions. Visual alternatives to search interfaces attempt to address some of the problems of information retrieval within large document sets. While research and devel- opment of visual search interfaces began well before the advent of the Web, current research into visual Web interfaces has continued to expand.3 Within librarianship, the most visual interface research seems to focus on those that could be applied to large-scale digital library projects.4 Although library products often have more metadata and organizational structure than the Web, search engine-style interfaces adapted for field searching and Boolean opera- tors are still the most frequent approach to information retrieval.5 Yet research has shown that visual interfaces to digital libraries offer great benefit to the user. Zaphiris emphasizes the advantage of shifting the user’s mental load “from slow reading to faster perceptual processes such as visual pattern recognition.”6 According to Borner and Chen, visual interfaces can help users better under- stand search results and the interrelation of documents within the result set, and refine their search.7 In their dis- cussion of the function of “overviews” in visual interfaces, Greene and his colleagues say that overviews can help users make better decisions about potential relevance, and “extract gist more accurately and rapidly than traditional hit lists provided by search engines.”8 Several library database vendors are implement- ing visual interfaces to navigate and display search results. Serials Solutions’ new federated search product, CentralSearch, uses technology from Vivisimo that “orga- nizes search results into titled folders to build a clear, concise picture for its users.”9 Ulrich’s Fiction Connection Web site has used AquaBrowser to help one “discover titles similar to books you already enjoy.”10 The Queens Library has also implemented AquaBrowser to provide a graphical interface to its entire library’s collections.11 XReferPlus maps search results to topics by making visual connections between terms.12 ComAbstracts, from CIOS, uses a similar concept map, although one cannot launch a search directly from the tool. Groxis chose a circular style for its concept-mapping software, Grokker. Partnerships between Groxis and Stanford University began as early as 2004, and Grokker is now being implemented at Stanford University Libraries Academic and Information Resources.13 EBSCO and Groxis announced their partnership in March 2006.14 The EBSCOhost interface now features a Visual Search tab as an option that librarians can choose to leave on (by default) or turn off in EBSCO’s administrator module. Figure 1 shows a screenshot of the Visual Search interface. Within the context of library research databases, visual searching likely provides a needed alternative from tradi- tional, text-based searching. To test this hypothesis, James Madison University Libraries (JMU Libraries) decided to conduct eight usability sessions with EBSCOhost’s new Visual Search, in coordination with EBSCO and Groxis. While this is by no means the first published usability test of vendor interfaces, the literature understandably reveals a far greater number of usability tests on in-house projects such as library Web sites and customized catalog interfaces than on library database interfaces.15 It is hoped that by observing users try both the EBSCO Basic Search and Visual Search, more understanding will be gained about user search behav- ior and the potential benefits of a visual approach. ฀ Method The usability sessions were conducted at JMU, a large liberal arts university whose student population is mostly drawn from Virginia and the northeastern region. Only 10 percent of the students are from minority groups. JMU requires that all freshmen pass the online Information Skills Seeking Test (ISST) before becoming a sophomore, and the Libraries developed a Web tutorial, “Go For the Gold,” to prepare students for the ISST. Therefore, usabil- Jody Condit Fagan Usability Testing of a Large, Multidisciplinary Library Database: Basic Search and Visual Search Jody Condit Fagan (faganjc@jmu.edu) is Digital Services Li- brarian at Carrier Library, James Madison University, Har- risonburg, Virginia. USABILITY TESTING OF A LARGE, MULTIDISCIPLINARY LIBRARY DATABASE | FAGAN 141 ity-test participants were largely white, from the northeast- ern United States, and had exposure to basic information literacy instruction. JMU Libraries’ usability lab is a small conference room with one computer workstation equipped with Morae soft- ware.16 Audio and video recordings of user speech and facial expressions, along with “detailed application and computer system data,” are captured by the software and combined into a searchable recording session for the usability tester to review. A screenshot of the Morae analysis tool is shown in figure 2. The usability test script was developed in collabora- tion with representatives of EBSCO and Groxis. EBSCO provided access to the beta version of Visual Search for the test, and Groxis provided financial incentives for student participants. The test sessions and the results analysis, however, were conducted solely by the researcher and librarian facilitators. The Visual Search development team was provided with the results and video clips after analysis. Usability study participants were recruited by posting an announcement to the JMU students’ Web portal. A $25 gift certificate was offered as an incentive, and more than 140 students submitted a participation interest form. These were sorted by the number of years the student(s) had been at JMU to try to get as many novice users as possible. Because so much of today’s student work is conducted in groups, four groups of two, as well as four individual ses- sions, were scheduled, for a total of twelve students. JMU librarians who had received both human-subjects training and an introduction to facilitation served as facilitators to the usability sessions. Their role was to watch the time and ask open-ended questions to keep the student participants talking about what they were doing. The major research question it was hoped would be answered by the tests was, “To what extent does EBSCO’s basic search interface and visual search interface support student research?” Since the tests could not evaluate the entire research process, it was decided to focus on the development of the research topic. Specifically, the goal was to find out how well each interface supported the intellectual process of the students in coming up with a topic, narrowing their topic, and performing searches on their chosen subtopics. An additional goal was to deter- mine how well users were able to find and use the interface widgets and how satisfied the students felt after using the interfaces. The overall session was structured in this order: a pretest survey about the students’ research experience; a series of four tasks performed with EBSCOhost’s Basic Search; a series of three tasks performed with EBSCOhost’s Visual Search; and a posttest interview. Both Basic and Visual Search interfaces were used with Academic Search Premier. Each of the eight sessions was recorded in entirety by the Morae software, and each recording was viewed in entirety. To try to gain some quantitative data, the researcher measured the time it took to complete each task. However, due to variables such as facilitator involvement and interaction between group members, the numbers did not lend themselves to comparison. Also, it would not have been clear whether greater numbers indicated a positive or negative sign. Taking longer to come up with subtopics, for example, could as easily be a sign of exploration and interested inquiry as it might be of frustration or failure. As such, the data are mostly qualitative in nature. Figure 1. Screenshot of EBSCOHost’s Visual Search Figure 2. Screenshot of Morae Recorder Analysis Tool 142 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 ฀ Results The student participants were generally underclassmen. Two of the students, Group 2, were in their third year at JMU. All others were in their first or second year. While students were drawn from a wide variety of majors, it is regrettable that there was not stronger representation from the humanities. When asked, “What do you normally use to do research?” six students answered an unquali- fied “Google.” Three other students mentioned Internet search engines in their response. Only two students gave the brand or product names of library research databases: one said, “PubMed, WilsonOmniFile, and EBSCO,” while the other, a counseling major, mentioned PsycINFO and CINAHL. When shown a screenshot of Basic Search, half of the students said they had used an EBSCO database before. All of the participants said they had never before used a visual search interface. The full results from the individual pretest interviews are shown in figures 3 and 4. To begin the usability test, the facilita- tor started Internet Explorer and loaded the EBSCOhost Basic Search, which was set to have a single input box. The scripts for each task are listed in figure 5. Note that Task 4 was only featured in the Basic Search portion of the test. For Task 1 on the Basic Search—coming up with a general topic—all of the par- ticipants began by using their own topics rather than choosing from the list of ideas. Also, although they were asked to “spend some time on EBSCO to come up with a possible general topic,” all but Group 6 fulfilled this by simply thinking of a topic (sometimes after some discussion within the groups of two) and typing it in. With the exception of Group 6, the size of the result set did not inspire topic changes. Figure 6 summarizes the students’ searches and relative success on Task 1. In retrospect, the tests might have yielded more straightforward findings if the students had been directed to choose from the provided list of topics, or even to use the same topic. However, part of the intention was to determine whether either interface was helpful in guiding the students’ topic development. It was hoped that by defining the scenario as writing a paper for class, their topic selection would reflect the realities of student research. However, it probably would have been better to have used the same topic for each session. Task 2 asked participants to identify three subtopics, and Task 3 asked them to refine their search to one subtopic and limit it to the past two years. A summary of these tasks appears in figure 7. A surprising finding during Task 2 was that students did go past the first page of results. Four groups went past the first page of results, while two groups did not get enough results for more than one page. The other two groups did not choose to look past the first page of results. This contrasts with Jansen and Spink’s findings, Figure 3. Results from pretest interview, groups 1–4 Figure 4. Results from pretest interview, groups 5–8 USABILITY TESTING OF A LARGE, MULTIDISCIPLINARY LIBRARY DATABASE | FAGAN 143 in which 73 percent of Web searchers only view the first results page.17 Another pleasant surprise was that students spent some time actually reading through results when they were searching for ways to narrow their topic. Five groups scanned through both titles and abstracts, which requires clicking on the article titles to display the citation view. One of these five additionally chose to open full-text articles and look at the references to determine relevance. Two groups scanned through the results pages only, but looked at both article titles and the subjects in the left-hand column. Group 5 seemed to only scan the titles in the results list. This user behavior is also quite different than that found with Web search-engine users. In one recent study by Jansen and Spink, more than 90 percent of the time, search-engine users viewed five or fewer documents per query.18 The five groups that chose to view the citation/abstract view by clicking on the title (Groups 1, 2, 3, 4, and 6) identi- fied subtopics that were significantly more interesting and plausible than the general topic they had come up with. From looking at their results, these groups were clearly identifying their subtopics from reading the abstracts and titles rather than just brainstorming. Although Group 2 had the weakest subtop- ics, going from the world baseball classic to specific players’ relationships to the classic and the home-run derby, they were working with a results set of but eleven items. The three groups that relied on scanning only the results list succeeded to an extent, but as a whole, the new subtopics would be much less satisfying to the scenario’s hypo- thetical professor. After scanning the titles on two pages of results, Group 5 (an individual) ended up brainstorming her subtopics (pre- vention, intervention, and what an eating disorder looks like) based on her knowledge of the topic rather than drawing from the results. Group 7 (a group of two) identified their subtopic (sand dunes) from the left- hand column on the results list. Group 8 (an individual) picked up his subtopics (steroids in sports, President Bush’s stance on steroids, and softball) from reading keywords in the article titles on the first page of results. Since the subjects in the left-hand column were a new addition to Basic Search, the use of this area was also noted. Four groups used the subjects in the left-hand column without prompting. Two groups saw the subjects (i.e., ran the mouse over them) but did not use them. The remaining two groups made no action related to the subjects. A worrisome finding of Tasks 2 and 3 was that most students had trouble with the default search being set to phrase-searching rather than to a Boolean AND. This can easily be seen in looking at the number of results the students came up with when they tried to refine their topics (figure 7). Even though most students had some limiter still in effect (full text, last two years) when they first tried their new refined search, it was the phrase- searching that really hurt them. Luckily, this Figure 6. Task 1, coming up with a general topic using Basic Search Figure 5. Tasks posed for each portion of the usability test. 144 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 is a customizable setting in EBSCO’s administrator module, and it is recommended that libraries enable the “Proximity” expander to be set “On” by default, which will automatically combine search terms with AND. Task 4, finding a “recent article in the Economist about the October earthquake in Kashmir,” was designed to test the usability of the EBSCOhost publication search and limiter. It was listed as optional in case the facilitator was worried that time was an issue. Four of the student groups—1, 2, 5, and 7—were posed the task. Of these four groups, three relied entirely on the publication lim- iter on the Refine Search panel. Group 1 chose to use the Publication search. All four groups quickly and success- fully completed this task. ฀ ฀Additional questions during Basic Search tasks At various points during the three tasks in EBSCO’s Basic Search, the students were asked to limit their results set to only full-text results, to find one peer-reviewed article, and to limit their search to the past two years. Seven out of the eight student groups had no problem finding and using the EBSCOhost “Refine Search” panel, including the full-text check box, date limiter, and peer- reviewed limiter. Group 7 did not find the Refine Search panel or use its limiters until specifically guided by the facilitator near the end. This group had found other ways to apply limits: they used the “Books/Monographs” tab on the results list to limit to full text, and the results-list sorting function to limit to the past two years. After having seen the Refine Search panel, Group 7 did use the “Peer Reviewed” check box to find their peer-reviewed article. Toward the end of the Basic Search portion, students were asked to “save three of their results for later.” Three groups demonstrated full use of the folder. An additional three groups started to use the folder and viewed the folder but did not use print, save, or e-mail. It is unclear whether they knew how to do so and just did not follow through, or whether they thought they had safely stored the items. Two students did not use the folder at all, act- ing individually on items. One group used the “Save” function but did not save each article. ฀ Visual Search Similar to Task 1, when using the Basic Search, students did not discover general topics by using the interface, but simply typed in a topic of interest. Only two groups, 1 and 8, chose to try the same topic again. In the interests of processing time, Visual Search limits the search to the first 250 results retrieved. Since JMU has set the default sort results to display in chronological order, the most recent 250 results were returned during these usability tests. Figure 8 shows the students’ original search terms using Visual Search, the actions they took while looking for subtopics, and the subtopics they identified. Additionally, if the subtopics they identified matched words on the screen, the location of those words is noted. Three of the groups (1, 2, and 5) identified subtopics when looking at the labels on topic and subtopic circles. Group 3 identified subtopics while looking at article titles as well as the subtopic circles. The members of Group 6 identified subtopics while look- ing at the citation view and reading the abstract and full text, as well as rolling over article titles with their mice. It was not entirely clear where the student in Group 4 got his subtopics from. Two of the three subtopics did not seem to Figure 7. Basic Search, Task 2 and 3, coming up with subtopics. USABILITY TESTING OF A LARGE, MULTIDISCIPLINARY LIBRARY DATABASE | FAGAN 145 be represented in the display of the results set. His third subtopic was one of the labels from a subtopic circle. Groups 7 and 8 both struggled with finding their subtopics. Group 7 simply had a narrow topic (“jackalope”), and Group 8 misspelled “steroids” and got few results for that reason. Lacking many clusters, both groups tried typing additional terms into the title keyword box on the Filter panel, resulting in fewer or zero results. For Task 3, students were asked to limit their search to the last two years and to refine their search to a chosen subtopic (figure 9). Particularly because the results set is limited to 250, it would have been better to have separated these two tasks: first to have them limit the content, then perhaps the date of the search. Three groups, all groups of two, used the date limit first (2, 6, and 8). Three groups (1, 3, and 6) narrowed the content of their search by typing a new search or additional keywords into the main search box. Groups 2 and 4 narrowed the content of their search by clicking on the subtopic circles. Note that this does not change the count of the number of results displayed in the filter panel. Groups 5 and 7 tried typing keywords into the title keyword filter panel and also clicking on circles. Both groups fared better with the latter approach. Group 8 typed an additional keyword into the filter panel box to narrow his search. While five of the groups announced the subtopic to which they wanted to narrow their search before beginning to narrow their topic, Groups 2, 7, and 8 began to interact with the interface and experi- ment with subtopics before choosing one. While groups 2 and 8 arrived at a subtopic and identified it, Group 7 tried many experiments, but since their original topic (jackalope) was already narrow, they were not ultimately successful in identifying or searching on a subtopic. As with Basic Search, students were asked to save three articles for later. Five of the groups (2, 4, 5, 6, and 8) used the “Add to folder” function which appears in the citation view on the right-hand side of the screen. Of these, three groups proceeded to “Folder Has Items.” Of these groups, two chose the “Save” function. Two groups used either “Save” or “e-mail” to preserve individual items, rather than using the folder. One group experienced system slowness and was not able to load the full-record view in time to deter- mine whether they would be able to save items for later. A concern that students may not realize is that in folder view or individually, the “Save” button really just formats the records. The user must still use a browser function to save the formatted page. No student performed this function. Figure 8. Visual Search, Task 1 and 2, coming up with a general topic Figure 9. Visual Search, Task 3, searching on subtopic (before date limit, if possible) 146 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 Several students had some trouble with the mechanics of the filter panel, shown in figure 10. Seven of the eight groups found and used the filter panel, originally hidden from view, without assistance. However, some users were not sure how the title keyword box related to the main search box. At least two groups typed the same search string into the title keyword box that they had already entered into the main search box. Also, users were not sure whether they needed to click the Search button after using the date limiter. However, in no case was a student unable to quickly recover from these areas of confusion. ฀ Results of posttest interview At the end of the entire usability session, participants were asked several questions while looking at screenshots of each interface. A full list of posttest interview questions can be found in figure 11. When speaking about the strengths of Basic Search, seven of eight groups talked about the search options, such as field searching and limiters. The individual in Group 1 mentioned “the ability to search in fields, especially for publications and within publications.” One of the students in Group 3 mentioned that “I thought it was easier to specify the search for the full text and the peer reviewed—it had a separate page for that.” The student in Group 4 added, “They give you all the filter options as opposed to the other one.” Five of the eight groups also mentioned familiarity with the type of interface as a strength of Basic Search. Since JMU has only had access to EBSCO databases for less than a year, and half of the students admitted they had not used EBSCO, it seemed their com- ments were with the style of interface more than their experience with the interface. The student in Group 1 commented, “Seems like the standard search engine.” Group 2 noted, “It was organized in a way that we’re used to more,” and Group 3 said, “It’s more traditional so it’s more similar to other programs.” Half of the groups mentioned that Basic Search was clear or organized. Group 6 explained, “It was nice how it was really clearly set out . . . like, everything’s in a line.” Not surprisingly, Visual Search’s strengths surrounded the grouping of subtopics: seven of eight groups made some comment about this. The student in Group 4 said, “It groups the articles for you better. It kinda like gives you the subtop- ics when you get into it and search it and that’s pretty cool.” The student in Group 8 stated, “You can look and see an outline of where you want to go . . . it’s easy to pinpoint it on screen like that’s where I want to go with my research.” Some of the other strengths mentioned about Visual Search were: showing a lot of information on one screen without scrolling (Group 7) and the colorful nature of the interface. A student in Group 2 added, “I like the circles and squares—the symbols register easily.” The only three weaknesses listed for Basic Search in response to the first question were: “not having a spot to put in words NOT to search for” (Group 1); that, like Internet search engines, Basic Search should have “a clip from the article that has the keyword in it, the line before and the line after” (Group 6); and that Basic Search might be too broad, because “unless you narrow it, [you have to] type in keywords to narrow it down yourself” (Group 7). Figure 10. Visual Search Filter Panel Figure 11. Posttest interview questions USABILITY TESTING OF A LARGE, MULTIDISCIPLINARY LIBRARY DATABASE | FAGAN 147 With regard to weaknesses of Visual Search, half of the groups had some confusion about the content, partially due to the limited number of results. A student from Group 7 declared, “It may not have as many results. . . . if you typed in ‘school’ on the other one, it might have . . . 8,000 pages [but] on this you have . . . 50 results.” The student in Group 5 agreed, saying that with Visual Search, “They only show you a certain number of articles.” The student in Group 1 said, “It’s kind of confusing when it breaks it up into the topics for you. It may be helpful for some other people, but for the way my mind works I like just having all my results displayed out like on the regular one.” Half of the groups also made some comment that they were just not used to it. Six of the groups were asked which one they would choose if they had class in one hour. (It is not clear why the facilitator did not ask this question of groups 3 and 8.) Four groups (1, 2, 5, and 7) indicated Basic Search. One student in Group 2 said, “I think it’s easier to use, but I don’t trust it.” The other in Group 2 added, “It’s new and we’re not quite sure because every other search engine is you just type in words and it’s not graphical.” Both stu- dents in Group 7 commented that the familiarity of Basic Search was the reason they would use it for class in one hour. Both Groups 2 and 7 would later say that they liked the Visual Search interface better. Two groups (4 and 6) chose Visual Search for the “class in one hour” scenario. The student in Group 4 commented, “Because it does cool things for you, makes it easier to find. Otherwise you’re going through by title.” Both these groups would later also say that they liked the Visual Search interface better. The students were also asked to describe two scenarios, one in which they would use Basic Search and one in which they would use Visual Search. Four of the groups (1, 3, 5, and 6) said they would use Basic Search when they knew what information they needed. Seven of the eight groups said they would use Visual Search for broad topics. All the students’ responses are given in figure 12. When asked which interface they preferred, the groups split evenly. Comments from the four who preferred Basic Search (1, 3, 5, and 8) centered on the familiarity of the interface. The student in Group 5 added, “The regular one . . . I like to get things done.” All four of these students had said they had used an EBSCO database before. The two students who could list library research databases by name were both in this group. Of the four who preferred Visual Search (2, 4, 6, and 7), three groups had never used EBSCO before, though one of the students in Group 7 thought he’d used it in the library Web tutorial. Group 2 commented, “It seemed like it had a lot more information . . . cool . . . futuristic.” The student in Group 4 said, “It’s kind of like a little game. . . . like you’re trying to find the hidden piece.” Group 7 commented that Visual Search was colorful and intriguing. The students in Group 6 both stated “the visual one” in unison. One stu- dent said that Visual Search was more “[Eye-catching] . . . it keeps you focused at what you are doing, I felt, instead of . . . words . . . you get to look at colors” and added later that it was “fun.” The other students in Group 6 said, “I’m a very visual learner. So to see instead of having to read the categories, and say oh this is what makes sense, I see the circles like ‘abilities test’ or ‘academic achievement’ and I automatically know that’s what it is . . . and I can see how many articles are in it . . . and you click on it and it zooms in and you have all of them there.” The second student went on to add, “I’ve been teaching my mom how to use technology and the visual search would be so much easier for her to get, because its just looks like someone drew it on there like this is a general category and then it breaks it down.” Other suggestions given during the free-comment portion of the survey were to have the filters from Basic Search appear on Visual Search (especially peer-reviewed); curiosity about when Visual Search would become available (at the time it was in beta test); and a suggestion to have general- education writing students write their first paper using Visual Search. Figure 12. Examples of two situations: one in which you would be more likely to use Visual Search, and one in which you would be more likely to use EBSCO 148 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 ฀ Discussion This evaluation is limited both because most students chose different topics for each search interface, and because they only had time to research one topic in each interface. Therefore, there could be an infinite number of scenarios in which they would have performed differently. However, this study does show that, for some students, or for some search topics, Visual Search will help students in a way that Basic Search may not. One hypothesis of this study was that within the con- text of library research databases, visual searching would provide a needed alternative from traditional, text-based searching. The success of the students was observed in three areas: the quality of the subtopics they identified after interacting with their search results; the improvement of the chosen subtopic over their chosen general topic, and the quality of the results they found for their subtopic search. The researcher made a best effort to compare top- ics and results sets and decide which interface helped the student groups to perform better. In addition, qualities that each interface seemed to contribute to the students’ search process were noted (figure 13). These qualities were deter- mined by reviewing the video recordings and examining the ways in which either interface seemed to support the attitudes and behaviors of the students as they conducted their research tasks. When considering all three of these areas, four groups did not, overall, require Visual Search as an alternative to Basic Search (1, 3, 4, and 7). Two of these groups (4 and 7) seemed to benefit from more focus when using the Basic Search interface. Although Visual Search lent them more interaction and exploration (which may be why they said they preferred Visual Search), it seems the focus was more important to their performance. For the other two groups (1 and 3), Basic Search really supported the depth of inquiry and high interest in find- ing results. These two groups confirmed that they preferred Basic Search. For two groups (6 and 8), Visual Search seemed an equally viable alternative to Basic Search. For Group 6, both interfaces seemed to support the group’s desire to explore; they said they preferred Visual Search. For the student in Group 8, Basic Search seemed to orient him to the goal of finding results, while Visual Search sup- ported a more exploratory approach. Since, in his case, this exploratory approach did not turn out well in the area of finding results, it is not surprising that he ended up preferring Basic Search. The remaining two groups (2 and 5) performed bet- ter with Visual Search, upholding the hypothesis that an alternate search is needed. Group 2 seemed bored and uninterested in the search process when using Basic Search even though they chose a topic of personal interest: “world baseball classic.” Visual Search caught their attention and sparked interest in the impersonal topic “global warming.” Group 2 spent more time exploring while using the Visual Search interface, and in the posttest survey admitted that they preferred the Visual Search interface. The student in Group 5 said she preferred Basic Search, and as a self- described PsycINFO user, seemed comfortable with the interface. Yet for this test scenario, Visual Search made her think of new ideas and supported more real exploration during the search process. Within each of the three areas, Basic Search appeared to have the upper hand for both the quality of the subtopics identified by the students, and in the improvement of the chosen subtopics over the general topics. This is at least partially explained by the limitation of Visual Search to the most recent 250 results. That is, as the students explored the Visual Search results, choosing subtopics would not relaunch a search on that subtopic, which would have engendered more and perhaps better subtopics. In the third area, the quality of the results set for the chosen topic, Visual Search seemed to have the upper hand if only because of the phrase-searching limitation present in JMU’s administrative settings for Basic Search. That is, students were often finding few or no results on their chosen subtopics in Basic Search. This study also had findings that seem to transcend Figure 13: Strengths of Basic Search and Visual Search in quality of subtopics, most improved topic, and result sets USABILITY TESTING OF A LARGE, MULTIDISCIPLINARY LIBRARY DATABASE | FAGAN 149 these interfaces and the underlying database. First, librar- ies should strongly consider changing their database default searching from phrase searching to a Boolean AND, if possible. (This is possible in EBSCO using the administrative module.) Second, most students did not have trouble finding or using the interface widgets to perform limiting functions, with the one exception being some confusion about the relationship between the Visual Search filters and main Search box. Unlike some research into Web search behavior, students may well travel beyond the first page of results and view more than just a few docu- ments when determining relevance. Finally, the presence of subject terms in both interfaces proved to be an aid to understanding results sets. This study also pointed out some improvements that could be made to Visual Search. First, it would be great if Visual Search returned more than 250 results in the initial set, or at least provided an overview of the size, type, and extent of objects using available metadata.19 However, even with today’s high-speed connections, result-set size will need to be balanced with performance. Perhaps, as students click on subtopics, the software could rerun the search so that the results set does not stay limited to the original 250. On a minor note, for both Basic and Visual Search, greater care should be taken to make sure users understand how the Save function works and alert users to the need to use the browser function to complete the process. It should be noted that EBSCO has not stopped devel- oping Visual Search, and many of these improvements may well be on their way. EBSCO says it will be adding more support for limiters, display preferences, and contextual text result-list viewing at some point in the future. These feature sets can currently be viewed on Grokker.com. An important area for future research is user behavior in library subscription databases. While these usability tests provide a qualitative evaluation of a specific inter- face, it would be worthwhile to have a more reliable understanding about students’ searching behavior in library databases across similar interfaces. Since public service librarians deal primarily with users who have self-identified as needing help, their experience does not always describe the behavior of all users. Furthermore, studies of Web search behavior may not apply directly to searching in research databases. Specifically, students’ use of subject terms in both interfaces could be explored. Half of the student groups in this study chose to use the Basic Search subject clusters in the left-hand column on the results page, despite the fact that they had never seen them before (this was a beta-test feature). Is this typical? Would this strategy hold up to a variety of research topics? Another interesting question is the use of a single search box versus several search boxes arrayed in rows (to assist in constructing Boolean and field searching). In the EBSCO administrative module, librarians can choose either option. Based on research rather than anecdotal evidence, which is best? Another option is the default sort: historically, at JMU Libraries, this has been a chronological sort. Does this cause problems for relevance-thinking students? Finally, the issue of collaboration in student research using library research databases would be a fascinat- ing topic. Certainly, these usability recordings could be reviewed with a mind to capturing the differences between individuals and groups of two, but there may be better designs for a more focused study of this topic. ฀ Conclusion If you take away one conclusion from this study, let it be this: Do not hesitate to try Visual Search with your users! Information providers must balance investments in cut- ting-edge technology with the demands of their users. Libraries and librarians, of course, are a key user group for information providers. A critical need in librarianship is to become familiar with the newest technology solutions, particularly with regard to searching, in order to provide vendors with informed feedback about which technologies to pursue. By using and teaching new visual search alter- natives, librarians will be poised to influence the further development of alternatives to text-based searching. References and notes 1. Bernard J. Jansen and Amanda Spink, “How Are We Searching the World Wide Web? A Comparison of Nine Search Engine Transaction Logs,” special issue, Information Processing and Management 42, no. 1 (2006): 257. 2. Bernard J. Jansen and Amanda Spink, “An Analysis of Web Documents Retrieved and Viewed,” in Proceedings of the 4th Inter- national Conference on Internet Computing (Las Vegas, 2003), 67. 3. Aravindan Veerasamy and Nicholas J. Belkin, “Evaluation of a Tool for Visualization of Information Retrieval Results,” SIGIR Forum (ACM Special Interest Group on Information Retrieval) (1996): 85–93; Katy Börner and Javed Mostafa, “JoDL Special Issue on Information Visualization Interfaces for Retrieval and Analysis,” International Journal on Digital Libraries 5, no. 1 (2005): 1–2; Ozgur Turetken and Ramesh Sharda, “Clustering-based Visual Interfaces for Presentation of Web Search Results: An Empirical Investigation,” Information Systems Frontiers 7, no. 3 (2005): 273–97. 4. Stephen Greene et al., “Previews and Overviews in Digital Libraries: Designing Surrogates to Support Visual Information Seeking,” Journal of the American Society for Information Science 51, no. 4 (2000): 380–93; Panayiotis Zaphiris et al., “Exploring the Use of Information Visualization for Digital Libraries,” New Review of Information Networking 10, no. 1 (2004): 51–69. 5. Katy Börner and Chaomei Chen eds., Visual Interfaces to Digital Libraries, 1st ed. (Berlin; New York: Springer, 2003), 243. 150 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2006 6. Zaphiris et al., “Exploring the Use of Information Visual- ization for Digital Libraries,” 51–69. 7. Börner and Chen, Visual Interfaces to Digital Libraries, 243. 8. Greene et al., “Previews and Overviews in Digital Librar- ies,” 380–93. 9. “Vivisimo Corporate Profile,” in Vivisimo, http://vivi simo.com/html/about (accessed Apr. 19, 2006). 10. “AquaBrowser Library—Fiction Connection,” www.fic tionconnection.com/ (accessed Apr. 19, 2006). 11. “Queens Library—AquaBrowser Library,” http://aqua .queenslibrary.org/ (accessed Apr. 19, 2006). 12. “xrefer—Research Mapper,” www.xrefer.com/research (accessed Apr. 19, 2006). 13. “Stanford ‘Groks,’” http://speaking.stanford.edu/Back _Issues/ SOC67/library/Stanford_Groks.html (accessed Apr. 19, 2006); “Grokker at Stanford University,” http://library.stan ford.edu/catdb/grokker/ (accessed Apr. 19, 2006). 14. “EBSCO has Partnered with Groxis to Deliver an Inno- vative Visual Search Feature as Part of EBSCO,” www.groxis .com/service/grokker/pr29.html (accessed Apr. 19, 2006). 15. Michael Dolenko, Christopher Smith, and Martha E. Williams, “Putting the User into Usability: Developing Cus- tomer-Driven Interfaces at West Group,” in Proceedings of the National Online Meeting 20 (Medford, N.J.: Learned Information, 1999), 81–90; E. T. Morley, “Usability Testing: The SilverPlatter Experience,” CD-ROM Professional 8, no. 3 (1995); Ron Stew- art, Vivek Narendra, and Axel Schmetzke, “Accessibility and Usability of Online Library Databases,” Library Hi Tech 23, no. 2 (2005): 265–86; Nicholas Tomaiuolo, “Deconstructing Ques- tia: The Usability of a Subscription Digital Library,” Searcher 9, no. 7 (2001): 32–39; B. Hamilton, “Comparison of the Different Electronic Versions of the Encyclopaedia Britannica: A Usability Study,” Electronic Library 21, no. 6 (2003): 547–54; Heather L. Munger, “Testing the Database of International Rehabilitation Research: Using Rehabilitation Researchers to Determine the Usability of a Bibliographic Database,” Journal of the Medical Library Association (JMLA ) 91, no. 4 (2003): 478–83; Frank Cer- vone, “What We’ve Learned From Doing Usability Testing on OpenURL Resolvers and Federated Search Engines,” Computers in Libraries 25, no. 9 (2005): 10–14; Alexei Oulanov and Edmund F. Y. Pajarillo, “Usability Evaluation of the City University of New York CUNY+ Database,” Electronic Library 19, no. 2 (2001): 84–91; Steve Brantley, Annie Armstrong, and Krystal M. Lewis, “Usability Testing of a Customizable Library Web Portal,” College & Research Libraries 67, no. 2 (2006): 146–63; Carole A. George, “Usability Testing and Design of a Library Web Site: An Iterative Approach,” OCLC Systems & Services 21, no. 3 (2005): 167–80; Leanne M. VandeCreek, “Usability Analysis of Northern Illinois University Libraries’ Web Site: A Case Study,” OCLC Sys- tems & Services 21, no. 3 (2005): 181–92; Susan Goodwin, “Using Screen Capture Software for Web-Site Usability and Redesign Buy-In,” Library Hi Tech 23, no. 4 (2005): 610–21; Laura Cobus, Valeda Frances Dent, and Anita Ondrusek, “How Twenty-Eight Users Helped Redesign an Academic Library Web Site,” Refer- ence & User Services Quarterly 44, no. 3 (2005): 232–46. 16. “Morae Usability Testing for Software and Web Sites,” www.techsmith.com/morae.asp (accessed Apr. 19, 2006). 17. Jansen and Spink, “An Analysis of Web Documents Retrieved and Viewed,” 67. 18. Ibid. 19. Greene et al., “Previews and Overviews in Digital Librar- ies,” 381.