Evidence Summary

The Form of Search Tool Chosen by Undergraduate Students Influences Research Practices and the Type and Quality of Information Selected

A Review of:

Asher, A. D., Duke, L. M., & Wilson, S. (2012). Paths of discovery: Comparing the search effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and conventional library resources. College & Research Libraries, 74(5), p. 464-488. Retrieved from http://crl.acrl.org/content/early/2012/05/07/crl-374.full.pdf

Reviewed by:

Michelle Dalton

Liaison Librarian

University College Dublin

Dublin, Ireland

Email: michelle.dalton@ucd.ie

Received: 17 Jan. 2014 Accepted: 22 May 2014

2014 Dalton. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

Abstract

Objectives – To explore the effectiveness of different search tools (EBSCO Discovery Service (EDS), Summon, Google Scholar and traditional library resources) in supporting the typical research queries faced by undergraduate students and gain an understanding of student research practices.

Design – Mixed methods approach using quantitative data collected from grading of students’ selected resources combined with qualitative data from a search process interview with students.

Setting – Two university libraries in the United States of America (Bucknell University (BU) and Illinois Wesleyan University (IWU)).

Subjects – Eighty-seven undergraduate students across a range of disciplines.

Methods – Participants were assigned to one of five test groups and required to find two resources for each of four standardised research queries using a specified tool: EDS; Summon; Google Scholar; Library catalogue/databases; or “no tool” where no specific tool was specified and participants were free to choose. The resources submitted by students for each of the four queries were rated on a scale of 0-3 by four librarians using a rubric, to produce average ratings for each tool. The interview comprised two parts: the search task, followed by a reflective interview based on open-ended questions relating to search practices and habits. The search process interview was recorded using Camtasia screen capture and audio software, and the URLs used by participants were also recorded.

Main Results – Quantitative results indicated that students who used EDS selected slightly higher quality sources on average (scoring 2.54 out of 3), compared to all other groups. Those who used EDS also completed the queries in less time (747 seconds) than those using Summon (1,209 seconds), Google Scholar (968 seconds), library databases (963 seconds) or where no tool was specified (1,081 seconds). Academic journal articles also represented the relatively highest proportion of resources for this group (73.8% of resources chosen), whilst newspaper articles were chosen most frequently by those using Summon (20.6% of resources chosen). The qualitative findings suggest that students may over-rely on the top results provided by search systems, rather than using critical analysis and evaluation.

Conclusion – Although EDS performed slightly better overall, in some cases the tools produced relatively similar results, and none of the tools performed particularly poorly. Indeed the reasonably strong performance of both Google Scholar and traditional library tools/databases in some aspects (such as the relative proportion of books and journal articles chosen by students), may raise questions regarding the potential benefit of acquiring a new discovery product, given the possibly significant costs involved. As the study finds that most students do not go beyond simple searches and the first page of results, regardless of the tool they are using, this suggests that discovery services do not substantially lessen the need for information literacy instruction, although it may provide some opportunity to redirect teaching time away from retrieval and towards higher-order skills such as evaluating information and critical thinking.

Commentary

Discovery often “tops the charts as the foremost issue” in library systems and automation (Breeding, 2010, p. 31). Faced with increasing competition from web-scale search tools such as Google, many academic libraries have looked towards discovery services as a solution that can match their users’ needs and preferences. However, no clear consensus has yet emerged regarding the best product available in this space, or indeed whether these tools are ultimately more efficient than using traditional library tools such as individual databases. In this context, the study addresses an important and emerging question, by comparing the efficacy of a suite of tools in dealing with undergraduate information queries.

Trying to make direct comparisons between search platforms that are used across two different institutions is both complex and potentially problematic due to the likely variation in holdings and resources. However, the authors clearly acknowledge this weakness and openly refer to aspects which may undermine any inferences. For instance, the EDS product in IWU did not automatically index LexisNexis whilst Summon in BU did. As elements and defaults can usually be customised to reflect institutional preferences, the results may have been more meaningful if both tools were configured in a similar way for the purposes of the study. Other possible differences, including variation in the information skills levels of students between institutions, are also flagged.

However, the level of detail in the study is exceptional, offering break downs by page views, number of searches, time taken and resource type. This kind of information provides granular and detailed data which can inform usability analysis and information literacy instruction. That EDS outperformed Google Scholar in terms of length of time taken (as well as the quality of material selected), may indicate that discovery platforms can potentially compete with the single search box experience of Google when it comes to ease and speed of use, and indeed user experience more generally.

In those instances where significant differences were found between products (such as the typical proportion of resources selected which are scholarly articles, books or websites), it highlights the influence that the tools that libraries provide, promote and recommend may have on the information and content ultimately selected and used by students. Websites were selected much more frequently by those given no explicit directions regarding which resource to use and this suggests that undergraduate students still require significant guidance on where to start searching.

The qualitative data in particular provides rich insight into students’ thought processes and how they select and evaluate sources – an aspect that is often overlooked in favour of retrieval. In this context, the study offers a valuable perspective that extends beyond many of the existing studies surrounding discovery which are purely quantitative (Lown et al., 2013; Chapman et al., 2013). From the comments included, in many cases there is a generally good awareness of what constitutes a quality source of information. Notwithstanding this, the depth of knowledge in this respect appears limited, indicative that true understanding and deeper critical evaluation skills may be an opportunity for further development through library support. The intuitive nature and interface of many discovery products, means that time previously devoted to database instruction could be redirected towards other areas. Indeed, this is perhaps one of the biggest potential benefits that discovery may open up for libraries: by simplifying our retrieval systems for users, it provides them with more time to explore the full range of our resources and services, as well as the information that they find.

References

Breeding, M. (2010). The state of the art in library discovery 2010. Computers In Libraries, 30(1), 31-34. Retrieved from http://www.librarytechnology.org/ltg-displaytext.pl?RC=14574

Chapman, S., Desai, S., Hagedorn, K., Varnum, K., Mishra, S., & Piacentine, J. (2013). Manually classifying user search queries on an academic library web site. Journal of Web Librarianship, 7(4), 401-421. http://dx.doi.org/10.1080/19322909.2013.842096

Lown, C., Sierra, T., & Boyer, J. (2013). How users search the library from a single search box. College & Research Libraries, 74(3), 227-241. Retrieved from http://crl.acrl.org/content/74/3/227.full.pdf