University Library Exterior View, 1995. Copyright 2002-2004. The Trustees of Indiana University. IUPUI Image Collection: A Usability Survey Elsa F. Kramer The author, an editor of consumer horticulture books and scholastic materials, received her M.L.S. degree in December 2005 from the Indiana University School of Library and Information Science in Indianapolis. Research paper Purpose To measure functionality, content, and awareness of an online digital image collection by observing participants in a controlled search for a specific image and evaluating their responses to questions with the objective of improving the site. Methodology Participants were recruited from among faculty, staff, and students based on interests stated in online profiles or indicated by type of professional work or academic major. Participants were timed while searching for a target photo and interviewed afterward about their search experience and their use of digital images. Findings Most potential users are not aware of the IUPUI Image Collection and have some difficulty locating it, but when introduced to it find the site attractive and navigable and its content interesting and useful for research, instruction, publication, or class work. Research limitations There were only 5,100 images uploaded to the collection at the time of the study. The participant group was limited to 70. Practical implications User data and suggestions help the Archives staff work with CONTENTdm™ to make improvements to the site’s function and metadata; choose photos to scan for the collection; and develop site-marketing plans. Originality/value This study provides quantitative measurements of user habits and concerns. Observation, sometimes rejected as an obtrusive methodology, is shown here as an important tool to evaluate human-computer interaction for improved understanding of user perspectives on navigability and functionality. Keywords Archival materials -- Digitization Human-computer interaction IUPUI (Campus) Web sites -- Evaluation 3 Introduction Digital image collections accessible via the Internet are still a relatively new library resource, most created no more than 15 years ago and many in just the last few years. Books and journal articles proliferate on technical aspects of the digitization process and image quality assessment. Until recently, however, comparatively little research had been undertaken with respect to the access needs of targeted users, and which factors influenced their use of digital image delivery systems. But as consumer expectation of Web-accessible images has grown exponentially along with technological advances in their delivery, libraries have hurried to take advantage of a new way to increase use of their photographic holdings while simultaneously identifying and preserving them—and perhaps also protecting them from repeated hands-on use. A decade ago librarians might have asked if they should digitize photo collections and why the task might be important. Today librarians are trying to decide which images to digitize first and where to get the money to fund the projects. Just a few years ago photographs might have been scanned only as low-resolution JPEG files in acknowledgement of long download times for patrons using 56K dialup modems. Now, high-resolution TIFF files are increasingly desired and available as more users have high-speed online service and demand top-quality images. The role of catalogers in collection access used to be hidden from public view. Now, in an era when most computer users understand the importance of Google keyword searches, the primary issue facing managers of cutting-edge digital image collections is how to provide bibliographic access of equally high quality that is also portable enough to accommodate the inevitable changes in technology and likely need for future collaborative interoperability. 4 The IUPUI Image Collection provides online access to campus photos of historical interest. Indianapolis Public Library Hospital Service, ca. 1922. Copyright 2002- 2004. The Trustees of Indiana University. The IUPUI Image Collection Indiana University-Purdue University Indianapolis (IUPUI) was created in 1969 as a partnership between Indiana and Purdue universities. IUPUI is a campus of Indiana University that offers degrees in more than 180 programs from both IU and Purdue. The IUPUI University Library's Ruth Lilly Special Collections and Archives include the Manuscript Collections, University Archives, and Rare Books. The University Archives preserves the official records of IUPUI and its various predecessor institutions, including materials related to the history and unique structure of the institution. The Archives includes approximately 250,000 photographic images. 5 Many photos in the IUPUI Image Collection document development of the campus. Two-Story Outhouse on Campus Grounds, 1941. Photo by Howard W. Fieber. Copyright 2002-2004. The Trustees of Indiana University. In an effort to increase use of photographs in the Archives, improve preservation of the materials by reducing the amount of manual contact with them, and reduce the number of staff- assisted photo searches, Archivist Brenda L. Burk created the IUPUI Image Collection, an online resource launched in October 2002 (http://indiamond6.ulib.iupui.edu/IUPUIphotos/). The author observed and timed 70 test subjects while they searched for a target photo, and interviewed them afterward about their search experience and their use of digital images. Based on results from a pilot test of the survey conducted in July 2004, questions were posed that were designed to elicit information needed to improve the site. Baseline Data At the time of the survey there were approximately 5,100 images uploaded to the IUPUI Image Collection. All photos—people, buildings, and events—pertain in some way to the IUPUI campus or the development of the urban area it occupies in downtown Indianapolis. According to statistics kept by the Archives, faculty and staff are the most frequent users of IUPUI’s archived images. Since the launch of the online resource, requests for printed copies of photographs have declined from a high of 232 in 1998-99 to a low of 6 in 2003-04. Requests for printed images are leveling off as users learn to retrieve or request digital images through the Web site. (Source: University Photograph Collection Reprint Statistics, Ruth Lilly Special Collections & Archives, IUPUI University Library, 1997-2004. Brenda L. Burk, archivist.) 6 WebTrends data are available only as an aggregate for the entire period of site operation and not by month. Because the images reside within CONTENTdm™ proprietary software, it is not possible to link directly to specific IUPUI Image Collection photos from a search engine such as Google. Users must first locate the IUPUI Image Collection site in order to perform a specific photo search. Also because of the software, it is not possible to display a feedback button or other survey mechanism when users complete their searches. These limitations prompted the Archives’ request for a usability study with locally collected statistics. Protocol The author conducted interviews with selected users and potential users about their online experience with the IUPUI Image Collection after observing them while they conducted a controlled search. Participants were recruited from among IUPUI faculty, staff, and students based on interests stated in their online profiles, type of professional work, or academic major. The author gathered data in October and November 2004 to evaluate the functionality of, content satisfaction with, and awareness of the online resource through observation interviews of current, past, and potential users from within the IUPUI community. The author provided each participant with written instructions to complete an online search for a designated photo, recorded their search path, and noted the time required to reach the target. After the participants completed the search, simple questions were asked about their experience to obtain descriptive answers and constructive criticism. The author provided the search scenario to each participant on a printed card, stopped participants who had not reached the target photo within 4 minutes, and helped them find the IUPUI Image Collection home page, http://indiamond6.ulib.iupui.edu/IUPUIphotos/, to try the search again. The scenario was: ••• Please search the IUPUI Image Collection to find a 1938 photo of Franklin D. Roosevelt talking with hydrotherapy patients at the Union Building. I will take notes while you search but cannot answer questions until you have finished. ••• The scenario was constructed so that the most important keywords, “IUPUI Image Collection,” were provided first, followed by terms to narrow the search, including two intentionally incorrect terms, “patients” and “Union Building,” to see what participants who tried to narrow their searches too early would do when confronted with no results. Several photographs in the collection met the search criteria; any of them was considered a target. 7 A specific photo search must start from this page. Accepted results for the targeted search for a photo of Franklin D. Roosevelt. 8 The author took step-by-step notes regarding where participants began the search, what search terms they used, what path they took to the site, and how long it took them to find the target photograph. Search engines used were also recorded, if any, and which search option on the site itself was chosen. Demographic data and interview questions Participants were asked their age (noted in spans that correspond with U.S. Census age groups) and university affiliation (undergraduate or graduate student and school; faculty and school; staff and department). After they completed the search, they were also asked if they had ever used or seen the IUPUI Image Collection previously; if so, how they found out about the site and if they had found what they were looking for; and whether they would use this resource in the future. They were then asked to describe changes that could be made to the site that would encourage them to use it again. While discussing possible changes, participants were asked if they found the site visually appealing and easy to navigate, using follow-up queries to elicit details. Their personal and professional uses of digital images were also queried, and the types of images they would like to see included on the site. With the goal of understanding how the site might best be marketed to potential users, participants were asked about their reading habits, with a focus on publications of local interest. RESULTS Total participants: 70 Age Group # of participants % of total 18-24 13 18.6 25-34 14 20.0 35-44 12 17.1 45-54 21 30.0 55-64 10 14.3 Gender # of participants % of total Females 32 45.7 Males 38 54.3 Position # of participants % of total Faculty 20 28.6 Staff 16 22.9 Graduate students 17 24.3 Undergraduate students 17 24.3 Departmental affiliation # of participants Alumni Office 3 Anthropology 4 Business 1 9 Communications 2 Computer Science 3 Education 1 French 1 Geology 11 Herron School of Art 12 History 4 Informatics/New Media 2 Information technology 1 IU Foundation 2 Journalism 2 Law 2 Library & Information Science 10 Marketing admin. 2 Medicine 3 Nursing 2 Psychology 1 Tourism 1 Used the site previously # of participants No 51 Maybe 1 Yes 18 Of those who said they had used the site previously, 10 heard about it from someone in Special Collections; 6 from a student or faculty member; and 2 came across it by chance while searching other IUPUI Web sites. One person had it bookmarked on her computer. Of those who had used the site previously, 9 were faculty, 4 were staff, 2 were graduate students, and 4 were undergrads. Of the 52 who had not or might have used the site prior to the survey, 4 were at least aware of it. Will use in the future # of participants Yes 55 Maybe 12 No, but will recommend to others 2 No 1 Of those who had not used the site prior to this survey, 36 said they will use it in the future, 12 might, 2 will recommend it to students, and 1 will not use it. Ease of navigation # of participants Easy 45 OK 13 Not easy 12 10 Of those who said the site was easy to navigate, 5 commented that it was easy to navigate once it had been located. Those who did not find the site easy to navigate made suggestions for improving the search functions and metadata. Attractiveness of the site # of participants Thought it was OK 35 Liked it 25 Didn’t like it 10 Of those who liked the site design, several specifically approved of its “minimalist” or “uncluttered” layout and the size of the thumbnail images. Of those who did not like the design, several specifically mentioned the gray background as detracting from its appeal. Many of those who said the site design was acceptable also criticized the gray background color and simple design. Images of interest Participants indicated their desire to see the following photographs (in order of frequency mentioned): Anything historical; people; events; aerial views; buildings; landscapes; parks; neighborhoods; building interiors; classrooms; teacher-student interaction; professional-patient interaction; technological innovations; Indianapolis; art; athletics; extracurricular activities; medical- and nursing-related; African Americans and other minorities; close-ups of structures, especially pre-IUPUI; disability access; computer labs; Internet-related; women; anything unusual. Use of digital images # of participants (multiple responses) Personal or professional work 21 Publications 19 Instruction (faculty) 18 Web site 18 Class projects (students) 17 Research 11 Don’t use 11 Quantity and type of downloads # of participants Multiple, high-res 27 Multiple, low-res 5 One at a time, high-res 8 One at a time, low-res 7 Doesn’t download images 23 Publications read (in order of frequency mentioned) The local daily newspaper (especially online) The student newspaper The campus e-newsletter for faculty and staff Indiana Historical Society publications 11 The local alternative weekly newspaper IU and Purdue alumni magazines Departmental or specialty Listservs and newsletters Office of Professional Development e-newsletter Indiana State Library newsletter Indiana Historic Landmarks newsletter Remember seeing a photo from the site used somewhere After the survey, 48 participants did not recall seeing a photo from the site in print or electronic use, 1 might have, and 21 were able to name a specific place they had seen an image from the collection published. Approach to the search # of participants From the IUPUI homepage 44 From the University Library homepage 12 From Google 12 From IUPUI’s Philanthropy Library site 1 From a bookmark for the collection 1 Elapsed time for the target search # of participants Less than 1 minute 27 1 minute 1 1.5 minutes 6 2 minutes 7 2.5 minutes 7 3 minutes 3 3.5 minutes 3 4 minutes 2 5 minutes 8 Of the 19 participants who said they had or maybe had used the site previously, 13 (68.4%) reached the target in 1 minute or less, and all (100%) reached it 3 minutes or less. Of the 51 participants who had not previously used the site, 14 (27.5%) reached it in 1 minute or less, and 32 (62.8%) reached it in 3 minutes or less. Of all 70 participants, 51 (73%) reached the target photo in 3 minutes or less. The two most direct paths to the site are: From Google > search “IUPUI Image Collection” > click on first link From any IUPUI home page > search “IUPUI Image Collection” (or “image collection”) > click on first link. Of the 70 participants, 6 took the most direct path from Google and 16 took the most direct path from IUPUI. They were able to reach the target photograph in less than 1 minute. 12 Others also started at these points but followed paths through additional links (sometimes many) before reaching the target photo. From outside university home pages, a Google search provides the most direct path to the IUPUI Image Collection. The URL is unusual compared to most other IUPUI domain syntax. 13 The shortest path from the University Library’s home page to the IUPUI Image Collection is through the link to the Ruth Lilly Special Collections and Archives near the bottom right. Instead, most participants linked from here to the online catalog. 14 Only a few participants found this prominent link from the Ruth Lilly Special Collections and Archives home page. 15 Many survey participants looked in the Library’s resources listings for the word “image” and did not scroll down to “IUPUI,” where the link is listed. Conclusions Analysis of the responses suggests that most potential users are not aware of the IUPUI Image Collection but when introduced to it find the site attractive and navigable and its content interesting and useful. They appreciate the prominent inclusion of information about image copyrights and how to obtain permissions. Problems identified are more related to site design than content. Users are grateful for the site and enthusiastic about its potential. Use of the site will continue to increase as more photos and features are added and the URL is publicized to potential users. Users will find the site more quickly and navigate it more easily as enhancements are made to the CONTENTdm software. Recommendations When updates to the CONTENTdm software make it possible, online user surveys will be an important additional tool for evaluation of the IUPUI Image Collection. A feedback mechanism given to visitors as they exit the site can take them to such a survey and provide perspectives from a wide variety of users. 16 Including digital image Web site designers in a larger study would invite valuable feedback on functionality and content. Subjects with expertise in search syntax could provide suggestions for improvements in metadata. A more intuitive and easy-to-remember URL consistent in syntax with other IUPUI URLs might help to increase repeat use of the site. Adding the simple search link (or a “search again” link) at the top of each results page will increase navigability. Adding the word “or” between the simple search and browse search boxes will eliminate some confusion. Ease of searching will increase the number of return visitors to the site. While the image collection is logically catalogued using controlled language (Library of Congress subject headings so that records can be exported to OCLC and IUCAT), searches might be made simpler—especially for those with poor computer literacy skills—by extending the natural language search capability used in the descriptive metadata to the subject headings. (The National Library of Medicine’s medical subject headings have been added for some images in order to facilitate searches.) MeSH headings have been added to some photos to facilitate keyword searches. Anatomy Class with Cadaver, n.d. Copyright 2002-2004. The Trustees of Indiana University. Routine publicity of the site and its updates through intra-university and print and electronic communications will increase faculty awareness. Faculty should be encouraged to introduce students to the site at the beginning of each semester and to include mention of it in every class syllabus, if possible. Alumni and other specialized university audience publications also can be targeted for promotion of the site. The student newspaper could collaborate on projects that promote use of the site and student journalists can use historical images from the collection to illustrate feature articles. 17 Local publications that focus on or sometimes feature the history of downtown Indianapolis also should be targeted for promotion of the site and its updates. As more users become aware of the site, an automated mechanism for downloading and purchasing high- resolution photos could further reduce the need for Archives staff assistance. The author believes that the rapid development of digital libraries has dramatically outpaced the average user’s navigation and information literacy skills. Although the site search for this survey was not designed to test individual online research skills, most participants who became lost in the effort appeared to have poor understanding of the basics of search syntax. That lack of understanding, moreover, was not statistically more significant in any one group over another. The ongoing collaboration between academic librarians and others in the university community to promote information literacy and teach mastery of online research skills will undoubtedly go as far as any software update or design change in increasing use of and user satisfaction with digital resources. Recommended Reading The following articles can be helpful in constructing a framework for measuring user evaluations of a digital image collection; identifying delivery, content, quality, and support variables that affect user satisfaction and frequency of use; and setting benchmarks for measuring the success of changes after they are implemented. Rieger and Gay’s 1999 article in RLG DigiNews is especially helpful in understanding how much more meaningful evaluation of an electronic resource can be when investigators use data collection and analysis tools that take into account human-computer interaction. Instead of assessing a technological resource in isolation, which often leads to emphasis on simple measurements alone, a framework for evaluation based on a “social construction of technology” model allows for a richer assessment that combines statistics with expert evaluation of resource design and investigators’ observation of human use of the electronic resource. Aladwani, Adel M., and Prashant C. Palvia. 2002. “Developing and Validating an Instrument for Measuring User-Perceived Web Quality.” Information & Management 39, no. 6, 467–476. Baca, Murtha, ed. 1998, 2000. Introduction to Metadata: Pathways to Digital Information. Los Angeles: Getty Research Institute. Battleson, Brenda, Austin Booth, and Jane Weintrop. 2001. “Usability Testing of an Academic Library Web Site: A Case Study.” The Journal of Academic Librarianship 27, no. 3, 188–198. Besser, Howard. 1999. “Digital Image Distribution: A Study of Costs and Uses.” D-Lib Magazine 5, no. 10. http://www.dlib.org/dlib/october99/10besser.html. Bishop, Ann Peterson. 2002. “Logins and Bailouts: Measuring Access, Use, and Success in Digital Libraries.” The Journal of Electronic Publishing 4, no. 2. http://www.press.umich.edu/jep/04-02/bishop.html. 18 Bishop, Ann Peterson, and Bertram (Chip) Bruce. 2002. “Digital Library Evaluation as Participative Inquiry.” Graduate School of Library and Information Science. University of Illinois. http://www.isrl.uiuc.edu/~chip/pubs/02delos.pdf California Digital Library Evaluation Activity Reports, University of California, 2003. http://www.cdlib.org/inside/assess/evaluation_activities.html Dickstein, Ruth, and Vicki Mills. 2000. “Usability Testing at the University of Arizona Library: How to Let the Users in on the Design.” Information Technology and Libraries 19, no. 3, 144–151. Dillon, Andrew, and Min Song. 1997. “An Empirical Comparison of the Usability for Novice and Expert Searchers of a Textual and Graphic Interface to an Art Resource Database.” Journal of Digital Information 1, no. 1. http://jodi.ecs.soton.ac.uk/Articles/v01/i01/Dillon Ester, Michael. Digital Image Collections: Issues and Practices. Washington, D.C.: Commission on Preservation and Access, December 1996. Farley, Laine. “Digital Images Come of Age.” Syllabus Magazine, May 2004. http://www.syllabus.com/article.asp?id=9363. Garrison, William A. 2001. “Retrieval Issues for the Colorado Digitization Project’s Heritage Database.” D-Lib Magazine7, no. 10. http://www.dlib.org/dlib/october01/garrison/10garrison.html. Gaynor, Edward. “Cataloging Digital Images: Issues.” Proceedings of the Seminar on Cataloging Digital Documents. Washington, D.C.: Library of Congress, October 13, 1994. http://www.loc.gov/catdir/semdigdocs/gaynor.html. Greenstein, Daniel. 2000. “Digital Libraries and Their Challenges.” Library Trends 49, no. 2, 290–303. Hill, Linda L., et al. 1997. “User Evaluation: Summary of the Methodologies and Results for the Alexandria Digital Library, University of California at Santa Barbara.” http://www.asis.org/annual-97/alexia.htm. Jeng, Judy H. 2004. “What Is Usability in the Context of Digital Libraries and How Can It Be Measured?” Information Technology and Libraries 24, no. 2 (June 2005), 3-12. Jones, Michael L. W. et al. 1999. “Project Soup: Comparing Evaluations of Digital Collection Efforts.” D-Lib Magazine 5, no. 11. http://www.dlib.org/dlib/november99/11jones.html. Kenney, Anne R., and Oya Y. Rieger. 2000. Moving Theory Into Practice: Digital Imaging for Libraries and Archives. Mountain View, CA: Research Libraries Group. 19 ———. 2000. “Preserving Digital Assets: Cornell’s Digital Image Collection Project.” First Monday 5, no. 6. http://www.firstmonday.org/issues/issue5_6/kenney/index.html. Kilker, Julian, and Geri Gay. 1998. “The Social Construction of a Digital Library: A Case Study Examining Implications for Evaluation.” Information Technology and Libraries 17, no. 2, 60-69. Long, Holley. 2002. “An Assessment of the Current State of Digital Library Evaluation.” Master’s Paper. University of North Carolina at Chapel Hill. “Subject Access to Images.” MARBI Discussion Paper no. 2005-DP01. Washington, D.C.: Library of Congress, December 10, 2004. http://www.loc.gov/marc/marbi/2005/2005-dp01.html. MIT Libraries Web Site Usability Test, March 1999. http://macfadden.mit.edu:9500/webgroup/usability/results/process.html. Moyo, Lesley M. 2002. “Collections on the Web: Some Access and Navigation Issues.” Library Collections, Acquisitions, & Technical Services 26, 47-59. Ostrow, Stephen. Digitizing Historical Pictorial Collections for the Internet. Washington, D.C.: Commission on Preservation and Access, February 1998. http://www.clir.org/pubs/reports/ostrow/pub71.html. Payette, Sandy D., and Ola Y. Rieger. 1997. “The User’s Perspective.” D-LibMagazine 2, no. 2. http://www.dlib.org/dlib/april97/cornell/04payette.html. Rieger, Robert, and Geri Gay. 1999. “Tools and Techniques in Evaluating Digital Imaging Projects.” RLG DigiNews 3, no. 3. http://www.rlg.org/preserv/diginews/diginews3-3.html. Saracevic, Tefko, and Marija Dalbello. 2001. “Digital Library Evaluation: Toward an Evolution of Concepts.” Library Trends 49, no. 2, 350–369. Stokes, John R. 1999. “Imaging Pictorial Collections at the Library of Congress.” RLG DigiNews 3, no. 2. http://www.rlg.org/legacy/preserv/diginews/diginews3-2.html#feature. Talbot, Dawn, Gerald R. Lowell, and Kerry Martin. 1998. “From the User’s Perspective: The UCSD Libraries User Survey Project.” Journal of Academic Librarianship 24, no. 5, 357–364. Visual Image User Study, Pennsylvania State University, December 2003. http://www.libraries.psu.edu/vius/reports.html. 20 Visual Resources Association. February 2005. Cataloging Cultural Objects: A Guide to Describing Cultural Works and Their Images. Washington, D.C.: Digital Library Federation. http://www.vraweb.org/CCOweb/index.html. Weibel, Stuart and Eric Miller. 1997. “Image Description on the Internet: A Summary of the CNI/OCLC Image Metadata Workshop.” D-Lib Magazine (January 1997). http://www.dlib.org/dlib/january97/oclc/01weibel.html.