Find that photo!: interface strategies to annotate, browse, and share COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 69 FIND THAT PHOTO! Interface Strategies to Annotate, Browse, and Share As digital photos become the standard media for personal photo taking, supporting users to explore those photos becomes a vital goal. Dominant strategies that have emerged involve innovative user interfaces that support annotation, browsing, and sharing that add up to rich support for exploratory search. Successful retrieval is based largely on attaching appropriate anno- tations to each image and collection since automated image content analysis is still lim- ited. Therefore, innovative techniques, novel hardware, and social strategies have been pro- posed. Interactive visualization to select and view dozens or hundreds of photos extracted from tens of thousands has become a popular strategy. And since the goal of photo search is to support sharing, storytelling, and remi- niscing, experiments with new collaborative strategies are being examined. While digital photographic databases and retrieval systems have been in use for many years, these systems were typically designed for professionals in museums, libraries, advertis- ing, and journalism, to name a few specíálitíes. Such systems employed a cadre of financially motivated individuals to hand-annotate the pictures with metadata such as keywords, dates, and locations, often using fixed vocabu- laries, to support traditional search techniques. By contrast, consumers typically put little effort into photo annotation; they are more focused on exploratory search and serendipi- tous discovery of photos with a stronger emphasis on entertainment. This leads to a very different set of requirements for personal photo use where ease of annotation, support for exploratory browsing, and convenient sharing is crucial. Annotate. In textual exploratory search, users can enter key phrases from a docu- ment to retrieve similar content. But for images, retrieval based on content through automated analysis is often limited to some forms of shape analysis (such as finding the presence of faces in an image) and color matching to find sunrises or determine whether an image was taken inside or outside. To support effective exploratory search on photos, appropriate annotations must be asso- B y Ben Shneiderman, Benjamin B. Bederson, a n d Steven M. Drucker 70 April 2006/Vol. 49, No. 4 COMMUNICATIONS OF THE ACM ciated with the images either by the camera or by users of the images, such as the photographer or potentially a larger community of users. Cameras are increasingly recording information about the photo- graph including time and date stamps, tilt sensors for orientation, light levels, focal distances, and even global position. Barcodes, RFID tags, or other label- ing methods could enable a higher percentage of pho- tos to be annotated automatically. Many interfaces enable manual annotation of photographs by “painting” keywords [3] or dragging and dropping names onto images. Commercial tools such as Adobe PhotoShop Album make tags drag-able onto photo borders. Other tools perform temporal clustering to create a more man- ageable set of photo groups [1]. As with many tasks, manual annotation can be improved by designing interfaces that support faster and easier annotation as well as making the future benefits more apparent. Automatic and manual annotations are valuable in supporting both searching and browsing. Browse. Users browse for fun and to find a spe- cific photograph. They may be looking for photos of their grandfather, their hike down the Grand Canyon, or a friend’s wed- ding. They also may be looking for a great photo to accompany a story of a sun- rise hike or memorable baseball game. Clearly, if the photo col- lection has been extensively annotated, techniques such as faceted search (see Hearst’s article in this sec- tion) can help users filter down a collection and show potential targets for brows- ing. User-controlled visualization of photos grouped by date, location, or annotation can greatly facilitate browsing and increase enjoyment [4]. Different lay- outs of photos can exploit this metadata to help peo- ple find desired photos and discover new ones. In particular, geo-tagging of photos and interfaces, like WWMX, allows people to find all those photographs associated with a particular area (see Figure 1). Chronological displays work well for dates as well, but large numbers of photos can be overwhelming, so groups of photos can be clustered by date and represen- tative photos can be manually or automatically chosen for each cluster [1, 2]. These representative photos again help to provide landmarks in order for users to locate photos from particular events. Interfaces such as PhotoMesa use powerful filtering tools, plus flexible grouping and rapid zooming, to enable users to explore thousands of photos fluidly (see Figure 2). Figure 1. The WorldWide Media Exchange (WWMX) interface showing map and calendar views along with images as published in ACM Multimedia 2003; wwmx.org. Consumers typically put little effort into photo annotation; they are more focused on exploratory search and serendipitous discovery of photos with a stronger emphasis on entertainment. Share. Sharing photos by email, instant messag- ing, Web sites, and cell phones is a growing suc- cess story. When users select photos and make them available to others, they seem to be willing to invest more effort in annota- tion. Also by making them public, they invite others to comment and add annotations. More elaborate story-generating tools invite users to provide slideshow sequences with text captions and audio narration. Recent innovations in social experiences on the Web have sought to encourage annotation by increas- ing satisfaction and making the benefits immediately apparent. A game-like approach to image annotation gets players to cooperate with anonymous, remotely located partners in assigning keywords for pho- tographs [5]. This surprisingly addictive game has succeeded in labeling over 10 million images as of August 2005 (since its introduction in 2003). Other communities, such as Flickr, allow users to share and annotate images on a Web site using tags. These “folksonomies” have now gone past photos to Web pages and blogs as well (such as technorati and deli.cio.us). The trend toward annotating, browsing, and sharing your photos via Web sites such as Flickr, Ofoto, and Shutterfly is perhaps one of the biggest changes enabled by the transformation from analog to digital photogra- phy. Photos no longer sit unattended in shoeboxes stored in attics, but are available for ready viewing by friends and family distributed around the world. SUMMARY A combination of annota- tion, browsing, and sharing of photos can support the special exploratory search needs of personal digital photo users by getting around the fact that direct search of image content con- tinues to be beyond the capa- bilities of current systems. The special needs of amateur digital photogra- phers are pushing the photo industry to support users with their desired activities. Social networking, in com- bination with innovative user interfaces and visualiza- tion, is just beginning to support everyday photogra- phers. However, we see significant work remaining, especially in metadata standardization to help users cope with their rapidly growing and increasingly val- ued collections. References 1. Graham, A., Garcia-Molina, H., Paepcke, A., and Winograd, T. Time as essence for photo browsing through personal digital libraries. In Pro- ceedings of the 2nd ACM/IEEE-CS Joint Conference on Digital Libraries (2002). ACM Press, NY, 326–335. 2. Huynh, D., Drucker, S., Baudisch, P., and Wong, C. Time Quilt: Scal- ing up zoomable photo browsers for large, unstructured photo collec- tions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2005). ACM Press, NY, 1937–1940. 3. Kuchinsky, A., Pering, C., Creech, M., Freeze, D., Serra, B., and Gwiz- dka, J. FotoFile: A consumer multimedia organization and retrieval sys- tem. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1999). ACM Press, NY, 496–503. 4. Kustanowitz, J. and Shneiderman, B. Meaningful presentations of photo libraries: Rationale and applications of bi-level radial quantum layouts. In Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (2005). ACM Press, NY, 188–196. 5. van Ahn, L. and Dabbish, L. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2004). ACM Press, NY, 319–326. Ben Shneiderman (ben@cs.umd.edu) is a professor and the founding director of the Human-Computer Interaction Lab, Computer Science Department, at the University of Maryland, College Park, MD. Benjamin B. Bederson (bederson@cs.umd.edu) is an associate professor and director of the Human-Computer Interaction Lab, Com- puter Science Department, at the University of Maryland, College Park, MD. Steven M. Drucker (sdrucker@microsoft.com) is lead researcher of the Next Media Research Group, Microsoft Research, Redmond, WA. © 2006 ACM 0001-0782/06/0400 $5.00 c COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 71 Figure 2. PhotoMesa showing 114 photos in six groups in a single view with integrated annotation and search tools as published in ACM UIST 2001; (courtesy of www.photomesa.com).