Identifying objects by touch: An “expert system” Perception & Psychophysics 1985. 37 (4). 299-302 Identifying objects by touch: An "expert system" ROBERTA L. KLATZKY University of California, Santa Barbara, California and SUSAN J. LEDERMAN and VICTORIA A. METZGER Queen's University, Kingston, Ontario, Canada How good are we at recognizing objects by touch? Intuition may suggest that the haptic system is a poor recognition device, and previous research with nonsense shapes and tangible-graphics displays supports this opinion. We argue that the recognition capabilities of touch are best as- sessed with three-dimensional, familiar objects. The present study provides a baseline measure of recognition under those circumstances, and it indicates that haptic object recognition can be both rapid and accurate. Many of you may remember a children's game in which the goal is to identify, by touch alone, several common objects plucked from a bag or a tray. The joy of playing lies in the discovery that success is possible. For those of us who have played the game and experienced that suc- cess, it may not be surprising that objects can be identi- fied through tactual exploration. However, the present report documents that this is no trivial observation. In a systematic study of haptic object identification, it was found that people are highly accurate and quite fast at iden- tifying a large number (100) of objects. We deem this observation surprising, because it has often been argued, both empirically and theoretically, that the haptic system (i.e., purposive touch, as defined in Gib- son, 1966) is inadequate for object identification, espe- cially when compared with vision. Two lines of research are used to support this claim. One compares haptic and visual identification of raised two-dimensional (and some- times fully three-dimensional) nonsense shapes (e.g., Bryant & Raz, 1975; Cashdan, 1968; Rock & Victor, 1964). The second uses tangible graphics displays-raised line drawings of objects, maps, graphs, etc.-in which raised symbols represent spatial and structural informa- tion (e.g., Lederman & Campbell, 1982; Lederman, Klatzky, & Barber, 1985; Magee & Kennedy, 1980). Such displays 'are intended to be read by hand rather than by eye. Although the research assesses the level of per- formance that can be achieved by touch alone, compari- sons with vision are nevertheless implicit. These studies demonstrate either that touch is ineffective for reading and This work was based on an honor psychology thesis conducted by the third author under the supervision of the second author at Queen's University, 1982. It was supported by Natural Sciences and Engineer- ing Research Council of Canada Grant A9854 to the second author. The first author acknowledges the support of the Center for Advanced Study in the Behavioral Sciences. Reprint requests may be sent to either the first or second author: R. Klatzky, Psychology-UCSB, Santa Barbara, CA 93106; or S. Lederman, Psychology, Queen's University, King- ston, Ontario, Canada K7L 3N6. identification or that touch is so dominated by vision that its contribution to pattern perception is minimal. However, we will suggest reasons why performance with arbitrary, unfamiliar three-dimensional or raised two- dimensional stimuli might underestimate the capacity for haptic object recognition. Further, we offer the present study as an "existence proof' that haptics can be highly effective for object identification. Our argument is simi- lar to one made by Reed, Durlach, and Braida (1982), who have been developing a sensory substitution system for the deaf. They suggest that the success of "Tadoma"-a method of understanding speech in which the "listener's" fingers and thumb(s) are placed on the cheeks, lips, and jaws of the speaker-is proof that the tactual system can process complex spatiotemporal pat- terns at rates close to those of auditory speech percep- tion. Similarly, we argue by demonstration that touch can perform a very common identification task with consider- able competence, despite its poor performance in other tasks (particularly those with less naturalistic stimuli). One reason for caution in generalizing results from studies with artificial objects or raised graphics displays to haptic performance as a whole is that such studies generally require pattern "apprehension" -obtaining in- formation about volumetric, topographical, and other at- tributes of the stimuli-as opposed to categorization. Even when artificial displays depict real objects and a categor- ical response is required, the stimuli generally fail to re- tain many of the properties of the objects themselves, such as temperature, size, or texture. The cues that these dis- plays do provide are usually dictated by the original visual master from which a raised replica was derived. It there- fore becomes necessary to determine the shape of the stimulus, perhaps even to form a visual image, in order to identify it. In contrast, real objects might be recognized on the basis of nonstructural cues. A kitchen sponge, for example, could be identified by its texture, without regard for its shape or size. As mentioned above, vision-touch comparisons have often been interpreted as evidence that haptic performance 299 Copyright 1985 Psychonomic Society, Inc. 300 KLATZKY, LEDERMAN, AND METZGER is poor. Several considerations, however, suggest that these comparisons are inappropriate to assess haptic ob- ject recognition. One concern is the degree of practice, which has been found to improve haptic discrimination performance (Gibson, 1966; Simons & Locher, 1979). A lack of familiarity with haptic identification of artifi- cial displays might be critical to its inferiority relative to vision. Another issue is whether the displays that have been used in previous research adequately allow for fun- damental differences between the visual and haptic sen- sory systems (Berla, 1982; Ikeda & Uchikawa, 1978; Lederman, 1979). For example, the resolving power of the fingertip is much less than that of the eye (Weinstein, 1968), and the tactile system may have inherent difficulty in representing stimulus orientation (Pick, Klein, & Pick, 1966). Both of these factors could influence haptic per- formance profoundly; yet stimulus construction cannot easily compensate for either. Consider that changing the size of a stimulus to accommodate the poor resolution of touch also changes the rate at which it can be explored and, accordingly, changes the temporal-integration and memory demands of the task (Berla, 1982). Similarly, us- ing explicit orientation cues may aid tactual apprehension (Lederman, 1979; Lederman & Campbell, 1982), but may also add irrelevant information that interferes with the per- ception of structural properties of the display. The foregoing suggests that in order to assess haptic object identification, one should avoid artificial objects or two-dimensional displays and, instead, use real objects. The cues that real objects provide are ecologically deter- mined, rather than based on a visual replica. Haptic manipulation of objects is commonplace and therefore familiar. Real objects maintain in full scale the attributes that contribute to haptic identification, and their proper orientation is determined by such intrinsic characteristics as principal axes, flat surfaces, and center of gravity. Thus, objects seem ideally suited to recognition through haptic exploration. Some extant studies suggest, in agreement with this reasoning, that haptic identification of common objects is quite accurate (Bigelow, 1981; Hoop, 1971; Schiff & Dytell, 1972; Simpkins, 1979). However, those investi- gations do not provide a general assessment of haptic iden- tification capabilities, because of various limitations- using very young subjects, a small sample of objects, or a task other than identification per se. The present study directly assessed adults' haptic identification of hand-size common objects that were readily identifiable through vi- sion. Its goal was to provide baseline measures of speed and accuracy. METHOD The subjects were students at Queen's University, 20-23 years of age. Three females and 2 males took part in a visual identifica- tion task, and 10 females and 10 males participated in a haptic iden- tification task. The stimuli were 100 common objects, of a size that could be held within the hands (see Appendix). None made a noise, either functionally or in the course of manual exploration, and none had a clear identifiable odor. Forty-one of the stimuli were selected from a list of objects (Snodgrass & Vanderwart, 1980) as having high (>70%) name agreement when they were presented pictorially; the remainder were selected by the experimenters as being unam- biguously identifiable by name. The objects were roughly classifi- able as personal articles, articles for entertainment, foods, cloth- ing, tools, kitchen supplies, office supplies, and household articles. There were from 8 to 23 items in each class. The visual identification task served as a pretest for verifying the nameability of the stimuli by sight. In addition to the 100 stimuli described above, an additional 10 were included, but were discarded when they were incorrectly named by a subject. A name was con- sidered correct if it was commonly applied to objects of the given type, was not commonly applied to distinctly different objects, and was not the name of a relatively abstract category. (Thus, for ex- ample, "thread" or "spool" would be acceptable for that object, but "dowel" or "sewing implement" would not.) Each object was placed, in turn, on a table before the subject, who was asked to name it. All of the stimuli ultimately used were named correctly by all five subjects. Although reaction times were not measured, they were generally brief, on the order of a second. For the haptic identification task, each subject sat at a table that had been padded with a towel to reduce noise. He or she was blind- folded and wore headphones through which white noise was deli- vered, in order to mask inadvertent noise from the exploration. The subject's chin was placed in a chinrest clamped to the table, and he or she made responses into a microphone that triggered a voice key for purposes of timing responses. The subject was free to use both hands and to pick up the objects while exploring. On each trial, the experimenter set an object on the padded table and turned on a tape recorder. (The order of objects was determined randomly for each subject.) She then tapped the subject's hand, to indicate that the object was available for exploration. When the sub- ject first touched the object, the experimenter pressed a finger switch, starting a timer that terminated with the subject's vocal response. The subject's task was to identify each object as quickly and ac- curately as possible, or, if he or she could not do so, to say, "I don't know." In addition, following vocalization of a name, the subject was asked to describe the properties that had been used to identify the object. RESULTS The principal dependent variables were reaction time (between the first tactile contact with the object and vocali- zation) and errors, either misnaming or omission ("1- don't-know" responses). Misnaming errors were further categorized as superordinate (giving the name of a higher order category, e.g., "vegetable" for a pumpkin), cate- gorically related (e.g., "sock" instead of "sweater"), corrected superordinate or related (following an initial su- perordinate or categorically related name with a correct response, e.g. "clothing ... sweater"), and categorically unrelated (e.g., "rock" for a potato). Of the 2,000 responses, only 83 (4.2 %) were errors. Four errors were omissions. There were 22 superordinate errors, 29 categorically related, 14 corrected superor- dinate or related, and 14 unrelated. Males and females did not differ in their error rates, by t test. Analysis of the reaction times for correct responses in- dicated that the model response latency was 1-2 sec and that 68 % of responses occurred within 3 sec of contact. Only 6% of responses took longer than 5 sec of contact. These data differed little by gender of subject. When mean response latencies were computed for each stimulus, all but two items had mean values of 5 sec or less. The devi- ant items were rice (mean latency = 6.6 sec) and T-shirt (mean latency = 10.6 sec). These items also accounted for relatively high proportions of the errors (6 per object). Another type of data was subjects' phenomenological reports of the object properties that had led to their iden- tifications. These were divided into 16 categories, derived from an initial sampling of the data, which distinguished among reports of a distinctive component and of the size, shape, texture, temperature, and function of the whole object or a component. Two independent raters assigned the reports to the 16 categories, with 85 % agreement. An average of 1.9 comments was made about each item. Global shape (e.g., of a whistle), global texture (e.g., of sandpaper), and presence of a distinct component (e.g., cap on a pen) were predominantly mentioned as the basis for identification (on 46%, 36%, and 35% of trials, respectively), with component texture (16%), global size (15%), and component shape (7%) next most often reported. DISCUSSION The principal finding from this study is that haptic iden- tification of a wide range of objects can be remarkably fast and accurate. Given the present scoring system, 96% of the naming responses were correct. With a more lenient system, allowing superordinate category names and related categorical responses, and permitting false starts, the accuracy rate would be 99 %. Moreover, 94 % of cor- rect (under strict scoring) names were given within a 5-sec interval. This performance is all the more remarkable when it is compared with haptic reading and identifica- tion performance observed in past research. Difficulties with raised drawings are informally confirmed in our laboratories as well; frequently, people are unable to iden- tify even simple outlines after 2-3 min of exploration! The observed levels of haptic identification are also con- siderably better than the level of identification of com- mon odors (Cain, 1979, 1982; Desor & Beauchamp, 1974; Engen & Ross, 1973)-generally estimated at about 40%-50%. An important factor underlying the relatively low level of identification by odor is label availability; thus, performance is subject to considerable improvement with practice (Cain, 1979; Desor & Beauchamp, 1974). One might argue, then, that by equating stimuli for the natural-language frequency of their labels, identification rates for touch and olfaction would be comparable. However, even after accounting for difficulties in associa- tive retrieval of names, a residual error in odor identifi- cation remains, apparently reflecting discrimination failures imposed by the sensory system itself (Cain, 1979). Odor identifications also have considerably longer laten- cies than the responses in the present experiment. In the Desor and Beauchamp study, unpracticed subjects took approximately 10 sec to give correct responses and highly HAPTIC OBJECT IDENTIFICATION 301 practiced subjects still averaged over 2.5 sec to respond. The rapid and accurate responses by unpracticed subjects in the present experiment suggest that the superiority of haptic over olfactory identification cannot be explained solely by superior accessibility to labels. Earlier it was suggested that studies with arbitrary con- figurations or two-dimensional simulations might underes- timate the capacity for haptic object identification, because these types of stimuli deprive the haptic system of some of its most effective cues. The present data provide sup- port for this suggestion. Most bases for identification that are emphasized in the phenomenological reports of these subjects-global shape, texture, and size cues-are not readily available with simulated objects. Raised-line draw- ings might appear to provide shape information, but the usual nature of that information is a projection to the reti- nal plane, not the tangible, grasped shape of haptic ex- ploration. Texture cues in raised drawings are normally minimal and usually arbitrarily assigned to the concepts they are intended to represent. Moreover, often only rela- tive size can be portrayed. Also relevant to this argument is a study (Krantz, 1972) that used factor analysis to assess the basis for haptic ob- ject identification. It identified five factors-amount of exertion needed to explore (related to compliance), rough- ness, size, temperature, and sharpness-none of which is adequately represented by raised-line drawing tech- niques in current use. Although we have argued that the haptic system is well equipped to identify familiar objects, we do not mean to claim that the perception of form through touch is gener- ally accurate and efficient. It is one thing to assign an ob- ject to a known category and quite another to apprehend its structural characteristics. Studies using artificial ob- jects and graphics displays appear well suited to assess haptic perception in this latter sense-and they assess it . as poor. In contrast, past experience with objects-visual as well as tactual-might enable categorization on the basis of minimal cues, without full apprehension of surface and shape. The present study indicates that, in this latter task, the haptic system can be very competent indeed, more competent than might be expected from intuition or ex- trapolation from experiments on haptic perception of form. We see the results of the current study as a neces- sary first (phenomenological) step towards developing a model of haptic object identification, one component of which may be a representation that is readily accessed by both haptics and vision (as suggested by Garbin & Bern- stein, 1984; Gibson, 1966). The present work also has implications for applied con- cerns. One application that motivated this research was the design of effective tangible graphics for the visually handicapped. Our study suggests that effective graphic aids should eschew simple mimicry of two-dimensional visual displays. Instead, they might incorporate the three- dimensionality of real objects and retain critical proper- ties such as texture. A rather different area in which research on haptic object identification might be applied 302 KLATZKY, LEDERMAN, AND METZGER is the development of intelligent robotic systems that use tactile sensing (Harmon, 1982). By understanding the mediators between haptic exploration and identification, we may discern levels of information representation that are appropriate for robots to simulate human behavior, thus rendering them capable of highly accurate identifi- cation performance. REFERENCES BERLA, E. P. (1982). Haptic perception of tangible graphic displays. In W. Schiff & E. Foulke (Eds.), Tactual perceptions: A sourcebook (pp. 364-386). Cambridge: Cambridge University Press. BIGELOW, A. E. (1981). Children's tactile identification of miniaturized common objects. Developmental Psychology, 17,111-114. BRYANT, P., & RAZ, I. (1975). Visual and tactual perception of shape by young children. Developmental Psychology, 11, 525-526. CAIN, W. S. (1979). To know with the nose: Keys to odor identifica- tion. Science, 203, 467-469. CAIN, W. S. (1982). Odor identification by males and females: Predic- tions vs performance. Chemical Senses, 7, 129-142. CASHDAN, S. (1968). Visual and haptic form discrimination under con- ditions of successive stimulation. Journal of Experimental Psychol- ogy Monograph, 76(2, Pt. I). DESOR, J. A., & BEAUCHAMP, G. K. (1974). The human capacity to transmit olfactory information. Perception & Psychophysics, 16, 551-556. ENGEN, L. T., & Ross, B. (1973). Long-term memory of odors with and without verbal descriptions. Journal ofExperimental Psychology, 100, 221-227. GARBIN, C. P., & BERSTEIN, I. H. (1984). Visual and haptic percep- tion of three-dimensional solid forms. Perception & Psychophysics, 36, 104-110. GIBSON, J. J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin. HARMON, L. (1982). Automated tactile sensing. Robotics Research, 1, 3-32. Hoop, N. H. (1971). Haptic perception in preschool children. Ameri- can Journal of Occupational Therapy, 25, 340-344. IKEDA, M., & UCHIKAWA, K. (1978). Integrating time for visual pat- tern perception and a comparison with the tactile model. Vision Research, 18, 1565-1571. KRANTZ, M. (1972). Haptic recognition of objects in children. Journal of Genetic Psychology, 120,121-133. LEDERMAN, S. J. (1979). Tactual mapping from a psychologist's per- spective. Bulletin ofthe Association of Canadian Map Libraries, 32, 21-25. LEDERMAN, S., & CAMPBELL, J. (1982). Tangible graphs for the blind. Human Factors, 24, 85-100. LEDERMAN, S., KLATZKY, R. L., & BARBER, P. (1985). Spatial and movement-based heuristics for encoding pattern information through touch. Journal of Experimental Psychology: General, 114, 33-49. MAGEE, L. E., & KENNEDY, J. M. (1980). Exploring pictures tactu- ally. Nature, 283, 287-288. PICK, H. L., KLEIN, R. E., & PICK, A. D. (1966). Visual and tactual identification of form orientation. Journal ofExperimental Child Psy- chology, 4, 391-397. REED, C., DURLACH, N., & BRAIDA, L. (1982). Research on tactile com- munication of speech: A review. American Speech-Language-Hearing Association Monographs, No. 20, 1-23. ROCK, I., & VICTOR, J. (1964). Vision and touch: An experimentally created conflict between the two senses. Science, 143, 594-596. SCHIFF, W., & DYTELL, R. S. (1972). Deafand hearing children's per- formance on a tactual perception battery. Perceptual and Motor Skills, 35, 683-706. SIMONS, R. W., & LOCHER, P. J. (1979). Role of extended perceptual experience upon haptic perception of nonrepresentational shapes. Per- ceptual and Motor Skills, 48, 987-991. SIMPKINS, K. E. (1979). Tactual discrimination of household objects. Journal of Visual Impairment and Blindness, 73, 86-92. SNODGRASS, J. G., & VANDERWART, M. (1980). A standardized set of 260pictures: Norms for name agreement, image agreement. familiar- ity. and visual complexity. Journal ofExperimental Psychology: Hu- man Learning and Memory, 6, 174-215. WEINSTEIN, S. (1968). Intensive and extensive aspects of tactile sensi- tivity as a function of body pan, sex, and laterality. In D. R. Ken- shalo (Ed.), The skin senses (pp. 195-218). Springfield, II.: Thomas. APPENDIX Stimulus Objects Category Articles Personal comb, emery board, glasses, hair dryer, ring, swab, toothbrush, wallet, watch Entertainment balloon, baseball bat, baseball glove, birdie, crayons, golf ball, playing cards, record, tennis racket, whistle Clothing belt, boot, mitten, scarf, shoelace, sock, sweater, T-shirt, tie Foods carrot, cracker, egg, onion, potato, pump- kin, rice, tea bag Office Supplies binder, book, bow, calculator, clipboard, envelope, eraser, notebook, paper clip, paper pad, pen, pencil, pencil sharpener, ruler, sta- pler, tape Tools clamp, hammer, paintbrush, sandpaper, scis- sors, screw, screwdriver, twine, wrench Household ash tray, bandage, button, candle, clothespin. dust pan, electric cord, flashlight, flower pot, hook, key, light bulb, match book, padlock, rubber band, safety pin, scrub brush, sponge, thread, toilet paper, toothpicks, umbrella, watering can Kitchen Supplies baby bottle, bottle opener, bowl, fork, fun- nel, glass, kettle, knife, ladle, muffin pan, mug, plate, pot, spatula, strainer, wooden spoon (Manuscript received November 21. 1984: revision accepted for publication January 20, 1985.)