48 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 200748 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 2008 Touchable Online Braille Generator Wooseob Jeong A prototype of a touchable online Braille generator has been developed for the visu- ally impaired or blind using force feedback technology, which has been used in video games for years. Without expensive devices, this prototype allows blind people to access information on the Web by touching out- put Braille displays with a force feedback mouse. The data collected from user studies conducted with blind participants has pro- vided valuable information about the opti- mal conditions for the use of the prototype. The end product of this research will enable visually impaired people to enjoy informa- tion on the Web more freely. The United States has made some attempts to nationally address infor- mation access for those with disabili- ties. Section 508 of the Rehabilitation Act (www.section508.gov) requires federal agencies to make their elec- tronic information accessible to peo- ple with disabilities, mainly those who are visually impaired. The Library of Congress launched a Web- Braille service (www.loc.gov/nls/) for the blind in 1998, which continues today. With the upsurge in infor- mation stored on the Internet, the importance of these issues cannot be overemphasized. Many products have been devel- oped to help the visually impaired use technology. Several Braille out- put and input devices are available, such as the Braille Notetaker (www. artictech.com) and voice synthesizers for screen readers like JAWS (www. freedomscientific.com/fs_products/ software_jaws.asp). While these products are mainly for textual information, recent devel- opments put more focus on graphi- cal displays. The American National Institute of Standards and Technology proposed a “Pins” Down Imaging System for the Blind (www.nist.gov/ public_affairs/factsheet/visualdis- play.htm). Uniplan in Japan and KSG America (www.kgs-america.com/ dvs.htm) have produced other prod- ucts based on similar ideas. Software like the Duxbury Braille Translator (www.duxburysystems.com) can translate plain text into Braille out- put, which can then be used for embossed printing. However, such products are fairly expensive, rang- ing from hundreds to several thou- sands of dollars in addition to the cost of computers. Fortunately, there is a potentially promising solution. Based on the technology used in prior research, it is possible to develop an online Braille generator.1 The Braille could then be read either by touching the screen with a fingertip sensor or through the use of a force feedback mouse similar to the type used in some video games.2 This application has several advantages over existing devices. First, it does not require expensive special devices—only a $20 mouse, which is readily available. Also, the technology is available as long as there is access to the Internet. Another advantage is that this technology utilizes the existing Braille skills of visually impaired people. The same technology can be used for produc- ing image displays as well, allowing for the creation of a virtual museum for the blind where they can touch objects that are displayed alongside their Braille descriptions. Literature review Force feedback has been studied under the name of haptic perception. Haptic perception involves sensing the movement and position of joints, limbs, and fingers through kinesthe- sia and proprioception, and sensing information through the skin’s tactil- ity.3 Haptic output can be achieved through several techniques, including pneumatic, vibrotactile, electrotactile, and electromechanical stimulation.4 This study examines only vibrotac- tile haptic output methods because vibrotactile stimulation is easily cre- ated, manipulated, and delivered. It is also easily perceived by users through the use of commonly avail- able software and devices. Researchers have begun to develop various haptic input/output devices and software, such as Massachusetts Institute of Technology’s (MIT) fre- quently used Phantom haptic inter- face.5 Along with these developments, a number of studies have tried to apply haptic displays to real-world computing, including a force feed- back Braille system,6 force feedback Virtual Reality Modeling Language (VRML),7 a force feedback X Window System8, and GIS.9 Haptic studies have only recently become more mainstream, and there are few extensive studies with real subjects. Gillespie and others devel- oped the “virtual teacher,” a device for manual skill learning, which they tested with 24 participants and found that most profited from the “force feedback teacher.” 10 Langrana and others used the Rutgers Master II, a dexterous, portable master for virtual reality simulations for force feedback using four fingers. In their experiment of tumor detection in virtual livers with 32 subjects, the experimental group with force feedback training performed slightly better than the control group.11 This may mean that either the training methods need improvement or that the task did not require extensive training. Colwell and others confirmed that a hap- tic interface (Impulse Engine 3000) has considerable potential for blind computer users through their three- dimensional objects experiment with 22 subjects.12 Jeong tested ordering Communications Wooseob Jeong (wjj8612@uwm.edu) is Associate Professor at the School of Information Studies, University of Wisconsin–Milwaukee. aRtiCLE titLE | autHoR 49touCHaBLE onLinE BRaiLLE GEnERatoR | JEonG 49 tasks in auditory and haptic dis- plays with 23 subjects and found that subjects performed better with haptic-only displays than with audi- tory-only displays or with auditory/ haptic combination displays.13 Several studies already attempted to apply force feedback technology to assist blind people’s computing. Ramstein conducted a pilot study to apply haptics to Braille.14 Yu and Brewster compared the use of force feedback in multimodal virtual real- ity and printed medium in visu- alization for the blind.15 Tzovaras and others tried to implement a virtual reality interface with force feedback for blind people.16 Ramloll and others studied the use of haptic line graphs with sound for blind students.17 Emery and others tested a multi- modal haptic interface with 29 older adults to find that all participants per- formed well under auditory-haptic bimodal feedback.18 Jacko and others tested a multimodal interface with 29 normal vision older adults and 30 visually impaired older adults, finding that in some cases, nonvisual feedback forms—including auditory or haptic feedback—demonstrated significant performance gains over the visual feedback form.19 S. Jeong and others proposed an interactive system that combines an immersive virtual environment with a human- scale haptic interface.20 When conducting user studies with the visually impaired, it is nec- essary to separate the completely blind from the partially sighted. In spite of the different characteristics of these two groups, the literature on visually impaired people typi- cally does not distinguish between them. This distinction is especially important if the legally blind or those with low vision are included in the definition of visually impaired. The challenges to the partially sighted are different from those of the totally blind, demanding different assistance and considerations. In fact, the completely blind rep- resent a small portion of the visually impaired population. According to an advisor in Wisconsin’s Division of Vocational Rehabilitation, less than 5 percent of her advisees are totally blind and require very specialized attention quite different from par- tially sighted people. Purpose of study The purpose of this study is to explore the feasibility of using force feedback technology to facilitate blind people’s access to text information on the Web. Both quantitative and qualitative data were collected to identify the optimal conditions under which the prototype can best serve the blind. Significance of study Public libraries in the U.S., primar- ily through their main libraries, are providing special services for the visually impaired. Currently, the core service is the provision of audio- books. As digital libraries prevail, services for the blind should be online as well, with the the Library of Congress’s Web-Braille service as one of the leading examples. However, such services require the use of an expensive Braille output device. Upon refinement, this prototype would signifi- cantly improve the experience of the visually impaired using online ser- vices. This proto- type can be easily expanded to sup- port graphical dis- plays without any additional devices, making the use of touchable picture books possible for blind users in libraries. Prototype development Force feedback technology has been used for many years in video games. Its use has expanded to other areas such as surgical operations and dan- gerous mechanical processes. This technology was previously applied to GIS to solve the problem of ambiguous multicolor displays for multi-variable thematic maps.21 The same technique was used for this project. The online Braille generator translates text on the Web into a Braille display, letting the user feel the Braille dots with a vibrating mouse. The prototype interface was developed using Immersion Studio (www.immersion.com), JavaScript, Perl/CGI, and Active Server Pages (ASP). Logitech’s iFeel mouse, inex- pensive at a cost of $20, was used for force feedback output (figure 1). The interface has an input text box, which can be filled with any plain text. Once it is submitted, the text is instantly translated into Braille (fig- ures 2 and 3). When the user moves the mouse over each dot on the screen, it vibrates with a given force. While users explore the screen with the vibrating mouse, force feedback dots provide a tactile effect similar to Braille displays. In future projects, the manual con- Figure 1. Experimental setting 50 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 200850 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 2008 version programs will be upgraded to automatic conversion programs with which any texts on the Web can be grabbed by their URLs and converted into a touchable format for the blind. Participants To make this prototype more usable, user studies were conducted in Milwaukee, Wisconsin, with 21 par- ticipants who are completely blind and read Braille. The small sample size—due to the relatively small per- centage of visually impaired peo- ple who are completely blind and can read Braille—is comparable to or larger than those found in other research on the blind. The participants came from vari- ous age groups—teens (3), twenties (6), thirties (2), fifties (5), and sixties (5)—and included 9 females and 12 males. Nineteen of the 21 were born blind. Participants were recruited at several sites, including the University of Wisconsin–Milwaukee Student Accessibility Center, public librar- ies with centers for the blind, and nonprofit organizations for the physi- cally impaired. Vision teachers in local school districts were also contacted. Participants provided valuable i n f o r m a t i o n about the optimal con- ditions for the use of the prototype. This infor- mation will eventually lead to force feedback displays that enable visu- ally impaired people to access the vast amount of information on the Web without expensive devices. Experimental procedure Experiments were conducted in a number of settings, including in the organization’s offices, at the partici- pants’ homes, and at the site of a regional annual meeting for the blind. Each ses- sion lasted no more than 60 minutes. Participants were asked to try differ- ent interfaces of force feedback Braille out- puts with various dot sizes and magnitudes of force. They used a tactile mouse on a note- book computer; after exploring every option, they were asked to select the most comfort- able settings for their sense of touch, including what size the dots should be, how strong the force should be, what kind of force feedback should be used (vibration or friction), and their general opinions of the prototype (see figure 4). Interviews accompanied the experiments so that both quantita- tive and qualitative data could be col- lected. Interviews were transcribed for qualitative data analysis. Result Even though there were only 21 study participants, a number of issues were clearly identified. It is encouraging to see that all of the participants could identify Braille characters using the force feedback mouse with the guidance of the researcher. All the participants agreed that this prototype would be useful with training. The participants pre- ferred the largest dot size (30 pixels in diam- Figure 2. Touchable braille input screen Figure 3. Touchable braille output screen Figure 4. Inexpensive Force Feedback Mouse aRtiCLE titLE | autHoR 51touCHaBLE onLinE BRaiLLE GEnERatoR | JEonG 51 eter) and the strongest force possible for maximum perception of the force feedback effect. However, the prototype was less attractive to the participants than the currently dominant voice synthesizer software. At least two participants mentioned that their current Braille pads fulfill their needs. It seems that they are not motivated to invest their time and effort in a new device. When a potential graphical dis- play application was introduced at the end of a session, the participants became more receptive. At this time there is no practical solution for the visually impaired to feel graphics on computers. Experimental devices are available, but they are either quite expensive or still in the research phase. The blind participants also suggested that this graphical proto- type could be used for geometry and geography easily and effectively. Discussion Blind people’s navigation by mouse Because blind people do not use a mouse for computing, using the force feedback mouse itself was a challenge for the study participants. A sighted person uses a mouse with both hand and eye, moving the mouse while watching the mouse cursor on the screen. For the blind it is difficult to identify the mouse’s position. The direction of movement and the dis- tance between two points are difficult to grasp. Due to the lack of guidance, the blind encounter difficulties in moving the mouse in a straight line. These issues hinder the effectiveness of force feedback displays for the blind. However, this issue does not only affect the blind. Some sighted people, especially older adults, can- not move a mouse easily. One pos- sible solution may be to develop guardrails to help blind people to dif- ferentiate relevant areas of the screen from irrelevant ones. Due to their inexperience in using a mouse, the participants held the mouse too firmly to move it or to feel the force feedback. The only participant to use the mouse success- fully was a college student who is music major with 15 years of piano playing experience. This implies that a significant learning session will be required to allow blind people to use the mouse freely. Ignorance or suppression of graphical information need Even though the participants were more excited about the potential graphical displays, blind people’s graphical information needs are lim- ited. It is possible that their graphi- cal information needs are ignored or suppressed based on their life- time experiences. They tend to resort to Braille and, more recently, voice synthesizers instead of graphi- cal displays. This finding suggests the importance of studying the real information needs of the blind or visually impaired rather than the sighted researchers’ expectations of those needs. More research needed with sound Because the blind already use sound, particularly voice synthesiz- ers, more sound applications should be researched. For example, audio games have the potential to help blind children learn some skills in the same way that video games teach certain skills to sighted children. Audio games also provide a broader research area for future studies. Conclusion Numerous devices have been devel- oped to improve blind or visually impaired people’s access to informa- tion, including information on the Internet. However, such devices are quite expensive or limited in flex- ibility and mainly work in text-only environments. There is no suitable graphic display for the blind, except the laboratory level’s expensive and bulky pin-based external devices. This new prototype uses estab- lished force feedback technology with a minimal cost to existing PCs. It functions for both text and graph- ics. The final products derived from this study can be used for many pur- poses nationally and internationally. Information on the Web can be deliv- ered to the visually impaired without expensive devices. This touchable Braille also lets deaf-blind people, who cannot use screen reader soft- ware, access information on the Web, and it can help people learn Braille. The application of this force feed- back prototype to image displays has exciting and enormous potential because currently there is no practical, usable method for the blind to access images. For example, blind children are still using handmade 3-D picture books that are labor-intensive and time-consuming to produce. With this prototype, children’s books can be delivered easily to blind children, who will touch the books’ images via the force feedback mouse. Maps of local, state, national, or international interests can be delivered to the blind as well. This prototype will help to add yet another sense—touch—to already blossoming visual and auditory digi- tal libraries. Through force feedback technology, new multimodal digi- tal libraries will be accessible to the world. Acknowledgement This research was supported by a Diversity Research Grant from the American Library Association in 2005. 52 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 200852 inFoRmation tECHnoLoGY anD LiBRaRiEs | maRCH 2008 References and Notes 1. Wooseob Jeong and Myke Gluck, “Multimodal Geographic Information Systems: Adding Haptic and Auditory Display,” Journal of the American Society for Information Science and Technology 54, no. 3 (2003): 229–242. 2. Wooseob Jeong, “Touchable Online Braille Generator,” in Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (New York: ACM Press, 2005), 188–189. 3. Jack M. Loomis and Susan J. Leder- man, “Tactual Perception,” in Handbook of Perception and Human Performance, ed. K. R. Boff, L. Kaufman and J. P. Thomas (New York: John Wiley & Sons, 1986), vol. 2, chap. 31, 1–41. 4. R. Dan Jacobson, Robert Kitchen, and Reginald Golledge, “Multimodal Vir- tual Reality for Presenting Geographic Information,” in Virtual Reality in Geog- raphy, ed. P. Fisher and D. Unwin (New York: Taylor & Francis, 2000), 382–400. 5. J. Kenneth Salisbury and Man- dayam A. Srinivasan, “Phantom-Based Haptic Interaction with Virtual Objects,” IEEE Computer Graphics and Applications 17, no. 5 (1997): 6–10. 6. Christopher Ramstein, “Combining Haptic and Braille Technologies: Design Issues and Pilot Study,” in Proceedings of the 2nd Annual ACM Conference on Assis- tive Technologies (New York: ACM Press, 1996), 37–44. 7. A. Hardwick, S. Furner, and J. Rush, “Tactile Access for Blind People to Virtual Reality on the World Wide Web,” IEE Colloquium on Developments in Tactile Displays 1997, no. 012: 9/1–9/3. 8. Timothy Miller and Robert Zeleznik, “The Design of 3D Haptic Wid- gets,” in Proceedings of the 1999 Symposium on Interactive 3D Graphics (New York: ACM Press, 1999), 97–102. 9. R. Dan Jacobson, “Geographic Visualization with Little or No Sight: An Interactive GIS for Visually Impaired People (paper submitted to AAG-GIS spe- cialty group student paper competition). 10. R. Brent Gillespie and others, “The Virtual Teacher” in Proceedings of ASME Dynamic Systems and Control Division (New York: ASME, 1998), vol. 2, 171–78. 11. Noshira A. Langrana and others, “Human Performance Using Virtual Real- ity Tumor Palpation Simulation,” Com- puter & Graphics 21, no. 4 (1997): 451–458. 12. C. Colwell and others, “Haptic Vir- tual Reality for Blind Computer Users,” in Proceedings of the Third Annual ACM Conference on Assistive Technologies (New York: ACM Press, 1998), 92–99. 13. Wooseob Jeong, “Exploratory User Study of Haptic and Auditory Display for Multimodal Geographic Information Systems,” in CHI’01 Extended Abstracts on Human Factors in Computing Systems (New York: ACM Press, 2001), 73–74. 14. Ramstein, “Combining Haptic and Braille Technologies.” 15. Wai Yu and Stephen Brewster, “Multimodal Technologies: Multimodal Virtual Reality Versus Printed Medium in Visualization for Blind People,” in Proceedings of the 5th International ACM Conference on Assistive Technologies (New York: ACM Press, 2002), 57–64. 16. D. Tzovaras and others, “Multi- modal Technologies: Design and Imple- mentation of Virtual Environments Training of the Visually Impaired,” in Proceedings of the 5th International ACM Conference on Assistive Technologies (New York: ACM Press, 2002), 41–48. 17. R. Ramloll and others, “Construct- ing Sonified Haptic Line Graphs for the Blind Student: First Steps,” in Proceedings of the 4th International ACM Conference on Assistive Technologies (New York: ACM Press, 2000), 17–25. 18. V. Kathlene Emery and others, “Toward Achieving Universal Usability for Older Adults through Multimodal Feedback,” in Proceedings of the 2003 Con- ference on Universal Usability (New York: ACM Press, 2003), 46–53. 19. Julie A. Jacko and others, “Older Adults and Visual Impairment: What Do Exposure Times and Accuracy Tell Us About Performance Gains Associated with Multimodal Feedback?” in Proceed- ings of the SIGCHI Conference on Human Factors in Computing Systems (New York: ACM Press, 2003), 33–40. 20. Seongzoo Jeong, Naoki Hashimoto, and Sato Makoto, “A Novel Interaction System with Force Feedback between Real and Virtual Humans,” in Proceedings of the 2004 ACM SIGCHI International Con- ference on Advances in Computer Entertain- ment Technology (New York: ACM Press, 2004), 61–66. 21. Jeong and Gluck, “Multimodal Geographic Information Systems”; and Wooseob Jeong, “Multimodal Trivariate Thematic Maps with Auditory and Hap- tic Display” (paper contributed to ASIST 2005, Charlotte, North Carolina, October 28–November 2, 2005).