key: cord-0016024-bjpkz9hu authors: Trinh, Viet; Manduchi, Roberto title: A Multi-scale Embossed Map Authoring Tool for Indoor Environments date: 2020-08-10 journal: Computers Helping People with Special Needs DOI: 10.1007/978-3-030-58796-3_53 sha: 96710d5e2f2d9734c3d53ec361c9efe5006bc2d4 doc_id: 16024 cord_uid: bjpkz9hu We introduce a multi-scale embossed map authoring tool (M-EMAT) that produces tactile maps of indoor environments from the building’s structural layout and its 3D-scanned interiors on demand. Our tool renders indoor tactile maps at different spatial scales, representing a building’s structure, a zoomed-in of a specific area, or an interior of a room. M-EMAT is very easy to use and produces accurate results even in the case of complex building layouts. Multiple studies have shown that, when visually impaired individuals are given the opportunity to preview a route displayed in a tactile format, prior to an actual travel, they are able to follow it more accurately and with fewer errors [2, 9, 10] . Tactile maps give readers the layout of a venue and the spatial relationship between its landmarks, facilitating a confident self-orientation and a safer travel. Regardless of different techniques in producing tactile graphics (i.e., embossing [4, 27] , audio-tactile pairing [3, 6] , or 3D printing [8, 28] ), it is impractical to design a one-size-fits-all tactile map because different users have different needs of tactile feedbacks. Thus, automation of tactile map making tailored to user needs is crucial in helping people with visual impairments. One of the most practical challenges of automating tactile map production is generalization, a process in which the map maker needs to decide which level of detail should be rendered at a given scale, due to limited tactile resolution [14, 17] . This problem is particularly relevant for the generation of indoor tactile maps. This is compounded by the fact that detailed maps of indoor environments in digital format are not always publicly available. And even when they are, they usually contain only large-scale architectural information (e.g., entrances, staircases, offices), but not detailed layouts of interior spaces. Findings from the focus group discussed in [26] indicate that embossing the map of a building at different levels of detail could be provide useful spatial information for pre-journey learning. In this paper, we introduce a multi-scale embossed map authoring tool, or M-EMAT, that produces tactile maps of indoor environments at different spatial scales on demand, including the building's architectural structure, zoomed-in of specific areas (sections, and a small-scale layout of a room, highlighting the spatial relationship among objects in the room. This tool is an extension of our online application, named Semantic Interior Mapology or SIM 1 [25] , that allows users to trace the contour of a building and to construct the spatial layout of its interior structure, starting from the picture of a floor plan. The problems of cartographic generalization and automatic tactile map production have been investigated independently over the past few decades. However, the use of standard generalization techniques for the production of indoor tactile map that could be used by visually impaired travelers has received relatively little attention. Early works in automatic generation of tactile maps relied on data from geographical information systems (GIS) to render tactile elements of outdoor environments. For instance, TMAP [16] , TMACS [27] , Mapy [4] , and On Demand Tactile Map [23] use OpenStreetMap as their underlying data source to generate a tactile map file of a location around a physical address that could later be printed offline using Braille embossers. These tactile elements cover important outdoor landmarks (e.g., buildings, parks, stations, road networks, rivers, etc) but not interior layouts of buildings. Recently, there is a growing interest in technologies for automating indoor tactile map making. For example, the Audio-Tactile navigation system by Papadopoulos et al. generates audio-tactile maps from digital map files containing specific spatial information of a building [18] . Hybrid methods for the automatic generation of 3D indoor maps from AutoCAD architectural floor plans were proposed in [1, 22] . This prior work only focused on the structural elements of a building, such as walls, doors, or staircases. Small-scale description of furniture items or floor covering, which can be useful for navigation without sight, were not considered. Advances in machine learning techniques have prompted researchers to revisit the problem of cartographic generalization. Several neural-network models have been developed for the tasks of recognizing, grouping, and typifying buildings [7, 20, 24] . However, these models are only able to learn and predict a building's contour and the geographical distribution if groups of buildings, and may not generalize well for the representation of the layout of an indoor space. Our embossed map authoring tool produces a digital tactile map file at a desired scale, based on the building's structure represented in sim [25] and the semantic layout of an interior space encoded in a JSON map that are collectively acquired from our SIM toolbox described in [25] . More specifically, M-EMAT is an extension of the SIM web application (Fig. 1 ) that allows one to convert an architectural floor plan and its 3D-scanned contents into a standard format (JSON). Using SIM, one can quickly trace a floor plan from an image of it, and produce a vectorized map, stored in the sim format [25] . Small-scale elements (such as furniture items) that are normally not available in a map can be acquired using a 3D (RGBD) camera such as Occipital's Structure Sensor, 2 which registers and stitches multiple point clouds into one 3D mesh. SIM allows one to parse the 3D scan of a room, to segment out objects of interest, and to geo-register these objects within the building's spatial layout. The object segmentation component used in SIM, which is a web-based toolkit developed in [5] , lets users to manually select a connected set of mesh facets having a similar orientation, indicating the 3D scan's objects of interest. A JSON file containing all spatial information for the building is automatically generated. Following the study described in [21] , our map authoring tool renders segments with length of at least 0.5 in (12 mm), with a minimum distance of 0.2 in (5 mm) between two segments. For easy discrimination, symbols representing different features have a minimum diameter of 0.25 in (6 mm), with minimum distance between two symbols of 0.5 in (12 mm). Braille characters for annotation of objects and spaces have size of 0.16 in × 0.26 in (4 mm × 6 mm) [11] . The digital tactile map files produced by M-EMAT can be printed on a 11.5 in × 11 in embosser sheet, with resolution of 20 DPI and a 0.5 in (12 mm) margin on all sides. It is partitioned in two sections: the header (10.5 in × 3 in; 267 mm × 76 mm); and the body (10.5 in × 7 in; 267 mm × 178 mm). The header contains on the top-left the building name, the floor number, and the map scale, as well as an arrow pointing to the North on the top-right. The body has the tactile map at the desired scale. There are two types of embossed features in the map: structural and interior . Structural features are those traced from a building's floor plan, including entrances, staircases, elevators, escalators, walls, and doors. Interior features represent objects that are segmented and annotated from 3D scans, such as tables, cubicles, shelves, and other pieces of furniture. M-EMAT allows one to choose between three different scales: 1. Structure-scale: General building layout, consisting of rooms, corridors, and structural features. Wall are embossed as solid segments, while rooms are represented by untextured areas, enclosed by at least 4 walls. Corridors are rendered as textured areas. In this scale, the room number or door is usually not rendered due to space constrains. 2. Section-scale: Expanded view of a specific area inside a building. In addition to the features already considered in the structure scale, a section-scale map also displays room numbers, room doors, and any available interior features. Room numbers are embossed at the center of each room, and doors are represented as circles along walls [13] . 3. Room-scale: The layout of a small area (typically a room), including walls, doors, and all annotated interior features. The features to be embossed at different map scales when space permits are shown in the Fig. 2-a. Tactile symbols and patterns used to represent such features are also listed in the Fig. 2-b . Note that the staircase symbol embossed in our maps was suggested in [12] , while the other symbols were drawn from [13] . The texture patterns were proposed in [19] . Our tactile map authoring tool limits rendering of a room or a section of a building to a single embosser sheet. The building's general layout (structurescale) can span multiple pages; this allows for rendition of very elongated buildings. M-EMAT automatically selects the features to be rendered based on the selected scale, while adhering to tactile resolution constraints of Sect. 3.1. Figure 3 shows an example of M-EMAT's user interface at different map scales, along with the produced tactile maps. Interior features at room-scale are represented by their bounding boxes, which are shaped as vertical-oriented cuboids. These cuboids are shown as rectangles in the tactile map. M-EMAT includes a simple editor (only available at the room-scale) that allows users to translate, rotate, and scale features within a boundary. This can be useful when the segmentation tools produces overcrowded, unaligned, or overlapping embossed features. The editor also allows users to merge multiple rectangles into a single polygonal feature [15] , which can be helpful for objects with complex shapes. As shown in the Fig. 4 , the feature #1 (a couch) was translated and then merged with another couch (#2); the coffee table (#3) was rotated to its correct orientation; and a desk (#4) and a whiteboard (#5) were both scaled down to their correct dimension. Note that if two objects (e.g., a table and a printer) are physically on top of each other, they will be represented as two stacked cuboids, which will be mapped as two nested rectangles. In this case, the innermost rectangle can be removed using the editor. M-EMAT's user-friendly interface allows one to select any region, room, or type of indoor features to be embossed. For example, one might choose to render only features that are close to walls (countertops, benches, or shelves) vs. furnitures positioned randomly in the middle of a room (tables, chairs, etc). Sample multi-scale indoor tactile maps generated by M-EMAT are shown in Fig. 5 . The figure also show 3D room scans (a), their segmentation (b), and the results after manual editing (c), along with Braille annotations. These Braille annotations at the section-scale denote room numbers; whereas, in the room-scale tactile map, they annotate the interior features segmented from a 3D scan. We introduced M-EMAT, an add-on to our existing SIM application, that enables generation of multi-scale tactile maps of a building. Using SIM, one can easily trace an existing map to generate a digital representation of the building's structure (walls, rooms, doors, stairs, elevators). M-EMAT converts this information into a format that is amenable for embossing, at the scale specified by the user, with specific constraints on the density and distances of tactile features. In addition, M-EMAT facilitates the generation of room-scale layouts by from a 3D scan of a room. This feature is particularly useful because maps at the room level are usually not available. SIM with the M-EMAT add-on is available for anyone to use at https://sim.soe.ucsc.edu. 3D printing technology for buildings accessibility: the tactile map for mte museum in pavia Map use by adults with visual impairments Usage of multimodal maps for blind people: why and how Blind friendly maps: tactile maps for the blind as a part of the public map portal (mapy. cz) Scannet: richly-annotated 3D reconstructions of indoor scenes Exploring map orientation with interactive audio-tactile maps Learning cartographic building generalization with deep convolutional neural networks Tactile: a preliminary toolchain for creating accessible graphics with 3D-printed overlays and auditory annotations Exploring auditorily enhanced tactile maps for travel in new environments Pre-journey visualization of travel routes for the blind on refreshable interactive tactile displays Guidance onADA standards for accessible design An evaluation of tactile symbols in public environment for the visually impaired The use of environmental features on tactile maps by navigators who are blind How Maps Work: Representation, Visualization, and Design A new algorithm for computing boolean operations on polygons Talking tmap: automated generation of audiotactile maps using Smith-Kettlewell's tmap software How to Lie with Maps A university indoors audio-tactile mobility aid for individuals with blindness Consistency of a tactile pattern set Building generalization using deep learning Automated generation of tactile maps Automatic pre-journey indoor map generation using autoCAD floor plan Automatic derivation of on-demand tactile maps for visually impaired people: first experiments and research agenda Is deep learning the new agent for map generalization? Semantic interior mapology: a toolbox for indoor scene description from architectural floor plans Feeling your way around: Assessing the perceived utility of multi-scale indoor tactile maps Tactile map automated creation system using OpenStreetMap Interactiles: 3D printed tactile interfaces to enhance mobile touchscreen accessibility Research reported in this publication was supported by the National Eye Institute of the National Institutes of Health under award number R01EY029033. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.