key: cord-0264029-z3h7s83m
authors: Nair, Vishnu; Karp, Jay L.; Silverman, Samuel; Kalra, Mohar; Lehv, Hollis; Jamil, Faizan; Smith, Brian A.
title: NavStick: Making Video Games Blind-Accessible via the Ability to Look Around
date: 2021-09-02
journal: nan
DOI: 10.1145/3472749.3474768
sha: 5623390056160318ad2e57513bbc8a5714b6e58c
doc_id: 264029
cord_uid: z3h7s83m

Video games remain largely inaccessible to visually impaired people (VIPs). Today's blind-accessible games are highly simplified renditions of what sighted players enjoy, and they do not give VIPs the same freedom to look around and explore game worlds on their own terms. In this work, we introduce NavStick, an audio-based tool for looking around within virtual environments, with the aim of making 3D adventure video games more blind-accessible. NavStick repurposes a game controller's thumbstick to allow VIPs to survey what is around them via line-of-sight. In a user study, we compare NavStick with traditional menu-based surveying for different navigation tasks and find that VIPs were able to form more accurate mental maps of their environment with NavStick than with menu-based surveying. In an additional exploratory study, we investigate NavStick in the context of a representative 3D adventure game. Our findings reveal several implications for blind-accessible games, and we close by discussing these.

Video games act as a social and emotional outlet for many people -improving mood and fostering a sense of community among arXiv:2109.01202v1 [cs.HC] 2 Sep 2021 players [22] . However, most mainstream 3D video games remain largely inaccessible to visually impaired people (VIPs) due to a lack of accessibility tools within them [4, 35] . As a result, VIPs are restricted to playing audio-centric blind-accessible games that are greatly simplified compared to games created for sighted players [2, 37] . These games are designed to be easier for VIPs. In order to achieve this, they confine players to 2D grid-like worlds [29, 33] , guide players along strict predetermined paths [14] , present pointsof-interest (POIs) in linear menus or grids [34, 40, 44] , and output announcements or sounds continuously (and often distractingly) to keep the player updated [27] .

These techniques limit the freedom that visually impaired players have in exploring the game world -freedom that sighted players are granted by default within mainstream games. It is, thus, an important challenge to grant VIPs the same level of agency in gaming -in part, by allowing them to play the same mainstream 3D games that sighted players play with the same amount of agency afforded to sighted players.

A key aspect of a sighted player's experience within a 3D video game is the ability to use the in-game camera to look around -to find items, detect enemies, and otherwise interact with the world. Looking around gives the player agency by allowing them to understand their surrounding environment, develop a plan of action, and attempt to execute that plan [6] [7] [8] 47] . Existing games, however, do not offer VIPs an analogue to the in-game camera that allows them to "look around" on their own terms. We believe that providing VIPs with such a tool is critical to providing them with a higher level of freedom and agency in exploring 3D game worlds in a self-directed way -that is, completely on their own terms.

Hence, we introduce NavStick, a self-directed directional surveying tool for virtual environments that grants VIPs the ability to look around within virtual worlds. Its aim is to make existing 3D video games more blind-accessible. NavStick repurposes a game controller's right thumbstick to allow players to survey their immediate surroundings by "scrubbing" the thumbstick in a circular fashion. As the player scrubs in different directions (their 1 o'clock, 2 o'clock, etc.), NavStick uses spatialized audio to announce what lies in those directions via line-of-sight. Figure 1 shows an example of this within a prototype 3D adventure game we created called The Explorer.

NavStick aims to replicate the ability to look around by giving users on-demand, random access to information about their surroundings in all directions. That is, users can survey any direction instantly without having to traverse a list or other sequential interface. Furthermore, NavStick does not output any audio feedback unless the user is scrubbing. These properties stand in contrast to many tools in existing blind-accessible games, which have users survey their surroundings via linear menus or instructions, and which often generate unwanted and distracting sounds by outputting announcements automatically. In addition, NavStick does not require any special hardware -it is designed to work using a standard game controller and pair of headphones. We co-designed NavStick alongside a member of our research team who is a co-author on this paper and is blind. We leveraged their experience with games to identify design requirements, playing a crucial role in designing NavStick and our studies.

We evaluate NavStick through two user studies. In our first study, we compared NavStick to menu-based surveying tools found in existing blind-accessible games by having nine VIPs perform six types of virtual navigation tasks. We found that NavStick allowed VIPs to form more accurate mental maps of their environment with less effort and frustration than "NavMenu, " a baseline menubased surveying tool. Players preferred NavStick for tasks that rely on having a mental map of the surrounding environment, as well as for navigating within video games. Players also felt, however, that the ability to efficiently determine the presence of objects (as exemplified by NavMenu) and the ability to look around (as exemplified by NavStick) serve different purposes when surveying an area, and that the two tools should complement each other.

For our second, exploratory study, we use The Explorer (our 3D adventure game) as a case study to investigate the ways in which NavStick can make 3D video games more blind-accessible by having seven VIPs play through the game. We found that NavStick allowed participants to experience a sense of freedom, environmental understanding, and fun not found in present-day blind-accessible games. Participants completed all portions of The Explorer, which featured different environments that replicated the complexities of mainstream 3D adventure games. Still, some challenges in handling these complexities surfaced: NavStick had difficulty handling environmental occlusions gracefully, and players had differing opinions about NavStick's handling of moving enemies and situations involving time pressure.

We make four main contributions in this paper:

(1) We introduce NavStick, a directional surveying tool that gives VIPs the ability to look around 3D virtual environments in a self-directed way. (2) We investigate the advantages and disadvantages of using NavStick compared to menu-based surveying tools (as are common in existing games) for various navigation tasks. (3) We explore the ways in which NavStick can make existing 3D adventure game worlds more blind-accessible. (4) We describe NavStick's implications for future efforts on making 3D video games blind-accessible.

NavStick contributes to a rich history of audio navigation tools made for both video games and the real world.

The vast majority of mainstream video games are inaccessible to people with visual impairments. Hence, researchers have expressed a need for tools within games that let VIPs make decisions themselves rather than being guided and told what to do [2] [3] [4] 37] . Current means of navigating within games do not offer VIPs on-demand, random access to information about their surroundings in all directions, which prevents VIPs from being able to play 3D games with an experience equivalent to what sighted players enjoy.

In PowerUp [40] , players survey the environment by scrolling through a menu of objects in the area, each of which is presented with its name, distance, and relative orientation to the player. Terraformers [34, 44], a blind-accessible 3D adventure game, gives players an assortment of navigation tools, including a "GPS navigator" for determining their current position via grid coordinates and a beacon-based "sonar" for finding out what is directly in front of them. These tools do not, however, provide random access to information about the player's surroundings as NavStick does. In Study 1, we recreate a level from Terraformers to act as a testbed for comparing NavStick with menu-based surveying.

Many adventure games for VIPs, such as Shadowrine [29] and A Hero's Call [33] , also conform to grid-and menu-based surveying. They physically confine players to 2D grid-like worlds so that they can only move in the four cardinal directions. This simplifies the structure of these games -POIs are situated within a single "cell" of the world, and players can query that cell to see what is present. NavStick, by contrast, does not require a grid-based world and thus enables players to look around within arbitrary game worlds as they please using line-of-sight.

Some mainstream 3D games have added accessibility features for VIPs. The Last of Us Part II [13] , a 3D action-adventure/survival horror game released in 2020, is a prominent example. It features an "enhanced listen mode" to help VIPs navigate. When triggered, audio beacons appear at the locations of nearby enemies and other POIs, but they appear all at once and may overwhelm the player if there are many targets in their vicinity. This tool also does not allow players to scan in specific directions. NavStick, by contrast, allows players to "scan" any direction they wish, preventing them from being overloaded with sounds and allowing them to explore the environment at their own pace.

Although our focus is on virtual worlds, many sound-based methods for real-world navigation also apply within virtual worlds. Indeed, previous research has shown that game-based wayfinding tasks have ecological validity in assessing real-world navigation ability [9] . Existing audio navigation tools guide users along a series of waypoints connected by line segments, and they do so in two different ways. SWAN [45] , audioGPS [25] , and Microsoft Soundscape [30] place acoustic "beacons" at each waypoint. VIPs can follow those beacons by physically turning to "center" the beacons' spatialized sounds in front of them, and then walking forward. NavCog [1] and NavCog3 [36] , by contrast, give VIPs spoken directions to follow instead of beacons, and those directions include guidance on how much users should rotate to turn at each waypoint.

These tools suffer from the same issues that tools within games have. They simply guide users to where they need to go, and do not offer users much agency when navigating, nor do they offer users a direct way of "looking around" environments themselves. Talking Points 3 [46] features some degree of looking around, specifically a "Directional Finder" that provides a list of POIs in the general 45°-wide direction that a user points their phone. However, users cannot precisely look in a given direction and must physically turn their body toward a general direction to hear what lies that way. Our preliminary exploration of NavStick [32] proposed the concept of surveying environments directionally; in this paper, we perform studies to evaluate NavStick with respect to existing navigation paradigms and investigate the role of such a system in video games. Figure 2 : Representing a player's surroundings with NavPies. We show a player surrounded by five objects (A-E) of varying sizes. Top: Two different player positions among these objects. Bottom: Forward perspective NavPies. Here, each pie has been rotated such that it visualizes the pie from the player's forward perspective. The arrow outside each colored slice represents the direction that the thumbstick must be tilted in order to "look at" each object.

Another issue with real-world navigation tools is that their automatic announcements may cause users to miss important environmental sounds [11] -a common complaint among VIPs [18] . NavStick only plays announcements or tones while the user is actively scrubbing.

NavStick is an audio navigation tool that gives VIPs the ability to look around virtual environments in a self-directed way. Nav-Stick was co-designed with VIPs and centers around the idea that navigation involves much more than simply being guided along a path efficiently or scrolling through a list of POIs -that is, players should have on-demand, random access to information about their surroundings. Two major aspects of NavStick's design facilitate this kind of navigation: (1) the act of "scrubbing," in which the player can tilt a thumbstick circularly to "look" in different directions; and (2) a circular data structure that we call a "NavPie, " which represents the player's immediate surroundings.

NavStick repurposes a game controller's right thumbstick into a tool for looking around. To use it, players tilt the thumbstick in any direction to hear (via speech) what lies in that precise direction via a raycast. They can then "scrub" the thumbstick circularly around the edge to quickly survey many different directions. Figure 1 illustrates how to use NavStick. In the figure, the player is tilting the right stick in the one o'clock direction. NavStick announces what lies in that direction from the player's current perspective -in this case, Chomper #4 (an enemy) -using spatialized audio emanating from the selected item itself. In the current iteration of NavStick, raycasts are performed straight out from the eye-level of the ingame character that the player controls -that is, parallel to the ground. (We discuss controlling NavStick's pitch angle as a point of future work in Section 7.2.) NavStick operates via line-of-sight (like a laser pointer), only taking into account what is directly visible from the player's current position. NavStick will only announce the first object situated in a given direction and has an infinite scan radius. We map NavStick to the right thumbstick because most 3D games use the right thumbstick to control the in-game camera. Controlling the game's camera is not useful for VIPs since many VIPs lack vision, so NavStick repurposes this thumbstick to act as a navigation aid instead. Similar to how a sighted player can move and control the camera at the same time, VIPs can also move and use NavStick simultaneously.

NavStick's design is inspired by earPod [48] , which repurposed an early iPod's click wheel into a tool for traversing eyes-free menus such as telephone menus. Similar to earPod, any speech or tone played by NavStick at a slice is immediately truncated when the user moves off the slice. Even if the user scrubs quickly, they will still be able to hear the first syllable of an announcement, providing users with a strong feeling of reactivity.

We implemented NavStick in the Unity game engine (version 2020.1.16f1) [41] . Players interact with the world using a standard game controller and receive feedback via audio using a standard pair of headphones. To generate spatialized sound, we used the Steam Audio toolkit for Unity [42] , which provides spatialized audio via its built-in head-related transfer function (HRTF). We pre-synthesized speech output using the Google Cloud Text-to-Speech service [21] .

To represent a player's surroundings, NavStick uses a circular data structure called a NavPie. NavPies can be thought of as spatial versions of pie menus; each slice represents where an object is in relation to the player at that moment in time. Each of the NavPie's "slices" or "regions" represents an item that is visible from the player's point of view. The slice's direction and circular extent (i.e., arc length) represent the object's direction and angular extent (the angle that the object subtends with respect to the player's location). Figure 2 shows how NavStick computes NavPies from a player's surroundings. The figure depicts a player at two different positions (top-left and top-right) among five objects (labeled A-E). The circle surrounding the player is the NavPie, and its arcs ("slices") are formed via line-of-sight ray traces. Each arc represents an object. Note that the arcs do not cover the NavPies' full circumferences. "Empty" slices (gray arcs) indicate that there is nothing of interest in that direction in the player's immediate surroundings. NavStick can be configured to either play no audio at all (as in Study 1) or play a default tone (as in Study 2) when players tilt or scrub in these directions.

The NavPies on the bottom-left and bottom-right of Figure NavPies show how NavStick maps the NavPies' slices to thumbstick directions.

NavStick supports all three navigation tasks that Darken and Sibert [10] classify for virtual worlds: naïve search, primed search, and exploration. In addition, NavStick supports both novice and expert usage patterns. Novices, who are unfamiliar with their surroundings, might choose to scrub NavStick's entire circumference to survey their full surroundings, thereby performing an exhaustive search (naïve search). Experts, who are already somewhat familiar with their surroundings, can choose to scrub in specific directions of interest to perform a non-exhaustive search (primed search). Both groups of users can also use NavStick to form a cognitive map even when there is no target (exploration).

The COVID-19 pandemic prevented us from evaluating NavStick in-person, so we were faced with the task of designing a completely remote study. By its very nature, however, NavStick presents distinct advantages for a remote study: the only hardware required is a game controller and a standard pair of headphones. Our prototype was implemented in Unity, which can compile executables for all major computing platforms. Executing the study remotely also allowed us to reach VIPs who may not feel comfortable traveling too far from home or who may be otherwise isolated, due to challenges in navigating the real world.

To execute our two studies in an entirely remote manner, we packaged the Unity study environment into portable game executables for both Windows and macOS. Each participant had their own executable that was signed with a unique identifier. We sent each participant a link to download their executable before their study appointment. We held the study appointments over Zoom and asked participants to share their computer audio (and, optionally, video) with the facilitators. Figure 3 shows a Study 2 session in progress. Both studies were approved by the Columbia University Institutional Review Board (IRB).

Navigation Task  Repetitions  Goal  1 Exploration ( We designed the executables to connect with a cloud backend that synchronizes with the runtime state of participants' games. This allowed us to control the executables remotely as needed. We could, for example, enable or disable rendered graphics, advance to the next study condition/segment, repeat a trial, and tweak other options fully remotely using a custom-built control panel.

In Study 1, we compare NavStick with the "status quo" means of surveying virtual environments: using linear menus that list surrounding items. Prior work in spatial cognition [20] and navigation tools [10] show that navigation tools should be analyzed and evaluated with respect to navigation tasks such as exploration vs. search. As such, we will also use Study 1 as an opportunity to assess how NavStick performs across a broad range of navigation tasks.

To represent status quo surveying, we implemented "NavMenu, " a linear menu that alphabetically lists POIs in the player's current area. Players "open" the NavMenu by pressing the left shoulder button, then navigate up and down within the menu by using the D-pad. When the player opens the menu, the system announces the number of targets in the area. Each time the player presses up or down on the D-pad, they will hear the corresponding item announced via speech spatially rendered at the item's location. Recall that NavStick also announces items via spatialized speech, but does so when the player tilts the thumbstick toward an item.

Although we opted for an alphabetically-sorted NavMenu for this study, we also initially considered a proximity-ordered menu (where elements are sorted based on how far they are from the player) and a directionally-ordered menu (where elements are sorted based on their direction relative to the player). Consecutive elements in both alphabetical-and proximity-ordered menus can be located in scattered directions. However, proximity-ordered menus have an additional weakness where the menu's ordering will change as the player moves through the world. Traversing a directionally-ordered menu is equivalent to scrubbing continuously around with NavStick, but with more effort and without the ondemand, random access to items that NavStick affords. As a result, an alphabetically-sorted NavMenu provides the strongest possible comparison with NavStick.

We investigate the following research questions in Study 1:

(RQ1.1) What are NavStick's advantages and disadvantages compared to menu-based surveying (NavMenu)? (RQ1.2) Which forms of navigation is NavStick better suited for, and which forms is NavMenu better suited for? 

We asked participants to perform six types of navigation tasks using both NavStick and NavMenu. Table 1 shows an overview of these tasks. Tasks 1-5 involved navigating within an aisle in a virtual grocery store, and Task 6 involved completing a level from the video game Terraformers [34, 44]. These tasks, although highly-controlled, allowed us to compare NavStick and NavMenu for different virtual navigation purposes.

We asked participants to perform tasks that spanned Darken and Sibert's classification of wayfinding tasks for virtual worlds [10] . Exploration tasks do not involve a target in the initial search and require forming a cognitive map for later use. In naïve search tasks, the player has a target but does not know its location, possibly requiring an exhaustive search of the NavPie in the worst case. In primed search tasks, the player knows that the target items are laid out in front of them, and so does not have to scrub the entire NavPie to find the items. We chose a virtual grocery store for this study since it represents a place in which people routinely perform all of these different tasks. Figure 4 shows the grocery store aisle layout that we used. It features eight equal-length item "slots" (four on each side) that we placed items into to represent different aisles such as canned goods or beverage aisles. These targets were made large enough such that that they were "airtight" (i.e., they had no gaps between them). For the exploration tasks (Tasks 1 & 2), we first let participants explore and survey the aisle for as long as they wished, then we asked them one of two types of questions. For Task 1, we asked them to name an item that is located close to a randomly chosen item. The closer their answer was, the better. For Task 2, we gave the participant a "shopping list" of five items, and they had to say whether or not each of the items was present in the aisle.

For the naïve search tasks (Tasks 3 & 4), we asked participants the same questions as Tasks 1 & 2, respectively. The difference is that participants surveyed the aisle after hearing the question.

We refer to Tasks 1 & 3 as "directional" in Table 1 because they test participants' knowledge of how items are directionally situated with respect to one another. Similarly, we refer to Tasks 2 & 4 as "non-directional" in Table 1 because participants do not need to remember items' directions -only whether or not they are present.

In the primed search task (Task 5), we positioned participants in front of a group of items and asked them to find the cheapest one among that group. Players can hear an item's price by pressing the right bumper (RB) while selecting it with NavStick or NavMenu.

For all tasks, we informed participants that the aisle contained eight items, and all participants surveyed all items in the aisle before committing to an answer (for naïve search and primed search) or hearing the question (for exploration).

Task 6 is a video game level that we recreated from Terraformers, a blind-accessible video game [34, 44]. Figure 5 illustrates the level. As NavStick's goal is to make 3D games more blind-accessible, we wanted to use Task 6 to compare NavStick and NavMenu within a game context. We chose Terraformers because it is a classic audio game that features linear menus for navigating. 

No Response NavStick Figure 6 : Preferred navigation tool for each type of navigation task. Most participants preferred NavStick for directional tasks (in which the location of items is important) and for the video game task. Most preferred NavMenu, however, for non-directional tasks: ones in which it is only important to understand what items are present. Overall, participants preferred NavStick if forced to choose, but strongly felt the two interfaces worked hand in hand. One refused to respond for this reason.

To complete the level, players must navigate to four targets in sequence within the room: a computer terminal, a key, a keyhole, and an exit door. Unlike our virtual grocery store (Tasks 1-5), this game level includes ambient environmental sounds similar to the original sounds from Terraformers. When players focus on a target using NavStick or NavMenu, they can press the left bumper (LB) on their controller to teleport to that target's location instantly.

We recruited nine participants for Study 1, all of whom described themselves as male and having no usable vision. Five participants (P2-4, P7, P9) were 18-25 years old and the other four (P1, P5, P6, P8) were 26-35 years old. P6 reported minor hearing loss in their right ear. All but two participants (P7 and P8) reported themselves as experienced with video games (4+ on a 5-point Likert scale). We recruited these participants through posts on the AudioGames.net Forum, 1 an online discussion board that centers around audio-based games and is frequented by VIPs.

We began by giving participants unlimited time to experiment with NavStick and NavMenu within a trial aisle. Afterward, each participant performed Tasks 1-6 in that order, but they were assigned a counterbalanced order for using NavStick and NavMenu for the tasks. After every (tool, task) combination, we administered a post-task questionnaire that gauged participants' perceived effort, frustration, and impressions of using the assigned tool for the task. The questionnaire employed 20-point Likert scales similar to a NASA TLX form [24] . For exploration tasks (Tasks 1-2), we also asked participants how well they thought they were able to create a mental map of their surroundings. For the video game task (Task 6), we also asked how fun it was to play the level with the tool they used. All responses were given on a 20-point Likert scale. After performing each task with both NavStick and NavMenu, we asked participants to state which tool they preferred for the task.

At the very end, we administered a post-study questionnaire followed by a semi-structured interview to ask participants to reflect on what they liked and disliked about both tools and to choose their favorite in a forced ranking.

Below, we report the main takeaways from Study 1 in terms of answering our research questions: learning what NavStick's advantages and disadvantages are compared to NavMenu (RQ1.1), and learning which forms of navigation each tool is better suited for (RQ1.2). We analyzed our qualitative results (participant quotes) via qualitative coding. All authors collectively clustered data into common themes, debating among themselves until there was mutual agreement on the themes. Each theme presented below represents opinions shared by a majority of our participants. For statistical significance conclusions with respect to our quantitative results (Likert scale responses and in-game data), we ran Welch's t-tests on the data to obtain two-tailed p-values and assumed a 95% confidence interval. Below, we share our findings on the advantages and disadvantages that each of the two tools -NavStick and NavMenu -have over the other.

: NavStick's Advantages over NavMenu. As Figure 6 shows, participants preferred NavStick for directional tasks (Tasks 1 & 3) and the video game task (Task 6). Figure 7 shows participants' average distance from ideal answers for directional navigation tasks (Tasks 1 & 3) using NavMenu and NavStick. An ideal answer is an item that is adjacent to the item named in the prompt ("Name an item that is close to [item X]. "). The distances are measured in meters. They act as an objective measure of the "accuracy" of users' mental maps after using NavMenu and NavStick.

For directional naïve search (Task 3), we found a significant main effect of navigation tool (NavMenu vs. NavStick); answers given using NavStick ( = 2.7, = 0.7) were closer to the item in the prompt than ones given using NavMenu ( = 3.8, = 1.0) ( < 0.05). Figure 8 shows participants' perceived quality of their mental maps after exploring with NavMenu and NavStick. Participants felt that they were able to form significantly better mental maps with NavStick ( = 14.5, = 5.2) than with NavMenu ( = 9.9, = 7.8) ( < 0.05). This finding is consistent with our objective finding in Figure 7 . Figure 9 shows the amount of fun participants reported having while playing the Terraformers level (Task 6) using NavMenu and NavStick. Participants saw NavStick ( = 14.1, = 6.5) as significantly more fun to use than NavMenu ( = 6.7, = 7.3) ( < 0.05).

Since NavMenu's list-based format does not communicate information about directionality, participants struggled to understand where items were positioned relative to one another using Nav-Menu. Figure 10 shows how participants rated NavMenu and Nav-Stick in terms of frustration. We found that NavStick ( = 5.8, = 3.9) was significantly less frustrating to use than NavMenu ( = 7.6, = 6.5) for directional navigation tasks ( < 0.05). Verbal sentiments indicated that some players found NavMenu to be mentally demanding and confusing: "I'm so lost! I think some items are close to each other, but I keep scrolling and so it's confusing. " -P3 Indeed, participants often had to take multiple passes through the NavMenu in order to keep track of the items present. We originally expected NavMenu to be significantly less frustrating and effortful than NavStick for the "non-directional" tasks (Tasks 2 & 4) given that it specializes in listing out what items are present in a space. However, we did not find any such relationship between NavStick and NavMenu in either effort (NS: = 5. Indeed, Figure 6 shows that, when forced to choose between using NavStick and NavMenu for the "non-directional" tasks (Tasks 2 & 4) -ones in which it was only important to understand what was in the aisle and not where they were located -participants mostly chose NavMenu.

Furthermore, NavMenu presented POIs in discrete units accessible one-by-one with the D-pad. With NavStick, participants had to contend with small NavPie slices from items that were further away. Several participants expressed their preference for the NavMenu Figure 11 : A still shot from The Explorer, our prototype 3D adventure game. The main character ("Ellen") is aiming a laser gun at an enemy (a "Chomper"). There are three Chompers in this view: one to the left, another to the right, and one in the middle under the crosshairs. An unlighted (pink) checkpoint crystal can be seen in the distance toward the right.

for this reason, revealing a need to somehow optimize NavStick to handle thin slices better:

"What I liked about [the NavMenu] is the use of the D-pad. It made things easier to navigate with instead of using [NavStick] in a way because with [NavStick], you have to be really, really careful. " -P10

NavStick's ultimate goal is to make existing 3D video games -ones that sighted players play -more blind-accessible so that VIPs are not restricted to highly simplified games made just for them. Our core belief is that being able to "look around, " as NavStick affords, is integral to having an authentic experience while playing 3D games. In Study 2, we complement Study 1 with an exploratory study where we use The Explorer, a representative 3D adventure game we created, to understand NavStick's potential in making existing 3D games more blind-accessible. We investigate the following research questions in Study 2: Explorer is a 3D video game with graphics, akin to what sighted players play, that we are simply "attaching" NavStick to in order to make it blind-accessible. The Explorer is set within a fantasy world that the main character ("Ellen") must traverse while using a laser gun to defeat enemies. Figures 1, 3, and 11 show screenshots from The Explorer.

The Explorer consists of eight segments (short levels). In each segment, players must traverse the area from a start checkpoint to an end checkpoint. We designed the eight segments to serve as a vehicle for investigating RQ2.2; namely, to investigate the extent that NavStick works for complex situations found in mainstream 3D games. To this end, we identified three environmental characteristics that can potentially present a challenge for NavStickenemy movement, time pressure, and environmental occlusionsand designed the eight segments to feature every possible on/off combination (2 3 = 8 in total) of these characteristics. Table 2 summarizes the segments. Our supplementary material includes a visual overview of the segments with screenshots (see VisualSegmentOverview.pdf).

Segments 2, 5, 6, and 8 feature enemy movement. The Explorer features two types of enemies: "Roaming Chompers" and "Stationary Chompers. " The Roaming Chomper continuously walks backand-forth along a straight line, making a squeaking sound when it turns around. The Stationary Chomper stays at one position and does not move around. Segments with enemy movement feature one or more Roaming Chompers, and players must continuously track the Roaming Chompers' positions to aim their laser gun correctly.

Segments 3, 5, 7, and 8 feature time pressure. In these segments, we impose a pre-defined time limit, specified in Table 2 . During these segments, soft clock-ticking sounds play in the background, and a voice announces the time remaining at regular intervals. If the player does not complete the segment in time, we restart it and allow them to try again. We allow a maximum of three tries before moving on. Segments 4, 6, 7, and 8 feature environmental occlusions. In these segments, we placed trees in the environment that occlude NavStick's line-of-sight to POIs. Participants may need to move around the trees if they are not able to find a line-of-sight target using NavStick.

Players move Ellen with the left thumbstick. Tilting it forward and backward will allow the character to move forward and backward. Tilting it left and right will rotate the character left and right. NavStick is mapped to the right thumbstick as described previously.

The player's primary weapon is a laser gun, which they can aim and fire at enemies. The controls for aiming and firing are similar to those in mainstream action-adventure games such as Uncharted 4 [12] and Borderlands 3 [38] . Holding the left trigger (LT) enters "aiming mode," in which the left thumbstick aims the gun rather than moves Ellen. Pressing the right trigger (RT) while in aiming mode fires the gun. Only one hit is needed to defeat each enemy. Enemies do not attack the player.

NavStick considers enemies, checkpoints, pressure pads, and doors as POIs. All sounds in the game are spatialized, except for the time pressure sound effects. The game plays additional spatialized sound effects when the player defeats enemies, shoots a projectile and misses, or reaches the end checkpoint of a segment.

We performed initial pilot tests with three research team members and four external individuals. One person was blind, and the others were sighted but wore blindfolds. These tests comprised an initial test of the study's procedure and revealed that players needed supplementary tools to help them traverse The Explorer's segments. To this end, we added several quality-of-life enhancements to the game beyond NavStick. These additions are not intended to replace NavStick but rather supplement it. (It is important to note that Study 1 did not include these enhancements.)

Quest display: Like other adventure games, The Explorer features quest objectives such as reaching a target checkpoint that players must reach in order to proceed. VIPs can easily get lost without any indication of where they should be going. The quest display works to rectify this. It plays a short, spatialized tone every time the player turns 15°. The tone originates from a fixed distance relative to the player and from the direction of the current target checkpoint. This tool serves two purposes: to give players a general sense of where their current target is located, and to help them understand their current orientation with respect to the target.

Automatic vertical aim: NavStick in its current form cannot scrub vertically, and its binding to the right thumbstick removes the player's ability to move the camera up and down. Because of this, enemies that are at a higher or lower elevation than the player cannot be hit by the laser gun since the laser travels straight ahead. We, thus, implemented an automatic vertical aiming functionality that automatically controls the vertical component of the player's aim. Section 7.2 describes how future versions of NavStick can generalize to spherical surveying.

Auto-turn: Rotating Ellen to face a scrubbed target's direction can be quite challenging, so we implemented an "auto-turn" feature. Players can press the left bumper (LB) while scrubbing to automatically turn Ellen toward the scrubbed target. The scrubbed target will be at the 12 o'clock position once the auto-turn is complete.

Proximity notifier: VIPs cannot see our game's graphics, and thus cannot tell whether or not they are near a POI. We implemented a "proximity notification" feature that automatically plays appropriate ambient sound effects while a player is physically near each type of POI.

Increased collider sizes: We doubled the collider dimensions around POIs to make them easier to target with NavStick.

Footstep feedback: Visually impaired pilot participants requested acoustic feedback on whether or not they are moving. As such, we added velocity-dependent footstep sounds that play as Ellen moves through the world.

NavStick post-announcement tone: When a player scrubs to a POI (an enemy, checkpoint, pressure pad, or door), NavStick announces the name of the object and then plays a low-pitched (440 Hz) sustained sine tone as long as the player keeps pointing at the object. This allows players to continuously track objects such as Roaming Chompers.

Non-POI tone: By default, NavStick does not say anything if it is not being pointed at a POI. However, this may cause the world to sound "empty. " Thus, if a player scrubs in a direction where the first object in that direction is not a known POI (e.g., a wall or tree), NavStick will immediately play a higher-pitched (1,320 Hz) sustained sine tone originating from the point of the obstruction in space (with no verbal announcement).

NavStick's goal is to give VIPs the ability to look around within a game world, giving them more of the type of experience that sighted players have when playing 3D games. In our initial pilot tests of NavStick (Section 6.1.1), we found that NavStick by itself cannot make a 3D game entirely blind-accessible -there are times in which players need other mechanics as well, such as an indicator showing players what general direction their goal is (in our case, the quest display) and a means of aiming vertically (in our case, automatic vertical aim). These quality-of-life enhancements may not be available in every game, preventing our results from being applicable to any 3D game without some degree of modification. Nevertheless, by integrating these enhancements, we were able to learn about what NavStick enables and what it does not enable by itself, and in Study 2, we explore what NavStick can enable VIPs to do when it is complemented by these enhancements.

Study 2 involved seven participants, six of whom described themselves as having no usable vision and one (P1) who described themselves as "otherwise visually impaired." Five had participated in Figure 12 : Player movement trails within Segment 8. Players start at the glowing checkpoint (far left) and must reach the glowing pressure pad (lower-right) to open a large gate (partially visible on far right). Each color represents a different player's path. We can see that NavStick afforded these players a high degree of agency in how they played this segment. Trees are shown as black circles for clarity; these act as occluders and cannot be walked or seen through. White line segments represent the seven Roaming Chompers' patrol paths. No two players defeated the same set of Chompers. Study 1. All participants were male and have had their vision impairments from birth. Five participants (P1-5) were 18-25 years old; two participants (P6 and P7) were 26-35 years old. Two participants (P4 and P7) reported minor hearing loss in their right ear. All but one participant (P4) reported themselves to be experienced with video games (4+ on a 5-point Likert scale). We recruited these participants in the same way as Study 1.

We began by administering a pre-study questionnaire requesting demographic information. We then guided players through three trial levels as we explained the game's mechanics. Trial Level 1 introduced pressure pads, doors, and checkpoints as well as NavStick itself. Trial Level 2 introduced a Stationary Chomper and the laser gun. Trial Level 3 introduced a Roaming Chomper.

We began each of the eight segments by explaining the segment's objective ( Table 2) . After the player completed each segment, we paused the game and administered a post-segment questionnaire. The first part gauged participants' subjective measures of effort, frustration, fun, and impression in the segment using 20-point Likert scales similar to a NASA TLX form [24] . In the second part, we revealed the environmental characteristic(s) that the segment possessed (the checkmarks in Table 2 ) and asked participants to reflect on how NavStick performed with respect to those characteristics.

At the end of the session, we administered a post-study questionnaire and a semi-structured interview to ask participants to reflect on what they liked and disliked about NavStick as well as their impression of the game itself.

Below, we report the main takeaways from Study 2 in answering our research questions: learning the ways in which NavStick can make 3D games more blind-accessible (RQ2.1), and learning how well NavStick performs in the face of enemy movement, time pressure, and occlusion (RQ2.2).

We analyzed our qualitative results (participant quotes) identically to Study 1. Any quantitative data (Likert scale results) were analyzed as described in the following subsections.

6.4.1 RQ 2.1 Results: Making 3D Games More Blind-Accessible. All participants were able to complete all eight segments of The Explorer using NavStick to look around. In the timed segments, participants ran out of time in only three instances: P3 and P4 in Segment 8 and P5 in Segment 5. P3 and P4 did not expect the Segment 8 pressure pad to be close to the gate (they were far away from each other in Segment 4); P5 lost track of the Segment 5 checkpoint and forgot about using the quest display to reorient themselves. However, all three participants were able to complete the segments on their second try.

In addition to making The Explorer merely possible to play, we found evidence that NavStick afforded players agency; that is, letting them pursue their own strategies while playing. In-game analytics data from Segments 7 and 8 show that no two participants defeated the same batch of Chompers in either segment. Furthermore, no participant defeated all of the Chompers in these segments; they defeated some subset and decided to move on. Figure 12 shows the paths that participants took within Segment 8 -note how participants took many unique paths to get to the target checkpoint. Also note how the trails become random around the pressure pad -once players hit the pressure pad, many of them freely roamed around it while the gate opened. Some participants noted this freedom: The last sentence of this quote sheds light on how inequitable the current world of video games is for VIPs. This type of gameplay is already present within video games made for sighted players. However, due to the gap between games for sighted players and blind-accessible games, our participants were not used to playing a game that promotes such a high level of freedom and requires such a high level of strategy.

Five of the seven participants wished that NavStick was available in existing games, explicitly mentioning The Last of Us Part II [13] and Halo 2 [5] as candidates for NavStick. The former is especially interesting given that it has many accessibility features [14] and has received many accolades for them [31, 43] . Participants, however, agreed that NavStick offers much more freedom than the guidance-based tools that The Last of Us Part II provides. They added that NavStick promoted "seeing" in the form of providing an understanding of the environment:

"I mean, if you stop to think about it, in most games, if you've ever played first-person, the whole point is that you can move the camera. And to me, [NavStick] was the equivalent of a camera. You're able to see objects around you. You're able to know where they are. " -P6

By providing the equivalent of a camera for VIPs to look around, NavStick was able to provide a sense of freedom and environmental understanding that VIPs do not currently experience in existing games. We found this quote interesting because NavStick repurposes the game controller's right thumbstick, which is used by sighted players to control the game's camera.

Lastly, four particpants reported having fun in a way that they are not used to in existing games:

"It was very enjoyable. It's very fun to look around your environment, figure out where you're going, figure out what you have to do, and to blow stuff up. " -P2

This quote is insightful because it aligns with established definitions of player agency from the game development community (using the synonym intention) [6] [7] [8] and from the computer science community [47] . Specifically, it highlights how NavStick enables the three activities critical to agency: understanding the environment, developing a plan of action, and attempting to execute that plan.

Here, we present our findings on how well NavStick performs in the face of enemy movement, time pressure, and environmental occlusion. Figure 13 shows a box plot of TLX frustration scores across segments that featured each of these characteristics. Our analysis revealed that participants found segments with occlusion ( = 5.8, = 5.0) to be the most frustrating overall, followed by segments with time pressure ( = 4.3, = 3.9), enemy movement ( = 3.4, = 3.3), and none of the three characteristics ( = 2.3, = 2.1), in that order. This result was affirmed by participants' comments during our semi-structured interviews, in which most remarked that occlusion was the most difficult factor to deal with when using NavStick. NavStick communicates occlusions by rendering a high-pitched sine tone at the raycast's point of impact whenever that point is not an actual target. Participants remarked that this was helpful, but five out of seven participants felt that they needed more information than this, specifically a description of the obstruction and its dimensions: "This is what Navstick does especially well. It allows me to really quickly scan around the area. The time restriction made me really need the ability to scan around quickly. " -P7

The other three participants, however, felt that the speed required in a timed situation was incompatible with NavStick's behavior of truncating speech when moving off a slice:

"The problem with NavStick is that when I move just a little bit, the voice indicating what is in front of me stops, [and] I have to re-move the stick again. So it has a pretty small target area [and] it made it pretty difficult to use because of that. " -P5

As mentioned previously, this aspect of NavStick follows ear-Pod's design [48] . Its authors found that truncated audio playback gave users a much stronger feeling of responsiveness and was less confusing to use than simultaneous playback (where announcements are always played in their entirety). Our findings here, however, suggest that it may be necessary to allow for some "residual" information about the previous slice to remain after the player scrubs away from it, which is an expansion of the findings from earPod.

Participants also had differing opinions with respect to enemy movement. Roaming Chompers emit spatialized footstep sounds that are audible when the player is sufficiently close, but participants were split on whether NavStick was needed if enemies' footsteps were already spatially audible. Some believed that NavStick offered a useful supplement to the sound effects' spatialized audio: Others, however, felt that NavStick was unnecessary in these situations, since these players could tell which way the Roaming Chompers were moving from their footstep sounds alone:

"The Chomper already makes noise, so [using NavStick is] just extra effort and brain processing power. It's like I'm trying to align NavStick with something I can already hear and there's not really a benefit to it. " -P7

The difference between these groups may be that the latter group is already accustomed to discovering and monitoring POIs (such as enemies) purely using sound. P4, P5, and P7, who comprise the latter group, went on to describe their experiences with existing audio-based games such as Swamp [27] and BK3 [19] . These games announce the presence of objects by playing relevant ambient sound effects when the player gets near them.

We should note, however, that NavStick gives players the choice of whether to use it or not. That is, players could use it as much or as little as they wanted. Those who want or need to use NavStick to locate objects can do so, and those who prefer to locate objects from their ambient sound effects alone can simply not use NavStick for that purpose. NavStick's on-demand nature means that it will stay out of the way if the player does not wish to use it and is ready for whenever they do require it.

The findings from our two studies yielded several implications for future blind-accessible games. Here, we reflect on the findings and propose ideas for next steps.

In forced rankings within Study 1 (see Figure 6 ), participants strongly preferred NavStick for navigation tasks that rely on having a mental map of the surrounding environment (both "directional" tasks), as well as for navigation within video games (the Terraformers task). However, for other navigation tasks, their preference was split between NavStick and NavMenu. Players viewed the two tools as working hand-in-hand with each other. The ability to be efficiently guided toward POIs in the environment (as represented by Nav-Menu) and the ability to look around (as represented by NavStick) ultimately serve different purposes and should work in tandem with one another. As a result, we believe that future blind-accessible games and virtual navigation tools should offer both the ability to look around (as offered by NavStick) and guided navigation (as offered by existing audio navigation systems) so that users can choose which they want to use at any given moment.

In its current state, NavStick only operates in the user's horizontal direction. To facilitate exploration in three dimensions, future versions of NavStick should allow users to scrub vertically, possibly by using a game controller's gyroscope to control pitch. Games such as Splatoon [16] and The Legend of Zelda: Breath of the Wild [17] already use gyroscopic controls for aiming vertically. This would allow a player to, for example, scrub to high or low portions of grocery store shelves (like the ones in Study 1) or even scrub for objects on top of hills or other high points (within environments similar to The Explorer). It may also facilitate looking around in all directions within immersive mixed reality environments. Although we developed NavStick for virtual environments, we believe that this new approach could also apply to physically navigating in the real world. It is important to note that the real world brings new challenges, such as accuracy, precise tracking, and guaranteeing a user's safety. Still, there is promise in adding NavStick's scrubbing-based interaction technique to existing physical devices beyond game controllers to facilitate real-world navigation. Future researchers may consider adding a thumbstick to a white cane, for example, or letting VIPs scrub their finger on the perimeter of a smartwatch face so they can survey and explore the real world more independently.

7.3 Blind-accessible 3D games are possible, but more perceptual challenges await

Both studies showed that NavStick has the potential to make 3D video games more blind-accessible and offer VIPs a more fun and fulfilling experience in gaming. In Study 1, we found that NavStick allowed players a better sense of the space they were in compared to status quo tools for VIPs. In Study 2, all participants completed all segments of The Explorer, a representative 3D adventure game created for sighted players. To some of our participants, completing a game made for sighted players came as a surprise and felt different from blind-accessible games that they have played before. Indeed, one said that this was the first game where they could "just walk around and blow stuff up." These results are encouraging for the prospect of making mainstream 3D games that sighted players play -games such as The Legend of Zelda -blind-accessible in the future. We believe from our findings, however, that two significant perceptual challenges must still be overcome to give VIPs a fully equivalent experience playing video games. The first challenge is communicating the layouts of environments and the obstructions within them, and the second challenge is to communicate more of the environment's aesthetics to players. Both are related to participants' deep desire to "see" even more.

Regarding the first challenge, many participants desired some form of "x-ray vision" -the ability to see the world beyond their immediate line-of-sight. At its core, NavStick revolves around the principle of replicating line-of-sight, but sighted players have access to a world map or minimap that indicates the shape of the area they are in and routes to adjacent areas. As a result, an important future step for blind-accessible 3D game worlds will be to give players an "overview" sense of a space beyond what they can survey with line-of-sight, especially if future systems allowed players to survey portions of the game world on demand with these techniques.

Regarding the second challenge, participants wanted to know what the game environments looked like beyond merely knowing what objects they contained. 3D game worlds are often richly detailed and fantastical, and The Explorer was no exception. The game design community characterizes sensation as a major type of fun that games provide [26, 28] . We believe that allowing VIPs to appreciate the aesthetics of a 3D world is crucial to making 3D games equivalently accessible to VIPs. Sensory substitution techniques [23] and automatic image captioning techniques [15] are two promising ways of communicating aesthetics since they can translate images to sound or textual descriptions. Future systems might combine one of these approaches with NavStick-style scrubbing to give players aesthetic information in a self-directed way, allowing them to answer the question, "What does the environment in this direction look like?"

In this work, we introduce NavStick, an audio-based tool for looking around within virtual environments. Its aim is to make 3D video games -ones designed for sighted players -more blind-accessible by fostering agency in navigation. NavStick repurposes a game controller's thumbstick to allow VIPs to survey what is around them via line-of-sight.

Through two studies, we investigated NavStick with respect to status quo methods for surveying a game world (Study 1) and explored NavStick's potential in making existing 3D video games more blind-accessible (Study 2). We found that VIPs were able to form more accurate mental maps of their environment with NavStick than with menu-based surveying; we found that players wanted the two types of surveying to co-exist; we found that NavStick offers players a sense of agency not found in current blind-accessible games; and we revealed new challenges that must be overcome to make 3D games fully accessible to VIPs.

As with many studies that involve VIPs, we had a relatively low number of participants. Although we found several significant results through our surveys and interviews, there may be more insights that our user studies could not reveal regarding how Nav-Stick could better integrate into a video game or how it compares to menu-based surveying tools. In addition, while we are proud of the fact that our research team included VIPs as co-designers throughout the process, we regret not being able to recruit a more diverse group of visually impaired study participants.

We hope that, just as sighted people are able to look around and explore complex and rich game environments on their own terms, so too can visually impaired people. Whether in a single-player adventure game or a multiplayer role-playing game, it is our hope that visually impaired players can one day play the same games as sighted players in an equivalent fashion.

NavCog: A Navigational Cognitive Assistant for the Blind

Playing Blind: Revealing the World of Gamers with Visual Impairment

Introducing the Gamer Information-Control Framework: Enabling Access to Digital Games for People with Visual Impairment

Towards Generalised Accessibility of Computer Games

Halo 2. Microsoft Game Studios

Formal Abstract Design Tools. Gamasutra

Formal Abstract Design Tools

Formal Abstract Design Tools

Virtual navigation tested on a mobile app is predictive of real-world wayfinding navigation performance

Wayfinding strategies and behaviors in large virtual worlds

Indoor Navigation Challenges for Visually Impaired People

Uncharted 4: A Thief's End. Sony Interactive Entertainment

Naughty Dog. 2020. The Last of Us Part II

Naughty Dog and Sony Interactive Entertainment. 2020. The Last of Us Part II: Accessibility Features

Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge

The Legend of Zelda: Breath of the Wild

Indoor Human Navigation Systems: A Survey

Wayfinding Behavior: Cognitive Mapping and Other Spatial Processes

Google Cloud Text-to-Speech

The benefits of playing video games

I Always Wanted to See the Night Sky": Blind User Preferences for Sensory Substitution Devices

Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research

AudioGPS: Spatial Audio Navigation with a Minimal Attention Interface

MDA: A Formal Approach to Game Design and Game Research

Com: The Collected Game Design Rants of Marc

Audible Mapper & ShadowRine: Development of Map Editor Using only Sound in Accessible Game for Blind Users, and Accessible Action RPG for Visually Impaired Gamers

Last of Us Part II: Is this the most accessible game ever? BBC News

Toward Self-Directed Navigation for People with Visual Impairments

An empirical study of issues and barriers to mainstream video game accessibility

NavCog3: An Evaluation of a Smartphone-Based Blind Indoor Navigation Assistant with Semantic Features in a Large-Scale Environment

The RAD: Making Racing Games Equivalently Accessible to People Who Are Blind

Gearbox Software. 2019. Borderlands 3. 2K Games

Unity Technologies

PowerUp: an accessible virtual world

Unity Technologies. 2020. Unity. Unity Technologies

Valve Software. 2019. Steam Audio

2021. 2020 Video Game Accessibility Awards Announced. GameSpot

Game Accessibility Case Study: Terraformers -a Real-Time 3d

Graphic Game

SWAN: System for Wearable Audio Navigation

Supporting Spatial Awareness and Independent Wayfinding for Pedestrians with Visual Impairments

Game accessibility: a survey

earPod: Eyes-Free Menu Selection Using Touch Input and Reactive Audio Feedback

We would like to thank Brian (Shao-en) Ma and Emily Li for their assistance in preparing for and executing Study 2. We would also like to extend our sincere gratitude toward our study participants for their cooperation and for providing us with helpful feedback. This work was supported in part by a Junior Faculty Grant from Columbia University's Office of the Provost.