key: cord-0548631-qh0ps1nk
authors: Zhao, Xuan; Fan, Mingming; Han, Teng
title: "I Don't Want People to Look At Me Differently": Designing User-Defined Above-the-Neck Gestures for People with Upper Body Motor Impairments
date: 2022-02-12
journal: nan
DOI: nan
sha: 9503752baa96c243453f0181e386548e00b379fe
doc_id: 548631
cord_uid: qh0ps1nk

Recent research proposed eyelid gestures for people with upper-body motor impairments (UMI) to interact with smartphones without finger touch. However, such eyelid gestures were designed by researchers. It remains unknown what eyelid gestures people with UMI would want and be able to perform. Moreover, other above-the-neck body parts (e.g., mouth, head) could be used to form more gestures. We conducted a user study in which 17 people with UMI designed above-the-neck gestures for 26 common commands on smartphones. We collected a total of 442 user-defined gestures involving the eyes, the mouth, and the head. Participants were more likely to make gestures with their eyes and preferred gestures that were simple, easy-to-remember, and less likely to draw attention from others. We further conducted a survey (N=24) to validate the usability and acceptance of these user-defined gestures. Results show that user-defined gestures were acceptable to both people with and without motor impairments.

People with upper body motor impairments have difficulty touching an on-screen target accurately with fingers due to tremors, muscular dystrophy, or loss of arms [22, 32] . They were found to have difficulty entering and correcting texts, grabbing and lifting the phone, making multi-touch input, pressing physical buttons, and so on, especially outside of home [23] . While voice-based interfaces, such as Siri [9] , could be an alternative method, they suffer from low input speed and accuracy [9, 23] and may raise privacy and social acceptance concerns when used in public [21, 23] .

Recently researchers proposed eyelid gestures for people with motor impairments to subtly interact with mobile devices without finger touch or drawing others' attention [10, 11] . These eyelid gestures, though followed design principles, were created without involving people with motor impairments in the design process. Consequently, it remains questionable whether these gestures are the ones people with motor impairments preferred in the first place.

Indeed, the participants with motor impairments in their study [10] also suggested other eyelid gestures and expressed a desire to design their own eyelid gestures. Thus, there is a need to involve people with motor impairments to design gestures that will be used by them. In the meantime, prior research demonstrated that allowing users to define their preferred gestures would uncover more representative and preferred gestures [26, 27, 36] .

Motivated by this need and prior success of designing user-defined gestures in other contexts, we sought to engage people with motor impairments to design eyelid gestures they prefer. What's more, as recent research demonstrated the promise of gaze and head pose for hands-free interaction in addition to eyelid gestures [17, 24, 29, 37] , we extended the design space of user-defined gestures by inviting people with motor impairments to design above-the-neck gestures that include eyelids, gaze, mouth, and head.

We conducted an online user study, in which 17 participants with various upper body motor impairments designed above-the-neck gestures to complete 26 tasks that were commonly performed on mobile devices. These tasks included general commands (e.g., tap and swipe), app-related commands (e.g., open an app or a container with the app), and physical button-related commands (e.g., volume up and down). During the study, participants first watched a video clip explaining each task and its effect on a smartphone and then had time to design and perform an above-the-neck gesture.

Afterward, participants rated the goodness, ease, and social acceptance of the gestures they just created. Finally, they were interviewed to provide feedback on the gestures. We collected a total of 442 user-defined gestures. Our results

show that participants preferred to use gestures that were simple, easy-to-remember, and less attention-demanding.

Based on all the gestures obtained and the rating and frequency of use of each gesture, we assigned each command the most appropriate gesture or gesture.

To validate the usability and acceptance of these user-defined gestures, we conducted an online survey that asked participants to select the most appropriate gesture from three candidates to complete each of the that we did previously in the user study on mobile devices. These candidate options were chosen from the most frequently mentioned gestures from the user study. Results show that our gesture set was well accepted and recognized by people with and without motor impairments. In sum, we make the following contributions in this work:

• We present a set of user-defined above-the-neck gestures based on the gestures designed by people with upper body motor impairments to complete common interactions on mobile devices; • We show that these user-defined above-the-neck gestures are largely preferred by people with and without motor impairment.

Our work is informed by prior work on interaction techniques for people with motor impairments and user-defined gesture designs.

Brain-computer interfaces (BCIs) sense brain signals for people with motor impairments to communicate with the environment or control computer systems without using hands (e.g., [5, 18] ). However, BCIs often need long periods of training for users to control their brain rhythms well [25] , and people with motor impairments were reported to be concerned about fatigue, concentration, and also social acceptance [4, 30] .

Gesture-based interactions have been investigated as an alternative approach. Ascari et al. [3] proposed two machine learning approaches to recognize hand gestures for people with motor impairments to interact with computers. However, these approaches are not feasible for people with upper body impairments who could not use their hands freely. To overcome the limitations of body and hand gestures for people with upper body impairments, researchers investigated eye-based interactions.

Among all eye-based gestures, blink was probably the most widely studied for people with motor impairments.

Earlier work used EOG sensors to detect blink to trigger computer commands [14] . The duration of blink was also utilized as additional input information. For example, a long blink was detected and used to stop a moving onscreen target [12] . What's more, blink was also used along with eye movements to trigger mouse click [16] . Blink was also used along with the head motion. For example, the frequency of blink combined with head motion was used to infer five activities, including reading, talking, watching TV, math problem solving, and sawing [13] .

In addition to blink, wink was used for people with motor impairments. Shaw et al. constructed a prototype to detect the open and close states of each eye and used such information to infer three simple eyelid gestures: blink, wink the left eye, and wink the right eye [28] . Similarly, Zhang et al. proposed an approach to combine blinks and winks with gaze direction to type characters [38] . Recently, Fan et al. took a step further to investigate the design space of eyelid gestures and proposed an algorithm to detect nine eyelid gestures on smartphones for people with motor impairments [10, 20] . These eyelid gestures were designed based on eyelid states, in which two eyelids could be in, and the possible parameters that humans can control, such as the duration of closing or opening an eyelid and the sequence. Although the design of eyelid gestures followed a set of design principles, these gestures were designed by the researchers who did not have motor impairments themselves, and the design process did not involve people with motor impairments in the loop. Consequently, it remains unknown whether these eyelid gestures were the ones that people with motor impairments preferred in the first place. In fact, participants with motor impairments in their study could not well perform some eyelid gestures well, proposed new eyelid gestures, and expressed the desire to design their own gestures. Motivated by this need, we seek to explore user-defined eyelid gestures that people with motor impairments would want to create and use.

Other body parts, such as the head, have also been used to extend the interaction for people with motor impairments.

Kyto et al. [17] compared eyes and head-based interaction techniques for wearable AR and found that the head-based interactions caused less error than eye-based ones. What's more, they found the combination of eye and head resulted in a faster selection time. Sidenmark and Gellersen [29] studied the coordination of eye gaze and head movement and found this approach was preferred by the majority of the participants because they felt better control and less distracted.

Similarly, gaze and head turn were also combined to facilitate the control of onscreen targets [24] . Inspired by this line of work that shows the advantage of combining head motion with eye motions, we extend our exploration to include above-the-neck body parts, including both eyes, head and mouth, to allow people with motor impairments to better design a richer set of user-defined gestures.

User-defined gestures have been investigated by researchers in various contexts [7, 8, 15, 19, 26, 27, 31, 34, 36] . Wobbrock et al. [36] studied user-defined gestures for multi-touch surface computing, such as tabletop. They investigated what kind of hand gestures non-technical users would like to create and use by asking the participants to create gestures for 27 referents with one hand and two hands. Wobbrock et al. [36] also designed gestures for the 27 referents on their own and compared the gestures created by them with the ones created by the users. They found that they created far fewer gestures than participants, and many of the gestures they created were never tried by users. Kurdyukova et al. [15] studied the user-defined iPad gestures to transfer data between two displays, including multi-touch gestures, spatial gestures, and direct contact gestures. Piumsomboon et al. [26] worked on user-defined gestures for AR and asked people to perform hand gestures with a tabletop AR setting while Lee et al. [19] utilized an augmented virtual mirror interface as a public information display. Dong et al. [7, 8] worked on the user-defined surface and motion gestures for mobile AR applications. In the work of Ruiz et al. [27] , they utilized the user-defined method to develop the motion gesture set for mobile interaction. Weidner and Broll [34] proposed user-defined hand gestures for interacting with in-car user interfaces, and Troiano et al. [31] presented for interacting with elastic, deformable displays. These user-defined methods motivated our research. Specifically, our research investigates what upper-neck gestures people with motor impairments would like to create and how they would want to use such gestures to accomplish tasks on their touch-screen mobile devices.

The goal of this IRB-approved study was to gather user-defined above-the-neck gestures for common tasks on mobile devices from people with motor impairments and then identify the common user-defined gestures for each task. 

We recruited seventeen (N=17) participants through online contact with a disability organization for the study. Table 1 shows the demographic information. Twelve were males and five were females. Their average age was 29 years old ( = 6). All participants had some forms of motor impairments that affected their use of mobile phones. Ten participants had arm or hand problems. Specifically, eight had the loss or injury of their limbs, six had cerebral palsy who had shaky hands and difficulty controlling their hand movements. The remaining three had spinal cord injuries, two of whom needed to use a wheelchair, and one did not have hand sensation. Seven had their legs amputated and needed prosthetics or crutches. Some of them had difficulties speaking clearly or fluently due to the influence of cerebral palsy, but it did not affect our user study. None of them used upper-neck gestures to control devices prior to the study. The participants were compensated for the study.

Firstly, we studied the instructions on the official websites of iOS and Android 1 to learn about the commands and corresponding gestures designed for today's touchscreen smartphones. Moreover, we drew inspiration from recent work [10, 11] for the commands that were supported by eyelid gestures designed for people with motor impairments to interact with touchscreen devices. In the end, we identified 26 commands commonly used for smartphone interactions.

Based on their similarities, we further clustered these commands into three groups. Group 3 included four Physical Button-related commands, which were Volume Up, Volume Down, and Screenshot.

The general commands were obtained from mobile phone systems (iOS & Android), and the app-related commands were inspired by a recent study [10] , which proposed commands such as switching between apps, switching between tabs in an app, and switching between containers in a tab. The four physical button-based commands were also inspired by the commands supported by iOS and Android and were related to common button functions, such as turning the volume up & down, taking screenshots, and locking the screen. Figure 1 shows the study procedure. After answering the background questionnaire, participants were asked to watch a short video clip for a command. We created a short video clip to show each command and its effect on a smartphone. Figure 2 shows example video clip frames for the command Zoom In, Next Container, and Volume Up.

Showing video clips instead of explaining verbally was to ensure the consistent presentation of tasks to participants.

After watching the video clip, participants would create an above-the-neck gesture for it and perform the gesture to the moderator. To reduce any ordering effects, the task videos for the general commands and physical button-related commands were presented to participants in random order. Because there was a logical order among the commands in the app-related group, we kept the order of the tasks in this group to avoid confusions. During this process, we asked participants to think aloud so that the moderator could better monitor the design process. After performing the user-defined gesture, they were asked to rate the goodness, ease, and social acceptance of the gestures for the command using 7-point like-scale questions. Then, participants repeated this process until they created gestures for all commands.

To reduce the gesture conflicts happening during the study, we asked participants to design different gestures for each command within the same group (i.e., three groups for the general, app-related, and physical-button related commands). As for the commands that were not in the same group, we allowed the participants to perform the same gesture. However, due to a large number of commands, some participants might forget their previous gestures. Thus, the moderator monitored the gestures already created, and if she found a conflict gesture was proposed, she would remind the participants to change to a different gesture for either the current one or the earlier one that was conflicted with. In addition, participants were also allowed to change their minds if they later wanted to go back and change a previous one.

All study sessions were conducted remotely through a video conference platform in order to comply with the COVID-19 social distance requirements, and the whole process was video recorded. In total, we collected 442 user-defined above-the-neck gestures (17 participants x 26 commands).

Before we determined the final user-defined gestures for the commands, we took a step to understand the perceived complexity of the commands. To do so, we calculated the conceptual complexity of each command. Conceptual complexity of a command was a concept widely used in prior works (e.g., Wobbrock et al.'s work [36] , Arefin Shimo et al.'s work [2] , and Dingler et al.'s work [6] ), which measures the perceived difficulty of the command from HCI researchers' points of view. For example, Single Tap (e.g., tapping a button on the touchscreen) is a command that we, as HCI researchers, believed could be achieved easily with a tap and thus has low conceptual complexity. In contrast, Screenshot (e.g., taking a screenshot) requires more fingers and steps than a single tap, so it has a relatively higher conceptual complexity. To determine the conceptual complexity for each command, two HCI researchers rated the difficulty of completing each command on a 5-point Likert scale (1: the relatively easiest, 5: the relatively most difficult) independently to avoid influencing the other person's point of view. A score of 5 means that it was perceived by the researcher as the most difficult among the 26 commands. The scores of the two researchers were similar for most commands with only a few that had relatively bigger differences. Finally, we calculated the average of the scores assigned by the two researchers to get the conceptual complexity of the command. Figure 3 shows the conceptual complexity for each command.

We followed the analysis methods of prior user-defined gesture design papers (e.g., [36] ) to obtained the gesture taxonomy, the gesture agreement score, the participants' ratings of the gestures performed, and the final user-defined gesture set. We also analyzed the participants' interviews to understand the design rationales.

We collected 17×26=442 gestures for all 26 commands and classified them according to different body parts, the eyes, the head, and the mouth.

Gesture Categories. We grouped the gestures into seven categories based on the body parts involved. These seven categories included a single body part and the combinations of different body parts: only eyes, only head, only Table 2 shows the taxonomy of the user-defined above-the-neck gestures.

The only eyes category includes five basic types of eye gestures: blink, gaze, eye movement, eye size, eyebrow and the combination of these basic eye gestures. Blinks include single and double blinks, as well as different numbers of blinks. Gaze is the eyes on the screen, which may differ in the length of time. In addition to moving the eyes up and down, left and right, eye rotating and cross-eye are also considered eye movement. The wide opening, closing, and squinting actions are regarded as the scope of eye size. In addition, we counted eyebrow movements as eye scope, such as having participants squeeze their eyebrows. After combining two or more of these single eye movements, there will be dozens of combinations in total. We got rid of the ones that the participants did not perform and ended up with seven combinations of eye gestures.

The only head category includes head movement, head distance, head rotation, and the combination of different head gestures as well. The scope of head movement is turning and tilting the head in different directions. We separated head rotation from the head movement because the amplitude of the rotation is larger and more apparent. In addition, the tilted head included in the head movement is a bit similar to the half-circle rotation, so we thought that distinguishing them could be better understand the participants' preference for these two different amplitude gestures. Head distance means the distance between the head and the mobile phone screen that people would move heads closer or further from the screen. Among all the combination possibilities of these single head gestures, only the combination of head movement with head distance change was chosen by our participants.

The only mouth category includes pout, open mouth, close mouth, wry mouth, suck mouth, and smile. Distribution of Gesture Groups. There were many overlaps between the gestures designed by different participants.

After removing overlaps, we found 250 unique user-defined above-the-neck gestures. Among these unique gestures, 

To derive the final user-defined gesture set from all gestures proposed by all participants, we collated the gestures included in each command and counted the number of participants performing the same gesture. We resolved conflicts between gestures to obtain the final gesture set. We also calculated the agreement score for each command. Agreement score was initially proposed by Wobbrock et al. [35] and later widely used in studies uncovering user-defined gestures for various platforms (e.g., tabletop, phone, watch, and glasses [2, 6, 27, 33, 36] . It intuitively characterizes differences in agreement between target users for assigning a gesture to a given command. In general, the higher the agreement score of a command, the better the participants are in agreement with the gesture assigned to the command.

We categorized the gestures performed by the participants for each command and then counted how many people made the same gesture. These groups and the number of people in each group were used to calculate the agreement score of the commands. We adopted this method from prior user-defined gesture research [27, 35, 36] and used the following equation:

In Equation.1, is one of the commands, represents its agreement score based on participants' proposed gestures for this command. The value ranges from 0 to 1. is the total number of gestures proposed for , which is the number of participants in our case (N=17). represents a unique gesture. Because different participants proposed the same gestures, the number of unique gestures was smaller than the total number of proposed gestures. represents the number of participants who propose the unique gesture i. Take the Single Tap command as an example, 17 participants proposed 17 gestures in total, thus equals 17. Among these gestures, there were seven unique gestures. There were 7, 4, 2, 1, 1, 1, and 1 participants who proposed each of the seven unique gestures respectively. As a result, the agreement score of the Single Tap command was calculated as follows:

. Fig. 4 . The agreement scores of the 26 commands. The higher the score, the higher the participants' consensus on which gesture(s) should be assigned Figure 4 shows the agreement score of the gestures proposed for each command. The commands were arranged in the same order as the conceptual complexity as in Figure 3 . In general, the higher the agreement score, the higher the participants' consensus on which gesture(s) should be assigned. The agreement score of Double Tap was high, which indicated that participants more agreed on which gesture(s) should be allocated to this command. In contrast, the agreement score of Rotate was relatively lower, which indicated that participants proposed more diverse gestures for it and less agreed on which gesture should be allowed to it.

Complexity vs. Agreement Score. As we explained in Sec.3.4, the conceptual complexity is a measurement of the perceived complexity of commands from researchers' perspectives. In contrast, agreement score is a measure of perceived complexity of commands from end-user's perspectives. If the perception of researchers aligns with the perception of end-users (i.e., people with motor impairments), then we should expect to see a correlation between the two measures. We did Pearson Correlation Test and found no significant correlation between the agreement score and conceptual complexity score of each command (r=-.38, p=.05). In other words, the commands given a low conceptual complexity score by the researchers did not result in a high agreement score. This suggests that there was a discrepancy between the researchers' understanding of the complexity of the commands and gestures and that of the target users'. This finding further highlights the necessity to involve end-users into the design process to design user-defined gestures that they, instead of researchers, would perceive easy-to-use.

We found that in some cases participants proposed different gestures for a command. Thus, we needed to resolve the conflicts to assign one gesture for each command.

Our conflict resolution strategy was as follows. When the same gesture was allocated to both a single command (e.g., Drag) and a paired commands (e.g., Swipe Left & Swipe Right), we would prioritize the gesture to the paired ones because the cost for finding alternative gestures for the paired commands was higher than that for a single command.

After allocating the conflicted gesture to the paired commands, we allocated the gesture proposed by the second-highest number of participants for this single command to it. Figure 5 illustrates the process with an example. The same gesture "Turn Head to the Left and Look Left" was proposed by the same number of participants (N=6) for both the single command (i.e., Drag) and the paired commands (i.e., Swipe Left & Swipe Right). Our resolving strategy would assign this gesture to the Swipe Left command. Next, we allocated the gesture that was proposed by the second-highest number of participants to the Drag command. In this case, it was "Gaze, and Look At a Certain Direction. " 

Above-the-Neck Gesture Set. Table 3 shows the final gestures for each command. Most of the commands have only one allocated user-defined gesture (i.e., Gesture 1 in the table). However, there are three commands that had more than one gesture allocated. This is because these commands have more than one gesture proposed by the same number of participants. As shown in Table 3 , twenty-three commands were assigned 1 gesture, two commands had 2 gestures, and one command had 3 gestures. We found continuity in our final gesture set. When two commands were related, such as single tap and double tap, the final assigned gestures were blinking once and blinking twice, which were very strongly correlated. There is also a of the gestures in large groups with those in small groups. There was no significant difference between them. We also did Pearson Correlation Test and found that the conceptual complexity of the commands did not have a strong was many participants commented while doing the test that "I choose to make this gesture because I think it is good, " resulting in a high rating score, which proved why a gesture was performed by fewer participants and command had a high conceptual complexity score, but the rating was not low.

We present the following insights learned from participants' feedback about user-defined gestures.

After participants made a gesture, we asked them why they chose that gesture, the most common feedback was that it was simple to understand, easy to do, and understandable. P13 explained, "Because I feel that these gestures are good, simple, easy to perform, so I choose them. ", and P16 argued, "It is easy to understand and belongs to the normal range of head movement. "

In addition to the gesture itself was simple to do, but also with the previous use of technical devices. Many participants thought about creating gestures to complete commands in conjunction with how they usually used their mobile phones with their hands. For example, when I asked them to swipe to the next page, most of them turn their heads to the left because they also swiped to the left when operated by hand. P11 used his computer more often than his phone, so he considered his habits of using the computer when creating gestures. For example, when doing the single tap command, he blinked his left eye and said, "Operation is similar to clicking with the left mouse button"-P11. Using different mobile phone models also affected the gestures. Many Android phone users chose to blink three times when taking a screenshot because they usually used three fingers to pull down for taking a screenshot with their Android phones, so this gesture was better for them to understand, but it might not be for iPhone users.

During the testing process, many participants gave feedback that they forgot what gestures they had done before. When P14 was doing command 15, she said she couldn't remember what gestures she had done. In addition to not remembering, some participants deliberately chose to make the gestures they could easily remember.

There were some gaze or close-eyes gestures, and these gestures would need to take into account the issue of how long the eyes were gazed at or closed. Although some participants mentioned 5 seconds or 10 seconds, most were 2 seconds or 3 seconds. The time length should be moderate, and as long as the differences could be distinguished from the length of time, it would not have to take too long. "Would 2 seconds or 3 seconds be better? It is clearer to stop for 3 seconds. "-P8; "It would be inconvenient if closing the eyes for too long. "-P12

Many participants considered whether the action would be too subtle for the phone to recognize when they did it. For example, when considering the gaze time, the participants thought about whether the time was too short for the phone to recognize. Many participants preferred head gestures because they thought eye movements were too small to be identified. They also considered whether the gesture was confused with our natural movements or random small movements that a spontaneous movement would trigger a command. Thus, some participants made a distinction between the gestures they performed and what we usually did spontaneously. "I feel that the head forward may not be very sensitive to identify. "-P14; "An occasional small action may affect the phone recognition. "-P11 4.3.5 Self-condition. The participants were people with motor impairments, and their motor impairment problems could cause some eye or facial movement difficulties, especially eye closure difficulties. Some participants had difficulty closing a single eye, so they chose to make the gestures in both eyes together. Some participants had difficulty closing the left or right eye, and they chose to make the gesture with the eye that had no difficulty closing. P3 was left-handed.

He was more willing to use his left eye to make the gesture. "I can close both eyes, but cannot close only one eye, so I do it with both eyes. "-P13

4.3.6 Social Acceptance. In the comments and observations, we found that these participants were very concerned about the eyes of others. They would consider whether the gesture would be too exaggerated and did not want to attract attention. They would like to choose some simple gestures so that they would not look strange. "Doing eye gestures will still take into account the feelings of others. "-P3; "Simple, does not attract special attention or disturb others. "-P11; "I don't want people to look at me differently. "-P14

We identified user-defined above-the-neck gestures by resolving conflicts in the original gestures created by a group of 17 people with motor impairments as illustrated in the previous section. One follow-up question would be: what are the more appropriate gestures among these user-defined ones for the users who would use the gestures to interact with smartphones? Our gesture elicitation study was to create a gesture set for people with motor impairments to interact with the touchscreen smartphones without touch. However, able-bodied people might also encounter the same difficulties in many scenarios and find such gestures useful. For example, it would be hard to operate the phone by hand when people's hands are occupied, for example, while carrying bags or with wet or dirty hands. As a result, we included people without motor impairments in this survey study as well to understand their preferences for the user-defined gestures designed by people with motor impairments. To answer this question, we conducted an online survey to validate the agreement of the user-defined gesture set by people with and without motor impairments.

In the survey, participants were first asked questions about their physical condition and some general demographic questions. Then, we asked about each of the 26 commands proposed in the previous user study. In each question, we asked participants to choose the most appropriate gesture for that command from three candidate options. These candidate options were derived from previous studies and were the gestures that participants most frequently performed for each command. These options were arranged in random order to reduce potential order bias. We also provided an additional option for participants to write out what they thought was most appropriate if they felt none of the three options were suitable. Figure 7 shows the screenshot of the question for the Long Press command in the survey. The survey was published in an online format and took an average of 5 minutes to complete.

Twenty-four participants volunteered in the survey study, including 10 people with motor impairments and 14 people without motor impairments. Ideally, it would be best to recruit people with motor impairments who do not have any prior knowledge about the study. However, due to the difficulty in recruiting people with motor impairments during the pandemic, we recruited the 10 participants with motor impairments who took the previous gesture elicitation study. To mitigate the potential memory effect, we did intentionally conduct the survey study one month after the gesture elicitation study. With a one-month time gap between the two studies, it was unlikely that these participants would still remember the commands or how they assigned the gestures to the commands a month ago. For the 14 participants without motor impairments, we recruited them through both online and offline means. The average age of all participants was 29 years ( = 3).

In Figure 8 , we showed the results of the participants with motor impairments in green color and the results of the participants without motor impairments in orange color. As we mentioned that there were three most frequently performed gesture options for each question, and they were all performed by the participants in the previous user study.

These three options in all questions were collectively referred to as Gesture 1, Gesture 2, and Gesture 3, in descending order of the number of times participants performed them in the user study. Of the results for all participants in the survey study, Gesture 1 was the first choice for 17 out of 26 commands and the second choice for the other 6 commands. From the results of the ten motor-impaired individuals, Gesture1 was preferred in 15 commands and the second choice for 9 commands. From the results of fourteen non-motor-impaired individuals, Gesture1 was the first choice for 15 commands and the second choice for 8 commands. Furthermore, Figure 8 also includes a "consistency" column to indicate the commands for which both participants without and with motor impairments agreed on the most popular gesture options. Out of the 26 commands, both user groups agreed on 16 commands (62%), which indicates that our gesture set had a reasonably high agreement between participants with and without motor impairments.

The agreement of Gesture 1 by the motor-impaired population was most on general commands, and the non-motorimpaired population was most on app-related and button commands. A possible reason for this was that as the complexity of the command increased, the gestures became more complex, and the combination of different above-the-neck parts and gestures involved increased. Some app-related commands were more complicated than general commands, and people were more likely to create complex gestures. For participants with motor impairments, in addition to considering the appropriateness of the gestures, it was also essential to consider their physical conditions when creating complex gestures, making it more difficult to reach an agreement.

We also noticed that none of the people with motor impairments chose Others options in the survey. The possible reason for this was that they had already done the earlier user study. Therefore, the three options provided were either the same or similar to what they had demonstrated in the user study, which led to their agreement on these three options. While in the survey for non-motor-impaired participants, we found that one participant chose the option Others very frequently. We checked her responses to see that she indicated that she did not come up with more appropriate gestures but disagreed with our provided options. We then interviewed her briefly, and she told us, "some gestures are so complicated that no one would find it easier than using their hands unless they had paraplegia". Another participant also chose Others options for two of the commands, but did not suggest any gestures that she would prefer to use.

Through the survey, we found that the choices of gestures by people with and without motor impairment were mostly consistent with our gesture set. However, there were also some differences due to physical impairments or lack of understanding of people with motor impairments by people without motor impairments. In general, our gesture set is universally applicable to people with and without motor impairments and is accepted and recognized by the public.

We discuss the implications of the user-defined above-the-neck gestures design and users with motor impairments.

By involving people with motor impairments in the design process and resolving conflicts in the proposed gestures, we uncovered a set of user-defined above-the-neck gestures and how people with motor impairments would want to use these gestures to execute commands on a touchscreen mobile device. All of the participants of the user study mentioned that they would like to use these gestures to interact with mobile phones in the future. Moreover, we learned that people with motor impairments preferred gestures that were simple, easy to remember, and had high social acceptance.

Although they had the freedom to include eyes, mouth, and head into the design of the gestures, gestures involving eyes were still the most diverse and preferred, followed by the gestures that combined eyes and head. This finding is consistent with the prior works [17, 29] that people like to add head movements to eye-based gestures. Our survey study with people with and without motor impairments found that these user-defined gestures were generally agreed by people with and without motor impairments.

Compared to the eyelid gestures designed by the researchers [10, 11] , our user-defined gestures are unique in two aspects. First, our gestures are grounded in the preferences and creativity of people with motor impairments. Second, our gestures are more diverse, which not only include eyelids but also eye motion and other body parts (e.g., head and mouth), and are more expressive and can be used to accomplish more commands. However, one must be wary of the downside of a more diverse set of user-defined gestures.

First, as the number of user-defined gestures increases, the efforts for remembering the mapping between the user-defined gestures and the corresponding commands also increases. Indeed, some of the user study participants asked us in an apprehensive tone whether they would have to remember all the gestures they created throughout the study. Thus, it is worth investigating how best to help people with motor impairments make use of these user-defined gestures with the minimum burden of memorization. One possible solution might be to suggest relevant user-defined gestures based on the initial input of the user. For example, if the user starts to close one eye, then the system could recommend a much smaller set of gestures that start with "close one eye. "

Second, some participants were concerned about the long-term or frequent use of some gestures, which might Lastly, social acceptance was a key factor that people with motor impairments considered when designing the gestures. However, little is known whether people with motor impairments perceive as socially unacceptable are really unacceptable from the public's perspective and vice versa. Furthermore, it also remains unclear what gestures are more socially acceptable. Perhaps gestures that are small in amplitude and consistent with normal daily routine activities might be more socially acceptable. One interesting question would be to understand the relative social acceptability of our user-defined gestures.

Potential Effect of The Presentation Order of Commands. When defining gestures, the participants were given the commands randomly. Thus, they were not able to know in advance the complexity of all commands or whether there were similar or symmetrical commands. Our approach to alleviating this potential presentation order effect, as stated in Section 3.3, was allowing the participants to change the gestures assigned earlier anytime during the study.

If they had difficulty keeping track of the assigned gestures, for example, forgetting about what gestures had been already assigned, they could ask the moderator to remind them. Our approach was promising based on the final gesture set shown in Table 3 . For example, different commands had been assigned with different gestures, and symmetrical gestures were given to symmetrical commands. Nevertheless, it is interesting to investigate whether participants would adopt different strategies to allocate gestures to commands if they were allowed to know all the commands upfront.

Command Selection. We explored 26 commands commonly used on a touchscreen smartphone. However, these commands are not exhaustive. For example, participants mentioned other commands, such as returning to the main screen, unlocking the phone, and back to the previous page. In addition, our classification of commands was based on the prior work [10, 11, 20] , and we divided the 26 commands into three categories based on the task goals. Different classification methods may also affect the participants' choice of gestures. Future work could apply the same principles to define user-defined gestures for additional commands. However, one challenge is to resolve conflicts among the gestures allocated for different gestures, which has proved to be increasingly challenging with more commands to cater.

Common vs. Personal User-defined Gestures. Our study aimed to identify a set of common user-defined gestures for the commands often performed on a touchscreen smartphone based on the feedback from a group of people with motor impairments. Thus, we believe that this set of common user-defined gestures is a good starting point for people with motor impairments to interact with smartphones without touch. However, we acknowledge that people with motor impairments have different residual motor abilities and may be able to or prefer to perform different gestures. Thus, it is imperative to investigate how best to allow people with motor impairments to design a personal set of user-defined gestures tailed to their specific motor abilities, such as a recent work by Ahmetovic et al. [1] .

Differences Between People With and Without Motor Impairments. Our survey study revealed the preferences of people with and without motor impairments. However, it remains unknown why people without motor impairments preferred the same or different user-defined gestures for the same command or whether they care about social acceptance of these gestures as much as people with motor impairments do.

Potential Effects of Age and Culture. Our participants were primarily young and middle-aged people. It remains an open question of whether age plays a role in user-defined gestures. Future work could replicate the study with older adults with motor impairments and examine whether the user-defined gestures are applicable across different age groups and whether there are specific user-defined gestures that are more preferred by an age group.

Our participants lived in Asia, and they were used to the culture of the East. The creation and preference of userdefined gestures might be affected by culture, and the social acceptance of the gestures was also likely related to the social norms and cultures they lived in. Future work could explore user-defined gestures for people with motor impairments in different cultures and compare the similarities and differences to understand cross-culture and culture-specific user-defined gestures.

We have adopted a user-centered approach by involving people with motor impairments to design user-defined abovethe-neck gestures for them to interact with mobile phones. By analyzing the 442 gestures and resolving conflicts,

we have arrived at a set of user-defined gestures. The participants were excited about the convenience the gestures could bring to them. They preferred gestures that were simple, easy to remember, and had high social acceptance. Our follow-up survey study results found that the user-defined gestures were well received by both people with and without motor impairments. Finally, we also highlight the design considerations and future work.

RePlay: Touchscreen Interaction Substitution Method for Accessible Gaming

Exploring nontouchscreen gestures for smartwatches

Computer Vision-Based Methodology to Improve Interaction for People with Motor and Speech Impairment

Barriers to and mediators of brain-computer interface user acceptance: focus group findings

A P300-based brain-computer interface aimed at operating electronic devices at home for severely disabled people

Designing consistent gestures across device types: Eliciting RSVP controls for phone, watch, and glasses

A Comparison of Surface and Motion User-Defined Gestures for Mobile Augmented Reality

Surface vs Motion Gestures for Mobile Augmented Reality

Mobile Accessibility for People with Combined Visual and Motor Impairment: A Case Study

Eyelid Gestures on Mobile Devices for People with Motor Impairments

Eyelid gestures for people with motor impairments

Simple gaze gestures and the closure of the eyes as an interaction technique

In the Blink of an Eye: Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass

An eye tracking computer user interface

Studying User-Defined IPad Gestures for Interaction in Multi-Display Environment

EOG-based glasses-type wireless mouse for the disabled

Pinpointing: Precise head-and eye-based target selection for augmented reality

EEG-Based Brain-Computer Interfaces for Communication and Rehabilitation of People with Motor Impairment: A Novel Approach of the 21st Century

User Defined Gestures for Augmented Virtual Mirrors: A Guessability Study

IWink: Exploring Eyelid Gestures on Mobile Devices

Personalized, Wearable Control of a Head-Mounted Display for Users with Upper Body Motor Impairments

Understanding the Accessibility of Smartphone Photography for People with Motor Impairments

Accessibility in Context: Understanding the Truly Mobile Experience of Smartphone Users with Motor Impairments

Evaluation of HeadTurn: An interaction technique using the gaze and head turns

Evaluation of brain-computer interfaces in accessing computer and other devices by people with severe motor impairments

User-Defined Gestures for Augmented Reality

Extended Abstracts on Human Factors in Computing Systems

User-Defined Motion Gestures for Mobile Interaction

The eye wink control interface: using the computer to provide the severely disabled with increased flexibility and comfort

Eye&head: Synergetic eye and head movement for gaze pointing and selection

Are we there yet? Evaluating commercial grade brain-computer interface for control of computer applications by individuals with cerebral palsy

User-Defined Gestures for Elastic, Deformable Displays

Predicting Stroke Gesture Input Performance for Users with Motor Impairments

Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit

Interact with Your Car: A User-Elicited Gesture Set to Inform Future in-Car User Interfaces

Maximizing the guessability of symbolic input

User-Defined Gestures for Surface Computing

Headgesture: Hands-free input approach leveraging head movements for hmd devices

Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities