key: cord-0131168-cs8ro7fa authors: Pyun, Jaehoon; Cho, Younggeol; Kim, Seon Gyeom; Lee, Woohun title: Point&Select: Designing an Interaction Technique for Inputting Surrounding Point of Interests in Driving Context date: 2022-01-17 journal: nan DOI: nan sha: c4418bcbc44e590300bb5b53015795e4c0a8f9e4 doc_id: 131168 cord_uid: cs8ro7fa We propose an interaction technique called"Point&Select."It enables a driver to directly enter a point of interest (POI) into the in-vehicle infotainment system while driving in a city. Point&Select enables the driver to directly indicate with a finger, identify, adjust (if required), and finally confirm the POI on the screen by using buttons on the steering wheel. Based on a comparative evaluation of two conditions (driving-only and driving with input-task) on a simulator, we demonstrated the feasibility of the interaction in the driving context from the perspective of driver performance and interaction usability at speeds of 30, 50, and 70 km/h. Although the interaction usage and speed partially affected the driver's mental load, all the participants drove at an acceptable level in each condition. They carried out the task successfully with a success rate of 96.9% and task completion time of 1.82 seconds on average. Recently, new driving experiences based on the in-vehicle infotainment system (IVIS) have been explored extensively. This is because improvements in information accessibility have made it more convenient for drivers to use the information they require by connecting wireless networks to vehicles. The present method for inputting information to an IVIS does not permit drivers to utilize information of surrounding objects while driving" in the interest of clarity and readability. In general, methods such as dragging of a cursor on a map with a screen touch or text entry (e.g., by a touch keyboard or handwriting on a pad) that require the exact name of the object are used. However, these input methods distract the driver's attention; require the use of the hand, which the driver needs to hold the steering wheel; and impose constraints on the driver's behavior. In addition, these impose a significant cognitive load on the driving [25] . Recently, speech recognition has begun to be used as an input method that can supplement these aspects. However, it has the problem that the driver must know the exact name of the corresponding point of interest (POI), similar to other text entry methods. Therefore, a convenient and intuitive method to input data on external POIs into the IVIS during driving is required to utilize information on surroundings. Spatial direct selection using fingers is a natural interaction method that was studied in the early 1980s in the field of human-computer interaction (HCI). It is commonly used in dynamic vehicle scenarios [41] . If the surrounding POI can be entered into the IVIS through spatial direct selection, it can be associated with existing IVIS functions. This would yield new application scenarios such as "Guide me to the entrance of that building's parking lot," "Save that restaurant as a place to visit later," or 'What's that building?" Therefore, research on the selection of surrounding POI through finger pointing is also being conducted in the automotive field [16, 41, 43, 46] . However, these studies focus on technical improvement in terms of accuracy. Whether the driver could use the input method in the driving context has not been evaluated effectively. Moreover, the design of the overall driver experience and the user feedback process (such as whether the POI has been entered correctly after finger pointing, and how to rectify an incorrect entry) have not been addressed [43] . We proposed an input interaction technique called "Point & Select." It enables the driver to input surrounding external objects into an IVIS while driving in the city environment. Considering the dynamic characteristics of vehicles, it is designed as an interaction method in which the intended POI can be entered into the IVIS through pre-selection by rough pointing (i.e., to indicate by fingers) and finally verified by fine selection (i.e., to identify the object on the digital instrumental cluster (DIC) and confirm it). We examined the driver's experience of Point & Select in a simulated driving environment to evaluate 1) its effect on driving and 2) whether the external objects were being entered successfully. An analysis of the experimental results revealed that the input interaction technique is effective for use while driving, in terms of driver performance and interaction usability. Finally, additional considerations for application in real-world driving environments were discussed. Humans spatially refer to surroundings through several modalities (e.g., hand, chin) continually. This is because it is one of the most intuitive and rapid means to communicate regarding omitted intention [11, 13, 14] . Thus, many studies have been carried out within the HCI community on spatial direct selection using modality sensing techniques to identify a user's intention. Bolt recommended the spatial interaction technique "Put-that-there." It involves direct finger-pointing toward the intended object and can supplement voice input commands [3] . Katsuragawa et al. implemented the mid-air pointing and clicking interaction in a ubiquitous display environment using a smartwatch, taking advantage of the high degree of freedom of hands [22] . Mayer et al. utilized gaze tracking with the simultaneous use of the front and rear cameras of a smartphone, to obtain a user's intention [30] . They presented its applications in an ordinary context. A driver encounters various types of spatial information in the vehicle context and refers to these on a daily basis to determine directions and share information. Therefore, it is possible to develop new application scenarios within the detailed purposes of the driving context if the driver can conveniently input the surroundings to the IVIS [10, 39, 43, 46] . Research has used direct spatial selection in the driving context to consider external surrounding objects as inputs [20, 24, 47] . Gomaa et al. demonstrated an improved (with respect to accuracy) external object referencing method using finger pointing and gaze [16] . Sauras et al. and Tscharn et al. designed gesture and voice interaction for maneuver intervention in autonomous driving [43, 46] . They figured out that hand gestures are a natural modality within a driving context. However, the majority of the research on spatial direct selection as an input method in the driving context has focused on improving the accuracy of pointing sensing or has assumpted autonomous driving (where the driver is not mandatory). In addition, after-pointing should also be considered (e.g., methods for the driver to assess whether the POI has been pointed correctly or to rectify a POI) because it is performed mostly in dynamic scenarios [43] . The driver performs several tasks simultaneously while driving [8] . The tasks traditionally performed in nonautonomous driving can be divided into two groups [38] . Wierwille defined all the essential activities for maneuvering a car as the primary tasks and the remaining as secondary tasks [48] . The performance of a secondary task while driving can distract the driver, which implies that the cognitive resources that ought to be used for driving are distributed to other actions [12, 44] . Because this is an issue that can be directly related to safety, verification in terms of driver performance is necessary to design interaction as a secondary task. However, advances in autonomous driving technologies have resulted in alterations of the driver's role. The driver can withdraw the hand, eye, and mind from the primary tasks and thereby, have more cognitive resources to accomplish a secondary task [2, 34] . These technologies can facilitate the transition between primary and secondary tasks and enable the driver to perform secondary tasks more conveniently [26] . Accordingly, exploratory studies have been conducted on the application of natural modalities to interaction techniques in autonomous driving [9, 37, 47] . However, further advances in autonomous driving technology are required before a fully autonomous driving scenario is realized [1] . Therefore, in the near future, we need to include the primary task while designing secondary tasks. Rather, we should consider whether it is possible to perform secondary tasks while driving. Because a driver's cognitive resources are limited [48] , his/her interaction should be designed with a low cognitive load so that he/she can handle both primary and secondary tasks successfully. In addition, the interaction should minimize the workloads related to behavioral and visual activities so as not to disturb the primary task [35] . Moreover, the dynamic driving scenario affects the driver's cognitive resources and the capability to recognize surroundings, which prevents him/her from fully carrying out human performance [27] . The speed of a car is an important factor that determines human performance during driving [6] . The driver interaction technique needs to be evaluated in terms of both primary and secondary tasks. To evaluate a secondary task, the experimental task is designed according to the purpose of the interaction. Furthermore, the usability is generally measured in terms of the task completion time (TCT) and success rate (SR) and based on an analysis of gaze and behavior. Broy et al. measured the TCT, SR, and gaze duration [15] for a subject to perform a task in safety-critical applications to evaluate the effect of 3D display on driver performance. Jung et al. designed a voice and tactile I/O interface using PinPad and examined two possible scenarios to measure TCT and gaze behavior [21] . Buchhop et al. proposed a method that enhances the driver's awareness of the scenario in front [7] . In addition, they measured the TCT and response time by composing an unforeseen scenario to verify its effect. In addition, other measures of experience (such as the NASA task load index (TLX), AttrakDiff, system usability score (SUS), and INTUI questionnaire, which are generally used to evaluate how the user cognitively perceives an interaction or interface) can also be used to evaluate a driver's secondary tasks [5, 17, 18] . However, the primary task needs to be evaluated simultaneously when a non-autonomous driving scenario is considered. The methods for evaluating the primary task are largely divided into objective and subjective methods. Objective methods assess the vehicle movement to evaluate the driver distraction level. The methods for determining the lateral lane maintenance (LLM) [19, 31] and speed maintenance (SM) [9, 28] evaluate the distraction level of a driver's driving based on the standard deviation (SD) of the distance between the centers of the vehicle and lane, and the SD of the speed. A higher value implies that the driver is more distracted. These methods can be used to determine how interactions directly affect the primary task. Meanwhile, subjective evaluation methods are based on the cognition of the driver. Pauzié proposed the driver activity load index (DALI) metric (a modified version of the NASA TLX), which focuses on the driving [36] . It evaluates the cognitive load that the driver experiences while carrying out the primary task. The assessment was conducted by comparing two or more conditions based on six indicators. In this study, we evaluated in terms of both primary and secondary tasks because our objective was to design an input technique for a non-autonomous driving scenario. We designed a spatial input technique named Point & Select. It enables the driver to input external surrounding objects into the IVIS even while driving. Then, we developed a prototype and driving simulator to verify the feasibility of the technique with respect to driver's performance. For the interaction to be available while driving, the Point & Select interaction scenario was designed as shown in Figure 1 . In the rough pointing phase, the driver can indicate the external object with his/her finger to rapidly and intuitively select the intended object. However, driving scenarios impose certain constraints on drivers in the direct selection process. The driver is required to look ahead while driving, and the intended object may pass by rapidly. This hinders him/her from accurately pointing at the object with his/her finger. In addition, although the driver may consider the POI to have been entered correctly, it may not have been entered as intended. To solve these problems, we designed fine selection as the second phase of Point & Select. Fine selection enables the driver to clearly identify the initial input as a POI candidate, conveniently adjust if the candidate is not intended, and finally confirm. A series of Point & Select processes from rough This interface prototype was implemented in a simulated driving environment to experimentally verify the suitability of using Point & Select, in terms of driver experience. It was constructed with a focus on identifying the effect on driving and the experience of using interaction, rather than the accuracy and efficiency of the technology. This chapter describes the process by which the Point & Select interaction method designed above was configured for each feature. In the rough pointing phase (where the driver rapidly indicates the intended POI with a finger), it enables natural communication of the POI while driving, by permitting the direct indication of spatial information. The purpose of the rough pointing phase is to provide the first input based on an approximate perception of distance and direction, rather than for the driver to accurately point to the intended POI. To identify the intended POI from the finger pointing by the driver in the particular context, it is necessary to recognize the driver's finger direction and calibrate it. We used the Leap Motion controller to track the driver's hand movements. In addition, we increased its accuracy by developing a sufficiently bright lighting environment and activating the robust tracking mode to further enhance the performance of the interaction prototype. Finger pointing was performed with the index finger, as shown in Figure 2A . Furthermore, original ray was constructed by connecting the third joint from the index fingertip to index fingertip. The ray was corrected through two step processes so that the closest object could be selected even when the original ray did not accurately intersect any object. First, we constructed the calibrated ray with a method that maps a data value measured by Leap Motion to the angle of the direction in which the driver points. Second, to identify the closest object even when calibrated ray did not directly intersect any object, calibrated ray was complemented with the perception of distance and direction. Among the objects within a certain range around the origin of ray, the one with the smallest angle between calibrated ray and the vector that connects the origin and the center of each object was determined as the object pointed by the driver. In an ideal scenario, rough pointing (which can initially input POI through these operations) starts with finger pointing without a separate trigger. However, in the prototype for this experiment, it was started by pressing the activation button for an accurate usability evaluation. In the fine selection phase, the driver can identify whether the POI candidate that is first selected by rough pointing is what the driver intended, using the in-vehicle 2D screen. This process is highly important because the previous rough pointing phase uses only a mid-air gesture input method using a finger, which has no direct feedback [29, 33, 42] . A rapid and clear feedback method is important because the identification process of fine selection is performed while driving [43] . Therefore, it is performed on a screen that can be accessed conveniently by the driver while looking forward. Recently, design exploration was carried out on in-vehicle displays such as center fascia, head-up display (HUD), and digital instrumental cluster (DIC) [40] . The center fascia is generally the most wide and suitable for utilizing infotainment system functions. However, it requires the eyes to shift its gaze away from the front. Although it is convenient to observe an HUD while looking forward, it is unsuitable for displaying the information required during the identification process. This is because it is fabricated to be transparent to prevent it from blocking the front view. Meanwhile, DIC is suitable for displaying visual information for driving because it is a screen that drivers already use conveniently (e.g., instrumentation including the speedometer). Moreover, it has recently begun to be used to display various functions of infotainment systems. Accordingly, we decided that the identification process should be performed using the DIC. The first scene that was presented by the DIC to the driver after rough pointing was an image of the front view at the time of rough pointing. This was to enable the driver to identify the scene at a glance without spatial cognitive dissonance ( Figure 3 ). The POI candidate that was selected first by rough pointing from among the many objects in that area was marked by a visual cursor (which highlighted the outline of the object) so that the driver could recognize it rapidly and clearly. Finally, after the Identification process (which is a comparison of the object selected first with the originally intended POI on the DIC), the driver could move the cursor (if required) and finally confirm to input the object. Because the object selected first through rough pointing is not always the correct selection, an input method that can correctly locate the cursor is required for the confirmation process of fine selection. Because unforeseen scenarios can occur at any time while driving, the driver must hold the steering wheel with at zleast one hand to respond rapidly and must be capable of returning the hand to the steering wheel rapidly while performing secondary tasks. Accordingly, the on-screen cursor that is controlled using the physical button on the remote control on the steering wheel was used as an input method for cursor manipulation. This was because it can be manipulated conveniently without looking, while driving. In general, the input method for the physical user interface (PUI) of the steering wheel remote control involves the use of a button or directional pad (D-pad) with vertical and horizontal directions. To make it natural, the up-arrow, down-arrow, and left-and right-arrow keys were used to shift the cursor farther from the driver, closer to the driver, and across the road, respectively. After placing the cursor on the intended object, the driver could finally confirm it by clicking the confirmation button. In the case of the Logitech G29 steering wheel used in this experiment, the function for each button was assigned as shown in Figure 4 . However, objects pass rapidly in the dynamic characteristic of the driving context. This can cause the addition of irrelevant objects by rough pointing because the intended object would have been passed by. Therefore, we developed backwardnavigation-along-the-path ( Figure 5 ). It enables the driver to select objects that have been passed by, by retracing the path. The driver achieves this by moving the cursor back and forth using the remote control on the steering wheel. Animation was used to provide a spatial perception of the back and forth movement of the viewpoint along the path [45] . The 3D simulated driving environment was implemented in Unity (version 2020.1.3f1). Natural car movement was implemented using Vehicle Physics Pro (an automotive physics engine) to provide drivers with a driving experience similar to real scenarios. Tactile force feedback such as steering sensitivity, handle resilience, and vibration on road surfaces was implemented using the Logitech G29 steering wheel. The road was implemented by Bézier Path Creator, and the buildings were placed randomly on both sides of the road from 100 m after the starting point of the course. Data such as the position and speed of the vehicle, time, and target selection activities in the simulated driving environment were logged in real time. The front view of the vehicle was displayed on a 55 inch TV, and the vehicle digital cluster was displayed on the top half of a 22 inch monitor. The Leap Motion controller was installed above the driver using a holder made of aluminum profile so that the hand could be recognized effectively by the sensor without. To increase the recognition rate, a sufficiently bright environment was developed by installing LED lighting above the Leap Motion sensor. Figure 6 illustrates the software and hardware configurations of the simulated driving environment and interface. In this experiment, the experience of using the interaction method Point & Select was evaluated quantitatively and qualitatively in terms of driver performance and interaction usability. We attempted to determine the applicability of the method in a vehicle environment by focusing on whether it is suitable for use while driving, rather than evaluating the technological advances of the spatial input methods. Therefore, the experiment was designed to evaluate the effect of the method on driving for each speed range of the city and the success with which external objects could be entered. This was We recruited subjects with over three months of actual driving experience in the Korean road environment. This was to ensure that they were fully aware of the local road regulations and capable of explaining the experience of using Point & Select based on their local driving experience. In addition, only those who could distinguish colors normally (i.e., no color blindness) were recruited because the experiment required color discrimination. The recruitment was conducted online. The final set of subjects comprised 12 (six male and six female) drivers. Their average age was 25.6 years (σ = 1.7 years), and their average driving experience was 10.7 months (σ = 9.5 months). Prior to the experiment, we examined them for fever and symptoms related to COVID-19. They were permitted to participate in the experiment after a disinfection process. Ethical approval for the experimental process was issued by Institutional Review Board (IRB). We also received each participant's consent for recording the process and using it after anonymization. We paid 15,000 KRW as a reward after the experiment. The subjects were named P1, P2, …, P12 to describe the results. It is difficult to test the new interaction method because of the many unforeseen scenarios and uncontrolled variables in real-world road environments. Therefore, the experiment was conducted in a simulated driving environment installed in a small, closed room. The resilience and vibration feedback from the steering wheel as well as the positions of the chair, steering wheel, pedal, and screen of the simulated driving environment were adjusted for each subject before the experiment. This was to ensure that he/she could undergo an experience similar to that of the actual driving environment. The positions of Leap Motion and the light were also adjusted so that the driver's use of the main hand could be identified effectively. The cameras for video and sound recording were installed at a distance to prevent interference with the driver's vision and behavior. The road in the simulated driving environment was designed to have a width of 3.3 m and two lanes in accordance with the Korean city road regulations. This was to enable the drivers to adapt rapidly to the lane width and sense of speed. The course was constructed to be of medium difficulty by irregularly composing straight sections and smoothly curved sections. The vehicle was designed with a width of 1.86 m (the size of a mid-size sedan). The road width and building size were designed in proportion to that of the vehicle. The buildings were generated with random sizes in the range of 10 ×10 × 8 m 3 to 20 × 20 × 15 m 3 . The distance from the road was in the range of 5-8 m, the space between buildings was in the range of 10-30 m, and the color palette consisted of ten colors. The simulated driving course for the experiment is shown in Figure 7 . An alarm was provided (see Figure 8 ) when the vehicle speed exceeded the speed range specified according to the experimental conditions. To determine the applicability of Point & Select while driving in a city, the speed ranges of the experiment were set to the following three values with a permissible range of ±11 km/h: 30 km/h, the speed limit for residential roads; 50 km/h, the speed limit for general urban roads; and 70 km/h, the speed limit for expressways. Six experimental conditions were configured as follows: two scenarios (C1 and C2) for each speed range to identify how Point & Select affects driving for each speed range and the success with which external objects could be entered. For C1, the subjects were asked to drive on the driving course for 5 min in the center of the first lane (to the extent feasible) within a specified speed range. For C2, the subjects were asked to input target objects that appeared randomly while driving the course for 5 min in the center of the first lane within a specified speed range as in C1. The target objects were assigned randomly in the visible range of the driver among the buildings on both sides of the road and displayed with a red outline on the driver's front screen. The drivers were asked to select the target with Point & Select after initiating by pressing the D-pad on the steering wheel. As shown in Table 1 , the outcome was denoted as success if fine selection was successful, wrong if fine selection failed, and missed if fine selection was not performed within the time limit or rough pointing was missed. A new target was assigned within 3-6 s when the outcome was success or missed. The measurement was started after the specified speed was attained for the first time during 5 min of driving for each speed condition. Moreover, it was configured to sample at least 25 times. Each subject required a maximum of 120 min to complete. The subjects filled out the consent form to participate in the experiment after the first 5 min of the experiment. Subjects who agreed to the procedure took approximately 5 min to complete a brief, short-answer questionnaire on their actual driving experience, secondary tasks while driving, and IVIS usage behavior. Next, in approximately 10 min, the subjects were informed of the Point & Select interaction method and simulated driving environment, and asked to adjust the driving environment and calibrate the Leap Motion for optimized driving environment and experience of use. Subjects drove freely for approximately 10 min to become familiar with the perception of the steering wheel and speed in the simulated driving environment. This was followed by the use of Point & Select while driving for approximately 10 min to become familiar with the selection of external objects. They were instructed to stop the experiment immediately when they experienced dizziness. When they had adapted themselves to the interaction method and the use of the simulated driving environment, they evaluated the environment to identify problems (if any) in it. Then, the 5 min experiments, 3 min surveys, and 2 min breaks were repeated for each of the six experimental conditions configured above. They were asked to complete the DALI in C1 and the DALI and NASA TLX in C2 for the survey. The order of the speed conditions was shuffled randomly for each subject. Finally, post-interviews were conducted on the overall experience of using the interface and a comparison of the six experimental conditions, for approximately 20 min. The entire experimental process was recorded using a camera. Prior to the main experiment, the SSQ questionnaire was used to evaluate sensory abnormalities, e.g., whether the subjects encountered problems in the use of the 3D simulated driving environment and whether it was difficult to concentrate. We verified whether there was cyber-sickness based on nausearelated, oculomotor-related, or disorientation-related [23] . Quantitative data on the position and speed of the vehicle over time were collected in the simulated driving environment to determine the extent to which Point & Select interaction interfered with the driving performance. We calculated LLM [19, 31] and SM [9, 28] based on the data to determine whether there were statistically significant differences between C1 and C2 for each speed condition, and whether there were differences according to the speed within C2. The DALI questionnaire [36] was administered to determine the extent to which Point & Select interaction affected the cognitive burden of driving performance. For each of the six experimental conditions, the subjects were asked to fill out the questionnaire at the end of each experimental session. The questionnaire consisted of a 21-point Likert scale. We investigated whether there was a statistically significant difference between C1 and C2 for each speed condition, and whether there were any differences according to the speed within C2. The SR for the target object input and the TCT were obtained in the simulated driving environment to determine how successfully the driver used Point & Select interaction. The SR was calculated as the number of successful selections over the total number of attempts to provide an input, as below. TCT was calculated as the time from when the driver attempted to select (i.e., the time when the driver pressed the button) till the successful selection of the target object. Based on the data, we investigated whether there was a statistically significant difference among the speed conditions in C2. The NASA TLX questionnaire [17] was administered to determine the cognitive burden that Point & Select interaction imposed on the drivers. For each of the three experimental conditions of C2 wherein the subjects were asked to select the target objects, they were asked to fill out the questionnaire at the end of each experimental session. The questionnaire consisted of a 21-point Likert scale. We investigated whether there was a statistically significant difference for each speed condition based on the data. Table 2 shows the result of SSQ questionnaire taken after the participants had sufficient adaptation time to get used to the simulated driving environment. All participants reported that there was no problem in using the simulated driving environment, so they were able to proceed to the end without interruption. Because SSQ total score of all subjects in the simulation driving environment was significantly less than 32 (average: 6.50, standard deviation: 4.80), it is determined that the effect of cyber-sickness was not severe enough to affect the experimental results [23] . In addition, through the postinterview showed that the simulated driving environment was somewhat different from the actual real-world driving experience, but some experiences such as the force feedback of the steering wheel and the sounds were very similar to the actual driving experience on the road, thereby increasing the immersion: This implies that using Point & Select interaction while driving affected the drivers to keep the speed in the medium speed, whereas it was not as burdensome as it did not significantly interfere with it in the low and high speed. Meanwhile, SM in each speed condition showed a statistical difference in the C1, but in the C2 it did not (pC1, 30- Primary driving task Load evaluated by DALI questionnaire for six experimental conditions is summarized as shown in Figure 10 . Comparing C1 and C2 for each speed condition, there were differences from the items with significant differences for each speed condition for the six items of DALI questionnaire: effort of attention, visual demand, auditory demand, temporal demand, situational stress. C1 and C2 differed in one item (situational stress) in low-speed condition, in two items (effort of attention, situational stress) in medium-speed condition, and in three items (effort of attention, interference, situational stress) in high-speed condition. Through the post-interview, it was identified that 'situational stress' comes from the limitation of the speed, 'effort of attention' from the appearance of the targets, and 'interference' from turning the corners and keeping the car to the center of the lane: " (Evaluated situational stress of C1 and C2 as 1 and 17, respectively) The situational stress was high because 30 km/h was too slow to meet the speed. In fact, it was not difficult to point and select the targets, but it was hard to maintain the speed. So, I gave differences to the situational stress of C1 and C2, respectively." (P4) "Personally, I think the 50 km/h C2 condition was the easiest, because it had some speed, so it was natural as usual driving, and I also had to keep selecting something, so I was more focused". (P1) "In fact, it wasn't hard to select in all three speed conditions, There was no statistical difference in all six items of the DALI questionnaire when comparing by speed for each condition of C1 and C2. That is, it was identified that the cognitive load on the driving itself did not differ significantly on the speed when driving only (C1), and there was also no statistical difference for speed when selecting external objects while driving (C2). Meanwhile, three items, visual demand, auditory demand and temporal demand, were statistically equivalent, regardless of speed and the use of Point & Select interaction. "It wasn't too hard to do other things while driving because it was not too fast. The interaction method itself doesn't seem to interfere with driving much, because it is able to use quickly with a glance when once adapted." (P10) In summary, from the results of the DALI questionnaire, rather than the speed itself affect the drivers' cognitive load when driving only (C1), the cognitive load that the driver felt was increased much when carrying out other tasks while driving (C2). In order to evaluate the secondary task performance, the average TCT and the average SR for three speed conditions were summarized as shown in Figure 11 . The average TCT, the average time taken for selecting an external object while driving, was measured at 1.79 seconds (SD: 0.25 seconds) in low speed, 1.84 seconds (SD: 0.39 seconds) in medium speed, and 1.82 seconds (SD: 0.40 seconds) in high speed. The average SR was measured at 98.7% (SD: 2.38%) in low speed, 96.7% (SD: 6.73%) in medium speed, and 95.17% (SD: 5.35%) in high speed. The overall average for all three speed conditions of TCT and SR were 1.82 seconds (SD: 0.36 seconds), and 96.9% (SD: 5.35%), respectively. Both the TCT and the SR according to speed were not statistically different, so it was identified that the driver's secondary task performance of selecting external dynamic objects using Point & Select interaction was not affected much by speed. Rather, as the speed increased, the driver himself / herself felt burdened because the driver thought it could affect the primary driving task, not because of the difficulty of the Point & Select interaction. "I don't think the difficulty of selection depending on the speed itself changes. Rather (when using the interface), I felt the difficulty of driving differently." (P11) However, the drivers showed the use behavior of making quick pre-selection through rough pointing and switching actions flexibly between driving and carrying out fine selection, after getting familiar with the Point & Select interaction. "Once I pointed at it with my finger and the marker appeared on the screen, it wasn't a big burden after I got used to it because I could select it on the screen after turning the curve or maintaining the speed." (P9) Secondary Task Load evaluated by NASA TLX questionnaire for three speed conditions of C2 is summarized as shown in Figure 12 . There was no statistically significant difference by speed for four items (mental demand, physical demand, effort, frustration level), among a total of six NASA TLX items, but there were significantly differences in 'temporal demand' and 'performance'. From the previous results of TCT and SR that describes the speed did not significantly affect the performance of Point & Select interaction, it is identified that 'temporal demand' was increased as the speed increased because of the pressure to return to driving more quickly as the speed increased. It is also identified that the 'performance' was increased as the speed increased because of the regret that the driver could do better if he/she got more used to it, rather than the driver was really bad at selecting the target objects as the speed increased. However, many subjects had strong confidence that they would be able to use it without difficulty if they get more used to the Point & Select interaction and the simulated driving environment. In addition, we compared the secondary task load between the Point & Select interaction and the secondary task during normal driving which identified by the questionnaires written before the experiment. Comparing experiential cognitive loads between the use of the Point & Select interaction and input some information to the car among the various secondary tasks routinely used by the subject, it showed a consistent tendency to feel difficult in the order of input through physical buttons with clear functions, input through speech recognition, input external objects through Point & Select, input information through screen touch on maps or lists, and input text through on-screen touch keyboards. It was easy to input through physical buttons with clear functions because it could be done without looking at the button of which position was known even while driving. It was a bit more difficult to input through speech recognition because the driver had to focus on the voice guidance while the contents were exchanged by voice, comparing to input through buttons. Input external object through Point & Select interaction was showed a difference from speech recognition in that the drivers' hands left the steering wheel and their eyes are taken away for a while. Subjects said that they felt much more difficult to input information by touching screen, such as searching a point by panning on maps which is most used in in-vehicle infotainment system, or finding a music by scrolling playlist, because they have to use their hand a lot to control it, as well as their eyes were completely out of driving to focus on the information on the screen. In particular, subjects found it most difficult to input text by screen touch, such as enter the destination on the GPS navigator, among the input modalities above. Moreover, some said that they could do properly it only after they stop the car, because it was too difficult to do while driving. Based on the experimental results in the above simulated driving environment, the applicability of the Point & Select interaction method to various speed conditions was discussed. In terms of the cognitive load on driving, the faster the speed is, the larger is the number of items with significant difference between C1 and C2. There was a statistically significant difference in LLM in the medium-and high-speed conditions, and in SM in the medium-speed condition. However, the primary driving task performance in terms of the LLM and SM was reasonable during driving on the roads, although it may have been lower than that for only driving. In terms of LLM, the average of the standard deviations from the center of the lane was within the lane for all the three speed conditions. The average was the largest at 920 mm in the medium-speed condition. This implies that the vehicle designed based on medium-sized sedans (as described in the section on setup) occupied 2780 mm of the road on an average. The average for each subject was the largest at 1290 mm in the high-speed condition of P3. This implies that the vehicle occupied 3150 mm of the road on an average. These did not exceed the width of the lane, which was 3300 mm (designed to comply with the Korean city road regulations) ( Figure 13 ). The average for C2 differed by a maximum of 130 mm on both sides in the medium-speed condition from that for C1, and by 85 mm on both sides on an average. In terms of SM, there was a significant difference between C1 and C2 only in the medium-speed condition (50 km/h). However, the standard deviation of speed for each experimental condition was significantly less than ±11 km/h, which is the generally permitted speed range under Korean road regulations. The differences in standard deviation of speed between C1 and C2 were highly marginal: 0. 16 The impact of the interaction when used in real-world road environments would be significantly lower than that observed in the experiment. This is because it would be used occasionally rather than every few seconds as in the experiment. Furthermore, it is considered that if the drivers had used cars that they were familiar with, they would have displayed significantly higher performance in terms of many aspects, e.g., maintaining the center of the lane, turning a corner, accelerating, braking, and gear shifting. In conclusion, the experiment conducted in the simulated driving environment implies that the impact on the driver's performance in terms of LLM and SM was reasonable when the interaction was used in city speed conditions (30 km/h-70 km/h). The experiment with the simulated driving environment demonstrated Point & Select to be feasible as a secondary task. Although the driver's primary task performance and driving load were partially affected negatively, it was not to a critical level. Therefore, the driver could drive effectively while conducting Point & Select. However, the simulated driving environment in this study excluded certain elements that are present in real road environments (e.g., other cars or traffic lights) so as to measure the usability more precisely. However, considering that these elements can distract a driver, we should take these into account while applying the interaction technique to the real world. Therefore, we need to discuss the factors that should be considered and how these affect the driver. Drivers should be aware of the signals from the other drivers and predict what actions would be adopted. In addition, even when other drivers are not presenting signals, the driver should control the speed and path and be prepared to evade accidental events by observing and assessing the surrounding scenario. To comply with road traffic rules while driving, the driver should rapidly obtain information from signs, traffic lights, and markings of the road system. Pedestrians and bicycles are good examples of factors outside the roadway. In particular, in places where the distinction between sidewalks and roadways is unclear (such as residential roads and crosswalks), unanticipated scenarios occur frequently wherein pedestrians and bicycles appear abruptly out of alleys. As these are unanticipated events, drivers attempt to follow the low-speed limit and be always prepared cognitively to decelerate. Even when the driver is performing tasks that are not directly related to driving, the driver should be capable of returning to the primary task at any time to mitigate the various risk factors mentioned above [4] . Various warning technologies including blind-spot collision warning, forward collision warning, and lane departure warning (which are classified as Autonomous driving level 0 by SAE J3016) help the driver return to the primary task by alerting him/her if he/she appears distracted. Similarly, we consider that the technologies including the warning system mentioned earlier can support the driver's input process of Point & Select for it to be safer and more stable, and can promote its application [32] . In Point & Select, finger pointing is first used to make rapid and approximate selections and then, button manipulation is used for fine tuning. However, considering its use cases in a real driving context, there are a variety of modalities that can be combined with Point & Select in a natural manner, such as voice, gaze, and gesture [16, 43, 46, 47] . In particular, voice input is a potential type of interaction modality in the driving context that can be combined with Point & Select to clearly communicate a POI to the IVIS, during the entire process from entering the POI to its applications after input. Similarly, the exploration of other modalities that can be combined with Point & Select in a complementary manner and the evaluation of the resulting scenarios are potential future works. Although the simulator in this study simplified the surrounding environment and objects, the study did not consider various other objects that can also be targets in a real road. These include traffic signs, intersections as spatial information, and other objects such as cars and pedestrians. In addition, a pointing gesture can have diverse meanings via different hierarchical depths and variances of information. In the case of pointing at a building, the intention can be the building itself, one of the stores in it, or directional information. As mentioned earlier, further studies are required to enable detailed inputs on what is intended to be referred, through combinations with other input modalities. This study focused on validating the feasibility of Point & Select as an input method for drivers. However, to facilitate the use of surrounding object information, detailed application scenarios after the entry of the POI should be explored. This exploration can be started by combining the input method with the IVIS to introduce a new driving experience in a real road context. This study assumed a non-autonomous driving context to investigate the feasibility of the interaction technique with respect to the driver's performance. At the current level of autonomous driving (Level 2), technologies such as lane keeping assist system(LKAS), adaptive cruise control (ACC), and forward collision-avoidance assist (FCA) partially help the driver to steer, control the speed, and prevent emergency scenarios [2, 34] . It is considered that when these autonomous driving technologies are applied, the technologies would assist the driver to have more cognitive resources. This can result in better performance with both primary and secondary tasks [2] . Therefore, the driver's performance and the interaction usability of the external object input method can be reevaluated according to the level of autonomous driving technology. This can be followed by an exploration of additional possible application scenarios. In this study, we recommended an input interaction technique called Point & Select. It enables a driver to conveniently enter external surrounding objects to the IVIS while driving. Then, we verified the feasibility of Point & Select from the perspective of driver performance and interaction usability, using a simulated driving environment. Point & Select consists of two main phases. In the rough pointing phase, the driver spatially indicates the POI with the finger rapidly. In the subsequent fine selection phase, the driver identifies the POI candidate on the DIC and confirm it with a button on the steering wheel. Although the interaction partially affected the driver's mental load and the higher speed level marginally increased cognitive burdens for the driver, we could demonstrate that Point & Select is a well-designed input method that satisfies the design considerations. All the participants drove at an acceptable level in all the six conditions in terms of LLM and SM, with an SR of 96.9% and TCT of 1.82 s on an average in city speed levels. Additional factors such as other drivers or traffic systems need to be considered while applying the interaction technique to real driving situations. This study proposed a two-step input interaction model that is feasible for the driver. It is likely to be applied in various manners for driver interaction in future automobiles. From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI Driver distraction based lane-keeping assistance Put-that-there": Voice and gesture at the graphics interface Ready for Take-Over? A New Driver Assistance System for an Automated Classification of Driver Take-Over Readiness Sus: a "quick and dirty'usability. Usability evaluation in industry Focus on driving: how cognitive constraints shape the adaptation of strategy when dialing while driving -vehicle touchscreen interaction: Can a headdown display give a heads-up on obstacles on the road? AutomotiveUI 2017 -9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, Proceedings The Influence of the Secondary Tasks on Driving Performance and Mental Workload at Different Speeds Driving Performance and Driver Workload at High Speeds: Results from On-Road and Test Track Studies The HRI-CMU Corpus of Situated In-Car Interactions. Situated Dialog in Speech-Based Human-Computer Interaction Learning to interpret pointing gestures with a time-of-flight camera Principles of Cognitive Psychology Spatial gestures using a tactileproprioceptive display Using vision, acoustics, and natural language for disambiguation Eye Tracking in Usability Evaluation. A Practitioner's Guide. The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität Lane Keeping Under Cognitive Load: Performance Changes and Mechanisms Gaze Tracking Accuracy Maintenance using Traffic Sign Detection Voice+Tactile: Augmenting In-vehicle Voice User Interface with Tactile Touchpad Interaction Voice user interface; tactile feedback touchpad; in-vehicle user interface CCS Concepts CHI Conference on Human Factors in Computing Systems Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness Cross-Car Distraction from multiple in-vehicle secondary tasks: vehicle performance and mental workload implications Dynamics of Driver Distraction: The process of engaging and disengaging. Annals of advances in automotive medicine. Association for the Advancement of Automotive Medicine Computational Techniques Used in the Driver Performance Model of the Interactive Highway Safety Design Model Speed maintenance under cognitive load -Implications for theories of driver behaviour Agency in Mid-air Interfaces Enhancing Mobile Voice Assistants with WorldGaze Driver work load for various turn radii and speeds Tunneled In: Drivers with active secondary tasks need more time to transition from automation HaptoMime: mid-air haptic interaction with a floating virtual screen Secondary task engagement and vehicle automation -Comparing the effects of different automation levels in an on-road experiment The Importance of Interruption Management for Usefulness and Acceptance of Automated Driving A method to assess the driver mental workload: The driving activity load index (DALI) Multimodal interaction in the car -Combining speech and gestures on the steering wheel International Conference on Automotive User Interfaces and Interactive Vehicular Applications, In-cooperation with ACM SIGCHI -Proceedings driving-related activities in the car: Defining driver activities for manual and automated driving Gaze guidance reduces the number of collisions with pedestrians in a driving simulator Design methods for augmented reality in-vehicle infotainment systems Free-hand pointing for identification and interaction with distant objects Enabling always-available input with musclecomputer interfaces VoGe: A Voice and Gesture System for Interacting with Autonomous Cars. SAE Technical Papers Design guidelines for safety of in-vehicle information systems Spatial vision meets spatial cognition: Examining the effect of visual blur on human visually guided route learning Stop over there": Natural gesture and speech interaction for non-critical spontaneous intervention in autonomous driving Interact with your car: a user-elicited gesture set to inform future in-car user interfaces Demands on driver resources associated with introducing advanced technology into the vehicle