key: cord-0550987-y6nl7lry authors: Winiarski, Tomasz; Dudek, Wojciech; Stefa'nczyk, Maciej; Zieli'nski, Lukasz; Gieldowski, Daniel; Seredy'nski, Dawid title: An intent-based approach for creating assistive robots' control systems date: 2020-05-25 journal: nan DOI: nan sha: af9f269d51a8813d52a78f234c442b29a0d875b0 doc_id: 550987 cord_uid: y6nl7lry The current research standards in robotics demand general approaches to robots' controllers development. In the assistive robotics domain, the human-machine interaction plays a substantial role. Especially, the humans generate intents that affect robot control system. In the article an approach is presented for creating control systems for assistive robots, which reacts to users' intents delivered by voice commands, buttons, or an operator console. The whole approach was applied to the real system consisting of customised TIAGo robot and additional hardware components. The exemplary experiments performed on the platform illustrate the motivation for diversification of human-machine interfaces in assistive robots. The current demographical processes demand new technologies to support a large number of Elderly or disabled persons. Assistive and social robots respond to it [1] , [2] . Although some aspects of assistive robot development have a general nature (e.g. low-level motor control), others are more specific. Human-machine interfaces (HMIs) play a vital role in practical applications of assistive robots. Human-machine interaction is being investigated both for mobile robots [3] and stationary HMI devices [4] . One of the main problems encountered is to maintain reliability of communication. The goal of our work is to at first determine appropriate HMIs to provide robust multimodal interaction of a human and robot-based system. We concentrate on the armless systems, as they are expected to dominate the market, because of This work was funded within the INCARE AAL-2017-059 project "Integrated Solution for Innovative Elderly Care" by the AAL JP and co-funded by the AAL JP countries (National Centre for Research and Development, Poland under Grant AAL2/2/INCARE/2018). The work was initially submitted to the 25th International Conference on Methods and Models in Automation and Robotics (MMAR) that was cancelled due to COVID-2019. The authors would like to thank the MMAR reviewers for their effort to help to improve the paper and Maciej Bogusz for his work on robot database development. the relatively low price. Efficient research demands frameworks [5] that reduce the time to develop software by code reuse and utilisation of supporting tools. Hence, we propose an intent based general approach to assistive robot's controller development that takes into account the current technologies that support the efficient usage of the previously chosen HMIs. The research process started with the determination of the HMIs based on the analysis of the existing robotic platforms (sec. II). The general structure of the based system is depicted in sec. III-A, while the main aspects of its behaviour in sec. III-B. Sec. IV presents a practical application of our approach, i.e. the system build basing on TIAGo robot. The exemplary experiments illustrate the importance of humanmachine interface multimodality. The paper is concluded in sec. V. A review of 58 different robotic platforms was conducted. The functions of examined robots varied from entertainment platforms, through assistive services, to helping people with day-to-day activities and supporting Elderly (Tab. I). The study inspected the robots for the chosen features: 1) possession of the built in, interactive, touch-screen tablet, 2) ability to speak, 3) ability to recognise voice commands, 4) possession of physical buttons and their placement. Out of 58 examined robotic platforms: • 30 possessed the specified tablet, • 36 were able to speak, • 32 were capable of voice commands recognition, • 8 possessed some physical buttons (emergency buttons are not counted, because they are not the regular communication medium) and only one robot had buttons that could be accessed from all sides of the robot. TIAGo PAL Robotics X X * Emergency stop buttons are not regarded, -Robot possesses feature, X -Robot does not possess feature, ? -The possession of feature is suggested but could not be confirmed, R -Access to the physical button is restricted because of its location. Note: in some cases, some of the robot features were not described. Such robots were not included in the above juxtaposition. The above survey visualises that voice communication and tablets are the main HMIs, while the buttons are absent or used for emergency procedures only. To obtain the highest reliability of voice communication, nowadays the cloud-based services can be used. Although the usage of tablets touchscreen, as the different communication interface to voice, is versatile, we opt for buttons. In comparison to buttons, tablet is bigger, heavier, prone to physical damage. It needs to be charged often, its usage is more complicated, especially for Elderly [6] . The number of assistive robots applications can be handled with buttons, in order to, e.g., call the robot (the appropriate button is located in the home environment), ac-knowledge the command spoken by the robot, etc. The buttons should be mounted in the proper places to be accessed easily and intuitively. For the convenience of research, development, maintenance or even regular operation, an operator/caregiver console should be included in the system. Our approach to control system creation is based on the RAPP architecture [7] extended by the Agent types introduced in [8] -a Task Harmoniser Agent and a Task Requester Agent. It is described with usage of EARL [9] -Embodied Agent-based [10] cybeR-physical control systems modelling Language (version 1.2) [11] . The whole System is decomposed into various types of Agents derived from the general Agent «block» (Fig. 1) . The Agents instances are the parts of the System with particular names. The permissible communication in the System is shown in internal block diagram (Fig. 2) . 1) Core Agent: drives the hardware of the robot and provides the services to be called by other Agents: • navigation configuration and goal set, • control of neck and torso lift, • text-to-speech, • translation of a voice command into an intent. In the presented System there is one robot operating in the environment, so there is only one Core Agent named robot/aco. 2) Dynamic Agents: are parts of the System, which utilise services delivered by other Agents in order to perform a task that a particular Dynamic Agent is designed for. A Dynamic Agent interprets data possessed from other Agents and compute control for the robot, guiding it to perform the task implemented in the Dynamic Agent. An exemplary specification of a Dynamic Agent was described in [12] . There can be multiple Dynamic Agents in the System, however, at most one can be initialised at a time. 3) Platform Agent: aggregates System-wide computational services. As the Platform Agent from its definition operates in the cloud [13] and has huge computation power and storage space, most of the behaviours requiring high computation power are delegated here. However, some of the computationally complex behaviours have to be done by Agents located in the robot computer as they are latency-sensitive or process sensitive personal data. The method of a System behaviours delegation between Agents was presented in [14] . There is only one Platform Agent in the System. 4) Task Requester Agents: are parts of the System, which basing on the data from robot's or additional sensors request the Task Harmoniser Agent to perform tasks. There are three Task Requester Agents in the System: smart_home/atr , robot/atr and operator /atr . The first one is an interface for a smart home platform. It requests tasks basing on the defined rules. The robot/atr uses user intents captured by the robot hardware (e.g. microphone), and the operator /atr uses an application that is used for robot supervision. 5) Task Harmoniser Agent: receives task requests from the Task Requester Agents and schedules Dynamic Agents basing on a specified algorithm (e.g. priorities). The problem of Dynamic Agents scheduling was introduced in [8] . It communicates with RAPP Store Agent in order to download a proper task files implementing a Dynamic Agent. There is one Task Harmoniser Agent in the System named robot/ath. 6) RAPP Store Agent: keeps files with executable code that implement various tasks available for a System and provides a service allowing to download them. One of the main aspects of communication between Agents is shown in Fig. 3 . At first, an Elderly commands the robot Fig. 3 . Dynamic Agent call request and creation either with voice or with a button (Fig. 4) . In the first case, Fig. 4 . Dynamic Agent call request the voice is recorded by robot/aco and the recording is sent to /apl . The Platform Agent runs speech-to-text algorithms and sends back the detected intent to robot/aco. An intent represents an interpretation of the voice command. Next, the intent is sent to robot/ath via robot/atr . In the second case, a button is triggered, and smart_home/atr sends to robot/ath an intent related to the button. At this point there are two alternatives (Fig. 5 ): • task /ady can be created, so task harmoniser robot/ath downloads the code of Dynamic Agent task /ady from the RAPP Store Agent /ars. It is done by sending a message from robot/ath to /ars. In response, a message containing the code of task /ady is sent in opposite direction and robot/ath creates task /ady, • task /ady cannot be created, so robot/ath sends a message with human-readable description of failure reason to robot/aco. If needed, the Core Agent requests text-to- • another Dynamic Agent is already running, and it has higher priority than the requested one, • the detected intent is not a command for creation of a new Dynamic Agent. In order to avoid repeating text-to-speech /apl queries, the recordings are cached by robot/aco and they can be reused ('opt' frame in Fig. 5) . Another example of communication between agents is shown in Fig. 6 . In this case, the dynamic agent task /ady initiates a conversation with human by sending a message to robot/aco. Text-to-speech, if needed, is realized in /apl , thus robot/aco sends message with text to /apl and receives back a message that contains a wave file. The voice is played to human. The human says a response, and the recorded voice is processed by /apl to detect intent, similarly as in the previous example. The intent is then passed through robot/atr and robot/ath to task /ady. The experimental platform is presented in Fig. 7 . TIAGo robot was employed, as it is a robust, ROS-based [15] mobile platform used in modern research projects, e.g. [16] , especially when assistive robotics is concerned [17] . An important reason to chose TIAGo was the compatibility of Embodied Agent theory with mobile bases utilising ROS (e.g., a robot with various modes of locomotion [18] ). A detailed description of the TIAGo platform and its capabilities are presented in [19] . The robot with its internal PC and all software provided by PAL is specified as a single embodied Agent robot/aco Fig. 7) . Due to robot/aco internal PC performance limits, an additional notebook was required to perform voice acquisition and high-level task execution. Thus, a part of robot/aco Agent responsible for voice processing, and Agents robot/atr , robot/ath and all /ady are realised as software (ROS nodes) running on the notebook. The notebook and the internal PC of TIAGo communicate with each other through wireless network. ROS was used as the main implementation platform. Agent smart_home/atr is composed of devices of smart home that are connected through wireless network. Dynamic Agents /ady are implemented using SMACH [20] , however, any other framework compatible with ROS can be used. During the initial tests of the system, some audio capture problems were identified. TIAGo is equipped with a built-in microphone [21] , but it is mounted inside the robot's body, close to the head's motor. Stiff mounting of the microphone causes a high level of rumble noises during the movement, which made it almost impossible to recognise speech. To counteract this problem mounting was changed to soft sponge, but limited space inside the robot made it hard to eliminate all noises. Next problem was the long cable connecting the microphone with audio interface. This analogue cable is long and runs parallel to a bundle of high-current power cables, which induces high level of non-uniform noise. Another problem was a low dynamic range of the device. When the amplification was set too low, commands spoken from the bigger distance were ignored. Too high amplification, on the other hand, makes the noises even worse and effects with signal clipping when the user is near the robot. To cope with those problems an additional microphone was added to the system. It is an omnidirectional device, with digital interface and high dynamic range. To mitigate mechanical nosies, it was mounted using rubber bands. The recordings are much cleaner and the overall recognition accuracy went up. There was, however, another problem still occurring. Our testing environment is rather big (almost a 100 m 2 single room). There are also some places, where the high closets stand on the line-of-sight between the robot and the operator. In order to analyse the effect of the room shape and existing obstacles on the audio recognition accuracy word-spotting accuracy test was carried. During the experiment, the robot was static and different operators were trying to trigger the robot using the keyword from 12 different places (Fig. 8) . Each of the 5 users said the keyword 10 times in each spot. Keyword spotting was running on both internal and external microphone to compare the results. The first visible thing is the much higher overall accuracy of an external microphone. Not only the results are better in every spot, but also they decay less with the distance (see the movie 1 ). In some spots the accuracy dropped significantly. For example, the kitchen (spot 1) is separated from the robot with the closets. Structure elements (pillars) also can interfere direct path from the operator to the robot (spot 7). As the effective robot behaviours triggering is crucial, adding alternative resources for this purpose was a natural decision. It was achieved by utilising smart-home infrastructure and expanding it with a buttons. That kind of arrangement guarantees successful human-robot interaction. The recent problems in our world let to the conclusion that high effort should be put to assistive robots research not only because of the ageing population of the developed countries. The number of worldwide diseases showed a need to help people that have to be isolated or simultaneously are temporarily or continuously disabled. The robots can radically improve the situation even with simple applications like transportation or guarding. Such robots should communicate with patients in various ways to achieve reliable service of e.g. people that can not operate with hands or have problems with speaking. An approach proposed in this work, helps to develop control systems with the above assumptions. Currently, it used in the TIAGo robot based system to help the Elderly in the various scenarios that include transportation attendance, guarding, fall prevention and hazard detection [22] . Social robots for long-term interaction: a survey Socially assistive robots for older adults and people with autism: An overview Design for a robotic companion Long-term cohabitation with a social robot: a case study of the influence of human attachment patterns Survey of popular robotics simulators, frameworks, and toolkits Learning to use new technologies by older adults: Perceived difficulties, experimentation behaviour and usability Variable structure robot control systems: The RAPP approach Task harmonisation for a single-task robot controller EARL -Embodied Agent-Based Robot Control Systems Modelling Language A universal architectural pattern and specification method for robot control system design Agent-Based Robot Control Systems Modelling Language -version 1.2 -reference manual Distributed NAO robot navigation system in the hazard detection application Cloud computing support for the multi-agent robot navigation system Nao Robot Navigation System Structure Development in an Agent-Based Architecture of the RAPP Platform ROS: an open-source Robot Operating System Human activity recognition with convolution neural network using tiago robot Acceptance and long-term use of a social robot by elderly users in a domestic environment Control system design procedure of a mobile robot with various modes of locomotion Tiago: the modular robot that adapts to different research needs The SMACH high-level executive Adding audio capabilities to tiago service robot INCARE project page at WUT