key: cord-0058226-mpza6x3p authors: Aruanno, Beatrice; Caruso, Giandomenico; Rossini, Mauro; Molteni, Franco; Espinoza, Milton Carlos Elias; Covarrubias, Mario title: Virtual and Augmented Reality Platform for Cognitive Tele-Rehabilitation Based System date: 2020-08-10 journal: Computers Helping People with Special Needs DOI: 10.1007/978-3-030-58796-3_17 sha: a8748d272ab38f8f35de4b5e987d0addf3acf5ad doc_id: 58226 cord_uid: mpza6x3p Virtual and Augmented Reality systems have been increasingly studied, becoming an important complement to traditional therapy as they can provide high-intensity, repetitive and interactive treatments. Several systems have been developed in research projects and some of these have become products mainly for being used at hospitals and care centers. After the initial cognitive rehabilitation performed at rehabilitation centers, patients are obliged to go to the centers, with many consequences, as costs, loss of time, discomfort and demotivation. However, it has been demonstrated that patients recovering at home heal faster because surrounded by the love of their relatives and with the community support. The ambition of the Virtual and Augmented Reality Platform based on Tele-Rehabilitation is to combine assistance with appropriate feedback to close the loop in cognitive learning strategy for the home. The tele-rehabilitation system is based on the use of low cost sensors, connected with a gaming module for cognitive-motor rehabilitation, and integrated with an infrastructure connecting the patient at home with therapists at hospitals who monitor the rehabilitation exercises. Specifically, the ambition of the Platform is to strengthen and extend rehabilitation services, including community-based rehabilitation, and assistive technology. The platform addresses people with Cognitive Disability. Cognitive Disability (CD) is a broad concept encompassing intellectual or cognitive deficits derived from either specific neurodevelopmental disorders (e.g., Autism) or problems emerging later in life such as brain injuries or neurodegenerative diseases like dementia. Most individuals with CD often experience co-occurring difficulties in different areas such as language and speech, memory, learning, social behavior and motor skills [4] . Designed in cooperation with CD specialists (psychologists and therapists) of three Rehabilitation Center in the north of Italy, the VR/AR platform aims to promote the understanding and execution of some basic tasks that are typical of domestic routines, helping people with CD to improve their autonomy in everyday life, or at least to make them less dependent on caregivers. The VR/AR platform allows us to implement a set of exercises for cognitive rehabilitation. The interaction with virtual objects is performed through hand and body movements and gestures. The most innovative aspect of the platform is that the interaction is performed by using the Oculus rift, the LeapMotion, the Kinect sensors and the HoloLens in a unique platform. The VR/AR platform is used by the patients, by the neurologist in charge of the patient and by other clinical specialists (e.g., by the neuropsychologists and physiotherapists). Each of the users has proper functions: the patient can see the daily exercises therapy, the neurologist can assign, modify or evaluate the therapeutic plans, the medical staff can check how the rehabilitation is going. The interaction of the different users and the logical architecture of the platform are showed in Fig. 1 The Indoor Unity represents the patient's interaction with the platform. The main software tools we used in the implementation of the VR/AR platform are Unity and Visual Studio. The scripts are coded in C-Sharp, which is more supported than other programming languages by both Microsoft and Unity documentation. The application source consists of Unity assets and C-Sharp scripts. Unity assets are managed by the Unity engine and include the 3D models used for the different scenes and holograms, the UI elements, and the Mixed Reality Toolkit. The Mixed Reality Toolkit is a collection of components provided by Microsoft, which accelerates the development of applications for Microsoft HoloLens and other Windows Mixed Reality headsets. For instance, the input module contains scripts that interpret inputs such as gaze, gesture, and voice, and the spatial mapping module is used to map the real world into the MR environment. We have developed a visualization and interaction system integrated with the gaming module that visually renders the virtual environment for rehabilitation and also the user's virtual hands and body in real-time. This functionality requires tracking the user's hands and body in the physical space. In order to provide a realistic immersiveness, the tracking and the representation of the user's hands and body in the virtual environment should be accurate and timely. Figure 2 shows a training session. In fact, some studies have shown that if the users are able to see the virtual rendering of their hands and legs and their movements relative to the movements of other objects there is a much better chance that they will feel that the virtual hands embodies their intentions and actions [3] . The gesture interaction is performed through the LeapMotion and Kinect Sensors in the cognitive exercise who are supported by the Oculus Rift VR device. While for the HoloLens the hand gestures (air-tap) enable the interaction with holograms or other digital contents. Instantaneous air-tap selects and activates the gazed interactive element. Prolonged air-tap enables drag and drop. The exercise database consists of three different categories which are connected through a film database and a picture database. A film is used in order to catch the attention of the user. One of the strengths of the platform is its customizability and modularity. Customizability is particularly important in applications for people with CD, who have enormously different cognitive and motor skills and evolving therapeutic or educational needs. In each VR/AR activity it is possible to choose the "configuration" that is most suitable for the specific user, for instance the level of difficulty, the number and the kind of objects involved in the task, the film who is connected to the cognitive exercise. A high modularity also facilitates design extensions and technical improvements, like the addition of new levels and new objects, which can be performed with a little implementation effort. -Category 2: Exercise =⇒ Film: This category of exercises starts directly with the rehabilitation exercise and the video is seen by the patient only if the exercise has been completed. The film is used as an award. A set of 10 cognitive exercise have been developed as can be seen from Fig. 3 . The exercises have been designed in order to offer a more effective and customized training for cognitive deficiencies, allowing for an easy customization and definition of exercises; offering the possibility of performing training programs in autonomy, without the need of the continuous presence of caregiver/parents, but under their continuous monitoring, and which can be performed quietly at home; allowing CD specialist to use a complete and rich set of data recorded from the exercises executed by the users to improve knowledge on rehabilitation and therapies. There already exist, some projects that explore therapeutic approaches based on social stories [5] and storytelling [6] , mostly targeting young people with Autism [2, 8] . The benefits of MR in the CD arena are much less explored. To our knowledge, the only HoloLens application that addresses people with Cognitive Disability is the one reported in [1] , which addresses people with Alzheimer's Disease. The application consists of a set of tasks that aims to slow down mental decline by strengthening short-term memory and spatial memory (which are usually damaged by Alzheimer's Disease). In our research, a user study has been carried out in order to test the VR/AR platform initially with healthy people. In particular, we were interested in checking if any issues may arise during the use of the Virtual Reality application, in particular any issues related to gesture interactions. 10 healthy users, 6 female and 4 male, aged between 18 and 23 participated to the preliminary tests. Before the test, the participants were asked to fill in a pre-test questionnaire with their data, confidence to use games and also hand gestural technologies. In addition, we asked the participants to compile a symptoms check-list related to the sense of sight. Participants were instructed about the task, and they were allowed to use the hand gestural approach until they felt sufficiently confident. The task consisted in using the 'Cancel and go' exercise. The test lasted approximately 8 min. Figure 4 -a shows the user while starts with the exercise with the left hand. Figure 4 -b shows the instant in which the user is cancelling the image of the rehabilitation exercise. The questionnaire was organized in a 6 points Likert-scale, from 1 (which is the most negative value) to 6 (which is the most positive value). The charts on Fig. 5 show the results of the questionnaires. The VR application achieved a high evaluation rate relatively to the aspects concerning the system in general, a quite positive evaluation of the easiness in using it as a whole (chart a in Fig. 5 ). The knowledge acquisition section of the test intended to go more in details in understanding and evaluating the system from the user's perspective. Chart b in Fig. 5 shows the results. Overall, the collected data show a positive evaluation. Only one user assigned a very low rate for what concerned the easiness of using the system the first time. But the same user was convinced that the following time it would have been easier and more natural to use it. Do you think that the next Ɵme will be easier to use the system? Did you perceive the informaƟon you expected? Was it easy to start using the system? These anomalies however, were usually short-termed and did not represent a significant impact in the user's performance, and in general in the overall results. Nevertheless, these anomalies were known to happen [7] and therefore expected. This preliminary study was designed to cope with these issues. 15 CD subjects from the ASPOC association [9] aged between 19 and 22 participated to the SUS tests. Participants were instructed about the task, and they were allowed to use the hand gestural approach until they felt sufficiently confident. Also in this case, the task consisted in using the 'Cancel and go' exercise. The same exercise than in the user study with healthy subjects. The test lasted approximately 10 min. Figure 6 shows the SUS results. The paper describes a VR/AR platform based on hand and body gesture interaction. We have performed some tests in order to prove our approach. The preliminary test results reported in the paper are positive for what concerns the quality of the hand and body gesture interaction while executing the cognitive exercise. On this basis, we conclude that the gestural interaction system provides users with an effective and natural method to interact with the rehabilitation exercises. The study results, although preliminary, highlight a high degree of likeability. Still, for people with severe forms of CD. In the shorter term, we will add new cognitive exercise and more complexity levels; they will be inspired by stories, films and cartoons currently used during regular therapeutic activities, and partially designed by people with CD through a co-design process. We will also include more options for the current activities and will implement new types of tasks, already designed with the therapists in our team. HoloLens-based mixed reality experiences for subjects with alzheimer's disease A survey on virtual reality for individuals with autism spectrum disorder: design considerations Hand tracking and visualization in a virtual reality simulation American Psychiatric Association Diagnostic: Statistical manual of mental disorders Creating social stories as wearable hyper-immersive virtual reality experiences for children with neurodevelopmental disorders Wildcard: a wearable virtual reality storytelling tool for children with intellectual developmental disability An analisis of the precision and reliability of the leap motion sensor and its suitability for static and diynamic tracking Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism