key: cord-0680900-05av6yuo
authors: Sobrepera, Michael J.; Lee, Vera G.; Garg, Suveer; Johnson, Michelle J.; Ph.D,
title: Feasibility and Acceptability of Remote Neuromotor Rehabilitation Interactions Using Social Robot Augmented Telepresence: A Case Study
date: 2022-02-27
journal: nan
DOI: nan
sha: 82376d2dfb32d0e5fd7b175d8cb678945bc3ce77
doc_id: 680900
cord_uid: 05av6yuo

There is a growing need to deliver rehabilitation care to patients remotely. Long term demographic changes, geographic shortages of care providers, and now a global pandemic contribute to this need. Telepresence provides an option for delivering this care. However, telepresence using video and audio alone does not provide an interaction of the same quality as in-person. To bridge this gap, we propose the use of social robot augmented telepresence (SRAT). We have constructed a demonstration SRAT system for upper extremity rehab, in which a humanoid, with a head, body, face, and arms, is attached to a mobile telepresence system, to collaborate with the patient and clinicians as an independent social entity. The humanoid can play games with the patient and demonstrate activities.These activities could be used both to perform assessments in support of self-directed rehab and to perform exercises. In this paper, we present a case series with six subjects who completed interactions with the robot, three subjects who have previously suffered a stroke and three pediatric subjects who are typically developing. Subjects performed a Simon Says activity and a target touch activity in person, using classical telepresence (CT), and using SRAT. Subjects were able to effectively work with the social robot guiding interactions and 5 of 6 rated SRAT better than CT. This study demonstrates the feasibility of SRAT and some of its benefits.

Growing shortages of neuromotor rehabilitation clinicians [1] , [2] and changing demographics present a challenge to delivering effective rehab care. Frequent rehabilitation care is critical for convalescence from conditions such as stroke and cerebral palsy (CP), which can limit a person's ability to fully interact with the world around them. One option for overcoming these challenges is to provide rehab care using telepresence, i.e. telerehab, a subcategory of telehealth, the remote delivery of health care using telecommunications [3] . Telehealth has recently seen a large increase in use as a result of the COVID-19 pandemic and related regulatory adjustments [4] . However, telehealth with video or audio alone does not provide as rich of an interaction as inperson. Adding a social robot to interact with patients during telehealth interactions may be able to improve those interactions. Previous work has shown therapists believe adding a social robot to augment telehealth would significantly improve communication with patients, patient motivation, and patient compliance during telerehab interactions, compared to traditional telepresence [5] . Such a system could be deployed at primary care clinics, schools, or other locations in the community to provide access to rehabilitation care for patients who can not otherwise access it locally. This would enable patients to receive more frequent care without the cost and difficulty of traveling to a distant center of rehab care excellence. Understanding how patients react to such systems will allow further development and possibly lead to better care for patients. In this paper, we present a case series evaluating the feasibility and acceptability of social robot augmented telepresence (SRAT) for rehabilitation relevant activities, using the social robot, Flo, a small humanoid, with a torso, head, arms, and face. The humanoid rides on a mobile platform with a screen, cameras, and microphones ( fig. 1) .

This work examines the combination of telerehabilitation and socially assistive robots. Both have rich literature showing the promise of each for delivering rehabilitation care.

1) Social Robots for Upper Extremity Rehab: Social robots for rehabilitation are not new. They fall within the broader category of socially assistive robots [6] which combine both assistive robots, which support users with disabilities, and social robots, which are designed to interact and communicate with humans. Several systems have been developed for upper extremity rehabilitation, ex: The NT project uses a Nao robot to interact with patients, and a Microsoft Kinect sensor to track patients, allowing the robot to autonomously demonstrate and correct poses in a pose mirroring game and in a pose sequence recall game. In a longitudinal study of the system, with 13 subjects participating in on average 11.6 sessions, each approximately 24 minutes long, clinicians, parents, and children, found the system useful and wanted to continue to use it [7] . RAC CP Fun is another Nao based robotic platform designed to engage with preschool students who have CP by playing games and motivating physical activity. The robot interacts by singing songs, changing its position relative to patients, and providing feedback. Fridin et al. found that children with CP exhibited a higher level of interaction with the robot than their typically developing peers, as measured by eye contact as well as various facial, body, and vocal expressions of emotion [8] , [9] . The Bandit robot is a social robot which is targeted towards older adults and is designed to monitor, instruct, evaluate, and encourage patients while they do their rehab. In a multi-session study with 33 subjects, the robot was found to be effective at motivating physical exercise [10] .

2) The importance of presence in interactions: Robots provide challenges in cost and complexity when being added to a telepresence system. Why then are they worth pursuing?

Considerable evidence suggests that physical presence is important for motivation, compliance, understanding, and enjoyment in both rehab and non-rehab interactions. Fridin et al. demonstrated with RAC CP Fun that, comparing an in-person robot and robot projected onto a screen, pediatric subjects interacted significantly more with the in-person robot [11] .

The Bandit robot was also tested with subjects in physically present, virtually present, and simulated presence conditions. The physically present condition was preferred [10] . Bainbridge et al. showed that physical presence is important for trust and motivation, especially for uncomfortable tasks [12] , and Kiesler et al. showed that subjects co-located with a physical robot were more engaged with it and better followed diet advice when compared to a virtual agent [13] . Vasco et al. similarly found that stroke patients prefer using SARs for rehabilitation therapy over virtual agents shown on a screen, reporting higher engagement levels and exercise performance with the physical robot [14] Mann et al. demonstrated that subjects were more likely to trust, be engaged with, and follow instructions from a robot giving instructions and asking questions, compared to the same interface on a tablet [15] . And Céspedes et al. showed that simply adding a SAR onto a pre-existing neurorehabilitation device as a third agent for motivation and engagement purposes has the potential to increase performance [16] .

3) Telerehabilitation: Telepresence systems can be as simple as using a screen, camera, and Internet connection via a cellphone, tablet, or computer, equipment which patients and providers often already possess. Some systems include a mobile robotic base which can be remotely controlled, such as the commercialized systems from Double Robotics and VGo Communications. Others have robotic appendages to communicate the operator's intent [17] or screens which can actuate to face the direction the operator is looking [17] , [18] .

There have been reported successes in telerehabilitation. For example, Dodakian et al. presented a tabletop game system attached to a computer for rehabilitation of stroke patients. By prompting the patient to play physical games while monitoring movements over telepresence, patient compliance and motivation was increased [19] . Abel et al. found that patients' range of motion can be effectively assessed over telepresence, and some patients may prefer telehealth appointments for certain rehabilitation tasks, such as range of motion assessments and wound tracking [20] . Preference is driven largely by convenience. Bettger et al. compared a tele-physical therapy program for therapy post total knee arthroplasty (n=143) with a traditional in-person therapy program (n=144) for 12 weeks. The telerehab program had lower costs, lower rates of rehospitalization, and was non-inferior in measures of rehab outcomes when compared to the traditional program [21] .

However, limitations associated with this technology, including field of view of the operator (clinician), network latency, screen resolution, and projection of three-dimensional interactions into two dimensions, lessen the perception of the presence of the remote operator and reduce spatial reasoning for both users (clinician and patient) [18] , [22] . The resulting lack of physical presence, coupled with unclear instructions for movements over telepresence, may decrease patients' compliance and motivation to perform required motor assessment tasks and, as a result, limit the clinician's ability to assess the patient's current function and progress and motivate home therapy adherence. This highlights a need to develop platforms that have a physical presence and can perform both assistive and social functions. With the current pandemic, the need and call for telerehab systems has grown [23] , a need which will continue to grow with shifting demographics and future unforeseen care challenges.

Recognizing the broad potential impact from telerehab and that some of the challenges with telepresence could be overcome by using a socially assistive robot, we propose to augment telerehab interactions with social robots. In a previous study of 351 therapists in the United States, therapists reported they believe social robot augmented telepresence (SRAT) would significantly improve communication, motivation, and compliance during telerehab interactions, compared to classical telepresence. To explore this new direction in healthcare robotics, we have developed Flo, an example SRAT.

The system is comprised of a Kobuki drive base, two RealSense D415 cameras, a screen, speakers, and a fisheye camera ( fig. 1) . A humanoid with a body, head, arms, and a face can be placed onto the system and easily removed. This allows the system to be used in both classical telepresence (CT) and SRAT configurations. The system is controlled through a server via a remote web interface by a plays and script methodology with wizard of oz control. The humanoid can synthesize arbitrary speech using AWS Polly, and the face of the humanoid can be dynamically changed [24] . More design details can be found in [25] .

Here we explore the feasibility of social robot augmented telepresence for engaging in upper extremity rehabilitation activities, using the Flo platform to compare SRAT to CT.

The study was approved by the University of Pennsylvania Institutional Review Board. In addition to consenting to participate in the study, all subjects also provided an optional media release, allowing publication of their images.

After consenting, subjects were assessed using the box and block test [26] to measure unilateral gross manual dexterity, children's color trails test for pediatric subjects [27] and color trails test for adult subjects [28] to measure visual attention, graphomotor sequencing, psychomotor speed, and cognitive flexibility, as a proxy for executive cognitive function more generally, and grip strength test [29] to measure hand and forearm strength as a proxy for upper limb strength ( fig. 2 ).

Subjects, aided by a caretaker/parent if present, answered a survey asking: basic demographics, history of cognitive and motor impairment, technology usage, level of education, and current therapy practice and compliance, how they are feeling today, and how they feel about robots. Surveys were administered by a study team member, enabling data to be collected from subjects of all ages and cognitive levels as well as allowing follow-up questions to be asked.

Subjects were randomly assigned to an ordering group. With FTF signifying face-to-face ( fig. 3), SRAT (fig. 4) , and CT ( fig. 5 ), the groups are: FTF-SRAT-CT and FTF-CT-SRAT. Once the intake surveys were complete, the subject was seated in a room. For the FTF condition, the operator entered the room and sat on a chair, next to the telepresence system, which was not connected, but used to gather video data, in front of the subject. For the SRAT/CT conditions, the robot entered the room under remote control by the same operator. This ordering sequence was used to give each subject a baseline face-to-face experience to begin, as would likely happen in the real world, before moving on to the telepresence modalities. Previous results have suggested that when initial interactions are remote, results are poor [11] .

To begin each interaction, the operator introduced themselves and, in the SRAT condition, the robot. The operator asked the subject if they wanted to play a game. When the subject said yes: in the SRAT condition the robot said "great, let's play Simon Says" and explained how to play a game of Simon says, in the CT/FTF conditions, the operator said "great, let's play Simon Says" and explained how to play a game of Simon Says. The game was played with the subject moving through 20 motions with an additional 6 repeated motions where there was no Simon Says command, with all instructions given by the robot in the SRAT condition. The motions were identical in all conditions, composed of bimanual activities: clapping, reaching overhead with hands apart, reaching out forward with arms, covering both eyes and unimanual activities: reaching across the body to touch the shoulder, touching the mouth, touching the top of the head, reaching out to the side, etc. For the first four subjects, the unimanual activities were done individually for each arm, randomly ordered. As the study evolved, for the final two subjects, activities were made more difficult and engaging using bilateral non-symmetrical motions constructed randomly (ex: "wave with your right arm and touch your shoulder with your left hand"). This game pushed the subjects to the ends of their range of motion, using motions which are relevant for activities of daily living (ADLs).

The operator then said that they want the subject to play another game. In the CT/FTF conditions, the operator explained the game and in the SRAT condition, the robot explained it: "In the target touch activity I will tell you to touch the dots on my [hands/board] ... Let's start in a ready position, return to this position after every touch". In the CT condition, the telepresence system had colored dots mounted on a board hanging on the robot. In the FTF condition, the same board was mounted on the platform next to the operator. In the SRAT condition, the robot had the same dots on its hands and moved its hands to the same points as those on the board. The first four subjects were instructed to touch each of three dots with each hand ten times, for a total of sixty point-to-point motions. The final two subjects were instructed to complete a series of 1 to 4 dot/hand sequences with two dots available, randomly selected. This activity tests the attention and motivation of the subjects by instructing them to do repetitive motions. In the modification made for the final two subjects, difficulty was increased, and more sustained attention was required rather than simple touching.

On completion of each condition, the robot/operator exited the room. Subjects were then given a survey asking questions from the NASA task load index (TLX) [30] , intrinsic motivation inventory (IMI) [31] , how well they enjoyed the interaction, if they would like to do the interaction again, and how safe they felt during the interaction. Once the surveys were completed, subjects rested until at least 15 minutes had passed since the end of the prior condition.

At the end of the study, subjects are asked questions about the entire experiment: which interaction modality they thought was best and several questions adapted from Telemedicine Satisfaction and Usefulness Questionnaire [32] on whether they thought telemedicine would change how they manage their healthcare, communicate with their clinicians, and if telehealth visits would be convenient to them. 

Eight subjects participated in the pilot trial. The surveys were dramatically edited between the first and second subjects, so the first subject was excluded from analysis. For one other subject, the power system on the robot failed, preventing that subject from completing the protocol. As a result, that subject was also excluded. Therefore 6 subjects' results were analyzed. Complete subject information can be seen in table I.

The subjects with stroke varied functionally -one had aphasia, another had spasticity and loss of function in one hand, preventing independent finger and wrist actuation. All but one subject (PD) performed 2 to 6 standard deviations below normal on the box and block test [26] , [33] . Subjects varied from severely impaired to typically functioning on the color trails test. At intake, all subjects reported being happy, excited to neutral, and dominant to neutral. All subjects reported positive feelings towards robots. All subjects had previously made video calls, only one had done so for healthcare (GS). Subjects initially were mixed on their feelings towards using video calls for healthcare.

AJ, BF completed the study in the FTF-CT-SRAT order. HL, GS, VM, PD completed it in the FTF-SRAT-CT order.

All subjects were able to fully participate with the robot. No subjects reported difficulty understanding the instructions given by the humanoid. AJ reported difficulty in the CT condition with seeing the operator due to the small screen size. The last two subjects to participate (PD and BF), who were the youngest and experienced more challenging activities, appeared to have more difficulty with the Simon Says activity and had significant difficulty with the target touch activity across conditions. PD was not comfortable with left and right hands and had difficulty with more than two step instructions.

Using the IMI (1 low to 7 high), enjoyment for each condition averaged between 5.1 and 5.8, competence averaged between 6.5 and 6.7, effort averaged between 3.8 and 4.4, and pressure averaged between 2.1 and 2.4.

All three pediatric subjects rated SRAT as the best modality. The three adult subjects rated SRAT as the second-best modality. Complete ratings are shown in table I.

Throughout the interactions, the subjects had opportunities to provide feedback and the study staff recorded notes. GS reported not liking telepresence, but liked working with the social robot, and stated that they felt less pressure as the robot could adjust its pacing to their needs (compared to a human who may be in a rush). AJ said that they thought the social robot would be useful for home practice. VM reported that the SRAT interaction was the best due to it being the funniest. BF was more energetic filling out surveys after the SRAT interaction than the other interactions. PD, the youngest subject, was very excited to interact with the social robot. After completing the experiment and surveys, they asked if they could play with the robot more. We drove it into the room and had the robot say hello, wave, and talk with them. They interacted with it and ended up petting it on its head as if it were a pet or a doll.

This case series covers users of a wide variety of ages and motor and cognitive function. All subjects were able to fully participate in three conditions: face to face, classical telepresence, and social robot augmented telepresence. Enjoyment was high for all conditions. All conditions required moderate effort, but subjects reported high competence and felt low pressure. With this size of a case series, comparisons between measures after conditions are not warranted. The high level of enjoyment and the ability of all subjects to interact with the social robot while it augmented the telepresence interaction demonstrates acceptability of the system by patients.

The social robot augmented telepresence system was rated higher than traditional telepresence by five of the six subjects (all but AJ). This is an early indicator of the benefit that the social robot can add. The pediatric subjects all rated the social robot augmented condition as best. This may be partly a novelty effect, which is well documented in the literature. But previous systems, such as the NT, have demonstrated the use of social robots for rehab longitudinally with good results [7] . Even if there is a large amount of novelty driven interest, by using evolving games and activities coupled with dynamic robot expressions, high engagement should be maintained over time. Seeing the youngest subject's (PD) high enthusiasm for the social robot was exciting to the study team. That subject clearly assigned a personality to the robot.

One of the adult subjects commented on the social robot being well suited for subjects who move slower, because it can adapt to their pace, reducing the pressure during visits while encouraging improved performance. This highlighted a point of utility that we had not anticipated and adds more rationale for developing social robots for rehabilitation and remembering to design with elders, not just children, in mind.

All subjects had positive views towards robotics at the start of the trial and had previous experience making video calls, although only one for health care purposes, which may bias the respondents towards accepting the social robot.

We completed a case series with 6 subjects interacting with the Flo robotic system, an example of social robot augmented telepresence. By multiple metrics, the system performed well, was accepted by subjects, and appeared feasible to use for upper extremity motor rehabilitation activities and assessments.

Occupational Therapy Workforce in the United States: Forecasting Nationwide Shortages

Physical Therapy Workforce in the United States: Forecasting Nationwide Shortages

Telehealth applications for outpatients with neuromuscular or musculoskeletal disorders: Telehealth Applications for Outpatients

Perspective on COVID-19: Finally

Perceived Usefulness of a Social Robot Augmented Telehealth Platform by Therapists in the United States

Defining Socially Assistive Robotics

A Socially Assistive Robotic Platform for Upper-Limb Rehabilitation: A Longitudinal Study With Pediatric Patients

Robotics Agent Coacher for CP motor Function (RAC CP Fun)

Kindergarten social assistive robot: First meeting and ethical issues

A Socially Assistive Robot Exercise Coach for the Elderly

Embodied Robot versus Virtual Agent: Involvement of Preschool Children in Motor Task Performance

The Benefits of Interactions with Physically Present Robots over Video-Displayed Agents

Anthropomorphic Interactions with a Robot and Robot-like Agent

Train with Me: A Study Comparing a Socially Assistive Robot and a Virtual Agent for a Rehabilitation Task

People respond better to robots than computer tablets delivering healthcare instructions

Social Human-Robot Interaction for Gait Rehabilitation

MeBot: A Robotic Platform for Socially Embodied Presence

Motion and Attention in a Kinetic Videoconferencing Proxy

A Home-Based Telerehabilitation Program for Patients with Stroke

Can Telemedicine Be Used for Adolescent Postoperative Knee Arthroscopy Follow-up?

Effects of Virtual Exercise Rehabilitation In-Home Therapy Compared with Traditional Care After Total Knee Arthroplasty: VERITAS, a Randomized Controlled Trial

Can You See Me Now? How Field of View Affects Collaboration in Robotic Telepresence

COVID-19: Maintaining essential rehabilitation services across the care continuum

Designing and Evaluating the Face of Lil'Flo: An Affordable Social Rehabilitation Robot

The design of Lil'Flo, a socially assistive robot for upper extremity motor assessment and rehabilitation in the community via telepresence

Norm scores of the box and block test for children ages 3-10 years

Children's Color Trails

Evaluation of two new neuropsychological tests designed to minimize cultural bias in the assessment of HIV-1 seropositive persons: A WHO study

Grip and pinch strength: Norms for 6-to 19-year-olds

Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research

Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: A confirmatory factor analysis

Development, Validation, and Use of English and Spanish Versions of the Telemedicine Satisfaction and Usefulness Questionnaire

Adult Norms for the Box and Block Test of Manual Dexterity

We thank the subjects who participated in this work.