key: cord-0154780-sip987au authors: Kodur, Krishna Chaitanya; Rajpathak, Kaustubh; Rajavenkatanarayanan, Akilesh; Kyrarini, Maria; Makedon, Fillia title: Towards a Multi-purpose Robotic Nursing Assistant date: 2021-06-07 journal: nan DOI: nan sha: cc96c433d5574af92e88d5993bb7ea39079a51f6 doc_id: 154780 cord_uid: sip987au Robotic nursing aid is one of the heavily researched areas in robotics nowadays. Several robotic assistants exist that only focus on a specific function related to nurses assistance or functions related to patient aid. There is a need for a unified system that not only performs tasks that would assist nurses and reduce their burden but also perform tasks that help a patient. In recent times, due to the COVID-19 pandemic, there is also an increase in the need for robotic assistants that have teleoperation capabilities to provide better protection against the virus spread. To address these requirements, we propose a novel Multi-purpose Intelligent Nurse Aid (MINA) robotic system that is capable of providing walking assistance to the patients and perform teleoperation tasks with an easy-to-use and intuitive Graphical User Interface (GUI). This paper also presents preliminary results from the walking assistant task that improves upon the current state-of-the-art methods and shows the developed GUI for teleoperation. The role of robotics in healthcare has been growing in the last decade [1] . This is due to the need to improve the quality and safety of care while controlling expenses [2] . Moreover, the shortage of nurses and healthcare personnel significantly impacts the quality of care [3] , [4] . The COVID-19 pandemic demonstrated how vulnerable the healthcare workers are and at risk to be exposed to the virus [5] , [21] . Several robotic systems are introduced to assist with hospital logistics, disinfection of spaces, and patient screening [15] , [16] . Robots have the potential to support caregivers in several routine tasks, while caregivers can focus on the actual patient care. For example, there is a variety of commercially available robots that are currently used in hospitals to take care of delivery tasks, such as the mobile robots TUG [6] and Relay [7] , or fetching objects, such as Moxi [8] , a mobile robot equipped with a robotic arm. Moreover, there are robots that focus on patient care, such as rehabilitation [9] , [10] , walking This work was supported by the National Science Foundation (NSF) grant IIP 1719031. assistance [11] or monitoring vital signs [12] . A recent review of robots in healthcare [1] makes it obvious that each robotic platform focuses on a specific care-giving task. In a recent single-site cohort study [22] , 40 patients interacted with a mobile robotic system, which was controlled by a clinician in an emergency department and used to facilitate triage and telehealth tasks. After the interaction, the patients completed an assessment to measure their satisfaction. A total of 37 patients (92.5%) reported that the interaction with the robotic system was satisfactory and 33 patients (82.5%) reported that their experience of a mobile robotic system facilitated the interview was as satisfactory as an in-person interview with a doctor. These findings are very promising for the acceptance of robotic telehealth. In this work, we propose a solution designed for a multipurpose robotic platform that has a high potential to perform several tasks in healthcare. The MINA robotic system is a collaborative robot that fetches objects (as shown in [29] ), assists patients with walking, and is teleoperated by nurses in an easy and intuitive manner. This work aims to assist and protect nurses with a general-purpose robot that can be employed for many applications. The main contributions of this work are the following: (a) an improved leg detection model using laser scanner data during walking assistance tasks, (b) a multimodal approach for gait detection, and (c) an easyto-use GUI that enables teleoperation. The paper is organized as follows; Section 2 discusses the related work, Section 3 presents the proposed multi-purpose robotic system MINA, and Section 4 presents the preliminary results. Finally, Section 5 concludes the work and discusses future steps. In recent years, several robotic platforms have been developed and deployed in healthcare. There are robotic platforms that assist with rehabilitation [9] , [10] , walking [13] , and surgery [14] . During the COVID-19 pandemic, robots took over logistic tasks in hospitals, patient screening, and disinfecting tasks [15] , [16] . Most of the robotic platforms are designed in an application-oriented fashion, requiring stakeholders to acquire multiple robotic platforms to assist with different tasks. Abubakar et al. [23] proposed an Adaptive Robotic Nursing Assistant (ARNA) robot, which is a custom-made robot by the University of Louisville in Kentucky and consists of an omnidirectional mobile base with an instrumented handlebar, and a 7-Degrees of Freedom (DoFs) robotic arm. ARNA has two primary tasks, a patient monitoring task, wherein the robot monitors a patient and responds to remote commands, and a patient walking task, wherein the patient controls the robot's motion while walking behind it. A cohort trial with 24 human subjects was conducted to evaluate ARNA, and the results of this preliminary user study indicate good usefulness and ease to use for the two primary robotic tasks. However, in other industries, such as warehouses and manufacturing, multi-purpose robotic systems research has advanced. An example is the Mobile Collaborative Robotic Assistant (MOCA) [17] , conceived and realized by the Italian Institute of Technology (IIT), which consists of a commercially available omnidirectional mobile base (3-DoFs) and a commercially available robotic arm (7-DoFs) equipped with a SoftHand as end-effector. MOCA has been employed in several tasks in warehouses, such as assisting humans with comanipulation of objects [18] , transportation and positioning of a pallet jack [19] , and teleoperation [17] . Inspired by the MOCA system, the MINA system incorporates many functionalities present in application-oriented systems and aims at combining them in a user-friendly manner to assist with several healthcare tasks. A. Hardware The MINA system consists of the following components; (a) the Robotnik SUMMIT-XL STEEL, which is a 3-DoFs omnidirectional mobile base and it is equipped with two UST 20LX Hokuyo laser scanners, (b) the Franka Emika Panda, which is a 7-DoFs robotic arm equipped with a 2-finger gripper, and (c) the Intel RealSense D435i camera, which integrates an RGB camera and a Depth sensor. Fig. 1 shows the MINA system and the coordinate frames of its components. One of MINA's primary purposes is to assist a user with walking and, while doing so, the system calculates the gait parameters that is a valuable information of the patient's progress. An overview of the walking assistance framework is shown in Fig. 2 . The camera captures the user's mobility and the OpenPose algorithm [24] detects the body keypoints, which are then used to calculate the gait parameters. The OpenPose provides body keypoints as pixel coordinates. These pixel coordinates are then de-projected to get the 3D coordinates of each keypoints. In our setup, the robotic arm is used as a support while walking. During this task, the user's lower body is occluded by the mobile base, which makes it difficult to detect keypoints related to the legs and ankles for gait estimation. To address this issue, we make use of the rear laser scanner that is available on the mobile base. To detect the legs from the laser scan, we generate an occupancy grid which is used as input to a Convolutional Neural Network model (CNN) to segment the legs. An occupancy grid is a binary grayscale image of size 256x256 generated to detect the leg's keypoints from the laser scanner using algorithm 1. This image is then fed into a U-Net model [28] , a commonly used CNN for segmentation tasks. In this scenario, we try to segment the user's legs from the environmental clutter. For instance, Fig. 3a shows the occupancy grid, and Fig. 3b shows a de-cluttered image showing only the detected legs. As a baseline model for leg detection, we use the U-Net model proposed by [25] . For our system, we propose the MINA model that improves upon the model proposed by [25] by retraining the U-Net model with dataset available at [26] and augmenting it with several transformations of the original dataset. We also utilize the weighted binary cross-entropy loss [27] to train the model. Once the model segments the legs, each leg's midpoint is found by calculating the respective leg's blob midpoint. This midpoint is in the form of pixel coordinates, which can be converted to 3D coordinates, as shown in (1). Result: occupancy grid (256x256 binary grayscale image) matrix length = 256; l = matrix length/2; occupancy grid = zeros(256, 256); for angle i, obstacle distance from range of angles scanned by laser do // Calculate x and y which are the real world coordinates in laser frame x = obstacle distance × cos(i); y = obstacle distance × sin(i); pixel x = x × 100 + l; pixel y = y × 100 + l; if pixel x and pixel y between 0 and matrix length then occupancy grid[pixel x ][pixel y ] = 255; end end where matrix length is the required image shape, pixel x and pixel y are the detected legs' midpoint in the x and y axes. Here, z is 0 because the laser scanner in use is 2D, and its scan spans only in the x-y plane. The 3D body keypoints are with the respect to the camera coordinate frame (C), and 3D leg keypoints from the laser sensor is in the laser coordinate frame (L). All the keypoints data need to be with respect to the same coordinate frame in order to fuse the multimodal data for gait estimation and robot control. To achieve this, each keypoint is transformed to the robot's base coordinate frame (R) using (2) and (3). In the aforementioned equations x P is an 3D point in coordinate frame x where x ∈ (C, L). x y M is a conversion matrix that converts the 3D points from frame x to frame y where y ∈ R. After the legs are segmented and validated, gait parameters, such as stride velocity and stride length are calculated. This stride velocity acts as input to the mobile base to set in which direction and at what speed the robot should move forward, thus automatically adjusting to the user's speed and direction. Another important task of a robotic system that assists nurses is the ability to enable nurses to care for patients that are in isolation (e.g., patients infected by an easily transmissible disease). For example, during the COVID-19 pandemic, nurses are required to gear up with personal protective equipment when they are in contact with an infected patient. Robotic teleoperation provides an easy alternative where a nurse controls the MINA system to provide a drink to the patient. In this work, we have developed a GUI, which is easy to use and requires minimal training for the nurses. Fig. 4 shows the GUI that consists of two parts. The first part (Fig. 4 -Right) shows the output of object detection system YOLO (You Only Look Once) [20] , which is the camera feed, the bounding boxes, and the name of the detected objects. The second part (Fig. 4 -Left) has several buttons that enable the control of the robot's end-effector position in 3-dimensional space and its orientation and gripper actuation. It also enables the user to select the detected objects by YOLO and to enable an automatic pick and place of the selected objects. Figure 5 shows an image of the system while it is teleoperated through the GUI to provide a drink to a user. [25] . The laser scan from the MINA mobile base is converted into an occupancy grid image and is used as input for both the models, and the resulting leg segmentation results are compared. The nine locations around the robot are chosen to compare the model outputs and generate statistics. From table I, the baseline and our MINA model are evaluated on two scenarios. One scenario is more cluttered than the other. The Scenario with more clutter has a number of small boxes on the ground, which are usually mistaken as human legs by the model. In each scenario the model is evaluated while a human stands at nine different locations as shown in Fig. 6a . Let n t be the total number of times the model is evaluated by giving different inputs by varying the locations and scenarios, n s be the total number of times when the model correctly segmented the legs, the accuracy (acc) can be calculated using (4) . For the baseline model, the acc is 77.7%, and for our model (MINA), it is 94.4%. Another evaluation metric is called False Positive rate (FP); that measures if any other object except legs is segmented. Let n f be the total number of times when the model incorrectly segments other environment clutter as legs, the FP can be calculated using (5) . For the baseline model, the FP is 38.8%, and for our model MINA, it is only 5.5%. It is clear that our model performs better than the baseline model in both acc and FP metrics. One factor that contributed reducing the FP is that the U-Net is trained with data augmentations, such as random rotation. Segmentation results from two different locations are shown in Fig. 6b . In this work, we proposed a novel Multipurpose Intelligent Nurse Aid (MINA) collaborative robot system that is capable of assisting users with walking and is teleoperated in an easy and intuitive manner. We presented preliminary results of our leg detection algorithm that showed improved accuracy and false positive rates in comparison to the state-of-the-art leg detection algorithm. We also show the GUI developed for teleoperation of robotic manipulation that is easy to use and requires minimal training. We are currently working to verify our leg detection algorithm in several different scenarios to ensure safe operation. As future work, we propose to study an impedance control strategy in conjunction with model predictive control to ensure an optimal control strategy that keeps the robot within a strict distance and matched velocity to the patient. Sensing will be provided by embedded force/torque sensing in the robot arm in conjunction with the laser sensors. We also intend to add more functions to the MINA system like disinfection of a room, carrying an IV pole, and many others. We believe that such a robotic system will be useful in several hospital and nursing home scenarios to provide safe care, ensure nurse's safety, and reduce nurse burden. A Survey of Robots in Healthcare Health care robotics: qualitative exploration of key challenges and future directions Effective policies for eliminating nursing workforce shortages: a systematic review Caregiver shortage reaches critical stage Update: Characteristics of Health Care Personnel with COVID-19 -United States Mobile Robots Improve Patient Care, Employee Satisfaction, Safety, Productivity and More Relay Autonomous Service Robot for Hospitals Routine activities completed with ease through automation by Moxi A Review on Upper Limb Rehabilitation Robots A review on lower limb rehabilitation exoskeleton robots Research on Driving Control Technology of a Multifunctional Elderly-Assistant & Walkingassistant Robot Mobile app for human-interaction with sitter robots The i-Walk Assistive Robot: a Multimodal Intelligent Robotic Rollator Providing Cognitive and Mobility Assistance to the Elderly and Motor-Impaired Autonomy for surgical robots: Concepts and paradigms Healthcare robots to combat COVID-19 The implementation of social robots during the COVID-19 pandemic A teleoperation interface for loco-manipulation control of mobile collaborative robotic assistant MOCA-MAN: A MObile and reconfigurable Collaborative Robot Assistant for conjoined huMAN-robot actions A Collaborative Robotic Approach to Autonomous Pallet Jack Transportation and Positioning You Only Look Once: Unified, Real-Time Object Detection Covid-19: control measures must be equitable and inclusive Assessment of the Acceptability and Feasibility of Using Mobile Robotic Systems for Patient Evaluation ARNA, a Service robot for Nursing Assistance: System Overview and User Acceptability Open-pose: Realtime multi-person 2d pose estimation using part affinity fields Tracking people in a mobile robot from 2d lidar scans using full convolutional neural networks for security in cluttered environments Benchmark dataset for evaluation of range-based people tracker classifiers in mobile robots The real-world-weight cross-entropy loss function:Modeling the costs of mislabeling U-net: Convolutional networks for biomedical image segmentation MINA: A Multitasking Intelligent Nurse Aid Robot ACKNOWLEDGMENT This work has been partially supported by National Science Foundation (NSF) grant IIP 1719031. This material is based upon work by the authors. Any opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the NSF.