key: cord-0674620-byqhqbd3
authors: Lenz, Christian; Behnke, Sven
title: Bimanual Telemanipulation with Force and Haptic Feedback and Predictive Limit Avoidance
date: 2021-09-27
journal: nan
DOI: nan
sha: 5536db2fe4aabb588ae5fbf5af61c506c34f8630
doc_id: 674620
cord_uid: byqhqbd3

Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting of an anthropomorphic avatar robot and an operator station providing force and haptic feedback to the human operator. The avatar arms are controlled in Cartesian space with a 1:1 mapping of the operator movements. The measured forces and torques on the avatar side are haptically displayed directly to the operator. We developed a predictive avatar model for limit avoidance which runs on the operator side, ensuring low latency. Only off-the-shelf components were used to build the system. It is evaluated in lab experiments and by untrained operators in a small user study.

Teleoperation is a very powerful method to control robots. It enables humans to explore remote locations and to interact there with objects and persons without being physically present. Although state-of-the-art methods for autonomous control are improving rapidly, the experience and instincts of humans, especially for solving unpredictable problems is unparalleled so far. The current COVID-19 pandemic is a great example where remote work is highly desirable. Further possible applications for teleoperation include disaster response and construction where humans can operate remotely without risking their lives as well as maintenance and healthcare to allow experts operating in remote locations without the need of travel. Robotic teleoperation is a popular research area which is advanced by multiple robotic competitions like the DARPA Robotics Challenge [1] , RoboCup Rescue [2] and the ANA Avatar XPRIZE Challenge 1 .

In addition to immersive visualization of the remote location, one important aspect is telemanipulation which enables the operator to physically interact with the remote environment. This capability is critical for many applicationswithout it, we are constrained to mere telepresence.

In this work, we present a humanoid bimanual telemanipulation system built from off-the-shelf components, which allows a human operator to interact and manipulate in remote locations. Our contributions include: 1) An upper-body operator exoskeleton and a bimanual robotic avatar, 2) an arm and hand controller with force and haptic feedback, and 3) a model-based arm movement prediction to haptically display position and velocity limitations of the remote avatar in real time.

Teleoperation is a widely investigated research area. A leading device (in our context called the Operator Station), often with haptic feedback is used to control a following device (Avatar Robot) in a remote location. The DARPA Robotics Challenge (DRC) 2015 [1] required the development of mobile teleoperation systems. Several research groups, such as DRC-HUBO [3] , CHIMP [4] , Ro-boSimian [5] , and our own entry Momaro [6] presented teleoperation systems with impressive manipulation capabilities. The focus was on completing as many manipulation tasks as possible using a team of trained operators. In addition, the robots were not required to communicate or interact Operator (left) and avatar (right) arm with used hardware components. For simplicity, only the right arm is shown. The axes depict the common hand frame which is used for control commands and feedback.

with other humans in the remote location and thus did not feature respective capabilities. In contrast, our developed avatar solution was designed for human interaction in the remote location and the operator interface is designed to give intuitive control over the robot to a single, possibly untrained operator.

Some recent approaches use teleoperation interfaces which only send commands to the robot without providing any force or haptic feedback to the operator [7] , [8] . The advantage of such systems is clearly the light weight of the capture devices which hinder the operator only marginally. The downside is missing force or haptic feedback, especially for tasks that cannot be solved with visual feedback alone such as difficult peg-in-hole tasks.

Other methods use custom-developed operator interfaces including force and haptic feedback [9] - [12] . In the latter, the operator station is comparable to our approach, but the focus is placed on haptic feedback for balance control of the bipedal humanoid avatar robot.

Wearable haptic feedback devices [13] overcome the workspace constraints generated by stationary devices but are limited to displaying contact since they cannot create any force towards the operator.

Other research projects focus on controlling a teleoperation system under time delays [14] or with two different kinematic chains on the avatar side [15] .

In contrast to the highlighted related research, our approach focuses on off-the-shelf components which allow for easy replication and maintenance. Furthermore, the used robotic arms are replaceable with any other appropriate actuators with different kinematic chains, since the whole communication between the systems uses only the 6D endeffector pose.

The developed robotic teleoperation system consists of an operator station and an avatar robot, as shown in Fig. 1 . The avatar robot is designed to interact with humans and in human-made indoor environments and thus features an anthropomorphic upper body. Two 7 DoF Franka Emika Panda arms are mounted in slightly V-shaped angle to mimic the human arm configuration. The shoulder height of 110 cm above the floor allows convenient manipulation of objects on a table, as well interaction with both sitting and standing persons. The shoulder width of under 90 cm enables easy navigation through standard doors.

The Panda arms have a sufficient payload of 3 kg and a maximal reach of 855 mm. The extra degree of freedom gives some flexibility in the elbow position. While the arm measures joint torques in each arm joint, we mounted additional OnRobot HEX-E 6-Axis force/torque sensors at the wrists for more accurate force and torque measurements close to the robotic hands, since this is the usual location of contact with the robot's environment (see Fig. 2 ). The avatar is equipped with two anthropomorphic hands. A 20 DoF Schunk SVH hand is mounted on the right side. The nine actuated degrees of freedom provide very dexterous manipulation capabilities. The left arm features a 5 DoF Schunk SIH hand for simpler but more force-requiring manipulation tasks. Both hand types thus complement each other.

The avatar's head is equipped with two RGB cameras, a microphone, and a small screen for communication purposes. It is attached to the upper body using a 6 DoF UFACTORY xArm for free head movement. Further details on the VR remote visualization system are provided in [16] . The humanoid upper body has been mounted on a movable base.

The operator controls the avatar through the Operator Station from a comfortable sitting pose. The human hand movement is captured with a similar setup as already described for the avatar robot: The left and right Panda arms are equipped with an OnRobot HEX-E force/torque sensor and connected to the operator hand using a SenseGlove haptic interaction device. The Panda arm thus serves dual purposes: A precise 6D human hand pose measurement for avatar control, as well as the possibility to induce haptic feedback measured on the Avatar on the human wrists. The force/torque sensor is used to measure the slightest hand movement in order to assist the operator in moving the arm, reducing the felt mass and friction to a minimum.

The SenseGlove haptic interaction device features 20 DoF finger joint position measurements (four per finger) and a 1 DoF haptic feedback channel per finger (i.e. when activated the human feels resistance, which prevents further finger closing movement).

For visual and audio communication, the operator is wearing a head mounted display equipped with eye trackers, audio headset and a camera viewing the lower face part (for more details see [17] ). The avatar locomotion will be controlled using a 3D Rudder foot paddle device.

The Panda arms feature built-in safety measures and will stop immediately if force, torque, or velocity limits are exceeded. This ensures safe human-robot interactions both on the operator and the avatar side.

The operator station and the avatar robot are controlled with a standard desktop computer (Intel i9-9900K @ 3.60 GHz, NVidia RTX 2080) each. The communication between these computers is achieved by a single Gigabit Ethernet connection. 

The control architecture for the force feedback teleoperation system consists of two arm and hand controllers (one for each the operator and the avatar side). For the right and the left arm each controller pair is running separately. The hand controller for the right and left hand are slightly different since different robotic hands are used. An overview of the control architecture is shown in Fig. 3 . The arm controllers run with an update rate of 1 kHz and the force-torque sensor measurements are captured with 500 Hz. The force-torque measurements are smoothed using a sensor-sided low-pass filter with a cut-of frequency of 15 Hz.

Since the robot arms attach from outside to the operator's wrists (see Fig. 3 ), the kinematic chains of avatar and operator station differ, and thus a joint-by-joint mapping of the operator and avatar arm is not possible. Consequently, the developed control concept does not rely on similar kinematic chains. Instead, a common control frame is defined in the middle of the palm of both the human and robotic hands, i.e. all necessary command and feedback data are transformed such that they refer to this frame before being transmitted. The controllers for the operator and avatar arm and both hands are described in the following.

The operator arm controller commands joint torques to the Panda arms and reads the current operator hand pose to generate the commanded hand pose sent to the avatar robot. The goal is to generate a weightless feeling for the operator while moving the arm-as long as no haptic feedback is displayed. Even though the Panda arm has a quite convenient teach-mode using the internal gravity compensation when zero torques are commanded, the weightless feeling can be further improved by using precise external force-torque measurements. For simplicity, just one arm is mentioned in the following, since the left and right arms are controlled equally.

Let us denote with τ o ∈ R 7 the commanded joint torques for a particular time step. Then

describes the used torque components (command, haptic feedback, operator limit avoidance, avatar limit avoidance, null space, and Coriolis) which will be explained in the following. Note that the gravity compensation is not taken into account here, since it is done by the Franka Control Interface (FCI) itself. The commanded joint torques τ cmd to move the Panda arm based on the force/torque sensor measurements are defined as

with J being the body Jacobian relative to the hand frame and F ∈ R 6 the measured 3D forces and 3D torques. Note that F has to be corrected taking sensor bias and attached end-effector weight into account, as well as transformed into the common hand control frame. τ h denotes the force feedback induced by the avatarside force-torque sensor. Since the measurements are already bias-corrected and correctly transformed, Eq. (2) can be directly applied analogously to compute induced joint torques.

Humans can achieve high speeds moving their arm which can exceed the Panda joint velocity limits (up to 150 • /sec). In order to prevent the operator from exceeding joint position or velocity limits of the Panda arm, the term τ lo ∈ R 7 is introduced to apply torques pushing the arm away from those limits. For a single joint i the torque to avoid its position limit is defined as

with β p being a constant scalar, d i p is the distance for joint i to its closer position limit, and t p = 10 • is a threshold how close a joint has to be at a limit to activate this behavior. τ i lo−velocity is calculated analogously with t v = 40 • /sec. Together, τ lo is defined as

The torques τ lo exhibit hyperbolical growth when getting closer to respective limits. Since the operator-side forcetorque sensor will measure the generated limit avoidance (1)) is scaled using α to reduce oscillations when getting close to joint position and velocity limits. The scalar decreases linearly if the distance to a joint position limit dp exceeds the threshold tp. Velocity limits are handled analogously.

torques, the arm can end up oscillating, especially being close to one or multiple position limits. Thus, the torques τ cmd , which are influenced by the force-torque sensor, are scaled per joint by α, which is defined as

The scalar α is designed to decrease linearly and reach zero when the limit is approached half way after activating the limit avoidance (see Fig. 4 ). This reduces the commanded torques τ cmd enough when approaching a position or velocity limit and prevents the oscillation. As already mentioned, the operator station and avatar robot have different kinematic arm chains. Therefore, avoiding position and velocity limits on the operator side does not guarantee limit avoidance on the avatar side. Calculating the joint torques preventing joint limits on the avatar robot in a similar way is not beneficial, since the feedback information would arrive with high latency. To overcome this problem, we use a model of the avatar inside the operator arm controller to predict the avatar arm movement for the next time cycle and calculate the needed joint torques to prevent joint limit violations in advance. The model is updated with the current measured avatar joint positions each time cycle. After calculating the desired 6D avatar arm pose, the model is used to approximate the next joint positions using inverse kinematics. The current joint velocities are approximated by a low-pass filtered version of the joint position first derivative. Having estimated the joint positions and velocities, we can apply the same avoidance strategy as described above (see Eq. (4)). Finally, the resulting joint torques can be transformed into the common 6D hand frame using the pseudoinverse of the Jacobian transpose (J T A ) + and back to joint torques for the operator arm with J T O . This results in

The torque component τ no is a null space optimization term which pulls the elbow towards a defined convenient pose in the null space of the Jacobian. The result is a more human-like elbow pose to maximize the operator workspace by pushing the arm away from singularities. The last torque component is the Coriolis term τ co obtained by the Panda model.

The avatar arm controller commands the Panda arm on the avatar robot to follow the commanded 6D pose by sending joint torques τ a ∈ R 7 to the Franka Control Interface. The commanded torque is defined as

The components τ na and τ ca are the null space optimization and Coriolis terms similar to τ no and τ co as described in Section IV-A. The convenient elbow poses used for the nullspace optimization is defined such that the elbows are slightly stretched out. This generates a more human-like arm configuration and keeps the arm away from singularities.

The goal torques τ cmd ∈ R 7 and τ init ∈ R 7 are generated using a Cartesian impedance controller that emulates a spring-damper system. Its equilibrium point is the 6D goal pose commanded by the operator station in the common hand frame:

where J denotes the zero Jacobian, S ∈ R 6×6 and D ∈ R 6×6 denote the stiffness and damping matrix, ∆p ∈ R 6 is the error in translation and rotation between the current and goal end-effector pose andq ∈ R 7 denotes the current joint velocities. τ init is only used for a safe initialization procedure (see below) and generated similar using the current end-effector pose. The stiffness and damping parameters are empirically tuned to achieve some compliance while still closely following the operator command. When no goal pose command is received (after a communication breakdown or after the initial start), the controller keeps commanding the current arm pose to remain in a safe state. After receiving a command, the controller performs an initialization procedure which fades linearly between the current and new received goal pose. This prevents the robot from generating high torques in order to suddenly reach the new, possibly distant pose. This initialization process takes about 3 sec.

The Panda arm stops immediately when excessive forces are measured, for example when there is unintended contact that exceeds force/torque thresholds. This feature is necessary to operate in a safe way. After notification of the human operator, the avatar arm controller can restart the arm automatically. After performing the initialization procedure, normal teleoperation can be resumed.

The operator finger movements are captured using two SenseGlove haptic interaction devices. Four separate finger joint measurements are provided per finger. Since the Schunk robotic hands on the avatar have nine and five actuated joints, respectively, only the corresponding joint measurements are selected and linearly mapped to the avatar hands. While this mapping does not precisely replicate hand posturesthis is impossible anyways due to the different kinematic structure-it gives the operator full control over all hand DoFs. Both hands provide feedback in the form of motor currents, which is used to provide per-finger haptic feedback to the operator. The SenseGlove brake system is switched on or off depending on a pre-defined current threshold.

Different end-effectors (SenseGloves, Schunk SIH, Schunk SVH hand, and corresponding 3D printed mounting adapters) are mounted on each of the four involved forcetorque sensors. In addition, sensor bias results in barely usable raw sensor data. Thus, each sensor is calibrated separately to compensate these effects. 20 data samples from different sensor poses are collected. Each sample includes the gravity vector in the sensor frame and the mean of 100 sensor measurements from a static pose. A standard least squares solver [18] is used to estimate the force-torque sensor parameters, i.e. the force and torque bias and the mass and center of mass of all attached components. The same parameters including the additional mass and center of mass transformation resulting by the force-torque sensor itself is used to configure the built-in gravity compensation of the Panda arms. The calibration is performed once after hardware changes at the end-effectors or if the bias drift is too large. This method does not compensate for bias drift during usage but is sufficient for our application. 

We performed multiple experiments along with a small user study to evaluate the developed teleoperation system in our lab environment.

In a first experiment, we evaluated the operator arm workspace. 2959 different 6D left hand poses were captured from a sitting person performing typical arm motions with a VR tracker on their wrist. In addition to hand poses with a fully extended arm, most of the poses are directly in front of the person, likely to be performed during manipulation tasks. First, the initial arm mounting pose (motivated by the avatar configuration) of the operator arm was evaluated. Each captured hand pose was marked as reachable if an inverse kinematic solution for the arm was found. In a second step, different arm mounting poses were sampled to find an optimal pose, maximizing the number of reachable hand poses (see Fig. 5 ). Table I shows quantitative results. The optimized arm mounting pose drastically increases the overlap between the human operator's and the avatar's arm workspace, but requires a more complicated mounting setup.

In a second experiment, we evaluated the predictive limit avoidance module. Avatar arm position and velocity limits are haptically displayed via joint forces to the operator. Since measured joint positions and velocities are afflicted with latency generated by network communication (<1 ms) and motion execution using the Cartesian impedance controller, which can reach up to 200 ms (see Section IV-B), the operator control predicts the avatar arm joint configuration. Fig. 6 shows the measured and predicted joint position of the first right arm joint. The prediction compensates the delays, which allows for instantaneous feedback of the avatar arm limits to the operator.

In a third experiment, we investigated the forces and torques required to move the operator station arm, since this directly affects operator fatigue. We measured the forces and torques applied to the arm by reading the force-torque sensor measurements. The arm was moved in a comparable manner once with only the Panda gravity compensation enabled (i.e. τ cmd = 0 see Section IV-A) and a second time with our arm force controller running. In Fig. 7 , the forces in the direction of one exemplary axis are shown. The results show the advantage of using an external force-torque sensor to generate a more unencumbered feeling for the operator while using the system.

Our goal was to create an immersive and intuitive feeling for operation at remote locations using our system. Since humans have their very own preferences and subjective feelings of how good or intuitive certain control mechanisms perform, we carried out a user study with untrained operators, comparing different telemanipulation approaches. Due to the current COVID-19 pandemic, we were limited to immediate colleagues as subjects, which severely constrained the scope of our study. A total of five participants were asked to perform a bimanual peg-in-hole manipulation task. First, two different objects had to be grasped: a small aluminum bar and a 3D printed part with a hole. Afterwards, the bar should be inserted into the hole (see Fig. 8 ). The task was challenging due to very little friction between the finger and objects and tight tolerances which required precise insertion alignment. Each participant performed the task three times with the following control modes: 1) Operator station with haptic feedback enabled, 2) Operator station with haptic feedback disabled, and 3) VR controllers. Two HTV Vive VR controllers acted only as input devices: As long as the trigger button was pressed, the corresponding avatar arm followed the controller movement. A different button was programmed to toggle between a defined closed and open hand pose. The tested control mode sequence was randomized for each participant. A maximum of 5 min were granted to solve the task before it was marked as a failure. The participants were allowed to test each control mode about 1 min before starting the measured test. Table II shows the quantitative results of the user study. The time needed to successfully solve the task is quite similar over the different telemanipulation modes. From these experiments, we realized that the completion time was highly influenced by external factors, such as losing the object due to not enough finger friction or different grasping and object handling solutions, which are unrelated to the used operator In addition to these quantitative measurements, we asked the participants to answer a short questionnaire about each telemanipulation mode with answers from the 1-7 Likert scale (see Fig. 9 ). The results show that especially the feeling of handling the objects and intuitive finger control was subjectively much better using the Operator Station. Enabling the force and haptic feedback gives the highest advantage when picking up the objects from the table. This can be explained by the additional feedback indicating contact between the hand and the table which cannot be perceived visually due to occlusions. All participants reported to feel save and comfortable using the system. Although the experiment time was limited, this suggests non-excessive cognitive load on the operator.

Overall, the user study showed that our developed system is intuitive to use for untrained operators. Even though the force and haptic feedback did not increase the success rate of solving the task, it increases the immersive feeling as shown by the questionnaire.

This work presented a bimanual telemanipulation system consisting of an exoskeleton-based operator control station and an anthropomorphic avatar robot. Both components communicate using our force and haptic feedback controller which allows safe and intuitive teleoperation for both the operator and persons directly interacting with the avatar. The control method is invariant to the kinematic parameters and uses only a common Cartesian hand frame for commands and feedback. Using the predictive limit avoidance avatar model, arm limits for both the operator and avatar side can be force displayed to the operator with low latency. We evaluated the Did you feel safe and comfortable?

Did you feel like you were handling the objects directly?

Was it easy to control the robot?

Was it intuitive to control the arms?

Was it intuitive to control the fingers?

Did you find and recognize the objects?

Was it easy to grasp the objects?

Was it easy to fit the objects together? VR Without FF With FF better Fig. 9 . Statistical results of our user questionnaire. We show the median, lower and upper quartile (includes interquartile range), lower and upper fence, outliers (marked with •) as well as the average value (marked with ×), for each aspect as recorded in our questionnaire.

system using a user study on untrained operators as well as lab experiments, which demonstrated the intuitiveness of our control method. In future work, the workspace and haptic feedback will be further improved.

The DARPA Robotics Challenge finals: Results and perspectives

RoboCup Rescue: Search and rescue in large-scale disasters as a domain for autonomous agents research

Technical overview of team DRC-Hubo@UNLV's approach to the

DARPA Robotics Challenge finals

CHIMP, the CMU highly intelligent mobile platform

Team RoboSimian: Semi-autonomous mobile manipulation at the 2015 DARPA Robotics Challenge finals

NimbRo Rescue: Solving disaster-response tasks with the mobile manipulation robot Momaro

A teleoperation interface for loco-manipulation control of mobile collaborative robotic assistant

A motion retargeting method for effective mimicry-based teleoperation of robot arms

Remote mobile manipulation with the Centauro robot: Full-body telepresence and autonomous operator assistance

Human-oriented control for haptic teleoperation

Haptic based teleoperation with master-slave motion mapping and haptic rendering for space exploration

Humanoid teleoperation using task-relevant haptic feedback

Teleoperation in cluttered environments using wearable haptic feedback

Telemanipulation with time delays

Evaluation of an exoskeleton-based bimanual teleoperation architecture with independently passivated slave devices

Low-latency immersive 6D televisualization with spherical rendering

NimbRo Avatar: Interactive immersive telepresence with forcefeedback telemanipulation

On-line rigid object recognition and pose estimation based on inertial parameters