key: cord-0057774-47gda1z4
authors: Zhao, Dan; Qu, Ming Fei
title: Simulation Training Remote Control System of Industrial Robot Based on Deep Learning
date: 2020-10-30
journal: e-Learning, e-Education, and Online Training
DOI: 10.1007/978-3-030-63955-6_21
sha: 921be07a55e8319bcab6261deeb451f6d22919b2
doc_id: 57774
cord_uid: 47gda1z4

In order to improve the remote control performance of industrial robot simulation training, deep learning algorithm is used to optimize the design of traditional remote control system. On the basis of traditional remote control system, the configuration of hardware system is modified, and the database of control system is established. With the support of hardware system and database, the remote control of two training items of industrial robot simulation mobile training and simulation picking training are realized respectively. Through the system test experiment, the conclusion is drawn: compared with the traditional industrial robot remote control system, the control function of the design control system is improved, and the system can save about 12.5 s response time in the control process.

Industrial robot is a multi joint manipulator or multi degree of freedom robot facing the industrial field. It is a kind of machine which can automatically perform work and realize various functions by its own power and control ability. It can be commanded by human beings or run according to the pre arranged program. Modern industrial robots can also act according to the principle program formulated by artificial intelligence technology [1] . Nowadays, industrial robots can replace people to do some monotonous, frequent and repeated long-term operations, or work in dangerous and harsh environments. Before the industrial robot is put into application, it needs to carry out simulation training. The simulation training can be roughly divided into two parts, namely, the mobile performance training and the picking performance training of the industrial robot [2] . The above-mentioned simulation training projects of the industrial robot need to be realized with the help of the corresponding control system. Industrial robot control system is the brain of robot, which is the main factor to determine the function and performance of robot. Under the influence of information technology, the remote control system of industrial robot simulation training is formed by integrating network communication technology and industrial robot control technology. The remote control system is generally divided into two parts: client program and server program. Before use, the client program needs to be installed on the main control computer, and the server program needs to be installed on the controlled computer. Remote control can only be carried out through the network. The local computer is the sending end of the operation instruction, which is called the main control end or the client, and the non local control computer is called the control end or the server. "Remote" is different from "remote". The main control end and the controlled end can be in the same room of the same LAN, or two or more computers connected to the Internet at any position.

The traditional industrial robot simulation training remote control system mostly uses robot vision technology, Internet of things technology or cloud control technology. However, after a long time of application research, it is found that the traditional remote control system can not achieve the control of robot movement and picking at the same time, and there is a time delay phenomenon in the control process.

In order to solve the above problems in the traditional remote control system, this paper uses the deep learning algorithm to design the industrial robot simulation training remote control system. Deep learning is a new field in machine learning. Its motivation is to build and simulate neural network of human brain for analysis and learning. It imitates the mechanism of human brain to interpret data, such as image, voice and text. The development of deep learning has greatly promoted the innovation of visual perception technology. The method of extracting features from massive data through deep neural network has also shown strong advantages. This also promotes the research upsurge of artificial intelligence in the field of robot. This paper designs and realizes the remote control system of industrial robot from four aspects: hardware, database, human-computer interface and software function, so as to realize the efficient control of industrial robot. And the development of deep learning is expected to solve the challenge of traditional remote control system.

The simulation training remote control system of industrial robot is based on client and server architecture. The system is composed of computer control terminal, public network management forwarding server and mobile robot. The computer control terminal controls the mobile robot through the public network management forwarding server. The control principle of the industrial robot simulation training remote control system is: the public network management forwarding server starts and waits for the computer control terminal to connect with the mobile robot terminal. After the computer control terminal starts, it sends the registration request to the server. Similarly, after the mobile robot terminal starts, it sends a registration request to the server [3] . Then the computer control terminal selects the mobile robot to be controlled and sends the control request to the service. After receiving the request, the server establishes a communication connection between the computer control terminal and the mobile robot terminal. Then the mobile robot terminal sends the video taken by the mobile robot to the computer control terminal through the server. The computer control terminal receives the video, displays the video, and sends the control instruction to the mobile robot through the server. According to the above-mentioned remote control principle of industrial robot simulation training, the remote control system of industrial robot is designed and implemented from four aspects of hardware, database, human-computer interface and software function, so as to achieve efficient control of industrial robot. The hardware system of industrial robot simulation training remote control system provides hardware support for the realization of control function and the compilation of related programs. The specific hardware system structure is shown in Fig. 1 

The industrial robot terminal is mainly responsible for collecting video and sending it to the transponder, and receiving the mobile control instructions from the transponder. According to the established mobile control protocol, the robot makes the corresponding mobile control motion. In the design of the remote control system, the Toshiba four axis thl400 robot is used. The robot is a fast plane grabbing and placing robot commonly used in the industrial field. The binocular robot assembly system adopts ABB six axis industrial robot. Six axis robot is also widely used in automatic sorting, welding, spraying and other fields in the industrial field. The two kinds of robots use pneumatic suction nozzle to grab the workpiece. The industrial computer and robot controller are connected through Ethernet and communicate through socket protocol.

Wireless local area network is needed between the simulation training control server and the application server of industrial robot, so that the industrial robot can get a larger range of movement. Select two wireless network cards with USB interface, and install them on the USB port of host a and host B respectively, then install the driver of wireless network card and supporting application program. Just make sure that A and B can be connected, and only use the utility provided by the network card to set up the WLAN. When you are finished, you can view the connection instructions to the system.

In the remote wireless communication network environment, the servers that need to be accessed include Internet server and robot server. The Internet server consists of three parts: Web service program, user database and Winsock communication program. The Internet server of the remote robot is a server with a static IP address. It runs the Apache 3.12 HTTP service program under Windows NT 4.0. When the user connects to the operation home page of the remote robot through a web browser, he starts the web service program of the Internet server [4] . The web service program provides users with static pictures and robot parameters of the operation site by operating the HTTP file of the home page. Users manipulate the PUMA robot on the site by changing the robot parameters. Robot server is an industrial computer running Windows 2000, which is equipped with robot communication control card, image acquisition card, gripper controller and other hardware. In the remote control system, the main work is to modify the robot's motion in the tool coordinate system and read the transfer matrix in the end world coordinate system.

The remote control of industrial robot is composed of two five phase stepping motors and corresponding motor drivers. Two stepper motors drive two driving wheels. By changing the frequency of the pulse signal acting on the stepper motor controller, the stepper motor can be adjusted with high precision. At the same time, applying the same or different pulse signals to the two stepping motors can control the motion of the industrial robot conveniently.

In addition to the remote control equipment and remote communication equipment of the above remote control system, the hardware system also includes the minimum system, debugging interface, network interface, external power supply, flash memory, SDRAM and expansion interface. Among them, the minimum system refers to the minimum configuration system in which the microprocessor can run programs and complete the simplest tasks. The minimum system is a necessary part of any complex system. The minimum system includes: power circuit, crystal oscillator circuit, reset circuit, etc., debugging test circuit, etc. [5] . The power circuit is used to supply power to each module to ensure its normal operation. The power supply of the system is 5 V, the power supply voltage of the chip pin is 3.3 V, and the power supply of WiFi module is 5 V, so necessary power conversion is needed. The design circuit of the system power supply circuit is as follows (Fig. 2 ). In addition, the reset circuit will initialize the processor state to a state that can make it run normally, so that the microcontroller can start to work from that specific state every time it is powered on. This reset logic requires a signal to work properly. The function of crystal oscillator circuit is to provide power for the system. Almost all microcontrollers are sequential circuits, which requires a clock pulse signal to make them work normally. The general microcontroller itself has a crystal oscillator. But some special occasions need to use external oscillator to provide clock signal.

3.3V

The database of industrial robot simulation training remote control system is mainly used for user management and remote control data storage. So the design of database includes two parts: user table and basic operation data table of industrial robot. Among them, the user table is used to manage the users logged in by the client, which is mainly implemented in two aspects: entity contact diagram, attribute and database logic design [6] . The database uses MySQL database to create remote control system user According to the same structure, the basic data information of industrial robot to be simulated and trained and the user authority information of remote control system can be constructed corresponding database table. During the operation of the system, the links between the database and the system interface, between the database and the software program are formed to ensure that the remote control function can directly call the data information in the database and display it on the human-computer interface in time [7] . At the same time, the real-time remote control information can also be stored in the system database in time.

The simulation training remote hardware system of industrial robot is the hardware foundation of software function realization, and the system database provides sufficient data support for software function realization. In this case, the mobile function and picking function of industrial robots are simulated respectively, and the remote control of the two simulated training processes is realized [8] . In the process of software function realization, deep learning algorithm is applied to realize the remote control function of industrial robot simulation training. The specific software function realization structure is shown in Fig. 3 .

Human-computer interaction interface 

In order to build the kinematic model of industrial robot, the following assumptions are made as follows. First, the structure of mobile robot is rigid; Secondly, the motion plane of mobile robot is plane; Third, the rotation center and the mass center of the mobile robot are coincident; Fourth, the resistance of the left and right wheels of the mobile robot is the same. There is no relative sliding between the wheels and the ground. Some factors affecting the speed, such as the quality of the mobile robot, the friction resistance of the ground, and the load-bearing load, are ignored [9] . Then in XY coordinate system, the mobile coordinate of industrial robot is defined as (x R , y R ), the heading angle of mobile direction is expressed as θ R , and the initial pose of industrial robot is expressed as (x R , y R , θ).

The relationship between linear velocity and angular velocity of mobile robot is expressed as follows:

Where, [v, ω] T represents the control vector of industrial robot, and x.y.θ T represents the state vector of industrial robot simulation training.

The collected simulated mobile training environment image is divided into two parts: foreground and background. By modeling the background, the difference between the current frame and background image is used to determine the foreground target area, which is particularly suitable for foreground obstacle detection. The specific detection process is shown in Formula 3.

Where, f (x, y) and f 0 (x, y) are the current image and background image respectively, D(x, y) is the difference image between the current image and background image, and t is the binary threshold. After background subtraction is used to remove the interference of the ground and the perspective, the general contour of the foreground obstacles can be obtained. However, due to the noise and hollowness of the obtained difference image, the edge of the obstacles is not smooth, so it needs to carry out morphological processing to denoise. After three steps of contour extraction, area filtering and convex hull processing, the accurate obstacle area is finally obtained.

According to the principle of deep learning, decision function, evaluation function, reward function and external environment are the basic components of deep learning system [10] . The decision function of agent is the core of the whole deep reinforcement learning system, which directly determines the action of agent. Decision function π(a|s ) represents a mapping from state to behavior. The decision expression of agent at time t is as follows:

Deep learning seeks the best decision by continuously training agents. According to the above theory, the mobile route of industrial robot is planned according to the deep learning framework shown in Fig. 5 . According to the learning framework shown in Fig. 4 , the straight-line interpolation algorithm in Cartesian coordinate system is used to plan the captured and placed straight-line tracks. The middle track is still planned by quintic polynomial interpolation algorithm, and the workpiece is located in three-dimensional real-time. Using the binocular vision of ABB Robot, the working area images of left and right cameras are collected respectively, and then the workpiece is located by threshold segmentation and Hough transform Finally, the height information is obtained by stereo matching. The coordinates of the workpiece are recorded as A 0 (x 0 , y 0 , z 0 ), and the end point of the trajectory, i.e. the placement point, is known and set as C(x 2 , y 2 , z 2 ). Using the background subtraction method, the image area of the obstacle can be determined by subtracting the background image from the working area image of the left camera, and the center point coordinate is recorded as B(x 1 , y 1 , z 1 ) . The height of grasping is determined to avoid obstacles. The straight-line trajectory is parallel to z-axis of robot, including two tracks of grasping and placing. A point is inserted every 0.2 s by using the timing interpolation method. The straight-line trajectory moves at a constant speed with a speed of 5 cm/s. all interpolation points are inverse solved. The intersection coordinates of the straight-line trajectory and the middle trajectory are determined and the inverse solution is obtained. Linear interpolation and quintic polynomial are combined to fit the trajectory of each joint. Every joint of the robot is controlled regularly. All the joint angles of the robot obtained above are transformed with time and sent to the robot at the same time every 0.2 s to control the motion of each joint.

The transmission of control instructions can be divided into two parts. One is to collect mouse coordinates under the supervision and control mode, and use the mouse to set the destination. The mouse coordinates to be transmitted at this time are the control instructions. The other is the acquisition of the key code clicked by the mouse in the direct control mode. The key code to be transmitted is the control instruction. What we need to achieve now is to transfer the control instructions of the server to the robot control server through the wireless network through the middleware technology based on.

According to the same way, first use the deep learning algorithm to detect the target, and the specific principle is shown in Fig. 6 . The output of the target detection algorithm can be expressed as follows:

Among them, n represents n targets after detection from the image and 0.7 threshold filtering. p b i represents the probability of the target with index I. Through the position information of the output box of the object detection results to be grabbed, n depth regions of interest can be obtained from the corresponding depth estimation results. Then control and control industrial robot serialization grab, control flow as shown in Fig. 7 . 

In the test experiment, the simulated training industrial robot is aubo-i5, a light manmachine cooperative robot developed by Beijing intelligent. The model of industrial robot is based on the concept of modularization and adopts an open software architecture. Aubo-i5 provides a variety of interfaces based on ROS. For the inverse solution and trajectory planning of manipulator kinematics, we can use not only the integrated moveit package of ROS, but also the interface of manipulator motion provided by Aobo company. Robotiq-140 claw is used as mechanical claw, which is developed by robotiq company of Canada.

In order to verify the operation effect of the robot obstacle avoidance module, the simulation test is carried out on the simulation software stage provided by ROS. Stage is a relatively simple 2D simulator, which can simulate a single or multiple robots, and add robots by modifying the file at the end of. World. It can load a variety of manually designed maps and add the desired obstacles. The designed test environment is a 10 × 10 area, the actual length of one grid is 1 m, the robot is set as a circle with a radius of 0.5 m, the maximum moving speed of the robot is set as 1 m/s, and the trajectory arc is set as red. The gain coefficient of gravitational field is set to 1, the gain coefficient of repulsive field is set to 6, and the influence range of obstacles is set to 2. Aiming at the problems of collision, target inaccessibility and local minimum point caused by too Table 2 .

From the experimental results shown in Table 2 , it can be seen that compared with the traditional remote control system, under the control of the designed remote control system, the success rate of mobile control of industrial robot has reached 100%, and the number of collisions with environmental obstacles has been reduced by 62.5%. Figure 8 shows the specific scene of detecting the picking control function of the remote control system. The experiment flow is controlled by code on the computer side. Firstly, Kinect collects the color and depth map of the scene, sends the color map into the network to predict the capture center point and the capture direction, then calculates the values of the capture point and the capture direction in the base coordinate system according to Kinect's conversion relationship, and finally sends these values into the control program of the manipulator, which is executed by the manipulator. Compared with the traditional remote control method, it is found that under the simulation training remote control system of industrial robot based on deep learning, the speed of industrial robot picking up the designated objects is faster and the rate of empty grasping is lower.

Real time is an important performance index of any remote control system. There are many reasons that affect the real-time performance of remote control, such as hardware performance, software performance and network transmission. In the remote control, due to the existence of many factors, the industrial robot may not receive the control command from the client immediately, resulting in the robot can not respond to the control of the client quickly. To some extent, this causes the response delay of mobile robot, which affects the user control experience and the normal operation of the whole remote control system. Because of the relationship between response time and real-time, the shorter the response time is, the higher the real-time control is. The response time of the control command of the remote control system is calculated by recording the time when the control command is sent out and the time when the industrial robot executes the command. The comparative experimental results are obtained by statistical calculation, as shown in Table 3 . From the data in the table, it can be found that the average time delay of traditional remote control system is 33.5 s, while the average time difference of the designed remote control system is 21 s. It can be concluded that the design method can save about 12.5 s on average in the aspect of control delay performance.

The simulation training remote control system of industrial robot based on deep learning is the product of the integration of the development of machinery, computer technology and communication technology. This kind of industrial robot can adapt to the working environment of various adverse conditions, which can not only improve the working efficiency, but also reduce the cost and improve the working quality. So in all walks of life, this kind of networked, integrated and intelligent industrial robot will be more and more widely used. The system has high reliability and secondary development of communication, which can save about 12.5 s running time on average. It can not only monitor the working state of industrial robot in real time, but also improve the running efficiency, so it has high application value. 

Radar emitter identification algorithm based on deep learning

Enhancing images of shale formations by a hybrid stochastic and deep learning algorithm

An integrated design method coupling structure and control for industrial robot

Recurrent fuzzy wavelet neural networks based on robust adaptive sliding mode control for industrial robot manipulators

Analysis and implementation of a force control strategy for drilling operations with an industrial robot

Direct adaptive robust tracking control for 6 DOF industrial robot with enhanced accuracy

Multimodal sensor-based whole-body control for human-robot collaboration in industrial settings

Remote control of an omnidirectional mobile robot with time-varying delay and noise attenuation

A hybrid joystick with impedance control for a stable remote control of a mobile robot

Web environment for programming and control of a mobile robot in a remote laboratory