Authors: Duan Yanting, Cai Chensheng, Wang Pengfei, Wang Ning, Chen Ping

This paper summarizes the robot visual servo technology, introduces the concept, development history and classification of the robot visual servo system, and focuses on the position-based visual servo system and the image-based visual servo system. The frontier issues involved in robot vision are summarized, and the existing problems and future development directions are pointed out.

At present, in the manufacturing industry all over the world, industrial robots have played an increasingly important role in production. In order for robots to be competent for more complex tasks, robots not only need better control systems, but also need to be more aware of changes in the environment. Among them, robot vision has become the most important robot perception function because of its large amount of information and complete information.

Robot vision servo system is an organic combination of machine vision and robot control. It is a nonlinear and strongly coupled complex system. Its content involves research fields such as image processing, robot kinematics and dynamics, and control theory. With the improvement of the performance-price ratio of camera equipment and the speed of computer information processing, as well as the improvement of related theories, visual servoing has the technical conditions for practical application, and related technical issues have become the focus of current research.

This paper summarizes the robot visual servo technology, introduces the concept, development history and classification of the robot visual servo system, and focuses on the position-based visual servo system and the image-based visual servo system. The frontier issues involved in robot vision are summarized, and the existing problems and future development directions are pointed out.

Robot Vision Servo System

Definition of Visual Servo:

Humans obtain most of the external information through their eyes. For thousands of years, human beings have been dreaming of creating intelligent machines. Such intelligent machines first have the function of human eyes and can recognize and understand the external world. There are many organizations in the human brain that are involved in the processing of visual information, so they can easily deal with many visual problems. However, as a process, human beings know very little about visual cognition, which makes it difficult to realize the dream of intelligent machines. With the development of camera technology and the emergence of computer technology, intelligent machines with visual functions began to be manufactured by humans, and gradually formed the discipline and industry of machine vision. The so-called machine vision is defined by the Machine Vision Branch of the American Society of Manufacturing Engineers and the Automation Vision Branch of the American Robotics Industries Association (ria robotic industries association):

“Machine vision is a device that automatically receives and processes an image of a real object through optical devices and non-contact sensors to obtain the required information or to control the motion of a robot.”

Machine vision, as a machine bionic system similar to the human eye, from a broad perspective, all the information obtained from real objects and the processing and execution of related information through optical devices are machine vision, which includes visible vision and invisible vision, and even includes The acquisition and processing of internal information of objects that cannot be directly observed by human vision.

Robot Vision Development History

In the 1960s, due to the development of robotics and computer technology, people began to study robots with vision functions. But in these studies, the vision of the robot and the action of the robot are strictly open-loop. The vision system of the robot obtains the target pose through image processing, and then calculates the pose of the machine’s motion according to the target pose. During the whole process, the vision system “provides” information at one time, and then does not participate in the process. In 1973, the vision system was applied to the robot control system, and the process was called visual feedback during this period. Until 1979, hill and park proposed the concept of “visual servo”. Obviously, the meaning of visual feedback is only to extract feedback signals from visual information, while visual servoing includes the whole process from visual signal processing to robot control, so visual servoing can reflect robot vision and control more comprehensively than visual feedback. related research content.

Since the 1980s, with the development of computer technology and camera equipment, the technical problems of robot visual servo system have attracted the attention of many researchers. In the past few years, robot visual servoing has made great progress in both theory and application. In many academic conferences, visual servo technology is often listed as a topic of the conference. Visual servoing has gradually developed into an independent technology across technical fields such as robotics, automatic control and image processing.

Robot visual servo system classification:

At present, the robot visual servo control system has the following classification methods:

●According to the number of cameras, it can be divided into monocular visual servo system, binocular visual servo system and multi-eye visual servo system

The monocular vision system can only obtain a two-dimensional plane image, and cannot directly obtain the depth information of the target; the multi-eye visual servo system can obtain the image of the target in multiple directions, and the obtained information is rich, but the information processing capacity of the image is large, and the more cameras there are. The more difficult it is to ensure the stability of the system. The current visual servo system mainly adopts binocular vision.

●According to the location of the camera, it can be divided into eye in hand and fixed camera system (eye to hand or stand alone).

In theory, the hand-eye system can achieve precise control, but it is sensitive to the calibration error of the system and the motion error of the robot; the fixed camera system is not sensitive to the kinematic error of the robot, but the accuracy of the target pose information obtained under the same conditions is not as good as that of the hand-eye system. Therefore, the control accuracy is relatively low.

●According to the spatial position or image features of the robot, the visual servo system is divided into a position-based visual servo system and an image-based visual servo system

Figure 1 Dynamic look and move system based on position control

In the position-based visual servo system (as shown in Figure 1), the pose of the target relative to the camera and the robot is calculated after processing the image, so this requires calibration of the camera, target and robot models, and the calibration accuracy It affects the control accuracy, which is the difficulty of this method. During control, the pose that needs to be changed is converted into the rotation angle of the robot joint, and the joint controller is used to control the rotation of the robot joint.

Fig.2 Direct visual servo system based on image control

In an image-based visual servoing system (as shown in Figure 2), the control error information comes from the difference between the target image features and the desired image features. For this control method, the key problem is how to establish the image Jacobian matrix that reflects the relationship between the image difference change and the robot hand pose speed change; another problem is that the image is two-dimensional, and the calculation of the image Jacobian matrix requires estimation Object depth (three-dimensional information), and depth estimation has always been a difficult point in computer vision.

The calculation methods of Jacobian matrix include formula derivation method, calibration method, estimation method and learning method. The former can be obtained by model derivation or calibration, and the latter can be estimated online.

●According to the robot with closed-loop joint controller, the visual servo system is divided into dynamic observation-moving system and direct visual servoing

The former uses the robot joint feedback inner loop to stabilize the robotic arm, and the image processing module calculates the speed or position increment that the camera should have, and feeds it back to the robot joint controller; the latter uses the image processing module to directly calculate the motion control of each joint of the robot arm. quantity.

The main problems faced by visual servoing

The research of visual servoing has a history of nearly 20 years, but because there are many disciplines involved in visual servoing, its development depends on the development of these disciplines. At present, there are still many problems in the research of visual servoing that have not been well solved. .

●The method of image processing is the biggest difficulty of image servo in terms of theoretical and practical calculation processing speed;

After the image processing is completed, the establishment of the model between the image features and the robot joint motion is another difficulty in image servoing;

●Many of the current control methods can not guarantee that the system is stable in a large range during operation, so the research on related control methods is also necessary.

The Development Prospect of Visual Servo

The future research directions of visual servoing mainly include the following aspects:

●Fast and robust acquisition of image features in practical environments is a key issue for visual servo systems

Due to the large amount of information in image processing and the development of programmable device technology, the recent method of hardwareizing general algorithms to speed up information processing may make progress in the study of this problem.

●Establish relevant theory and software suitable for robot vision system

Many current image processing methods of robot vision servo systems are not aimed at robot vision systems. If there is such a dedicated software platform, the workload can be reduced when completing vision servo tasks, and it can even be achieved by hardware vision information processing. Improve the performance of the visual servo system.

●Applying various artificial intelligence methods to robot vision servo system

Although neural networks have been applied in robot visual servoing, many intelligent methods have not been fully applied in robot visual servoing system, and current research tends to rely too much on mathematical modeling and mathematical calculation, which makes robot visual servoing The system requires too much calculation when it is working, and the current processing speed of the computer is difficult to meet the requirements of the rapidity of the system, but humans do not complete the relevant functions through a large number of calculations, which enlightens whether we can use artificial intelligence. The method reduces the amount of mathematical calculation to meet the requirements of the rapidity of the system.

●Applying active vision technology to robot vision servo system

Active vision is a hot spot in the field of computer vision and machine vision research. Here, the visual system can actively perceive the environment and actively extract the required image features according to certain rules, which makes the problems that are difficult to solve in general to be solved.

● Combining vision sensors with other external sensors

In order to enable the robot to perceive the environment more comprehensively, especially to supplement the information of the robot vision system, a variety of sensors can be added to the robot vision system, which can overcome some difficulties of the robot vision system, but the introduction of multiple sensors needs to solve Information fusion and information redundancy in robot vision systems.


In recent years, robot vision servo technology has made great progress, and there are more and more practical applications of robot vision systems at home and abroad. Many technical problems are expected to make progress in recent research. In the future, robot vision servo system will occupy a prominent position in robotics, and robot vision servo system will be more and more used in industrial production.

Responsible editor: gt

Leave a Reply

Your email address will not be published.