With the improvement in the cost-performance ratio of imaging equipment and the speed of computational information processing, together with the improvement of related theories, servo vision technology has met the technical conditions for practical application, and related technical issues have become the current research point.
Servant Vision Definition
Most of the information that humans acquire from the external world is obtained through the eyes. For centuries, humanity has dreamed of creating intelligent machines, whose primary function is to imitate the human eye, enabling recognition and understanding of the external world.
Numerous structures of the human brain participate in the processing of visual information, thus effortlessly dealing with many vision-related problems. However, as a process, our understanding of visual cognition remains limited, making the dream of intelligent machines a challenge to realize.
With the development of camera technology and the emergence of computer technology, intelligent machines with visual functions began to be manufactured by humans, gradually forming the machine vision discipline and industry.
Machine vision, as defined by the Mechanical Vision Division of the Society of Manufacturing Engineers (SME) and the Automated Imaging Association of the Robotic Industries Association (RIA), is “the automatic reception and processing of an image of a real object by using optical devices and non-contact sensors to obtain the necessary information or to control the movements of a robot.”
As a biomimetic system similar to the human eye, machine vision broadly covers the acquisition of information from real objects through optical devices and the processing and execution of related information. This includes visible and non-visible vision, and even the acquisition and processing of information within objects that human vision cannot directly observe.
Robotic Vision Development Journey
In the 1960s, due to the advancement of robotics and computer technology, people began researching robots with visual capabilities. However, in these studies, the robot's vision and movement were, strictly speaking, open-loop.
The robot's visual system processes the image to determine the position and orientation of the target, calculates the robot's motion posture based on this, and provides this information in a single instance without additional involvement.
In 1973, the concept of visual feedback was introduced when someone applied a visual system to robot control.
It was not until 1979 that Hill and Park proposed the concept of “visual servo-servo”. Unlike visual feedback, which only extracts visual information signals, visual servoing covers the entire process from visual signal processing to robot control, offering a more comprehensive reflection of relevant research on robot vision and control.
Since the 1980s, with the advancement of computer technology and camera equipment, technical issues of robotic visual servo systems have attracted the attention of many researchers. In recent years, significant progress has been made in both the theory and application of robotic visual servo.
This technology is often a special topic at many academic conferences and has gradually developed into an independent technology covering the fields of robotics, automatic control and image processing.
Classification of Servo Robotic Vision Systems
Currently, robotic vision servo control systems can be categorized in the following ways:
● Based on the number of cameras, they can be divided into monocular vision servo systems, binocular vision servo systems and multi-eye vision servo systems.
Monocular vision systems can only obtain two-dimensional images and cannot directly obtain target depth information.
Multi-eye vision servo systems can capture images from multiple directions of the target, providing valuable information. However, they require large amounts of image data processing and the more cameras involved, the more challenging it will be to maintain system stability. Currently, binocular vision is mainly used in servo vision systems.
● Depending on camera positioning, systems can be classified as eye-in-hand systems and fixed camera systems (eye-in-hand or stand-alone).
In theory, eye-in-hand systems can achieve precise control, but they are sensitive to calibration errors and robot motion errors.
Fixed camera systems are not sensitive to the robot's kinematic errors, but under the same conditions, the accuracy of the target pose information obtained is not as good as that of the eye-in-hand system, resulting in relatively lower control accuracy. .
● Based on the spatial position or image characteristics of the robot, servo vision systems are divided into position-based servo vision systems and image-based servo vision systems.
In position-based servo vision systems after image processing, the pose of the target relative to the camera and the robot is calculated.
This requires calibration of the camera, target and robot models, and the calibration accuracy affects the control accuracy, which is the difficulty of this method. During control, the required pose change is converted into the rotation angle of the robot joint, which is controlled by the joint controller.
In image-based vision servo systems, control error information comes from the difference between the features of the target image and the features of the desired image.
For this control method, the main question is how to establish the image Jacobian matrix that reflects the relationship between changes in image differences and changes in the pose speed of the robotic hand; Another issue is that the image is two-dimensional, and calculating the image's Jacobian matrix requires estimating the depth of the target (three-dimensional information), which has always been a challenge in computer vision.
The methods for calculating the Jacobian matrix include formula derivation, calibration, estimation and learning methods, etc. The former can be derived or calibrated based on the model, and the latter can be estimated online. Learning methods mainly use neural network methods.
● For robots using closed-loop joint controllers, vision servo systems are divided into dynamic observation-motion systems and direct vision servos.
The first uses feedback from the robot joint to stabilize the robotic arm, and the image processing module calculates the speed or position increment that the camera should have, which is then fed back to the robot joint controller. The latter makes the image processing module directly calculate the amount of joint movement control of the robot arm.
Primary challenges in visual service
Visual servo research has spanned nearly two decades. However, with its multidisciplinary nature, its evolution depends heavily on advances in these different areas. There are still many problems in visual servo research that remain unsolved.
- The biggest challenge in image servo is the theoretical and practical calculation speed of image processing methods.
- After image processing, creating a model that links image features to robotic joint movement represents another significant hurdle in image servoing.
- Many current control methods cannot guarantee large-scale stability during operation, necessitating further research into these control techniques.
Future perspectives for visual servoing
Key future research directions in visual servoing include:
• Fast and robust capture of image features in real environments is a critical issue for servo visualization systems.
Given the large amount of information in image processing and the development of programmable device technology, the implementation of general hardware-based algorithms to accelerate information processing can advance this issue.
• Establish relevant theories and suitable software for robotic vision systems.
Many current image processing methods in robotic vision systems are not adapted for these systems. If specialized software platforms were available, this could reduce workload and even improve system performance through hardware visualization information processing.
• Application of various artificial intelligence methods to robotic vision servo systems.
Although neural networks have already been implemented in robotic vision, many intelligent methods have not yet been fully utilized.
Over-reliance on mathematical modeling and calculations can lead to excessive computational demands during operation that current computer processing speeds may have difficulty meeting.
However, humans do not achieve related functions through extensive calculations, suggesting that artificial intelligence methods could reduce mathematical calculations and meet system speed requirements.
• Implementation of active vision techniques in robotic vision servo systems.
Active vision, a hot topic in current computer and machine vision research, allows vision systems to actively perceive their environment and extract necessary image features based on defined rules. This approach can solve problems that are normally difficult to solve.
• Integration of visual sensors with other external sensors.
To enable robots to perceive their environment more fully, especially to complement the information from robotic vision systems, various sensors could be added to robotic vision systems.
This could solve some of the difficulties in robotic vision systems, but the introduction of multiple sensors would require solving problems of information fusion and redundancy in robotic vision systems.
Conclusion
In recent years, significant progress has been made in robotic vision servo vision technology, with practical applications increasing both domestically and abroad. Many technical challenges are expected to be overcome in short-term research.
In the coming period, robotic vision systems will occupy a prominent position in robotic technology and their industrial applications will continue to expand.
