Search

With the rapid development of artificial intelligence and computer vision, numerous technologies have been introduced to automate manufacturing in the industry. Typical metal workpieces in the industry often have highly reflective surfaces, come in various sizes, and are positioned irregularly. The motor rotor presented in this paper is one such representative workpiece. Traditional grasping methods for workpiece loading and unloading are pre-programmed and often struggle to cope with complex and disordered situations. In this paper, we introduce a structured light (SL) sensor as the visual guide for the triaxial robot. Furthermore, we propose a high-precision hand-eye calibration method for the non-orthogonal coordinate system of the triaxial robot. Additionally, a motor rotor center localization method based on U-Net image segmentation is proposed. By combining the high-precision hand-eye calibration and localization, we can accurately and automatically locate and grasp the rotor. We have conducted sufficient experiments to verify the effectiveness and accuracy of our system. [ABSTRACT FROM AUTHOR]

Hand-eye calibration, a fundamental task in vision-based robotic systems, is commonly equipped with collaborative robots, especially for robotic applications in small and medium-sized enterprises (SMEs). Most approaches to hand-eye calibration rely on external markers or human assistance. We proposed a novel methodology that addresses the hand-eye calibration problem using the robot base as a reference, eliminating the need for external calibration objects or human intervention. Using point clouds of the robot base, a transformation matrix from the coordinate frame of the camera to the robot base is established as “I=AXB.” To this end, we exploit learning-based 3D detection and registration algorithms to estimate the location and orientation of the robot base. The robustness and accuracy of the method are quantified by ground-truth-based evaluation, and the accuracy result is compared with other 3D vision-based calibration methods. To assess the feasibility of our methodology, we carried out experiments utilizing a low-cost structured light scanner across varying joint configurations and groups of experiments. The proposed hand-eye calibration method achieved a translation deviation of 0.930 mm and a rotation deviation of 0.265 degrees according to the experimental results. Additionally, the 3D reconstruction experiments demonstrated a rotation error of 0.994 degrees and a position error of 1.697 mm. Moreover, our method offers the potential to be completed in 1 second, which is the fastest compared to other 3D hand-eye calibration methods. We conduct indoor 3D reconstruction and robotic grasping experiments based on our hand-eye calibration method. Related code is released at . [ABSTRACT FROM AUTHOR]

In industrial environments, cameras are heavily influenced by light and perspectives, making traditional vision-based parallel robots struggle with object-sorting tasks. Two-dimensional vision lacks depth information, which limits parallel robots to pick up objects at various heights. This paper proposes a 3D pickup estimation method for parallel robots based on point cloud simplification and registration for multi-objective pickup tasks. Firstly, a point cloud segmentation method based on the spatial domain is proposed to separate incomplete point clouds of object from the robot's workspace. The object dataset is generated by scanning complete point clouds of object using a 3D scanner. Secondly, according to the fast point feature histogram (FPFH) and the weight locally optimal projection (WLOP) algorithms, fusing the FPFH and WLOP (FF-WLOP) method is proposed to simplify the incomplete point cloud and extract more distinctive edge features of objects. The complete point cloud from the dataset is aligned with the simplified incomplete point cloud, and the calculated barycenter's coordinate information is given to the incomplete point cloud. Next, a dynamic weight singular value decomposition (D-SVD) hand-eye calibration method and an optimal projection point strategy are proposed to transform the barycenter coordinates of the object to the optimal pickup coordinates. Experimental results show that the point cloud registration error is 0.38 mm, the pickup rate is 92%, and the robot positioning error is 4.67 mm, which meets the basic pickup requirements. [ABSTRACT FROM AUTHOR]

Marker-less hand–eye calibration permits the acquisition of an accurate transformation between an optical sensor and a robot in unstructured environments. Single monocular cameras, despite their low cost and modest computation requirements, present difficulties for this purpose due to their incomplete correspondence of projected coordinates. In this work, we introduce a hand–eye calibration procedure based on the rotation representations inferred by an augmented autoencoder neural network. Learning-based models that attempt to directly regress the spatial transform of objects such as the links of robotic manipulators perform poorly in the orientation domain, but this can be overcome through the analysis of the latent space vectors constructed in the autoencoding process. This technique is computationally inexpensive and can be run in real time in markedly varied lighting and occlusion conditions. To evaluate the procedure, we use a color-depth camera and perform a registration step between the predicted and the captured point clouds to measure translation and orientation errors and compare the results to a baseline based on traditional checkerboard markers. [ABSTRACT FROM AUTHOR]

This study addresses the problem of hand–eye calibration in robotic systems by developing Continual Learning (CL)-based approaches. Traditionally, robots require explicit models to transfer knowledge from camera observations to their hands or base. However, this poses limitations, as the hand–eye calibration parameters are typically valid only for the current camera configuration. We, therefore, propose a flexible and autonomous hand–eye calibration system that can adapt to changes in camera pose over time. Three CL-based approaches are introduced: the naive CL approach, the reservoir rehearsal approach, and the hybrid approach combining reservoir sampling with new data evaluation. The naive CL approach suffers from catastrophic forgetting, while the reservoir rehearsal approach mitigates this issue by sampling uniformly from past data. The hybrid approach further enhances performance by incorporating reservoir sampling and assessing new data for novelty. Experiments conducted in simulated and real-world environments demonstrate that the CL-based approaches, except for the naive approach, achieve competitive performance compared to traditional batch learning-based methods. This suggests that treating hand–eye calibration as a time sequence problem enables the extension of the learned space without complete retraining. The adaptability of the CL-based approaches facilitates accommodating changes in camera pose, leading to an improved hand–eye calibration system. [ABSTRACT FROM AUTHOR]

[Objective] Designing a apace grid-based hand-eye calibration method to reduce robot positioning errors and calibration object Recognition errors in automated food production lines. [Methods] Based on the composition of the attitude matrix, the attitude and position of the end of the robot relative to the camera were obtained by translating and rotating the end of the robot, and a registration method using known corresponding point pairs was proposed to reduce the influence of the robot positioning error on the attitude solution in the translation method. Based on the space circle fitting method, a two-point vector method was proposed to solve the end position, which reduced the amount of data collection. A method of fitting the center of the sphere was proposed to compensate for the result of hand-eye calibration. [Results] The average measured error after calibration is 0.8 mm. [Conclusion] The hand-eye system based on the Eye-In-Hand type, can realize high-precision object positioning.

Aiming at solving the problems such as long calibration time, low precision, and complex operation in hand-eye calibration of welding robot, an automatic hand-eye calibration algorithm based on linear structured light was proposed to solve the calibration matrix X by using AX=ZB calibration equation. Firstly, a square calibration plate is customized to effectively constrain the structured light. The α-shape algorithm was adopted to extract the contour of the 3D point cloud model of the calibration plate. Secondly, an improved random sampling consistency algorithm which could determine the optimal iterative number was proposed to fit the contour point cloud, the contour point cloud model fitted was obtained. Finally, the 3D coordinates of the target points were determined with the linear structured light to complete the hand-eye calibration. In order to prevent the calibration plate from deviating from the acquisition range of the vision sensor during the calibration process, the distance between the linear structural light and the inner circle in the calibration plate was set to limit the motion range of the robot. In order to eliminate the error transfer of the robot body, an optimal solution of the rotation matrix R and the translation vector t of the calibration data was calculated with the singular value decomposition (SVD) and the least square rigid transpose method. The experimental results show that the calibration accuracy reaches 0.3 mm without compensating the robot body error, and the calibration speed is improved by 36% than the existing automatic calibration method. Therefore, the algorithm proposed can automatically complete the calibration only by establishing the user coordinates in advance, which improves the working accuracy and efficiency of the welding robots greatly. [ABSTRACT FROM AUTHOR]

The hand-eye calibration problem is an important application problem in robot research. Based on the 2-norm of dual quaternion vectors, we propose a new dual quaternion optimization method for the hand-eye calibration problem. The dual quaternion optimization problem is decomposed to two quaternion optimization subproblems. The first quaternion optimization subproblem governs the rotation of the robot hand. It can be solved efficiently by the eigenvalue decomposition or singular value decomposition. If the optimal value of the first quaternion optimization subproblem is zero, then the system is rotationwise noiseless, i.e., there exists a "perfect" robot hand motion which meets all the testing poses rotationwise exactly. In this case, we apply the regularization technique for solving the second subproblem to minimize the distance of the translation. Otherwise we apply the patching technique to solve the second quaternion optimization subproblem. Then solving the second quaternion optimization subproblem turns out to be solving a quadratically constrained quadratic program. In this way, we give a complete description for the solution set of hand-eye calibration problems. This is new in the hand-eye calibration literature. The numerical results are also presented to show the efficiency of the proposed method. [ABSTRACT FROM AUTHOR]

At present, the HUD calibration of the whole vehicle enterprise adopts the traditional physical Master Gauge to calibrate the camera, and the scheme has low flexibility, occupies a large space, and is unsuitable for the mixed production of different types of vehicles. A flexible calibration approach is presented based on collaborative robots for HUD inspection of passenger cars. They are considering that the general HUD (Head-Up Display) virtual image distance is 2–3 m. A 2.3 m hand-eye calibration method is investigated by mounting a monocular camera on a collaborative robot, considering the effect of ambient light on long-distance hand-eye calibration, and employing customized alumina calibration plates and lighting through appropriate angles. Using the area_scan_polynomial model for camera calibration, the camera’s internal reference is solved, and the average error of the resulting camera’s internal reference is about 0.15. Compared with the master gauge camera calibration, it shows higher flexibility. The hand-eye calibration results show that the Translation part’s maximum error is about 1mm, the maximum error of the Rotational position is about 0.1 degree, and the camera calibration error is within 0.08 pixels. For repeatability, the median of Translation RMS was 0.5 with an interquartile range of 0.171, and the median of Translation Maximum was 0.972 with an interquartile range of 0.216. The median of Rotation RMS was 0.0325 with an interquartile range of 0.0125. The median of the Rotation Maximum was 0.059, with an interquartile range of 0.016. The data of translation is relatively scattered, with an extensive range of changes, but it still maintains a certain repeatability. The Rotation data is relatively stable, with a small degree of change and good repeatability, further meeting the use requirements in the final inspection HUD scene of the whole vehicle.

In the process of industrial production, manual assembly of workpieces exists with low efficiency and high intensity, and some of the assembly process of the human body has a certain degree of danger. At the same time, traditional machine learning algorithms are difficult to adapt to the complexity of the current industrial field environment; the change in the environment will greatly affect the accuracy of the robot's work. Therefore, this paper proposes a method based on the combination of machine vision and the YOLOv5 deep learning model to obtain the disk porous localization information, after coordinate mapping by the ROS communication control robotic arm work, in order to improve the anti-interference ability of the environment and work efficiency but also reduce the danger to the human body. The system utilizes a camera to collect real-time images of targets in complex environments and, then, trains and processes them for recognition such that coordinate localization information can be obtained. This information is converted into coordinates under the robot coordinate system through hand–eye calibration, and the robot is then controlled to complete multi-hole localization and tracking by means of communication between the upper and lower computers. The results show that there is a high accuracy in the training and testing of the target object, and the control accuracy of the robotic arm is also relatively high. The method has strong anti-interference to the complex environment of industry and exhibits a certain feasibility and effectiveness. It lays a foundation for achieving the automated installation of docking disk workpieces in industrial production and also provides a more favorable choice for the production and installation of the process of screw positioning needs. [ABSTRACT FROM AUTHOR]

Three dimensional (3D) robot measurement systems, with binocular planar structured light cameras (3D cameras) installed at the terminal flange of robots, are widely used to measure complete workpieces by stitching point clouds obtained from various sampled poses. To ensure accurate stitching, hand–eye calibration is necessary. However, 3D cameras often demonstrate low precision in measuring jumping edges and vertices, resulting in challenges when selecting appropriate calibrators and calibration methods for hand–eye calibration. We propose a simple and high‐precision hand–eye calibration method for 3D robot measurement systems. Initially, a single sphere is utilized as the calibrator, and its poses are observed using specific robotic motions. Subsequently, the hand–eye calibration problem is formulated as a well‐known equation, AX=XB$A X = X B$, relying solely on the high‐precision relative pose of the robot. Finally, a novel simultaneous solution for AX=XB$A X = X B$ and the corresponding closed‐form initial solution are introduced. Simulations and experiments confirm that the proposed method exhibits high noise resistance and achieves high‐precision calibration, even with a limited calibration data quantity. Compared with traditional methods, the fitting error of the proposed method can be reduced from over 0.9 mm to less than 0.6 mm. [ABSTRACT FROM AUTHOR]

Marker-less hand–eye calibration permits the acquisition of an accurate transformation between an optical sensor and a robot in unstructured environments. Single monocular cameras, despite their low cost and modest computation requirements, present difficulties for this purpose due to their incomplete correspondence of projected coordinates. In this work, we introduce a hand–eye calibration procedure based on the rotation representations inferred by an augmented autoencoder neural network. Learning-based models that attempt to directly regress the spatial transform of objects such as the links of robotic manipulators perform poorly in the orientation domain, but this can be overcome through the analysis of the latent space vectors constructed in the autoencoding process. This technique is computationally inexpensive and can be run in real time in markedly varied lighting and occlusion conditions. To evaluate the procedure, we use a color-depth camera and perform a registration step between the predicted and the captured point clouds to measure translation and orientation errors and compare the results to a baseline based on traditional checkerboard markers.

Three dimensional (3D) robot measurement systems, with binocular planar structured light cameras (3D cameras) installed at the terminal flange of robots, are widely used to measure complete workpieces by stitching point clouds obtained from various sampled poses. To ensure accurate stitching, hand–eye calibration is necessary. However, 3D cameras often demonstrate low precision in measuring jumping edges and vertices, resulting in challenges when selecting appropriate calibrators and calibration methods for hand–eye calibration. We propose a simple and high‐precision hand–eye calibration method for 3D robot measurement systems. Initially, a single sphere is utilized as the calibrator, and its poses are observed using specific robotic motions. Subsequently, the hand–eye calibration problem is formulated as a well‐known equation, AX=XB, relying solely on the high‐precision relative pose of the robot. Finally, a novel simultaneous solution for AX=XB and the corresponding closed‐form initial solution are introduced. Simulations and experiments confirm that the proposed method exhibits high noise resistance and achieves high‐precision calibration, even with a limited calibration data quantity. Compared with traditional methods, the fitting error of the proposed method can be reduced from over 0.9 mm to less than 0.6 mm.

Hand-eye calibration is one of the important problems in the field of robot vision and control, aiming to determine the pose transformation between the robot end-effector and the visual system. The analytical solution of this problem has a more explicit numerical computation process and a more stable solution time compared with iterative or deep learning methods, and theoretically provides more accurate results. When using quaternion parameterization for rigid body motion rotation, the hand-eye calibration problem can be transformed into solving the eigenvector corresponding to the maximum eigenvalue of a symmetric matrix. This paper proposes an analytical solution for hand-eye calibration based on quaternion parameterization. Compared with previous methods, the advantage of this method is that the solution for the required eigenvector is transformed into solving cubic equations through the eigenvector-eigenvalue identity, without SVD or Eigendecomposition, thus making the proposed method a complete analytical solution, and avoids solving irrelevant information. The performance of the proposed method was evaluated through multiple experiments conducted on a synthetic dataset as well as two real-world datasets, and compared to representative analytical methods. The experimental results unequivocally demonstrate that our proposed method achieves comparable accuracy with significantly shorter solution times. Therefore, our complete analytical solution offers an efficient and accurate alternative for addressing the hand-eye calibration problem without SVD or Eigendecomposition. [ABSTRACT FROM AUTHOR]

With the promotion of Industry 4.0 reform, the trend of intelligent and precise production in the production workshop is gradually highlighted. This directly leads to higher requirements for robot hand eye coordination accuracy in automated workshops. In order to achieve more precise robot hand eye coordination control, this study designed a new mean calculation method based on the probability density theory, and designed a new mean robot hand eye calibration algorithm based on this. After the test, it is found that the translation error and rotation error calculated by the new mean algorithm are 0.26 and 0.92 respectively, which are significantly lower than other comparison algorithms when using all test samples of normal distribution. And the calculation time of the algorithm when using all the test samples is 2115 ms, which is also significantly lower than the comparison algorithm. The simulation results show that the new mean hand eye calibration method designed in this study can achieve more accurate hand eye coordination control of robots, and has certain application potential in high-precision industrial production scenarios. [ABSTRACT FROM AUTHOR]

In the functioning of the hand–eye collaboration of an apple picking robot, the accuracy of the hand–eye relationship is a key factor affecting the efficiency and accuracy of the robot's operation. In order to enhance the low accuracy of traditional hand–eye calibration methods, linear and nonlinear solving methods based on mathematical tools such as quaternions are commonly adopted. To solve the loss of accuracy in decoupling during the linearization solution and to reduce the cumulative error that occurs during nonlinear solutions, a hand–eye calibration method, based on the ICP algorithm, is proposed in this paper. The method initializes the ICP matching algorithm with a solution derived from Tsai–Lenz, and substitutes it for iterative computation, thereby ascertaining a precise hand–eye conversion relationship by optimizing the error threshold and iteration count in the ICP matching process. Experimental results demonstrate that the ICP-based hand–eye calibration optimization algorithm not only circumvents the issues pertaining to accuracy loss and significant errors during solving, but also enhances the rotation accuracy by 13.6% and the translation accuracy by 2.47% compared with the work presented by Tsai–Lenz. [ABSTRACT FROM AUTHOR]

For a robot to pick up an object viewed by a camera, the object’s position in the image coordinate system must be converted to the robot coordinate system. Recently, a neural network-based method was proposed to achieve this task. This methodology can accurately convert the object’s position despite errors and disturbances that arise in a real-world environment, such as the deflection of a robot arm triggered by changes in the robot’s posture. However, this method has some drawbacks, such as the need for significant effort in model selection, hyperparameter tuning, and lack of stability and interpretability in the learning results. To address these issues, a method involving linear and nonlinear regressions is proposed. First, linear regression is employed to convert the object’s position from the image coordinate system to the robot base coordinate system. Next, B-splines-based nonlinear regression is applied to address the errors and disturbances that occur in a real-world environment. Since this approach is more stable and has better calibration performance with interpretability as opposed to the recent method, it is more practical. In the experiment, calibration results were incorporated into a robot, and its performance was evaluated quantitatively. The proposed method achieved a mean position error of 0.5 mm, while the neural network-based method achieved an error of 1.1 mm.

Hand-eye calibration is crucial for vision-based robots using open-loop visual control. Based on the product of exponentials model, this study introduces a novel hand-eye calibration algorithm to improve calibration accuracy. As opposed to the traditional method of solving the AX = XB type in the hand-eye calibration using the marker, we transform the solution into an AX = B type without requiring accurate camera parameters. Firstly, the nominal marker pose in the robot base is described by the nominal kinematic parameters of the robot, camera, and end-effector-to-camera. The actual marker pose is obtained by physically contacting the marker with the probe. Subsequently, the error parameter of the hand-eye pose is defined as linear to the error between the nominal and actual marker poses and can be solved by an iterative least-squares algorithm. Finally, the calibrated hand-eye pose is acquired by multiplying its nominal pose and the exponential mapping of the error parameter. The simulations and real experiments illustrate the effectiveness and stability of the proposed algorithm. Noteworthy, the mean position accuracy of the vision-based robot is improved nearly 20 times after calibration. Furthermore, the comparison experiment demonstrates that the proposed algorithm is preferable for improving hand-eye calibration accuracy. [ABSTRACT FROM AUTHOR]

With the increasing automation of mobile phone assembly, industrial robots are gradually being used in production lines for loading and unloading operations. At present, industrial robots are mainly used in online teaching mode, in which the robot's movement and path are set by teaching in advance and then repeat the point-to-point operation. This mode of operation is less flexible and requires high professionalism in teaching and offline programming. When positioning and grasping different materials, the adjustment time is long, which affects the efficiency of production changeover. To solve the problem of poor adaptability of loading robots to differentiated products in mobile phone automatic assembly lines, it is necessary to quickly adjust the positioning and grasping of different models of mobile phone middle frames. Therefore, this paper proposes a highly adaptive grasping and positioning method for vision-guided right-angle robots. [ABSTRACT FROM AUTHOR]

For a robot to pick up an object viewed by a camera, the object's position in the image coordinate system must be converted to the robot coordinate system. Recently, a neural network-based method was proposed to achieve this task. This methodology can accurately convert the object's position despite errors and disturbances that arise in a real-world environment, such as the deflection of a robot arm triggered by changes in the robot's posture. However, this method has some drawbacks, such as the need for significant effort in model selection, hyperparameter tuning, and lack of stability and interpretability in the learning results. To address these issues, a method involving linear and nonlinear regressions is proposed. First, linear regression is employed to convert the object's position from the image coordinate system to the robot base coordinate system. Next, B-splines-based nonlinear regression is applied to address the errors and disturbances that occur in a real-world environment. Since this approach is more stable and has better calibration performance with interpretability as opposed to the recent method, it is more practical. In the experiment, calibration results were incorporated into a robot, and its performance was evaluated quantitatively. The proposed method achieved a mean position error of 0.5 mm, while the neural network-based method achieved an error of 1.1 mm. [ABSTRACT FROM AUTHOR]

This paper studies the effect of reference frame selection in sensor-to-sensor extrinsic calibration when formulated as a motion-based hand–eye calibration problem. As the sensor trajectories typically contain some composition of noise, the aim is to determine which selection strategies work best under which noise conditions. Different reference selection options are tested under varying noise conditions in simulations, and the findings are validated with real data from the KITTI dataset. The study is conducted for four state-of-the-art methods, as well as two proposed cost functions for nonlinear optimization. One of the proposed cost functions incorporates outlier rejection to improve calibration performance and was shown to significantly improve performance in the presence of outliers, and either match or outperform the other algorithms in other noise conditions. However, the performance gain from reference frame selection was deemed larger than that from algorithm selection. In addition, we show that with realistic noise, the reference frame selection method commonly used in the literature, is inferior to other tested options, and that relative error metrics are not reliable for telling which method achieves best calibration performance. [ABSTRACT FROM AUTHOR]

The measurement system proposed in this paper, using a measuring arm and line structured light, has a wide range of applications. To improve the scanning efficiency, the system outlined in this paper uses two single-line structured lights to form crosshair structured light, which we combine with a measuring arm to form a comprehensive scanning measurement system. The calibration method of Zhengyou Zhang and a calibration board are used to complete parameter calibration of the sensors and cameras, as well as hand–eye calibration of the measuring arm. For complex curved-surface objects, this system extracts the cross-line structured light optical center location, which suffers from ambiguity. Therefore, we introduce the use of periodic control of the two line structured light sources in order to resolve the light extraction polysemy. Our experimental results indicate that the proposed system can effectively satisfy the function of crosshair structured light scanning of large, complex surfaces. [ABSTRACT FROM AUTHOR]

In the process of industrial production, manual assembly of workpieces exists with low efficiency and high intensity, and some of the assembly process of the human body has a certain degree of danger. At the same time, traditional machine learning algorithms are difficult to adapt to the complexity of the current industrial field environment; the change in the environment will greatly affect the accuracy of the robot’s work. Therefore, this paper proposes a method based on the combination of machine vision and the YOLOv5 deep learning model to obtain the disk porous localization information, after coordinate mapping by the ROS communication control robotic arm work, in order to improve the anti-interference ability of the environment and work efficiency but also reduce the danger to the human body. The system utilizes a camera to collect real-time images of targets in complex environments and, then, trains and processes them for recognition such that coordinate localization information can be obtained. This information is converted into coordinates under the robot coordinate system through hand–eye calibration, and the robot is then controlled to complete multi-hole localization and tracking by means of communication between the upper and lower computers. The results show that there is a high accuracy in the training and testing of the target object, and the control accuracy of the robotic arm is also relatively high. The method has strong anti-interference to the complex environment of industry and exhibits a certain feasibility and effectiveness. It lays a foundation for achieving the automated installation of docking disk workpieces in industrial production and also provides a more favorable choice for the production and installation of the process of screw positioning needs.

The Euclidean distance error of calibration results cannot be calculated during the hand–eye calibration process of a manipulator because the true values of the hand–eye conversion matrix cannot be obtained. In this study, a new method for error analysis and algorithm optimization is presented. An error analysis of the method is carried out using a priori knowledge that the location of the augmented reality markers is fixed during the calibration process. The coordinates of the AR marker center point are reprojected onto the pixel coordinate system and then compared with the true pixel coordinates of the AR marker center point obtained by corner detection or manual labeling to obtain the Euclidean distance between the two coordinates as the basis for the error analysis. We then fine-tune the results of the hand–eye calibration algorithm to obtain the smallest reprojection error, thereby obtaining higher-precision calibration results. The experimental results show that, compared with the Tsai–Lenz algorithm, the optimized algorithm in this study reduces the average reprojection error by 44.43% and the average visual positioning error by 50.63%. Therefore, the proposed optimization method can significantly improve the accuracy of hand–eye calibration results.

Multi-line LiDAR and GPS/IMU are widely used in autonomous driving and robotics, such as simultaneous localization and mapping (SLAM). Calibrating the extrinsic parameters of each sensor is a necessary condition for multi-sensor fusion. The calibration of each sensor directly affects the accurate positioning control and perception performance of the vehicle. Through the algorithm, accurate extrinsic parameters and a symmetric covariance matrix of extrinsic parameters can be obtained as a measure of the confidence of the extrinsic parameters. As for the calibration of LiDAR-GPS/IMU, many calibration methods require specific vehicle motion or manual calibration marking scenes to ensure good constraint of the problem, resulting in high costs and a low degree of automation. To solve this problem, we propose a new two-step self-calibration method, which includes extrinsic parameter initialization and refinement. The initialization part decouples the extrinsic parameters from the rotation and translation part, first calculating the reliable initial rotation through the rotation constraints, then calculating the initial translation after obtaining a reliable initial rotation, and eliminating the accumulated drift of LiDAR odometry by loop closure to complete the map construction. In the refinement part, the LiDAR odometry is obtained through scan-to-map registration and is tightly coupled with the IMU. The constraints of the absolute pose in the map refined the extrinsic parameters. Our method is validated in the simulation and real environments, and the results show that the proposed method has high accuracy and robustness. [ABSTRACT FROM AUTHOR]

Many approaches have been developed to solve the hand–eye calibration problem. The traditional approach involves a precise mathematical model, which has advantages and disadvantages. For example, mathematical representations can provide numerical and quantitative results to users and researchers. Thus, it is possible to explain and understand the calibration results. However, information about the end-effector, such as the position attached to the robot and its dimensions, is not considered in the calibration process. If there is no CAD model, additional calibration is required for accurate manipulation, especially for a handmade end-effector. A neural network-based method is used as the solution to this problem. By training a neural network model using data created via the attached end-effector, additional calibration can be avoided. Moreover, it is not necessary to develop a precise and complex mathematical model. However, it is difficult to provide quantitative information because a neural network is a black box. Hence, a method with both advantages is proposed in this study. A mathematical model was developed and optimized using the data created by the attached end-effector. To acquire accurate data and evaluate the calibration results, a tablet computer was utilized. The established method achieved a mean positioning error of 1.0 mm. [ABSTRACT FROM AUTHOR]

Aiming at the stability of hand-eye calibration in fruit picking scene, a simple hand-eye calibration method for picking robot based on optimization combined with TOF (Time of Flight) camera is proposed. This method needs to fix the TOF depth camera at actual and calculated coordinates of the peach the end of the robot, operate the robot to take pictures of the calibration board from different poses, and record the current photographing poses to ensure that each group of pictures is clear and complete, so as to use the TOF depth camera to image the calibration board. Obtain multiple sets of calibration board depth maps and corresponding point cloud data, that is, “eye” data. Through the circle center extraction and positioning algorithm, the circle center points on each group of calibration plates are extracted, and a circle center sorting method based on the vector angle and the center of mass coordinates is designed to solve the circle center caused by factors such as mirror distortion, uneven illumination and different photographing poses. And through the tool center point of the actuator, the coordinate value of the circle center point on the four corners of each group of calibration plates in the robot end coordinate system is located in turn, and the “hand” data is obtained. Combined with the SVD method, And according to the obtained point residuals, the weight coefficients of the marker points are redistributed, and the hand-eye parameters are iteratively optimized, which improves the accuracy and stability of the hand-eye calibration. the method proposed in this paper has a better ability to locate the gross error under the environment of large gross errors. In order to verify the feasibility of the hand-eye calibration method, the indoor picking experiment was simulated, and the peaches were identified and positioned by combining deep learning and 3D vision to verify the proposed hand-eye calibration method. The JAKA six-axis robot and TuYang depth camera are used to build the experimental platform. The experimental results show that the method is simple to operate, has good stability, and the calibration plate is easy to manufacture and low in cost. work accuracy requirements. [ABSTRACT FROM AUTHOR]

The vision system provides an important way for construction robots to obtain the type and spatial location information of the object. The characteristics of the construction environment, construction object, and robot structure are jointly examined in this paper to propose an approach of object segmentation by spraying the robot based on multi-layer perceptron. Firstly, the hand-eye system experimental platform is built through establishing the mathematical model of the system and calibrating the parameters of the model. Secondly, effort is made to carry out research on image preprocessing algorithms and related experiments, and compare the effects of different binocular stereo-matching algorithms in the actual engineering environment. Finally, research and an experiment are conducted to identify the applicability and effect of the depth image object segmentation algorithm based on multi-layer perceptron. The experimental results prove that the application of multi-layer perceptron to object segmentation by spraying robots can meet the requirement on solution accuracy and is suitable for the object segmentation of complex projects in real life. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots. [ABSTRACT FROM AUTHOR]

We propose a novel high-accuracy active hand-eye calibration approach. In our method, the robot movement and camera view selection are both driven by and targeting the improvement of calibration accuracy. During the calibration process, the data acquisition is guided by an online estimated discrete viewing quality field (DVQF), representing the calibration quality of different views in various 3D locations. The view quality is measured by how much it reduces the uncertainty of calibration results and increases the diversity of robot poses, contributing to the calibration precision. Based on DVQF, we select the next-best-view as the target moving pose for each time step. A fully automatic system is presented to perform the overall hand-eye calibration process without any human intervention. Numerous experiments are conducted both in real-world and simulated scenarios. The proposed algorithm outperforms other approaches and shows much superiority in accuracy and robustness. [ABSTRACT FROM AUTHOR]

This paper presents a modified model for hand–eye calibration based on dual quaternion algebra. By using dual quaternions to represent the rotations and translations of a rigid body simultaneously in the task space, the formulation is elegant for the analysis of the hand–eye equation. The hand–eye transformation derived in this study is represented in a compact manner, which uses a combination of the dual part and the real part of the dual quaternion. Although the hand–eye equation can be solved by using six elements of a dual quaternion without using its scalar parts, the scaler numbers in both the real and dual parts of a dual quaternion contain part of the pose information. The originality is based on the derivation of the construct of the identification algorithm of external parameters of the camera by using all eight elements of a dual quaternion. Then, the data transformation between the cameras of the dual-arm hand–eye robot system is presented. The corresponding results demonstrate that the proposed hand–eye calibration algorithm can process measurement data with noise and can also improve the identification accuracy to verify its efficiency. [ABSTRACT FROM AUTHOR]

This paper investigates the direct application of standardized designs on the robot for conducting robot hand–eye calibration by employing 3D scanners with collaborative robots. The well-established geometric features of the robot flange are exploited by directly capturing its point cloud data. In particular, an iterative method is proposed to facilitate point cloud processing towards a refined calibration outcome. Several extensive experiments are conducted over a range of collaborative robots, including Universal Robots UR5 & UR10 e-series, Franka Emika, and AUBO i5 using an industrial-grade 3D scanner Photoneo Phoxi S & M and a commercial-grade 3D scanner Microsoft Azure Kinect DK. Experimental results show that translational and rotational errors converge efficiently to less than 0.28 mm and 0.25 degrees, respectively, achieving a hand–eye calibration accuracy as high as the camera's resolution, probing the hardware limit. A welding seam tracking system is presented, combining the flange-based calibration method with soft tactile sensing. The experiment results show that the system enables the robot to adjust its motion in real-time, ensuring consistent weld quality and paving the way for more efficient and adaptable manufacturing processes. [Display omitted] • A novel hand–eye calibration method with intrinsic design features of robot. • An iterative algorithm for effective optimization of hand–eye calibration. • Quantitative evaluation of the calibration method on various cobots and 3D scanners. • A safe and adaptable weld seam tracking system with the calibration method. [ABSTRACT FROM AUTHOR]

In the functioning of the hand–eye collaboration of an apple picking robot, the accuracy of the hand–eye relationship is a key factor affecting the efficiency and accuracy of the robot’s operation. In order to enhance the low accuracy of traditional hand–eye calibration methods, linear and nonlinear solving methods based on mathematical tools such as quaternions are commonly adopted. To solve the loss of accuracy in decoupling during the linearization solution and to reduce the cumulative error that occurs during nonlinear solutions, a hand–eye calibration method, based on the ICP algorithm, is proposed in this paper. The method initializes the ICP matching algorithm with a solution derived from Tsai–Lenz, and substitutes it for iterative computation, thereby ascertaining a precise hand–eye conversion relationship by optimizing the error threshold and iteration count in the ICP matching process. Experimental results demonstrate that the ICP-based hand–eye calibration optimization algorithm not only circumvents the issues pertaining to accuracy loss and significant errors during solving, but also enhances the rotation accuracy by 13.6% and the translation accuracy by 2.47% compared with the work presented by Tsai–Lenz.

Pedicle screw insertion with robot assistance dramatically improves surgical accuracy and safety when compared with manual implantation. In developing such a system, hand–eye calibration is an essential component that aims to determine the transformation between a position tracking and robot-arm systems. In this paper, we propose an effective hand–eye calibration method, namely registration-based hand–eye calibration (RHC), which estimates the calibration transformation via point set registration without the need to solve the AX = XB equation. Our hand–eye calibration method consists of tool-tip pivot calibrations in two-coordinate systems, in addition to paired-point matching, where the point pairs are generated via the steady movement of the robot arm in space. After calibration, our system allows for robot-assisted, image-guided pedicle screw insertion. Comprehensive experiments are conducted to verify the efficacy of the proposed hand–eye calibration method. A mean distance deviation of 0.70 mm and a mean angular deviation of 0.68° are achieved by our system when the proposed hand–eye calibration method is used. Further experiments on drilling trajectories are conducted on plastic vertebrae as well as pig vertebrae. A mean distance deviation of 1.01 mm and a mean angular deviation of 1.11° are observed when the drilled trajectories are compared with the planned trajectories on the pig vertebrae. [ABSTRACT FROM AUTHOR]

Robotic-assisted surgical system has introduced a powerful platform through dexterous instrument and hand-eye coordination intuitive control. The knowledge of laparoscopic vision is a crucial piece of information for robot-assisted minimally invasive surgery focusing on improved surgical outcomes. Obtaining the transformation with respect to the laparoscope and robot slave arm frames using hand-eye calibration is essential, which is a key component for developing intuitive control algorithm. We proposed a novel two-step modified dual quaternion for hand-eye calibration in this study. The dual quaternion was exploited to solve the hand-eye calibration simultaneously and powered by an iteratively separate solution. The obtained hand-eye calibration result was applied to the intuitive control by using the hand-eye coordination criterion. Promising simulations and experimental studies were conducted to evaluate the proposed method on our surgical robot system. We extensively compared the proposed method with state-of-the-art methods. Results demonstrate this method can improve the calibration accuracy. The effectiveness of the intuitive control algorithm was quantitatively evaluated, and an improved hand-eye calibration method was developed. The relationship between laparoscope and robot kinematics can be established for intuitive control. [ABSTRACT FROM AUTHOR]

With the increasing automation of mobile phone assembly, industrial robots are gradually being used in production lines for loading and unloading operations. At present, industrial robots are mainly used in online teaching mode, in which the robot’s movement and path are set by teaching in advance and then repeat the point-to-point operation. This mode of operation is less flexible and requires high professionalism in teaching and offline programming. When positioning and grasping different materials, the adjustment time is long, which affects the efficiency of production changeover. To solve the problem of poor adaptability of loading robots to differentiated products in mobile phone automatic assembly lines, it is necessary to quickly adjust the positioning and grasping of different models of mobile phone middle frames. Therefore, this paper proposes a highly adaptive grasping and positioning method for vision-guided right-angle robots.

The problem of calibrating an extrinsic parameter between a camera and an inertial measurement unit (IMU) using an industrial robotic manipulator has been studied. This generates a result of hand–eye/robot–world/camera–IMU calibration in a simultaneous fashion. The developed method is free of inertial integration over time and, thus, is robust to uncertain IMU biases. It is derived that the problem can be solved via a simultaneous optimization of hand–eye/robot–world/camera–IMU transformations. The resulted optimization is highly nonconvex on the special Euclidean group, and we give globally optimal solutions. Experiments verify that the proposed method is capable of estimating accurate calibration parameters. Comparative studies between representatives show the global optimality of the proposed method. The new simultaneous method is capable of conducting calibration of a robot/camera/IMU combination. The designed method guarantees the global optimality; thus, the accuracy is ensured. The developed globally optimal solutions will also be computationally efficient on modern industrial computers. Finally, we show that the proposed method can give accurate calibration results for a stereo/IMU sensor combination. [ABSTRACT FROM AUTHOR]

Although robotic vision systems offer a promising technology solution for rapid and reconfigurable in-process 3D inspection of complex and large parts in contemporary manufacturing, measurement accuracy poses a challenge for its wide deployment. One of the key issues in adopting a robotic vision system is to understand the extent of its measurement errors which are directly correlated with the calibration process. In this paper, a possible source of practical and inherent measurement uncertainties involved in the calibration process of a robotic vision system are discussed. The system considered in this work consists of an image sensor mounted on an industrial robot manipulator with six degrees of freedom. Based on a series of experimental tests and computer simulations, the paper gives a comprehensive performance comparison of different calibration approaches and shows the impact of measurement uncertainties on the calibration process. It has been found from the error sensitivity analysis that minor uncertainties in the calibration process can significantly affect the accuracy of the robotic vision system. Further investigations suggest that inducing errors in image calibration patterns can have an adverse effect on the hand–eye calibration process compared to the angular errors in the robot joints. [ABSTRACT FROM AUTHOR]

The dynamic performance test of a hard disk head is the most effective method of evaluating hard disk performance, but it must be conducted in a dust- and electrostatic discharge (ESD)-free environment. With the rapid development of perpendicular magnetic recording (PMR) technology, the flying height of a magnetic head is reduced from 18 nm to less than 2 nm, which makes the testing conditions increasingly demanding, and the traditional way of manual testing by an operator cannot match the requirement of production testing. To meet the technical requirement of the dynamic performance test of a hard disk head, an automatic test system is developed using a CCD image sensor and an Epson robot. In this system, the vision system of the robot can accomplish, in real time and accurately, the image acquisition and image processing of the object in the magnetic head testing process and calculate the actual position of the object after hand-eye calibration. Visual guidance is provided to the robot to grasp and place the target accurately, and related sensors provide various state data during the system operation. Thus, the difficult problem in the development process from manual operation to robot automation in the dynamic performance test of a magnetic head is successfully solved. The results show that the consumption of test materials is considerably reduced, and the test precision and efficiency are markedly improved. [ABSTRACT FROM AUTHOR]

This paper studies the effect of reference frame selection in sensor-to-sensor extrinsic calibration when formulated as a motion-based hand–eye calibration problem. As the sensor trajectories typically contain some composition of noise, the aim is to determine which selection strategies work best under which noise conditions. Different reference selection options are tested under varying noise conditions in simulations, and the findings are validated with real data from the KITTI dataset. The study is conducted for four state-of-the-art methods, as well as two proposed cost functions for nonlinear optimization. One of the proposed cost functions incorporates outlier rejection to improve calibration performance and was shown to significantly improve performance in the presence of outliers, and either match or outperform the other algorithms in other noise conditions. However, the performance gain from reference frame selection was deemed larger than that from algorithm selection. In addition, we show that with realistic noise, the reference frame selection method commonly used in the literature, is inferior to other tested options, and that relative error metrics are not reliable for telling which method achieves best calibration performance.

To solve the problem of inflexibility of offline hand–eye calibration in "eye-in-hand" modes, an online hand–eye calibration method based on the ChArUco board is proposed in this paper. Firstly, a hand–eye calibration model based on the ChArUco board is established, by analyzing the mathematical model of hand–eye calibration, and the image features of the ChArUco board. According to the advantages of the ChArUco board, with both the checkerboard and the ArUco marker, an online hand–eye calibration algorithm based on the ChArUco board is designed. Then, the online hand–eye calibration algorithm, based on the ChArUco board, is used to realize the dynamic adjustment of the hand–eye position relationship. Finally, the hand–eye calibration experiment is carried out to verify the accuracy of the hand–eye calibration based on the ChArUco board. The robustness and accuracy of the proposed method are verified by online hand–eye calibration experiments. The experimental results show that the accuracy of the online hand–eye calibration method proposed in this paper is between 0.4 mm and 0.6 mm, which is almost the same as the offline hand–eye calibration accuracy. The method in this paper utilizes the advantages of the ChArUco board to realize online hand–eye calibration, which improves the flexibility and robustness of hand–eye calibration. [ABSTRACT FROM AUTHOR]

In view of the phenomenon that camera imaging will appear large up close and small from afar in the eye-to-hand hand-calibration system, one hand–eye calibration is carried out. The manipulator is only suitable for grasping objects of the same height, and the calibration results cannot be applied to grasping products with variable height. Based on the study of the pinhole camera model and the rigid transformation model between coordinate systems, the introduction of the calibration height parameters, the relationship between parameters of the rigid transformation matrix between image the coordinate system and the robot coordinate system, and sampling height are established. In the experiment, firstly, through the calibration of camera parameters, the influence of camera distortion on imaging quality is eliminated, and the influence of calibration height is ignored. Then, the machine coordinate system and image coordinate system of the calibration plate at different heights are calibrated using the four-point calibration method. The parameters of the rigid transformation matrix at different heights (H) are calculated. Finally, through experimental analysis, the high linear relationship between the parameters of the rigid transformation matrix from the image coordinate system to the robot coordinate system and the calibration height is fitted. By analyzing the random error of the experiment, the linear relationship between calibration height and pixel density is further established, and the systematic error of the experimental process is deeply analyzed. The experimental results show that the hand–eye calibration system based on this linear relationship is precise and suitable for grabbing products of any height, and the positioning error is less than 0.08%. [ABSTRACT FROM AUTHOR]

When a three-dimensional (3D)-vision sensor is adopted to achieve high-precision measurement-assisted robotic machining, hand–eye calibration is necessary to determine the position and orientation between the 3D-vision sensor and the robot end-effector. However, errors exist in hand–eye calibration data, including robot poses and sensor measurement, which can seriously affect the solution accuracy for hand–eye transformation parameters. The main objective of this paper is to present an accurate method based on 3D visual measurement considering the effects of errors on modeling in nonlinear optimization and initial value calculation to improve the solution accuracy. Compared with a model established in the robot base frame, a nonlinear calibration model with a more accurate reference point in the sensor frame is established, which offers the possibility of a more accurate solution. An iterative weight optimization solution combined with the random errors of sensor measurement and robot position is proposed. To obtain an accurate initial value for iteration, considering the measurement error of reference in the base frame, a calibration model based on surveying adjustment is proposed and is solved using Lagrange multipliers. Simulations and experiments are conducted to test the superiority of the proposed methods. Finally, an experiment on measurement-assisted robotic machining is carried out to further demonstrate the validation. [ABSTRACT FROM AUTHOR]

Due to the increase in the difficulty and diversity of tasks performed by robots, robot "hand-eye" collaborative operation has attracted widespread attention. This technology is widely used in aerospace, medical, automotive, and industrial fields. Recently, hand-eye calibration technology is developing towards high precision and high intelligence. However, it has much work to be done in terms of identifying robot and camera parameters. This article introduces in detail the methods and theories involved in hand-eye calibration. According to the structure of the algorithm and the type of the optimization method, this paper summarizes the hand-eye calibration method into four steps: camera pose, mechanical claw pose, mathematical model, and error metrics. The well-known open problems about hand-eye calibration are finally stated, and some new research interests are also pointed out. The results of this review are useful for robot technicians to choose the correct parameter identification method and for researchers to determine further research areas. [ABSTRACT FROM AUTHOR]

Hand-eye calibration enables proper perception of the environment in which a vision guided robot operates. Additionally, it enables the mapping of the scene in the robots frame. Proper hand-eye calibration is crucial when sub-millimetre perceptual accuracy is needed. For example, in robot assisted surgery, a poorly calibrated robot would cause damage to surrounding vital tissues and organs, endangering the life of a patient. A lot of research has gone into ways of accurately calibrating the hand-eye system of a robot with different levels of success, challenges, resource requirements and complexities. As such, academics and industrial practitioners are faced with the challenge of choosing which algorithm meets the implementation requirements based on the identified constraints. This review aims to give a general overview of the strengths and weaknesses of different hand-eye calibration algorithms available to academics and industrial practitioners to make an informed design decision, as well as incite possible areas of research based on the identified challenges. We also discuss different calibration targets, which is an important part of the calibration process that is often overlooked in the design process.