Search

Hand-eye calibration aims to relate what the camera sees to where the robot moves, which is crucial for vision-guided robot systems and has received much attention in the robotics community. The classical hand-eye calibration flow estimates camera extrinsic poses first, then uses homogeneous pose alignment to solve the hand-eye calibration problem. Since this two-step procedure is trivial and prone to error propagation, point-alignment-based hand-eye calibration is currently gaining popularity. The point alignment approach is promising but still immature, as evidenced by the lack of a unified formulation for different calibration targets, the requirement of pose for initialization, and the reliance on an optimization solver and its numerical differentiation. These issues are addressed item by item in this article. We first formulate the hand-eye calibrations for multi-point, single-point, and patterned targets via 2D-3D generative point alignment. Then, we propose a generic initialization on a single-point sequence covering all the above target cases. Following that, we infer the analytical Jacobian matrix in detail and develop the sparsity for the pose-perturbation refinement. Finally, both numerical simulations and real-world experiments are provided to verify that our approach is more accurate, efficient, and robust than state-of-the-art methods. The codes and datasets are open-source and can be found at https://github.com/MatthewJin001/GPA-HEC. Note to Practitioners—Vision-guided robots have been widely deployed in flexible automation, and only through hand-eye information can visual perception be used to guide robotic movement. However, the classical hand-eye calibration methods perform the hand-eye parameter estimation with beforehand camera poses, which is time-consuming and prone to error propagation. This work investigates one-step hand-eye calibrations based on direct point alignment. We provide simultaneous hand-eye and target estimation methods for multi-point, single-point, and patterned calibration targets by solving 2D-3D generative point alignment problems. Generally speaking, the single-point method achieves the least time consumption, the patterned method achieves the highest accuracy with a precise pattern, and the multi-point method achieves the strongest robustness and adaptability. With reliable robot information, the proposed methods are more accurate even in pattern deformation cases compared with the classical two-step methods. With the analytical Jacobian refinement, our methods are solver-free, lightweight, and suitable for flexible deployment. Furthermore, without the range requirement of pose acquisition, our methods offer the ability to integrate path planning into autonomous hand-eye calibration.

This article studies robust adaptive Kalman filtering and smoothing problems for the linear state-space model with heavy-tailed multiplicative (measurement) noise and additive (process and measurement) noises. First, to model the heavy-tailed noises, the state transition and measurement likelihood densities are modeled as two generalized t distributions. Then, the unknown covariance matrices of process and measurement additive noises are modeled as inverse Wishart distributions, and the multiplicative noise covariance is modeled as an inverse Gamma distribution. To further improve the estimation performance and robustness to outliers, a one-step smoothing strategy is employed. Finally, robust adaptive Kalman filters with corresponding smoothers are proposed using variational Bayesian inference. A target tracking example is provided to verify the effectiveness and robustness of the proposed filters and smoothers.

The bin picking system integrates manipulators with cameras to automatically pick randomly piled objects. It has the commercial potential to help streamline production and is evaluated by its work efficiency and purchase cost. For universal object picking, the system requires the manipulator to cover six degrees of freedom (DoFs). If the picking objects have a uniform shape, this requirement may be relaxed to achieve higher efficiency with cheaper low-DoF manipulators. On the one hand, the pose of revolution-symmetry (RS) objects is 5-DoF for rotating around their revolution axes without changing their poses. On the other hand, the widely used selective compliance assembly robot arm (SCARA) can achieve 4-DoF kinematics due to four joint actuators. Inspired by the fact that the DoF of the RS pose is exactly the same as adding one DoF to SCARA, we develop a SCARA+ system of a SCARA with an additional revolute joint and explore the possibility of integrating it with a 3-D camera to achieve bin picking of RS objects. Toward this end, we first discuss the SCARA+ kinematics with the modified Denavit–Hartenberg (DH) parameters. Then, to calibrate the additional DH and the hand–eye parameters in the kinematics, we construct an axis-point model and provide an iterative solution without singularity. Finally, comprehensive experiments verify the superiority of the SCARA+ system. When compared with the state-of-the-art systems, our system achieves a significant efficiency improvement with relatively lower costs. It has also been successfully applied in the spinning industry for practical bobbin loading.

In this article, the joint estimation of state and noise covariance for linear systems with unknown covariance of multiplicative noise is considered. The measurement likelihood is modeled as a mixture of two Gaussian distributions and a Student's t distribution, respectively. The unknown covariance of multiplicative noise is modeled as an inverse Gamma/Wishart distribution and the initial condition is formulated as the nominal covariance. By using robust design and choosing hierarchical priors, two variational Bayesian-based robust Kalman filters are proposed. The stability and convergence of the proposed filters and the covariance parameters are analyzed. The lower and upper bounds are also provided to guarantee the performance of the proposed filters. A target tracking simulation is provided to validate the effectiveness of the proposed filters.

Hand–eye calibration (HEC) aims to determine the geometrical transformation between the camera and the robot, which is essential for vision-guided robotic (VGR) systems. On the one hand, the classic pose-based and the recent pixel-based HEC methods mainly focus on multipixel research for initializing or solving. However, in these multimarker cases, additional connector assembly and wide coverage area are not user-friendly in practical data collection operations. Furthermore, both accuracy and efficiency are not ideal due to the transition calculation of multimarker poses. On the other hand, with the development of 3-D sensing technology, 3-D position data can be obtained in increasing scenes. However, HEC research on 3-D observation is relatively insufficient, and previous position-based formulations ignore the relationship among parameters. Motivated by the above shortcomings, this article thoroughly investigates the hand–eye parameter estimation based on 3-D observation of a single marker. First, a uniform single-marker formulation is proposed. This formulation is unique in the optimization sense without factor-extraction variants and covers both the eye-in-hand and eye-to-hand configurations. Then, an analytical solution and an iterative solution are derived through different rotation treatments. It is worth noting that these solutions are in a consistent and compact form due to the precalculated variables and the equivalent transformation. Meanwhile, the solvability, estimation accuracy, and computational efficiency are discussed. Finally, comprehensive simulations and real-world experiments are provided to demonstrate the advantages of the proposed method over previous methods in terms of accuracy, computational efficiency, and operational efficiency. The codes and datasets are open source at: https://github.com/MatthewJin001/Single3D and https://github.com/MatthewJin001/3Ddata, respectively.

In this article, we focus on the scalable distributed data-driven state estimation problem using Gaussian processes (GPs). The framework includes two parts: 1) the data-driven training approach and 2) the state-estimation architecture. First, the objective is to obtain the transition and measurement functions of the considered state-space model by a data-driven training strategy via distributed GPs. In particular, to improve the training efficiency, we employ an online conditioning algorithm, which reduces the computational burden significantly. Then, all nodes exchange their own trained GPs with their respective neighbors. Furthermore, to achieve consensus on local trained GPs, we propose a Wasserstein weighted average consensus algorithm, which differs from the current Kullback–Leibler average consensus on probability densities. Second, based on the training results, we propose a distributed state estimation algorithm to perform fresh state estimation. After obtaining the state estimation results (mean and covariance) and exchanging them with their neighboring nodes, we then execute the Wasserstein weighted average to achieve consensus on state estimations. Also, we analyze the stability and robustness of the proposed distributed state estimation by using GP. Finally, numerical and real-world examples are provided to validate the effectiveness of the proposed training method and data-driven state estimation algorithms.

This article considers the problem of distributed robust state estimation for sensor networks in the presence of model uncertainty and multiplicative noise. More precisely, we assume that the modeling uncertainty, i.e., the actual state space model belongs to an ambiguity set or a set of convex polytopic uncertain parameters. Several robust Kalman filters are proposed based on projection theorem, variance-constrained optimization, and robust mean square error estimation with different types of ambiguity sets. Stability analysis and simulation example verify the presented distributed robust filters.