Search

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

High-resolution visuotactile sensors are often bulky and have flat contact shapes, making it difficult to integrate into dexterous multifinger fingertips and perform manipulation tasks smoothly. In this article, we present a soft, low-cost, biomimetic fingertip-sized visuotactile sensor, which can perceive high-precision 3-D surface deformation. This novel sensor, named GelStereo BioTip, adopts stereo vision system to robustly acquire 3-D deformation information on multicurvature contact surfaces. In particular, a self-calibrating refractive stereo ray tracing modeling method is proposed to solve the problem of ray refraction on multimedium and multicurvature refractive surfaces in the binocular reconstruction system. GelStereo BioTip has an overall spatial resolution of 1.2 mm, a shape measurement accuracy of less than 0.28 mm over a range of 0–5 mm for numerous distinct contacts with varying contact area. We integrate the bionic fingertips into a dexterous robotic hand for validation, and the proposed bionic fingertip concept can be seamlessly transferred to a variety of robotic manipulators.

Robots often interact with unstructured and cluttered environments, requiring a dexterous robotic hand that can adapt to unpredictability. In this article, we propose a 16-degree-of-freedom (DOF) thumb-inspired tactile four-finger hand (TIT hand) for high-precision dexterous fingertip manipulation, which can realize 6-DOF manipulation of the entire hand and 4-DOF control of a single finger in Cartesian space. Specifically, we design a novel 3-DOF spherical six-bar mechanism to imitate the human thumb’s carpometacarpal (CMC) joint, enabling finger flexion/extension (FE), abduction/adduction (AbAd), and pronation/supination (PS) motions. To realize active compliance interaction with objects, we equip the fingertip with a high-resolution visuotactile sensor and develop a compliance controller based on tactile feedback. Experiments show that the controller can adjust the interaction force within 0.6 s. Finally, the proposed hand is evaluated by extensive experiments, such as strength and stiffness tests, which shows that the hand has a fingertip repeatability of 0.0302 mm and a maximum grasp force of 36 N. The experimental results verify the effectiveness of this novel hand design and the robustness of the tactile-driven compliance controller.

Visuotactile sensors have significant advantages in high-resolution contact shape sensing, which is crucial for robots to locate and manipulate objects. However, high precision and robust 3-D geometry sensing is still a challenging problem. In this article, we propose an improved GelStereo sensor, named GelStereo 2.0, with a more stable and compact structure. A refractive stereo ray tracing modeling method is proposed to eliminate 3-D geometry reconstruction errors caused by multimedium light refraction. In addition, we introduce a novel calibration method named multiple contact depths to obtain refractive parameters, which only needs a few pressings using calibration boards. Extensive experimental results show that the measurement error of contact surface depth is less than 0.15 mm. Furthermore, two well-calibrated sensors mounted on the gripper are successfully applied to a fingertip bolt screwing task, which demonstrates the effectiveness of the GelStereo 2.0 sensor and the proposed tactile 3-D geometry sensing method.

Zero-shot imitation learning has demonstrated its superiority to learn complex robotic tasks with less human participation. Recent studies show convincing performance under the condition that the robot follows the demonstration strictly by the learned inverse model. However, these methods are difficult to achieve satisfactory performance in imitation when the demonstration is suboptimal, and the learning of the learned inverse models is vulnerable to label ambiguity issues. In this article, we propose self-optimal zero-shot imitation learning (SOZIL) to tackle these problems. The contribution of SOZIL is twofold. First, goal consistency loss (GCL) is designed to learn the multistep goal-conditioned policy from exploration data. By directly using the goal state as supervision, GCL solves the label ambiguity problem caused by trajectory and action diversity. Second, estimation-based keyframe extraction (EKE) is developed to optimize demonstrations. We formulate the keyframe extraction process as a path optimization problem under suboptimal control. By predicting the performance of the learned policy in executing transitions of any two states, EKE creates a directed graph containing all candidate paths and extracts keyframes by solving the graph’s shortest path problem. Furthermore, the proposed method is evaluated with various simulated and real-world robotic manipulating experiments, such as cable harness assembly, rope manipulation, and block moving. Experimental results show that SOZIL achieves a superior success rate and manipulation efficiency than baselines.

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this article, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3-D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a refractive stereo ray tracing model for GelStereo Palm is presented. Moreover, a 3-D tactile point cloud sensing pipeline is introduced to reconstruct the 3-D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3-D contact geometry sensing of our GelStereo Palm sensor.