Search

Unknown surface material classification can inform a robot about material properties, enabling it to interact with environments appropriately. Recent research has leveraged multimodal data using deep learning to improve the performance of surface material classification. In this paper, we present a deep learning model, multimodal temporal convolutional neural network (MTCNN), which integrates energy spectrum, dilated convolutions, and sequence poolings into a unified network architecture. The proposed model can learn material representations from auditory and multi-tactile (i.e., acceleration, normal force, and friction force) data generated by dragging a tool along surfaces, and distinguish unknown object surface materials into categories. For surface material data collection, a tool is also designed to detect different object surfaces. The performance of MTCNN is evaluated on a public dataset and the highest classification accuracy is 87.55%. A robotic curling example is provided to illustrate how the presented model helps the robot in manipulation.

Slip detection plays a vital role in robotic dexterous grasping and manipulation, and it has long been a challenging problem in the robotic community. Different from traditional tactile perception-based methods, we propose a Generalized Visual-Tactile Transformer (GVT-Transformer) network to detect slip based on visual and tactile spatiotemporal sequences. The main novelty of GVT-Transformer is its ability to address unaligned vision and tactile data in various formats captured by various tactile sensors. Furthermore, we train and test our proposed network on a public and our visual-tactile grasping datasets. The experimental results show that our method is more suitable for sliding detection tasks than previous visual-tactile learning methods and more versatile.

Next‐generation robots are being designed to function autonomously in complex and unstructured environments. In particular, based on the real‐time measurement and differentiation of normal pressure and shear force, robots can be equipped with the capabilities of damage‐free grasp within minimum force limits, as well as dexterous operation through surface roughness and slip information. Herein, a flexible tactile sensor with a small cylinder protrusion and four arc‐shaped protrusions is developed. Due to its center symmetry and axisymmetry characteristics, the normal pressure and shear force can be decoupled from the complex stress without any interference from torsion. The flexible tactile sensor exhibits good linearity and superior cycling stability and is capable of determining the magnitude and direction of the applied force accurately. The flexible tactile sensor is comfortable to wear, and it is integrated onto the manipulator to realize various delicate and dexterous tasks, such as pressure detection, interaction with fragile objects, and roughness identification. Moreover, intelligent recognition of the sliding and stationary states can be achieved by decoding signals of sliding friction and static friction from the feedback information, leading to real time and precise adjustment of the grasping state of the manipulator.

Humans can quickly determine the force required to grasp a deformable object to prevent its sliding or excessive deformation through vision and touch, which is still a challenging task for robots. To address this issue, we propose a novel 3D convolution-based visual-tactile fusion deep neural network (C3D-VTFN) to evaluate the grasp state of various deformable objects in this paper. Specifically, we divide the grasp states of deformable objects into three categories of sliding, appropriate and excessive. Also, a dataset for training and testing the proposed network is built by extensive grasping and lifting experiments with different widths and forces on 16 various deformable objects with a robotic arm equipped with a wrist camera and a tactile sensor. As a result, a classification accuracy as high as 99.97% is achieved. Furthermore, some delicate grasp experiments based on the proposed network are implemented in this paper. The experimental results demonstrate that the C3D-VTFN is accurate and efficient enough for grasp state assessment, which can be widely applied to automatic force control, adaptive grasping, and other visual-tactile spatiotemporal sequence learning problems., 7 pages, 9 figures, ICRA 2020 accepted

Zero-shot imitation learning has demonstrated its superiority to learn complex robotic tasks with less human participation. Recent studies show convincing performance under the condition that the robot follows the demonstration strictly by the learned inverse model. However, these methods are difficult to achieve satisfactory performance in imitation when the demonstration is suboptimal, and the learning of the learned inverse models is vulnerable to label ambiguity issues. In this paper, we propose Self-Optimal Zero-shot Imitation Learning (SOZIL) to tackle these problems. The contribution of SOZIL is twofold. First, Goal Consistency Loss (GCL) is designed to learn the multi-step goal-conditioned policy from exploration data. By directly using the goal state as supervision, GCL solves the label ambiguity problem caused by trajectory and action diversity. Second, Estimation-based Keyframe Extraction (EKE) is developed to optimize demonstrations. We formulate the keyframe extraction process as a path optimization problem under suboptimal control. By predicting the performance of the learned policy in executing transitions of any two states, EKE creates a directed graph containing all candidate paths and extracts keyframes by solving the graph’s shortest path problem. Furthermore, the proposed method is evaluated with various simulated and real-world robotic manipulating experiments such as cable harness assembly, rope manipulation, and block moving. Experimental results show that SOZIL achieves a superior success rate and manipulation efficiency than baselines.