Search

Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods., Comment: In CoRL 2022. Li and Zhang equal contribution; Gao and Wu equal advising. Project page: https://ai.stanford.edu/~rhgao/see_hear_feel/

Cloth in the real world is often crumpled, self-occluded, or folded in on itself such that key regions, such as corners, are not directly graspable, making manipulation difficult. We propose a system that leverages visual and tactile perception to unfold the cloth via grasping and sliding on edges. By doing so, the robot is able to grasp two adjacent corners, enabling subsequent manipulation tasks like folding or hanging. As components of this system, we develop tactile perception networks that classify whether an edge is grasped and estimate the pose of the edge. We use the edge classification network to supervise a visuotactile edge grasp affordance network that can grasp edges with a 90% success rate. Once an edge is grasped, we demonstrate that the robot can slide along the cloth to the adjacent corner using tactile pose estimation/control in real time. See http://nehasunil.com/visuotactile/visuotactile.html for videos., Comment: Accepted at CoRL 2022. Project website: http://nehasunil.com/visuotactile/visuotactile.html

This work details the design of a novel two finger robot gripper with multiple Gelsight based optical-tactile sensors covering the inner surface of the hand. The multiple Gelsight sensors can gather the surface topology of the object from multiple views simultaneously as well as can track the shear and tensile stress. In addition, other sensing modalities enable the hand to gather the thermal, acoustic and vibration information from the object being grasped. The force controlled gripper is fully actuated so that it can be used for various grasp configurations and can also be used for in-hand manipulation tasks. Here we present the design of such a gripper.

Accidental leakages in underwater natural gas pipelines pose significant risks for the environment. Therefore, it is essential to estimate the leakage rate and monitor the methane diffusion range outside the pipeline, which is challengeable due to the presence of water. The main contribution of this work is to establish the leakage rate calculation and gas dispersion models using the experimental and numerical methods. Firstly, the influences of the pressure, leakage orifice diameter and water depth on the leakage rate and gas dispersion process were revealed through experiments. Further, according to the established experimental facilities, the numerical models including the integral mathematical model and computational fluid dynamic model were established. According to the axial velocity, radius and spring height, the two model were compared and validated. In addition, a field simulation using computational fluid dynamic model was conducted to quantify the gas leakage rate and dispersion prediction, predicting the rising time and diffusion range. The results indicate that the quantifying estimation correlation can predict the leakage rate in the sea with errors less than 16.47%, and the computational fluid dynamic model could predict the dispersion process more accurately while the integral mathematical model could predict the dispersion process more quickly.