Search

Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual Try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. In this work, we leverage deep generative models to create a multi-sensory experience where users can touch and see the synthesized object when sliding their fingers on a haptic surface. The main challenges lie in the significant scale discrepancy between vision and touch sensing and the lack of explicit mapping from touch sensing data to a haptic rendering device. To bridge this gap, we collect high-resolution tactile data with a GelSight sensor and create a new visuotactile clothing dataset. We then develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We evaluate our method regarding image quality and tactile rendering accuracy. Finally, we introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device for an immersive experience, allowing for challenging materials and editable sketch inputs., Project website: https://visual-tactile-synthesis.github.io/ Code: https://github.com/RuihanGao/visual-tactile-synthesis

Manipulating cables is challenging for robots because of the infinite degrees of freedom of the cables and frequent occlusion by the gripper and the environment. These challenges are further complicated by the dexterous nature of the operations required for cable routing and assembly, such as weaving and inserting, hampering common solutions with vision-only sensing. In this paper, we propose to integrate tactile-guided low-level motion control with high-level vision-based task parsing for a challenging task: cable routing and assembly on a reconfigurable task board. Specifically, we build a library of tactile-guided motion primitives using a fingertip GelSight sensor, where each primitive reliably accomplishes an operation such as cable following and weaving. The overall task is inferred via visual perception given a goal configuration image, and then used to generate the primitive sequence. Experiments demonstrate the effectiveness of individual tactile-guided primitives and the integrated end-to-end solution, significantly outperforming the method without tactile sensing. Our reconfigurable task setup and proposed baselines provide a benchmark for future research in cable manipulation. More details and video are presented in \url{https://helennn.github.io/cable-manip/}

Pneumatic soft robots present many advantages in manipulation tasks. Notably, their inherent compliance makes them safe and reliable in unstructured and fragile environments. However, full-body shape sensing for pneumatic soft robots is challenging because of their high degrees of freedom and complex deformation behaviors. Vision-based proprioception sensing methods relying on embedded cameras and deep learning provide a good solution to proprioception sensing by extracting the full-body shape information from the high-dimensional sensing data. But the current training data collection process makes it difficult for many applications. To address this challenge, we propose and demonstrate a robust sim-to-real pipeline that allows the collection of the soft robot's shape information in high-fidelity point cloud representation. The model trained on simulated data was evaluated with real internal camera images. The results show that the model performed with averaged Chamfer distance of 8.85 mm and tip position error of 10.12 mm even with external perturbation for a pneumatic soft robot with a length of 100.0 mm. We also demonstrated the sim-to-real pipeline's potential for exploring different configurations of visual patterns to improve vision-based reconstruction results. The code and dataset are available at https://github.com/DeepSoRo/DeepSoRoSim2Real., 2023 International Conference on Robotics and Automation (ICRA)

The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs., Comment: Accepted by NeurIPS 2022 Track of Datasets and Benchmarks

Humans perceive the world by interacting with objects, which often happens in a dynamic way. For example, a human would shake a bottle to guess its content. However, it remains a challenge for robots to understand many dynamic signals during contact well. This paper investigates dynamic tactile sensing by tackling the task of estimating liquid properties. We propose a new way of thinking about dynamic tactile sensing: by building a light-weighted data-driven model based on the simplified physical principle. The liquid in a bottle will oscillate after a perturbation. We propose a simple physics-inspired model to explain this oscillation and use a high-resolution tactile sensor GelSight to sense it. Specifically, the viscosity and the height of the liquid determine the decay rate and frequency of the oscillation. We then train a Gaussian Process Regression model on a small amount of the real data to estimate the liquid properties. Experiments show that our model can classify three different liquids with 100% accuracy. The model can estimate volume with high precision and even estimate the concentration of sugar-water solution. It is data-efficient and can easily generalize to other liquids and bottles. Our work posed a physically-inspired understanding of the correlation between dynamic tactile signals and the dynamic performance of the liquid. Our approach creates a good balance between simplicity, accuracy, and generality. It will help robots to better perceive liquids in different environments such as kitchens, food factories, and pharmaceutical factories., Comment: Robotics Science and System 2022

Background and PurposeHyperglycemia (HG) is associated with increased postoperative complications. This study aims to evaluate the effect of HG during supplemental parenteral nutrition (SPN) on short-term prognosis in non-diabetic patients undergoing gastrectomy for cancer and to analyse the risk factors and prevention methods for HG.MethodsA total of 359 patients were divided into three groups according to blood glucose (BG) during SPN: normoglycemic patients ( ≤ 125 mg/dL), mild HG (125~200 mg/dL), and severe HG (>200 mg/dL). The effect of BG on postoperative short-term outcomes was analyzed. Multivariate regression was performed to investigate influencing factors for severe HG. The safety and efficacy of insulin addition to total nutrient admixture (TNA) for the prevention and management of HG were assessed by propensity score matching (PSM). In addition, regression analysis was performed in the noninsulin group to investigate the predictive factors of severe HG, and a nomogram was plotted.ResultsThe postoperative complication rate was 18.9%, but it was significantly higher in patients with severe HG than in mild HG and normoglycemic patients (25.2, 15.0, and 10.0%, respectively, p < 0.05). Multivariate logistic regression analysis showed that anemia, myosteatosis, higher postoperative capillary blood glucose (CBG) before TNA infusion, and insulin in the TNA were independent influencing factors for severe HG. Based on the above factors, 75 pairs of patients (insulin group and non-insulin group) with comparable baseline data were successfully matched by PSM. The HG incidence and the glycemic fluctuation were significantly improved through 1 U insulin/6 g glucose (1/6 scheme) to TNA. A nomogram containing hemoglobin, skeletal muscle radiodensity, pre-SPN CBG, and pTNM stage with good predictive efficacy (C-index: 0.750) was constructed based on the noninsulin group.ConclusionPoor postoperative glycemic control was related to worse outcomes in non-diabetic patients undergoing gastrectomy for cancer. Pre-operative anemia, myosteatosis, and high postoperative CBG before TNA infusion are risk factors for severe HG. Insulin in TNA can improve the blood glucose control of patients. Our proposed nomogram rendered an individualized predictive tool for HG during SPN, which helps screen high-risk patients requiring insulin therapy. Future studies with larger samples are needed to develop a complete insulin application protocol for SPN.

Based on comparative analysis on hydrochemical characteristics of geothermal water in the north part of Liaokao fault, this research focuses on studying the indicative significance of hydrochemical characteristics for the formation of geothermal water. The result shows that: (1) There is no obvious hydraulic connection between the karst geothermal water (occurred in the east part of the Liaokao fault) and the sandstone geothermal water (occurred in the west part of Liaokao fault). (2) In a close hydrological environment, caused by tectonic activities, geothermal water remains longer time in reservoir, hence the water-rock interaction is more complete, with high degree of concentrations, whereas the renewable capacity of the water is weaker. (3) There is no high temperature mantle source fluid mixed in the geothermal water. Karst geothermal water occurred deep circulatory convection along Liaokao fault and its secondary fault, therefore there is deep crust source fluid added into the geothermal water, closer to the Liaokao fault, the greater affected by the deep crust fluid. However, sandstone geothermal water has weak deep circulatory convection.

Geothermal energy has become increasingly important in our energy structure as it is a kind of clean and sustainable resource. Zhangjiakou, along with Beijing, is the host city of Winter Olympic Game 2022 and has the demand of geothermal resources. In the central-south part of Zhangjiakou area, integrated geophysical exploration was carried out in 2019, including regional and local high-precision gravity, magnetotelluric (MT), controlled source audio magnetotelluric (CSAMT) and seismic measurement. Combined with geothermal geological survey, new secondary faults, the contour maps of depth of geothermal reservoir and basement are achieved, which plays a vital role in delineating geothermal anomalous area. Based on the analysis of geophysical interpretation results, geological and hydrological information, the geothermal favorable zone and drilling target areas are determined. The work done in the year of 2019 is the beginning and basis of this geothermal resource survey project, and will provide technical support for the drilling work.

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance., 10 pages, accepted at the 1st Annual Conference on Robot Learning (CoRL)

With the development of industrialization and urbanization, heavy metals contamination has become a major environmental problem. Numerous investigations have revealed an association between heavy metal exposure and the incidence and mortality of gastric cancer. The mechanisms of heavy metals (lead, cadmium, mercury, chromium, and arsenic) contamination leading to gastric cancer are concluded in this review. There are four main potential mechanisms: (1) Heavy metals disrupt the gastric mucosal barrier by decreasing mucosal thickness, mucus content, and basal acid output, thereby affecting the function of E-cadherin and inducing reactive oxygen species (ROS) damage. (2) Heavy metals directly or indirectly induce ROS generation and cause gastric mucosal and DNA lesions, which subsequently alter gene regulation, signal transduction, and cell growth, ultimately leading to carcinogenesis. Exposure to heavy metals also enhances gastric cancer cell invasion and metastasis. (3) Heavy metals inhibit DNA damage repair or cause inefficient lesion repair. (4) Heavy metals may induce other gene abnormalities. In addition, heavy metals can induce the expression of proinflammatory chemokine interleukin-8 (IL-8) and microRNAs, which promotes tumorigenesis. The present review is an effort to underline the human health problem caused by heavy metal with recent development in order to garner a broader perspective.

Background With the wide use of antibiotics, antibiotic-resistant Helicobacter pylori strains are becoming increasingly prevalent. It has been hypothesized that culture-guided therapy might help to increase treatment success. But the effects and the costs still remain controversial. Aims To systematically review the efficacy and the cost of culture-guided triple therapy, compared to standard triple regimen for first-line treatment of Helicobacter pylori infection. Methods A search of the Cochrane Library, PubMed, EMBASE, Science Citation Index Expanded and CBM was performed. Randomized controlled trials comparing culture-guided triple therapy to standard triple therapy in the first-line treatment of Helicobacter pylori infection were selected for meta-analysis. Relative risk was used as a measure of the effect of two regimens mentioned above with a fixed-effects model using the methods of DerSimonian and Laird. Results Five randomized controlled trials totaling 701 patients were included. The meta-analysis showed that culture-guided triple therapy was superior referring to a higher eradication rate from intention-to-treat analyses (RR, 0.84; 95% CI,0.77, 0.90; p

Background Moxifloxacin-based triple therapy has been suggested as an alternative first line therapy to clarithromycin-based triple therapy for Helicobacter pylori infection. Aims To systematically review the efficacy and tolerance of moxifloxacin-based triple therapy, and to conduct a meta-analysis of studies comparing this regimen with clarithromycin-based triple therapy. Methods A search of The Cochrane Library, PUBMED, EMBASE, EBM Review databases, Science Citation Index Expanded, and CMB (Chinese Biomedical Literature Database) was performed. Randomized controlled trials comparing moxifloxacin-based triple therapy to gold standard triple therapy in the first-line treatment of Helicobacter pylori infection were selected for meta-analysis. Relative risk was used as a measure of the effect of the two above-mentioned regimens with a fixed-effects model using the methods of DerSimonian and Laird. Results Four randomized controlled trials totaling 772 patients were included. The meta-analysis showed that the mean eradication rate was 84.1 (318/378) in the moxifloxacin-based triple therapy group and 73.6 (290/394) in the clarithromycin-based triple therapy group; there was statistical significance between the two groups (RR, 1.13; 95% CI, 1.01, 1.27; P=0.04). There were no statistically significant difference in the overall side effects (RR, 0.61; 95% CI, 0.25, 1.48; P

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale.

Hardness sensing is a valuable capability for a robot touch sensor. We describe a novel method of hardness sensing that does not require accurate control of contact conditions. A GelSight sensor is a tactile sensor that provides high resolution tactile images, which enables a robot to infer object properties such as geometry and fine texture, as well as contact force and slip conditions. The sensor is pressed on silicone samples by a human or a robot and we measure the sample hardness only with data from the sensor, without a separate force sensor and without precise knowledge of the contact trajectory. We describe the features that show object hardness. For hemispherical objects, we develop a model to measure the sample hardness, and the estimation error is about 4% in the range of 8 Shore 00 to 45 Shore A. With this technology, a robot is able to more easily infer the hardness of the touched objects, thereby improving its object recognition as well as manipulation strategy.

Robust manipulation and insertion of small parts can be challenging because of the small tolerances typically involved. The key to robust control of these kinds of manipulation interactions is accurate tracking and control of the parts involved. Typically, this is accomplished using visual servoing or force-based control. However, these approaches have drawbacks. Instead, we propose a new approach that uses tactile sensing to accurately localize the pose of a part grasped in the robot hand. Using a feature-based matching technique in conjunction with a newly developed tactile sensing technology known as GelSight that has much higher resolution than competing methods, we synthesize high-resolution height maps of object surfaces. As a result of these high-resolution tactile maps, we are able to localize small parts held in a robot hand very accurately. We quantify localization accuracy in benchtop experiments and experimentally demonstrate the practicality of the approach in the context of a small parts insertion problem., National Science Foundation (U.S.) (NSF Grant No. 1017862), United States. National Aeronautics and Space Administration (NASA under Grant No. NNX13AQ85G), United States. Office of Naval Research (ONR Grant No. N000141410047)