Search

Visual navigation involves a movable robotic agent striving to reach a point goal (target location) using vision sensory input. While navigation with ideal visibility has seen plenty of success, it becomes challenging in suboptimal visual conditions like poor illumination, where traditional approaches suffer from severe performance degradation. We propose E3VN (echo-enhanced embodied visual navigation) to effectively perceive the surroundings even under poor visibility to mitigate this problem. This is made possible by adopting an echoer that actively perceives the environment via auditory signals. E3VN models the robot agent as playing a cooperative Markov game with that echoer. The action policies of robot and echoer are jointly optimized to maximize the reward in a two-stream actor-critic architecture. During optimization, the reward is also adaptively decomposed into the robot and echoer parts. Our experiments and ablation studies show that E3VN is consistently effective and robust in point goal navigation tasks, especially under nonideal visibility.

Purpose This study aims to process multi-agent system with kinds of limitations and constraints, and consider the robot in-hand manipulation as a problem of coordination and cooperation of multi-fingered hand. Design/methodology/approach A cooperative distributed model predictive control (MPC) algorithm is proposed to perform robot in-hand manipulation. Findings A cooperative distributed MPC approach is formulated for robot in-hand manipulation problem, which enables address complex limitation and constraint conditions in object motion planning, and realizes tracking trajectory of the object more than tracking position of the object. Originality/value This method to implement the moving object task uses the kinematic parameters without the knowledge of dynamic properties of the object. The cooperative distributed MPC scheme is designed to guarantee the movement of the object to a desired position and trajectory at algorithmic level.

The rich content in various real-world networks such as social networks, biological networks, and communication networks provides unprecedented opportunities for unsupervised machine learning on graphs. This paper investigates the fundamental problem of preserving and extracting abundant information from graph-structured data into embedding space without external supervision. To this end, we generalize conventional mutual information computation from vector space to graph domain and present a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graph and hidden representation. Except for standard GMI which considers graph structures from a local perspective, our further proposed GMI++ additionally captures global topological properties by analyzing the co-occurrence relationship of nodes. GMI and its extension exhibit several benefits: First, they are invariant to the isomorphic transformation of input graphs---an inevitable constraint in many existing methods; Second, they can be efficiently estimated and maximized by current mutual information estimation methods; Lastly, our theoretical analysis confirms their correctness and rationality. With the aid of GMI, we develop an unsupervised embedding model and adapt it to the specific anomaly detection task. Extensive experiments indicate that our GMI methods achieve promising performance in various downstream tasks, such as node classification, link prediction, and anomaly detection.

In this paper, we propose a novel Knowledge-based Embodied Question Answering (K-EQA) task, in which the agent intelligently explores the environment to answer various questions with the knowledge. Different from explicitly specifying the target object in the question as existing EQA work, the agent can resort to external knowledge to understand more complicated question such as "Please tell me what are objects used to cut food in the room?", in which the agent must know the knowledge such as "knife is used for cutting food". To address this K-EQA problem, a novel framework based on neural program synthesis reasoning is proposed, where the joint reasoning of the external knowledge and 3D scene graph is performed to realize navigation and question answering. Especially, the 3D scene graph can provide the memory to store the visual information of visited scenes, which significantly improves the efficiency for the multi-turn question answering. Experimental results have demonstrated that the proposed framework is capable of answering more complicated and realistic questions in the embodied environment. The proposed method is also applicable to multi-agent scenarios.

Because more and more solid waste is generated, in particular, in cities, the management of solid waste disposal has become a global challenge. A solution is to find an effective way to sort solid waste materials and recycle them into reusable products. In this article, we propose to use a material recognition method with vision-guided tactile to form a robotic system for waste sorting. The vision guidance module integrates an object detector and an affordance network together. It allows the robot to not only detect the desired containers and packaging from an assortment of the waste but also obtain a configuration to grasp the target and actively collect its tactile data. By classifying the object with the tactile data, the robot can sort containers and packaging into their respective categories according to the type of material. Our experimental results demonstrate the effectiveness of the proposed robotic waste sorting system in sorting containers and packaging various types of materials.

Tongue diagnosis is a convenient and noninvasive clinical practice of traditional Chinese medicine (TCM), having existed for thousands of years. Prickle, as an essential indicator in TCM, appears as a large number of red thorns protruding from the tongue. The term “prickly tongue” has been used to describe the flow of qi and blood in TCM and assess the conditions of disease as well as the health status of subhealthy people. Different location and density of prickles indicate different symptoms. As proved by modern medical research, the prickles originate in the fungiform papillae, which are enlarged and protrude to form spikes like awn. Prickle recognition, however, is subjective, burdensome, and susceptible to external factors. To solve this issue, an end-to-end prickle detection workflow based on deep learning is proposed. First, raw tongue images are fed into the Swin Transformer to remove interference information. Then, segmented tongues are partitioned into four areas: root, center, tip, and margin. We manually labeled the prickles on 224 tongue images with the assistance of an OpenCV spot detector. After training on the labeled dataset, the super-resolutionfaster-RCNN extracts advanced tongue features and predicts the bounding box of each single prickle. We show the synergy of deep learning and TCM by achieving a 92.42% recall, which is 2.52% higher than the previous work. This work provides a quantitative perspective for symptoms and disease diagnosis according to tongue characteristics. Furthermore, it is convenient to transfer this portable model to detect petechiae or tooth-marks on tongue images.

Robotic grasping has become increasingly important in many application areas such as industrial manufacturing and logistics. Because of the diversity and uncertainty of objects and environments, common grippers with one single grasping mode face difficulties to fulfill all the tasks. Hence, we proposed a soft gripper with multiple grasping modes in this study. The gripper consists of four modular soft fingers integrated with layer jamming structure and tendon-driven mechanism. Each finger's rotating shaft of the base uses a torsional spring to decouple the bending deformation and relative rotation. An octopus-mimicking vacuum sucker is installed in the fingertip to generate suction. The effectiveness of the bending deformation and variable stiffness of the design were proved by finite element simulation. Thus, the control model of the finger was built, and the control strategy of multimode grasping of the gripper was proposed. Three control modes were designed to realize the four anthropomorphic grasping modes, including wrap, pinch, hook, and suck. Furthermore, the grasping performance was evaluated to show the abilities. The experiments indicated the superior performance of the proposed gripper and the multimode grasping ability that satisfies various grasping tasks.

Purpose The purpose of this paper is to present a novel tactile sensor and a visual-tactile recognition framework to reduce the uncertainty of the visual recognition of transparent objects. Design/methodology/approach A multitask learning model is used to recognize intuitive appearance attributes except texture in the visual mode. Tactile mode adopts a novel vision-based tactile sensor via the level-regional feature extraction network (LRFE-Net) recognition framework to acquire high-resolution texture information and temperature information. Finally, the attribute results of the two modes are integrated based on integration rules. Findings The recognition accuracy of attributes, such as style, handle, transparency and temperature, is near 100%, and the texture recognition accuracy is 98.75%. The experimental results demonstrate that the proposed framework with a vision-based tactile sensor can improve attribute recognition. Originality/value Transparency and visual differences make the texture of transparent glass hard to recognize. Vision-based tactile sensors can improve the texture recognition effect and acquire additional attributes. Integrating visual and tactile information is beneficial to acquiring complete attribute features.

This article addresses the delay-dependent Takagi-Sugeno (T-S) fuzzy state feedback control and exponential admissibility analysis for a class of T-S fuzzy singular uncertain systems. First, the T-S fuzzy model is employed to approximate the singular uncertain system with time-varying delay, saturation input, and unmatched disturbance. Second, the delay-dependent T-S fuzzy state feedback controller is designed by employing the T-S fuzzy model. Third, the free-weighting matrices and delay-dependent Lyapunov-Krasovskii functional with multiple integral terms are employed to derive the delay-dependent exponential admissibility conditions and prescribed H-infinity performance is guaranteed. Compared with previous works, the delay-dependent T-S fuzzy state feedback controller is designed for the T-S fuzzy singular uncertain system to relax system design conditions. The convex hull lemma is employed to convert the closed-loop system with saturation input into the closed-loop system without saturation input to enhance controller design flexibility. The Schur complement lemma and Gronwall Bellman lemma are employed to derive the less conservative delay-dependent stability conditions for determining controller gain matrices. The exact invariant set with less conservativeness is employed to convert the controller design problem into linear matrix inequalities (LMIs) optimization constraints to reduce computation complexity of solving LMIs. Finally, simulation examples are presented to show the effectiveness of the proposed methods.

The interval type-2 Takagi-Sugeno (T-S) fuzzy dynamic output feedback and H-infinity stability analysis is studied for a class of networked control systems with multiple time-varying additive uncertainties, time-varying signal communication delay and external disturbance . Firstly, the interval type-2T-S fuzzy is employed to denote the system plant. Secondly, the multiple time-varying additive uncertainties are introduced in the controller design and the state variables depending on additive uncertainties. Thirdly, the delay-dependent Lyapunov-Krasovskii functional with double integral terms is designed to derive the less conservative stability conditions in terms of linear matrix inequalities (LMIs). The characteristic of the state variables are reflected effectively by employing the controller with multiple time-varying additive uncertainties. The closed-loop system is asymptotically stable with prescribed H-infinity performance index γ by employing the matrices transformation information. The less conservative stability conditions are derived and extended into the networked control system without additive uncertainties. Finally, simulations are presented to show the effectiveness of the proposed methods.

This article aims to studying how to solve dynamic Sylvester quaternion matrix equation (DSQME) using the neural dynamic method. In order to solve the DSQME, the complex representation method is first adopted to derive the equivalent dynamic Sylvester complex matrix equation (DSCME) from the DSQME. It is proven that the solution to the DSCME is the same as that of the DSQME in essence. Then, a state-of-the-art neural dynamic method is presented to generate a general dynamic-varying parameter zeroing neural network (DVPZNN) model with its global stability being guaranteed by the Lyapunov theory. Specifically, when the linear activation function is utilized in the DVPZNN model, the corresponding model [termed linear DVPZNN (LDVPZNN)] achieves finite-time convergence, and a time range is theoretically calculated. When the nonlinear power-sigmoid activation function is utilized in the DVPZNN model, the corresponding model [termed power-sigmoid DVPZNN (PSDVPZNN)] achieves the better convergence compared with the LDVPZNN model, which is proven in detail. Finally, three examples are presented to compare the solution performance of different neural models for the DSQME and the equivalent DSCME, and the results verify the correctness of the theories and the superiority of the proposed two DVPZNN models.

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN., Accepted by TPAMI 2022. Code is available at https://github.com/yikaiw/CEN. arXiv admin note: text overlap with arXiv:2011.05005

Tactile object recognition is crucial for effective grasping and manipulation. Recently, it has started to attract increasing attention in robotic applications. While there are many works on tactile object recognition and they also achieved promising performances in some applications, most of them are usually limited to closed world scenarios, where the object instances to be recognition in deployment are known and the same as that of during training. Since robots usually operate in realistic open-set scenarios, they inevitably encounter unknown objects. If automation systems falsely recognize unknown objects as one of the known classes based on the pre-trained model, it can lead to potentially catastrophic consequences. It motivates us to break the closed world assumption and to study tactile object recognition in realistic open-set conditions. Although several open-set recognition methods have been proposed, they focused on visual tasks and may not be suitable for tactile recognition. It is mainly due to that these methods do not take into account the special characteristic of tactile data in their models. To this end, we develop a novel Gaussian Prototype Learning method for robust tactile object recognition. Particularly, the proposed method converts feature distributions to probabilistic representations, and exploit uncertainty for tactile recognition in open-set scenarios. Experiments on the two tactile recognition benchmarks demonstrate the effectiveness of the proposed method on open-set tasks.

Embodiment is an important characteristic for all intelligent agents, while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real-world robotic platform for different scene description tasks, demonstrating the effectiveness and scalability of the developed method. Also, a mobile application is developed, which can be used to assist visually-impaired people to better understand their surroundings.

Existing precision design methods cannot directly guide the tolerance design. Therefore, in this study, an optimization design method of machine tool static geometric accuracy based on tolerance modeling is proposed. In this methodology, the mapping relationship between the geometric error of machine tools and tolerance design is established using the small displacement torsor to represent the tolerance information and the Monte Carlo simulation method is used to establish the response model of the torsor parameters and the tolerance variation bandwidths. An assembly accuracy model is then established by combining a machine tool topology analysis and the forming mechanism of the joint surface error. To calculate the tolerances of the component joint surface, a tolerance response model related to the component joint surface tolerance and torsor parameters is developed. Finally, according to the state function of assembly accuracy reliability, a function response model of the assembly accuracy, reliability, and tolerance is developed. Combining the assembly’s processing cost model with the accuracy, reliability, and tolerance principles, a tolerance optimization model of the static geometric accuracy of a CNC machine tool, a linear axis motion guide, is constructed as a case study. Using a simulated annealing genetic algorithm to solve the tolerance optimization model, the tolerance optimization value is obtained, thereby verifying the effectiveness of the proposed method.

Mobile vehicles require accurate localization and dense mapping for motion planning. In this paper, we propose a dense map construction algorithm based on a light-and-fast stereo visual-inertial navigation system (VINS). A tightly coupled nonlinear optimization method is used to calculate the position of adjacent keyframes. An optical flow tracking method fused with IMU information and ring matching constraints is used to improve the matching accuracy and speed of the feature points. In addition, we obtain the pose and depth values using the semi-global block matching (SGBM) method, which are used as the initial values of the depth filter to update the depth image and improve the convergence speed. Then, we further use the Truncated Signed Distance Function (TSDF) method to construct the dense map. We compare our algorithm with state-of-the-art algorithms on the EuRoc dataset and then compare the estimated depth image using the proposed algorithm and the point cloud construction with the probabilistic monocular dense reconstruction (REMODE). The experiments show that the proposed algorithm can obtain more accurate localization than VINS and OKVIS, as well as a faster tracking speed, a better depth map, a lower convergence time for the estimated image and a lower number of updated frames than REMODE.

In several epidemic diseases, one of the main symptoms exhibited by people is abnormal body temperature. Therefore, monitoring body temperature is crucial for preventing the spread of infectious diseases and facilitating timely responses. This study presents a wearable bracelet that can be used as a temperature monitoring and trajectory analysis system. The temperature sensor in the bracelet can effectively monitor the body temperature of the wearer in the target scene, and the data transmission between the bracelet and the data collector can effectively detect the movement range and trajectory of the wearer. Through these, the whole set can also detect the direct and indirect contact of the wearer in any period of time, which is very helpful for the prevention and control of infectious diseases and the isolation of potentially infected persons.

The high positioning accuracy of robotic systems is fundamental and critical for dexterous and precise manipulation. In this paper, a novel accurate positioning method based on vision and tactile sensors for object pose estimation in robotic manipulation is proposed to improve the positioning accuracy of robotic systems. The proposed methodology mainly includes two stages of vision-based rough positioning stage and tactile-based precise positioning stage which makes full use of the multimodal information for positioning and object pose estimation in robotic systems. In the vision-based rough positioning stage, the position and orientation of the target object can be obtained by RGB and depth images, and the robot approaches the target object considering a safe distance. In the tactile-based precise positioning stage, the tactile images are captured by the designed tactile sensor through the touch movement of the robot, and the position and orientation of the target object can be precisely calculated further to complete the positioning process. Finally, the experiments have been conducted to verify the proposed method in a real application platform, and achieved positioning accuracies of 0.138 mm and 0.203° have shown great improvement of 96.1% and 93.9% from 3.552 mm and 3.343° which can meet the demands for robotic precise positioning and manipulation.