Search

Visual navigation involves a movable robotic agent striving to reach a point goal (target location) using vision sensory input. While navigation with ideal visibility has seen plenty of success, it becomes challenging in suboptimal visual conditions like poor illumination, where traditional approaches suffer from severe performance degradation. We propose E3VN (echo-enhanced embodied visual navigation) to effectively perceive the surroundings even under poor visibility to mitigate this problem. This is made possible by adopting an echoer that actively perceives the environment via auditory signals. E3VN models the robot agent as playing a cooperative Markov game with that echoer. The action policies of robot and echoer are jointly optimized to maximize the reward in a two-stream actor-critic architecture. During optimization, the reward is also adaptively decomposed into the robot and echoer parts. Our experiments and ablation studies show that E3VN is consistently effective and robust in point goal navigation tasks, especially under nonideal visibility.

Purpose This study aims to process multi-agent system with kinds of limitations and constraints, and consider the robot in-hand manipulation as a problem of coordination and cooperation of multi-fingered hand. Design/methodology/approach A cooperative distributed model predictive control (MPC) algorithm is proposed to perform robot in-hand manipulation. Findings A cooperative distributed MPC approach is formulated for robot in-hand manipulation problem, which enables address complex limitation and constraint conditions in object motion planning, and realizes tracking trajectory of the object more than tracking position of the object. Originality/value This method to implement the moving object task uses the kinematic parameters without the knowledge of dynamic properties of the object. The cooperative distributed MPC scheme is designed to guarantee the movement of the object to a desired position and trajectory at algorithmic level.

The rich content in various real-world networks such as social networks, biological networks, and communication networks provides unprecedented opportunities for unsupervised machine learning on graphs. This paper investigates the fundamental problem of preserving and extracting abundant information from graph-structured data into embedding space without external supervision. To this end, we generalize conventional mutual information computation from vector space to graph domain and present a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graph and hidden representation. Except for standard GMI which considers graph structures from a local perspective, our further proposed GMI++ additionally captures global topological properties by analyzing the co-occurrence relationship of nodes. GMI and its extension exhibit several benefits: First, they are invariant to the isomorphic transformation of input graphs---an inevitable constraint in many existing methods; Second, they can be efficiently estimated and maximized by current mutual information estimation methods; Lastly, our theoretical analysis confirms their correctness and rationality. With the aid of GMI, we develop an unsupervised embedding model and adapt it to the specific anomaly detection task. Extensive experiments indicate that our GMI methods achieve promising performance in various downstream tasks, such as node classification, link prediction, and anomaly detection.

In this paper, we propose a novel Knowledge-based Embodied Question Answering (K-EQA) task, in which the agent intelligently explores the environment to answer various questions with the knowledge. Different from explicitly specifying the target object in the question as existing EQA work, the agent can resort to external knowledge to understand more complicated question such as "Please tell me what are objects used to cut food in the room?", in which the agent must know the knowledge such as "knife is used for cutting food". To address this K-EQA problem, a novel framework based on neural program synthesis reasoning is proposed, where the joint reasoning of the external knowledge and 3D scene graph is performed to realize navigation and question answering. Especially, the 3D scene graph can provide the memory to store the visual information of visited scenes, which significantly improves the efficiency for the multi-turn question answering. Experimental results have demonstrated that the proposed framework is capable of answering more complicated and realistic questions in the embodied environment. The proposed method is also applicable to multi-agent scenarios.

Because more and more solid waste is generated, in particular, in cities, the management of solid waste disposal has become a global challenge. A solution is to find an effective way to sort solid waste materials and recycle them into reusable products. In this article, we propose to use a material recognition method with vision-guided tactile to form a robotic system for waste sorting. The vision guidance module integrates an object detector and an affordance network together. It allows the robot to not only detect the desired containers and packaging from an assortment of the waste but also obtain a configuration to grasp the target and actively collect its tactile data. By classifying the object with the tactile data, the robot can sort containers and packaging into their respective categories according to the type of material. Our experimental results demonstrate the effectiveness of the proposed robotic waste sorting system in sorting containers and packaging various types of materials.

Tongue diagnosis is a convenient and noninvasive clinical practice of traditional Chinese medicine (TCM), having existed for thousands of years. Prickle, as an essential indicator in TCM, appears as a large number of red thorns protruding from the tongue. The term “prickly tongue” has been used to describe the flow of qi and blood in TCM and assess the conditions of disease as well as the health status of subhealthy people. Different location and density of prickles indicate different symptoms. As proved by modern medical research, the prickles originate in the fungiform papillae, which are enlarged and protrude to form spikes like awn. Prickle recognition, however, is subjective, burdensome, and susceptible to external factors. To solve this issue, an end-to-end prickle detection workflow based on deep learning is proposed. First, raw tongue images are fed into the Swin Transformer to remove interference information. Then, segmented tongues are partitioned into four areas: root, center, tip, and margin. We manually labeled the prickles on 224 tongue images with the assistance of an OpenCV spot detector. After training on the labeled dataset, the super-resolutionfaster-RCNN extracts advanced tongue features and predicts the bounding box of each single prickle. We show the synergy of deep learning and TCM by achieving a 92.42% recall, which is 2.52% higher than the previous work. This work provides a quantitative perspective for symptoms and disease diagnosis according to tongue characteristics. Furthermore, it is convenient to transfer this portable model to detect petechiae or tooth-marks on tongue images.

Autonomous indoor service robots are affected by multiple factors when they are directly involved in manipulation tasks in daily life, such as scenes, objects, and actions. It is of self-evident importance to properly parse these factors and interpret intentions according to human cognition and semantics. In this study, the design of a semantic representation framework based on a knowledge graph is presented, including (1) a multi-layer knowledge-representation model, (2) a multi-module knowledge-representation system, and (3) a method to extract manipulation knowledge from multiple sources of information. Moreover, with the aim of generating semantic representations of entities and relations in the knowledge base, a knowledge-graph-embedding method based on graph convolutional neural networks is proposed in order to provide high-precision predictions of factors in manipulation tasks. Through the prediction of action sequences via this embedding method, robots in real-world environments can be effectively guided by the knowledge framework to complete task planning and object-oriented transfer.

Robotic grasping has become increasingly important in many application areas such as industrial manufacturing and logistics. Because of the diversity and uncertainty of objects and environments, common grippers with one single grasping mode face difficulties to fulfill all the tasks. Hence, we proposed a soft gripper with multiple grasping modes in this study. The gripper consists of four modular soft fingers integrated with layer jamming structure and tendon-driven mechanism. Each finger's rotating shaft of the base uses a torsional spring to decouple the bending deformation and relative rotation. An octopus-mimicking vacuum sucker is installed in the fingertip to generate suction. The effectiveness of the bending deformation and variable stiffness of the design were proved by finite element simulation. Thus, the control model of the finger was built, and the control strategy of multimode grasping of the gripper was proposed. Three control modes were designed to realize the four anthropomorphic grasping modes, including wrap, pinch, hook, and suck. Furthermore, the grasping performance was evaluated to show the abilities. The experiments indicated the superior performance of the proposed gripper and the multimode grasping ability that satisfies various grasping tasks.

Purpose The purpose of this paper is to present a novel tactile sensor and a visual-tactile recognition framework to reduce the uncertainty of the visual recognition of transparent objects. Design/methodology/approach A multitask learning model is used to recognize intuitive appearance attributes except texture in the visual mode. Tactile mode adopts a novel vision-based tactile sensor via the level-regional feature extraction network (LRFE-Net) recognition framework to acquire high-resolution texture information and temperature information. Finally, the attribute results of the two modes are integrated based on integration rules. Findings The recognition accuracy of attributes, such as style, handle, transparency and temperature, is near 100%, and the texture recognition accuracy is 98.75%. The experimental results demonstrate that the proposed framework with a vision-based tactile sensor can improve attribute recognition. Originality/value Transparency and visual differences make the texture of transparent glass hard to recognize. Vision-based tactile sensors can improve the texture recognition effect and acquire additional attributes. Integrating visual and tactile information is beneficial to acquiring complete attribute features.

This article addresses the delay-dependent Takagi-Sugeno (T-S) fuzzy state feedback control and exponential admissibility analysis for a class of T-S fuzzy singular uncertain systems. First, the T-S fuzzy model is employed to approximate the singular uncertain system with time-varying delay, saturation input, and unmatched disturbance. Second, the delay-dependent T-S fuzzy state feedback controller is designed by employing the T-S fuzzy model. Third, the free-weighting matrices and delay-dependent Lyapunov-Krasovskii functional with multiple integral terms are employed to derive the delay-dependent exponential admissibility conditions and prescribed H-infinity performance is guaranteed. Compared with previous works, the delay-dependent T-S fuzzy state feedback controller is designed for the T-S fuzzy singular uncertain system to relax system design conditions. The convex hull lemma is employed to convert the closed-loop system with saturation input into the closed-loop system without saturation input to enhance controller design flexibility. The Schur complement lemma and Gronwall Bellman lemma are employed to derive the less conservative delay-dependent stability conditions for determining controller gain matrices. The exact invariant set with less conservativeness is employed to convert the controller design problem into linear matrix inequalities (LMIs) optimization constraints to reduce computation complexity of solving LMIs. Finally, simulation examples are presented to show the effectiveness of the proposed methods.

The interval type-2 Takagi-Sugeno (T-S) fuzzy dynamic output feedback and H-infinity stability analysis is studied for a class of networked control systems with multiple time-varying additive uncertainties, time-varying signal communication delay and external disturbance . Firstly, the interval type-2T-S fuzzy is employed to denote the system plant. Secondly, the multiple time-varying additive uncertainties are introduced in the controller design and the state variables depending on additive uncertainties. Thirdly, the delay-dependent Lyapunov-Krasovskii functional with double integral terms is designed to derive the less conservative stability conditions in terms of linear matrix inequalities (LMIs). The characteristic of the state variables are reflected effectively by employing the controller with multiple time-varying additive uncertainties. The closed-loop system is asymptotically stable with prescribed H-infinity performance index γ by employing the matrices transformation information. The less conservative stability conditions are derived and extended into the networked control system without additive uncertainties. Finally, simulations are presented to show the effectiveness of the proposed methods.

This article aims to studying how to solve dynamic Sylvester quaternion matrix equation (DSQME) using the neural dynamic method. In order to solve the DSQME, the complex representation method is first adopted to derive the equivalent dynamic Sylvester complex matrix equation (DSCME) from the DSQME. It is proven that the solution to the DSCME is the same as that of the DSQME in essence. Then, a state-of-the-art neural dynamic method is presented to generate a general dynamic-varying parameter zeroing neural network (DVPZNN) model with its global stability being guaranteed by the Lyapunov theory. Specifically, when the linear activation function is utilized in the DVPZNN model, the corresponding model [termed linear DVPZNN (LDVPZNN)] achieves finite-time convergence, and a time range is theoretically calculated. When the nonlinear power-sigmoid activation function is utilized in the DVPZNN model, the corresponding model [termed power-sigmoid DVPZNN (PSDVPZNN)] achieves the better convergence compared with the LDVPZNN model, which is proven in detail. Finally, three examples are presented to compare the solution performance of different neural models for the DSQME and the equivalent DSCME, and the results verify the correctness of the theories and the superiority of the proposed two DVPZNN models.

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN., Accepted by TPAMI 2022. Code is available at https://github.com/yikaiw/CEN. arXiv admin note: text overlap with arXiv:2011.05005

In this paper, a novel robotic grasping system is established to automatically pick up objects in cluttered scenes. A composite robotic hand composed of a suction cup and a gripper is designed for grasping the object stably. The suction cup is used for lifting the object from the clutter first and the gripper for grasping the object accordingly. We utilize the affordance map to provide pixel-wise lifting point candidates for the suction cup. To obtain a good affordance map, the active exploration mechanism is introduced to the system. An effective metric is designed to calculate the reward for the current affordance map, and a deep Q-Network (DQN) is employed to guide the robotic hand to actively explore the environment until the generated affordance map is suitable for grasping. Experimental results have demonstrated that the proposed robotic grasping system is able to greatly increase the success rate of the robotic grasping in cluttered scenes., Comment: has been accepted by IEEE/RSJ International Conference on Intelligent Robots and Systems 2019

Tactile object recognition is crucial for effective grasping and manipulation. Recently, it has started to attract increasing attention in robotic applications. While there are many works on tactile object recognition and they also achieved promising performances in some applications, most of them are usually limited to closed world scenarios, where the object instances to be recognition in deployment are known and the same as that of during training. Since robots usually operate in realistic open-set scenarios, they inevitably encounter unknown objects. If automation systems falsely recognize unknown objects as one of the known classes based on the pre-trained model, it can lead to potentially catastrophic consequences. It motivates us to break the closed world assumption and to study tactile object recognition in realistic open-set conditions. Although several open-set recognition methods have been proposed, they focused on visual tasks and may not be suitable for tactile recognition. It is mainly due to that these methods do not take into account the special characteristic of tactile data in their models. To this end, we develop a novel Gaussian Prototype Learning method for robust tactile object recognition. Particularly, the proposed method converts feature distributions to probabilistic representations, and exploit uncertainty for tactile recognition in open-set scenarios. Experiments on the two tactile recognition benchmarks demonstrate the effectiveness of the proposed method on open-set tasks.

Adversarial attacks can easily fool object recognition systems based on deep neural networks (DNNs). Although many defense methods have been proposed in recent years, most of them can still be adaptively evaded. One reason for the weak adversarial robustness may be that DNNs are only supervised by category labels and do not have part-based inductive bias like the recognition process of humans. Inspired by a well-known theory in cognitive psychology -- recognition-by-components, we propose a novel object recognition model ROCK (Recognizing Object by Components with human prior Knowledge). It first segments parts of objects from images, then scores part segmentation results with predefined human prior knowledge, and finally outputs prediction based on the scores. The first stage of ROCK corresponds to the process of decomposing objects into parts in human vision. The second stage corresponds to the decision process of the human brain. ROCK shows better robustness than classical recognition models across various attack settings. These results encourage researchers to rethink the rationality of currently widely-used DNN-based object recognition models and explore the potential of part-based models, once important but recently ignored, for improving robustness., Under review. Submitted to TPAMI on June 10, 2022. Major revision on September 4, 2022

Embodiment is an important characteristic for all intelligent agents, while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real-world robotic platform for different scene description tasks, demonstrating the effectiveness and scalability of the developed method. Also, a mobile application is developed, which can be used to assist visually-impaired people to better understand their surroundings.

Existing precision design methods cannot directly guide the tolerance design. Therefore, in this study, an optimization design method of machine tool static geometric accuracy based on tolerance modeling is proposed. In this methodology, the mapping relationship between the geometric error of machine tools and tolerance design is established using the small displacement torsor to represent the tolerance information and the Monte Carlo simulation method is used to establish the response model of the torsor parameters and the tolerance variation bandwidths. An assembly accuracy model is then established by combining a machine tool topology analysis and the forming mechanism of the joint surface error. To calculate the tolerances of the component joint surface, a tolerance response model related to the component joint surface tolerance and torsor parameters is developed. Finally, according to the state function of assembly accuracy reliability, a function response model of the assembly accuracy, reliability, and tolerance is developed. Combining the assembly’s processing cost model with the accuracy, reliability, and tolerance principles, a tolerance optimization model of the static geometric accuracy of a CNC machine tool, a linear axis motion guide, is constructed as a case study. Using a simulated annealing genetic algorithm to solve the tolerance optimization model, the tolerance optimization value is obtained, thereby verifying the effectiveness of the proposed method.

Although intelligent technologies has facilitated the development of precise orthopaedic, simple internal fixation, ligament reconstruction or arthroplasty can only relieve pain of patients in short-term. To achieve the best recover of musculoskeletal injuries, three bottlenecks must be broken through, which includes scientific path planning, bioactive implants and personalized surgical channels building. As scientific surgical path can be planned and built by through AI technology, 4D printing technology can make more bioactive implants be manufactured, and variable structures can establish personalized channels precisely, it is possible to achieve satisfied and effective musculoskeletal injury recovery with the progress of multi-layer intelligent technologies (MLIT).