Search

Sign Language Recognition (SLR) is an increasingly popular research topic due to its extensive potential applications, such as education, healthcare, emergency response, and social interaction. Sign language is a complex and dynamic language comprising hand gestures, facial expressions, and body motions. This high level of variability poses a significant obstacle for SLR tasks, which must accurately identify and respond to numerous gestures. To address these challenges, an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network (three multi information sharing network (TMS-Net)) is proposed. Specifically, in order to input more rich information to TMS-Net, we use joint feature pair with global features, bone feature pair with local features, and angle feature pair with scale invariance. In terms of network structure, to efficiently extract multiple features from inputs, we build a multi-stream structure and design a multi-level information sharing mechanism based on this structure to ensure the full utilization of skeleton feature information. From the experiment results of the WLASL-2000 dataset (56.4%), AUTSL dataset (96.62%) and MSASL(65.13%), TMS-Net surpass the state-of-the-art (SOTA) methods with single modality as input. In addition, a SLR-based human–robot interaction (HRI) experiment using our proposed TMS-Net is conducted, which proves the practical performance of the TMS-Net. [Display omitted] • A universal skeleton multi-feature has been proposed to characterize sign language motions from different perspectives. • An end-to-end multi-stream network architecture has been constructed to efficiently extract multiple feature information from the input. • A universal multi-level information sharing mechanism has been designed. [ABSTRACT FROM AUTHOR]

Background: Leg muscle fatigue is the most important factor that affects walking endurance. Considering the legs act as actuators in alternate contact with the ground during walking, the ground reaction force (GRF) of each leg can indirectly reflect the strength of leg muscles. However, it is not clear how the elastically-suspended backpack (ESB) affects GRF of each leg during human level walking.Research Question: How is ESB related in GRF of each leg during walking, and how do multiple variables (stiffness and damping of ESB, load mass, walking speed) affect GRF?Methods: An extended bipedal walking model (EBW) with a spring-mass-damping system was proposed to predict the GRF of each leg. In order to evaluate the prediction effect of the model, seven healthy subjects were recruited to attend the experiments using our backpack prototype and the GRFs data was compared. Each subject walked under 12 conditions (load states: locked or unlocked, walking speed: 3.6 km/h, 4.0 km/h, 4.5 km/h, 5.0 km/h, 5.5 km/h, 6.0 km/h).Results: Results showed that the model could quantitatively predict experimental GRFs over the whole gait cycle (R2≥0.9628) and the characteristic forces (two peak forces and one trough force) were close to the experimental data (average predicted accuracy 93.7 %). The model can reflect relationships between variables and GRF. The relationships showed that an apparent tradeoff exists among the three characteristic forces, and the ESB can produce positive or negative effect under different variables.Significance: This work could help us understand the experimental GRF phenomena, especially the contradictory experimental phenomenon caused by the different parameters. It could also help designers optimize structural parameters of ESB for excellent effects on human. The ESBs with excellent performance can be wildly used in military and tourism. [ABSTRACT FROM AUTHOR]

An increasing number of robotic systems involving lots of robotic individuals are used to serve human, such as intelligent terminal, intelligent storage, intelligence factories, etc. It is a trend of robotics technology that robotics system will become huger with more individuals. In these systems, they form the robotic societies and need establish some computing rules and mechanisms to ensure the operation like all biological social systems. In this paper, a novel system architecture for swarm robotic system, including three layers: human-computer interaction layer, planning layer and execution layer, is put forward, which is effective for task-oriented swarm robotic system. Then, a hierarchical organizational model for the system is presented, which is used to establish management relationship between different layers and individuals. Because task-oriented characteristic is required, this paper elaborates task description knowledge to explain the relationship between tasks for task decomposition and task logic. In addition, a method of behavior generation based on proposition/transition Petri networks is designed, which would effectively assist the system to construct combined behavior using simple individual behavior to solve a variety of tasks. At last, Illustration is shown to prove effectiveness and an implementation of the method based on SociBuilder system is introduced. [ABSTRACT FROM AUTHOR]

Human part detection has significant research and application in computer vision fields such as human–robot interaction, motion capture, facial recognition, and human key point detection. However, the current human body part detection method encounters challenges when detecting multi-scale objects and capturing the correlation relationship between human instances and human parts. To address these problems, a new anchor-free instance-level human part detection network (AIP-Net) is proposed. AIP-Net is a "two-level" structure that consists of two lightweight anchor-free detectors: a body detector and a parts detector. AIP-Net gradually focuses the human body on the human part from top to down, effectively avoiding the interference of extraneous background and enhancing the correlation relationship between human instances and body parts. Additionally, we design a body-part multidimensional context (BPMC) model in the parts detector branch to enhance the capability of the network. We trained the AIP-Ne end-to-end and achieved a state-of-the-art (SOTA) performance of 36.2 mean average precision (mAP) on COCO Human Parts Dataset. Moreover, we successfully utilized the AIP-Net in the human–robot interaction(HRI) platform and validated its practicality. [Display omitted] • We propose a new anchor-free instance-level human part detection net- work (AIP-Net). • Both BodyDetector and PartDetector adopt an fully convolutional networks (FCN) anchor-free network structure. • We validated the performance of AIP-Net on the COCO Human Parts dataset, achieving state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Human intent prediction (HIP) and human activity recognition (HAR) are important for human–robot interactions. However, human–robot interface signals are user-dependent. A classifier trained on labeled source subjects performs poorly on unlabeled target subjects. Besides, previous methods used a single learner, which may only learn a subset of features and degrade their performance on target subjects. Last, HIP and HAR require real-time computing on edge devices whose computational capabilities limit the model size. To address these issues, this paper designs an ensemble diverse hypotheses (EDH) and knowledge distillation (EDHKD) method. EDH mitigates the cross-subject divergence by training feature generators to minimize the upper bound of the classification discrepancy among multiple classifiers. EDH also maximizes the discrepancy among multiple feature generators to learn diverse and complete features. After training EDH, a lightweight student network (EDHKD) distills the knowledge from EDH to a single feature generator and classifier to significantly decrease the model size but remain accurate. The performance of EDHKD is theoretically demonstrated and experimentally validated. Results show that EDH can learn diverse features and adapt well to unknown target subjects. With only soft labels provided by EDH, the student network (EDHKD) can inherit the knowledge learned by EDH and classify unlabeled target data of a 2D moon dataset and two human locomotion datasets with the accuracy at 96.9%, 94.4%, and 97.4%, respectively, in no longer than 1 millisecond. Compared to the benchmark method, EDHKD lifts the target-domain classification accuracy by 1.3% and 7.1% in the two human locomotion datasets. EDHKD also stabilizes learning curves. Therefore, EDHKD significantly increases the generalization ability and efficiency of the HIP and HAR. [Display omitted] • A novel, accurate, and efficient unsupervised cross-subject adaptation method. • Ensemble diverse hypotheses can learn diverse features. • Knowledge distillation can increase both accuracy and efficiency. • Theoretic explanation of the ensemble learners and the knowledge distillation. • Comprehensive cross-subject evaluations on wearable sensor datasets. [ABSTRACT FROM AUTHOR]

In dynamic outdoor environments characterized by turbulent airflow and intermittent odor plumes, robotic odor plume tracking remains challenging, because existing algorithms heavily rely on manually tuning or learning from expert experience, which are hard to implement in an unknown environment. In this paper, a multi-continuous-output Takagi–Sugeno–Kang fuzzy system was designed and tuned with reinforcement learning to solve the robotic odor source localization problem in dynamic odor plumes. Based on the Lévy Taxis plume tracking controller, the proposed fuzzy system determined the parameters of the controller based on the robot's observation and guided the robot to turn and move towards the odor source at each searching step. The trained fuzzy system was tested in simulated filament-based odor plumes dispersed by a changing wind field. The results showed that the performance of the proposed fuzzy system-based controller trained with reinforcement learning can achieve a similar success rate and higher efficiency compared with a manually tuned and well-designed fuzzy system-based controller. The fuzzy system-based plume tracking controller was also validated through real robotic experiments. [ABSTRACT FROM AUTHOR]

Model-free reinforcement learning methods have successfully been applied to practical applications such as decision-making problems in Atari games. However, these methods have inherent shortcomings, such as a high variance and low sample efficiency. To improve the policy performance and sample efficiency of model-free reinforcement learning, we propose proximal policy optimization with model-based methods (PPOMM), a fusion method of both model-based and model-free reinforcement learning. PPOMM not only considers the information of past experience but also the prediction information of the future state. PPOMM adds the information of the next state to the objective function of the proximal policy optimization (PPO) algorithm through a model-based method. This method uses two components to optimize the policy: the error of PPO and the error of model-based reinforcement learning. We use the latter to optimize a latent transition model and predict the information of the next state. For most games, this method outperforms the state-of-the-art PPO algorithm when we evaluate across 49 Atari games in the Arcade Learning Environment (ALE). The experimental results show that PPOMM performs better or the same as the original algorithm in 33 games. [ABSTRACT FROM AUTHOR]

Planar surfaces are prevalent components of man-made indoor scenes, and plane extraction plays a vital role in practical applications of computer vision and robotics, such as scene understanding, and mobile manipulation. Nowadays, most plane extraction methods are based on reconstruction of the scene. In this paper, plane representation is formulated in inverse-depth images. Based on this representation, we explored the potential to extract planes in images directly. A fast plane extraction approach, which employs the region growing algorithm in inverse-depth images, is presented. This approach consists of two main components: seeding, and region growing. In the seeding component, seeds are carefully selected locally in grid cells to improve exploration efficiency. After seeding, each seed begins to grow into a continuous plane in succession. Both greedy policy and a normal coherence check are employed to find boundaries accurately. During growth, neighbor coplanar planes are checked and merged to overcome the over-segmentation problem. Through experiments on public datasets and generated saw-tooth images, the proposed approach achieves 80.2% CDR (Correct Detection Rate) on the ABW SegComp Dataset, which has proven that it has comparable performance with the state-of-the-art. The proposed approach runs at 5 Hz on typical 680 × 480 images, which has shown its potential in real-time practical applications in computer vision and robotics with further improvement. [ABSTRACT FROM AUTHOR]

• Human activities recognition (HAR) and human intent recognition (HIR) are important for medical diagnosis and human-robot interaction. HAR and HIR usually rely on the signals of some wearable sensors, such as inertial measurement unit (IMU), but these signals may be user-dependent, which degrades the performance of the recognition algorithm on new subjects. Traditional supervised learning methods require labeling signals and training specific classifiers for each new subject, which is burdensome. To deal with this problem, this paper proposes a novel non-adversarial cross-subject adaptation method called Gaussian-guided feature alignment (GFA). The proposed GFA metric quantifies the discrepancy between the labeled features of source subjects and the unlabeled features of target subjects so that minimizing the GFA metric leads to the alignment of the source and target features. The GFA metric is estimated by calculating the divergence between the feature distribution and Gaussian distribution, as well as the mean squared error of the mean and variance between source and target features. This paper analytically proves the effect of the GFA metric and validates its performance using three public human activity datasets. Experimental results show that the proposed GFA achieves 1% higher target classification accuracy and 0.5% lower variance than state-of-the-art methods in case of cross-subject validation. These results indicate that the proposed GFA is feasible for improving the generalization of the HAR and HIR. • The key contributions of the present paper include: The cross-subject validation problems are resolved by proposing a novel Gaussian-guided Feature Alignment (GFA) metric to extract the features of the two domains in a Gaussian distribution space without projecting the features to another vector space, e.g., Hilbert space. A better alignment is achieved by adjusting the mean and variance of the Gaussian distribution based on the mean and variance of the corresponding feature distribution and sorting the data sampled from the feature and Gaussian distributions. The effect of the GFA metric on aligning the target distribution with the source distribution and decreasing the upper bound of the target error is analytically proved. Moreover, the feature extractor is optimized to decrease the variance of the target classification accuracy on the training set to further improve the performance of the cross-subject adaptation. The experimental results show that the proposed method outperforms state-of-the-art methods in the case of the cross-subject validation on one motion-sensor-based HIR dataset, one motionsensor- based HAR dataset, and one skeleton-based HAR dataset. Human activities recognition (HAR) and human intent recognition (HIR) are important for medical diagnosis and human-robot interaction. HAR and HIR usually rely on the signals of some wearable sensors, such as inertial measurement unit (IMU), but these signals may be user-dependent, which degrades the performance of the recognition algorithm on new subjects. Traditional supervised learning methods require labeling signals and training specific classifiers for each new subject, which is burdensome. To deal with this problem, this paper proposes a novel non-adversarial cross-subject adaptation method called Gaussian-guided feature alignment (GFA). The proposed GFA metric quantifies the discrepancy between the labeled features of source subjects and the unlabeled features of target subjects so that minimizing the GFA metric leads to the alignment of the source and target features. The GFA metric is estimated by calculating the divergence between the feature distribution and Gaussian distribution, as well as the mean squared error of the mean and variance between source and target features. This paper analytically proves the effect of the GFA metric and validates its performance using three public human activity datasets. Experimental results show that the proposed GFA achieves 1% higher target classification accuracy and 0.5% lower variance than state-of-the-art methods in case of cross-subject validation. These results indicate that the proposed GFA is feasible for improving the generalization of the HAR and HIR. [ABSTRACT FROM AUTHOR]