Search

Sign Language Recognition (SLR) is an increasingly popular research topic due to its extensive potential applications, such as education, healthcare, emergency response, and social interaction. Sign language is a complex and dynamic language comprising hand gestures, facial expressions, and body motions. This high level of variability poses a significant obstacle for SLR tasks, which must accurately identify and respond to numerous gestures. To address these challenges, an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network (three multi information sharing network (TMS-Net)) is proposed. Specifically, in order to input more rich information to TMS-Net, we use joint feature pair with global features, bone feature pair with local features, and angle feature pair with scale invariance. In terms of network structure, to efficiently extract multiple features from inputs, we build a multi-stream structure and design a multi-level information sharing mechanism based on this structure to ensure the full utilization of skeleton feature information. From the experiment results of the WLASL-2000 dataset (56.4%), AUTSL dataset (96.62%) and MSASL(65.13%), TMS-Net surpass the state-of-the-art (SOTA) methods with single modality as input. In addition, a SLR-based human–robot interaction (HRI) experiment using our proposed TMS-Net is conducted, which proves the practical performance of the TMS-Net. [Display omitted] • A universal skeleton multi-feature has been proposed to characterize sign language motions from different perspectives. • An end-to-end multi-stream network architecture has been constructed to efficiently extract multiple feature information from the input. • A universal multi-level information sharing mechanism has been designed. [ABSTRACT FROM AUTHOR]

Predicting the next foot placement of humans during walking can help improve compliant interactions between humans and walking aid robots. Previous studies have focused on foot placement estimation with wearable inertial sensors after heel-strike, but few have predicted foot placements in advance during the early swing phase. In this study, a Bayesian inference-based foot placement prediction approach was proposed. Possible foot placements were modeled as a probability distribution grid map. With selected foot motion feature events detected sequentially in the early swing phase, the foot placement probability map could be updated iteratively using the feature models we built. The weighted center of the probability distribution was regarded as the predicted foot placement. Prediction errors were evaluated with collected walking data sets. When testing with the data from inertial measurement units, the prediction errors were (5.46 cm ± 10.89 cm, −0.83 cm ± 10.56 cm) for cross-velocity walking data and (−4.99 cm ± 12.31 cm, −11.27 cm ± 7.74 cm) for cross-subject–cross-velocity walking data. The results were comparable to previous works yet the prediction could be made earlier. For the subject who walked with more stable gaits, the prediction error can be further decreased. The proposed foot placement prediction approach can be utilized to help walking aid robots adjust their pose before each heel-strike event during walking, which will make human–robot interactions more compliant. This study is also expected to inspire additional probabilistic gait analysis works. [ABSTRACT FROM AUTHOR]

Human part detection has significant research and application in computer vision fields such as human–robot interaction, motion capture, facial recognition, and human key point detection. However, the current human body part detection method encounters challenges when detecting multi-scale objects and capturing the correlation relationship between human instances and human parts. To address these problems, a new anchor-free instance-level human part detection network (AIP-Net) is proposed. AIP-Net is a "two-level" structure that consists of two lightweight anchor-free detectors: a body detector and a parts detector. AIP-Net gradually focuses the human body on the human part from top to down, effectively avoiding the interference of extraneous background and enhancing the correlation relationship between human instances and body parts. Additionally, we design a body-part multidimensional context (BPMC) model in the parts detector branch to enhance the capability of the network. We trained the AIP-Ne end-to-end and achieved a state-of-the-art (SOTA) performance of 36.2 mean average precision (mAP) on COCO Human Parts Dataset. Moreover, we successfully utilized the AIP-Net in the human–robot interaction(HRI) platform and validated its practicality. [Display omitted] • We propose a new anchor-free instance-level human part detection net- work (AIP-Net). • Both BodyDetector and PartDetector adopt an fully convolutional networks (FCN) anchor-free network structure. • We validated the performance of AIP-Net on the COCO Human Parts dataset, achieving state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Human intent prediction (HIP) and human activity recognition (HAR) are important for human–robot interactions. However, human–robot interface signals are user-dependent. A classifier trained on labeled source subjects performs poorly on unlabeled target subjects. Besides, previous methods used a single learner, which may only learn a subset of features and degrade their performance on target subjects. Last, HIP and HAR require real-time computing on edge devices whose computational capabilities limit the model size. To address these issues, this paper designs an ensemble diverse hypotheses (EDH) and knowledge distillation (EDHKD) method. EDH mitigates the cross-subject divergence by training feature generators to minimize the upper bound of the classification discrepancy among multiple classifiers. EDH also maximizes the discrepancy among multiple feature generators to learn diverse and complete features. After training EDH, a lightweight student network (EDHKD) distills the knowledge from EDH to a single feature generator and classifier to significantly decrease the model size but remain accurate. The performance of EDHKD is theoretically demonstrated and experimentally validated. Results show that EDH can learn diverse features and adapt well to unknown target subjects. With only soft labels provided by EDH, the student network (EDHKD) can inherit the knowledge learned by EDH and classify unlabeled target data of a 2D moon dataset and two human locomotion datasets with the accuracy at 96.9%, 94.4%, and 97.4%, respectively, in no longer than 1 millisecond. Compared to the benchmark method, EDHKD lifts the target-domain classification accuracy by 1.3% and 7.1% in the two human locomotion datasets. EDHKD also stabilizes learning curves. Therefore, EDHKD significantly increases the generalization ability and efficiency of the HIP and HAR. [Display omitted] • A novel, accurate, and efficient unsupervised cross-subject adaptation method. • Ensemble diverse hypotheses can learn diverse features. • Knowledge distillation can increase both accuracy and efficiency. • Theoretic explanation of the ensemble learners and the knowledge distillation. • Comprehensive cross-subject evaluations on wearable sensor datasets. [ABSTRACT FROM AUTHOR]

Human–robot interaction of human augmentation robots presents a considerable challenge in achieving accurate and robust interaction force control. This paper proposes a novel softening nonlinear elastic mechanism with continuous adjustability (SNEMA) to address this challenge. The SNEMA achieves softening stiffness behavior through a nonlinear mapping relationship between the lengths of the diamond diagonals. This unique stiffness profile, featuring high stiffness for low output and low stiffness for high output, strikes a balance between the output force range and force resolution. Moreover, the continuous and convenient adjustment of the stiffness profile is realized by utilizing two antagonistic linear springs, enabling optimal stiffness matching for different output force ranges. Bench tests were conducted to validate the stiffness modeling and evaluate the force tracking and interaction performance of the developed SNEMA. Experimental results demonstrate the capability of the SNEMA to achieve precise force control and good collision safety in human–robot interaction. The proposed SNEMA is finally deployed on the Centaur robot to demonstrate its advantages in practical application. • A softening nonlinear elastic mechanism with continuous adjustability is proposed. • The analytical stiffness model is derived to guide the mechanical design. • Adjustable profiles facilitate stiffness matching across various output force ranges. • Experiments validate the stiffness model, ensuring accurate and robust force control. [ABSTRACT FROM AUTHOR]

• Human activities recognition (HAR) and human intent recognition (HIR) are important for medical diagnosis and human-robot interaction. HAR and HIR usually rely on the signals of some wearable sensors, such as inertial measurement unit (IMU), but these signals may be user-dependent, which degrades the performance of the recognition algorithm on new subjects. Traditional supervised learning methods require labeling signals and training specific classifiers for each new subject, which is burdensome. To deal with this problem, this paper proposes a novel non-adversarial cross-subject adaptation method called Gaussian-guided feature alignment (GFA). The proposed GFA metric quantifies the discrepancy between the labeled features of source subjects and the unlabeled features of target subjects so that minimizing the GFA metric leads to the alignment of the source and target features. The GFA metric is estimated by calculating the divergence between the feature distribution and Gaussian distribution, as well as the mean squared error of the mean and variance between source and target features. This paper analytically proves the effect of the GFA metric and validates its performance using three public human activity datasets. Experimental results show that the proposed GFA achieves 1% higher target classification accuracy and 0.5% lower variance than state-of-the-art methods in case of cross-subject validation. These results indicate that the proposed GFA is feasible for improving the generalization of the HAR and HIR. • The key contributions of the present paper include: The cross-subject validation problems are resolved by proposing a novel Gaussian-guided Feature Alignment (GFA) metric to extract the features of the two domains in a Gaussian distribution space without projecting the features to another vector space, e.g., Hilbert space. A better alignment is achieved by adjusting the mean and variance of the Gaussian distribution based on the mean and variance of the corresponding feature distribution and sorting the data sampled from the feature and Gaussian distributions. The effect of the GFA metric on aligning the target distribution with the source distribution and decreasing the upper bound of the target error is analytically proved. Moreover, the feature extractor is optimized to decrease the variance of the target classification accuracy on the training set to further improve the performance of the cross-subject adaptation. The experimental results show that the proposed method outperforms state-of-the-art methods in the case of the cross-subject validation on one motion-sensor-based HIR dataset, one motionsensor- based HAR dataset, and one skeleton-based HAR dataset. Human activities recognition (HAR) and human intent recognition (HIR) are important for medical diagnosis and human-robot interaction. HAR and HIR usually rely on the signals of some wearable sensors, such as inertial measurement unit (IMU), but these signals may be user-dependent, which degrades the performance of the recognition algorithm on new subjects. Traditional supervised learning methods require labeling signals and training specific classifiers for each new subject, which is burdensome. To deal with this problem, this paper proposes a novel non-adversarial cross-subject adaptation method called Gaussian-guided feature alignment (GFA). The proposed GFA metric quantifies the discrepancy between the labeled features of source subjects and the unlabeled features of target subjects so that minimizing the GFA metric leads to the alignment of the source and target features. The GFA metric is estimated by calculating the divergence between the feature distribution and Gaussian distribution, as well as the mean squared error of the mean and variance between source and target features. This paper analytically proves the effect of the GFA metric and validates its performance using three public human activity datasets. Experimental results show that the proposed GFA achieves 1% higher target classification accuracy and 0.5% lower variance than state-of-the-art methods in case of cross-subject validation. These results indicate that the proposed GFA is feasible for improving the generalization of the HAR and HIR. [ABSTRACT FROM AUTHOR]