Search

Sign Language Recognition (SLR) is an increasingly popular research topic due to its extensive potential applications, such as education, healthcare, emergency response, and social interaction. Sign language is a complex and dynamic language comprising hand gestures, facial expressions, and body motions. This high level of variability poses a significant obstacle for SLR tasks, which must accurately identify and respond to numerous gestures. To address these challenges, an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network (three multi information sharing network (TMS-Net)) is proposed. Specifically, in order to input more rich information to TMS-Net, we use joint feature pair with global features, bone feature pair with local features, and angle feature pair with scale invariance. In terms of network structure, to efficiently extract multiple features from inputs, we build a multi-stream structure and design a multi-level information sharing mechanism based on this structure to ensure the full utilization of skeleton feature information. From the experiment results of the WLASL-2000 dataset (56.4%), AUTSL dataset (96.62%) and MSASL(65.13%), TMS-Net surpass the state-of-the-art (SOTA) methods with single modality as input. In addition, a SLR-based human–robot interaction (HRI) experiment using our proposed TMS-Net is conducted, which proves the practical performance of the TMS-Net. [Display omitted] • A universal skeleton multi-feature has been proposed to characterize sign language motions from different perspectives. • An end-to-end multi-stream network architecture has been constructed to efficiently extract multiple feature information from the input. • A universal multi-level information sharing mechanism has been designed. [ABSTRACT FROM AUTHOR]

Human part detection has significant research and application in computer vision fields such as human–robot interaction, motion capture, facial recognition, and human key point detection. However, the current human body part detection method encounters challenges when detecting multi-scale objects and capturing the correlation relationship between human instances and human parts. To address these problems, a new anchor-free instance-level human part detection network (AIP-Net) is proposed. AIP-Net is a "two-level" structure that consists of two lightweight anchor-free detectors: a body detector and a parts detector. AIP-Net gradually focuses the human body on the human part from top to down, effectively avoiding the interference of extraneous background and enhancing the correlation relationship between human instances and body parts. Additionally, we design a body-part multidimensional context (BPMC) model in the parts detector branch to enhance the capability of the network. We trained the AIP-Ne end-to-end and achieved a state-of-the-art (SOTA) performance of 36.2 mean average precision (mAP) on COCO Human Parts Dataset. Moreover, we successfully utilized the AIP-Net in the human–robot interaction(HRI) platform and validated its practicality. [Display omitted] • We propose a new anchor-free instance-level human part detection net- work (AIP-Net). • Both BodyDetector and PartDetector adopt an fully convolutional networks (FCN) anchor-free network structure. • We validated the performance of AIP-Net on the COCO Human Parts dataset, achieving state-of-the-art performance. [ABSTRACT FROM AUTHOR]