Search

Sign languages generally consist of a sequence of upper body gestures and are cooperative processes among various parts such as the hands, arms, and face. Therefore, the dynamics of the parts as well as the holistic appearance of the upper body and individual parts are essential for robust recognition. In this paper, a global–local representation (GLR) module is proposed to boost the spatiotemporal feature modeling. The GLR module is composed of global shift and local shift along the height, width, and temporal dimensions. Specifically, the global shift is applied to the entire feature map for holistic representation, while the local shift restricts itself to local patches to capture detailed features. Furthermore, a novel cross-scale aggregation module is designed to combine the global and local information in different dimensions. Extensive experimental results on three large-scale benchmarks, including WLASL, INCLUDE and LSA64, demonstrate that the proposed method achieves state-of-the-art recognition performance. [ABSTRACT FROM AUTHOR]

Universal lesion detection (ULD) in computed tomography (CT) images is an important and challenging prerequisite for computer-aided diagnosis (CAD) to find abnormal tissue, such as tumors of lymph nodes, liver tumors, and lymphadenopathy. The key challenge is that lesions have a tiny size and high similarity with non-lesions, which can easily lead to high false positives. Specifically , non-lesions are nearby normal anatomy that include the bowel, vasculature, and mesentery, which decrease the conspicuity of small lesions since they are often hard to differentiate. In this study, we present a novel scale-attention module that enhances feature discrimination between lesion and non-lesion regions by utilizing the domain knowledge of radiologists to reduce false positives effectively. Inspired by the domain knowledge that radiologists tend to divide each CT image into multiple areas, then detect lesions in these smaller areas separately, a local axial scale-attention (LASA) module is proposed to re-weight each pixel in a feature map by aggregating local features from multiple scales adaptively. In addition, to keep the same weight, a combination of axial pixels in the height- and width-axes is designed, attached with position embedding. The model can be used in CNNs easily and flexibly. We test our method on the DeepLesion dataset. The sensitivities at 0.5, 1, 2, 4, 8, and 16 false positives (FPs) per image and average sensitivity at [0.5, 1, 2, 4] are calculated to evaluate the accuracy. The sensitivities are 78.30%, 84.96%, 89.86%, 93.14%, 95.36%, and 95.54% at 0.5, 1, 2, 4, 8, and 16 FPs per image; the average sensitivity is 86.56%, outperforming the former methods. The proposed method enhances feature discrimination between lesion and non-lesion regions by adding LASA modules. These encouraging results illustrate the potential advantage of exploiting the domain knowledge for lesion detection. [ABSTRACT FROM AUTHOR]

Skeleton data is widely used in human action recognition for easy access, computational efficiency and environmental robustness. Recently, encoding skeleton sequences into color images becomes a popular preprocessing procedure to make use of the spatial modeling ability of convolutional neural network (CNN). Furthermore, inspired by relevant work in other fields, attention mechanism has been introduced to CNN based skeleton action recognition. In this paper, we propose a two-branch hierarchical attention model (HAN) for skeleton based action recognition. The proposed model consists of a base branch for spatial-temporal feature extraction and an attention branch for feature enhancement. In attention branch, we utilize auxiliary feature instead of intermediate feature to generate attention maps. Specifically, variance vectors of skeleton sequences are fused as motion saliency matrices to determine the contributions of each joint. Then the motion saliency matrices are sent into the hierarchical attention branch to obtain multiple attention maps. In order to make better use of the attention maps, two distinct combination schemes are proposed to link the two branches at feature extraction blocks and the fully connected layer. The entire model is integrated into an end-to-end trainable network. The efficacy of the proposed HAN has been verified on three benchmark datasets: NTU RGB+D Dataset, UTD-MHAD Dataset and SYSU-3D Dataset. The comparison results show that our approach outperforms the state-of-the-art methods on all datasets. [ABSTRACT FROM AUTHOR]

Analysis of human interaction is one important research topic of human motion analysis. It has been studied either using first-person vision (FPV) or third-person vision (TPV). However, the joint learning of both types of vision has so far attracted little attention. One of the reasons is the lack of suitable datasets that cover both FPV and TPV. In addition, existing benchmark datasets of either FPV or TPV have several limitations, including the limited number of samples, participant subjects, interaction categories, and modalities. In this work, we contribute a large-scale human interaction dataset, namely FT-HID dataset. FT-HID contains pair-aligned samples of first-person and third-person visions. The dataset was collected from 109 distinct subjects and has more than 90K samples for three modalities. The dataset has been validated by using several existing action recognition methods. In addition, we introduce a novel multi-view interaction mechanism for skeleton sequences, and a joint learning multi-stream framework for first-person and third-person visions. Both methods yield promising results on the FT-HID dataset. It is expected that the introduction of this vision-aligned large-scale dataset will promote the development of both FPV and TPV, and their joint learning techniques for human action analysis. [ABSTRACT FROM AUTHOR]