Search

A novel mechanism for skeleton-based action recognition is proposed in this paper by enhancing and fusing diverse skeleton features from distinct levels. Graph convolutional neural networks (GCNs) have been proven to be efficient in skeleton-based action recognition. However, most graph convolutional networks tend to capture and fuse discriminative information from different forms of data in spatial neighborhoods. In that case, the deeper interactions among different forms of data as well as the extraction of information in the temporal and channel dimensions are limited. To tackle the issue, we propose the ternary adaptive graph convolution (TAGC) module to capture spatiotemporal information by graph convolution. A novel skeleton information, called parallax information, is explored from original joints or bones with little computation to further improve the performance of action recognition. In addition, in order to make better use of multiple streams, multi-stream feature fusion (MSFF) is proposed to mine deeper-level hybrid features supplementing the original streams. And a graph-based ternary enhance (GTE) module is proposed to further refine the extracted discriminative features. Finally, the proposed multi-stream ternary enhanced graph convolutional network (MS-TEGCN) achieves the state-of-the-art results through extensive experiments on three challenging datasets for skeleton-based action recognition, NTU-60, NTU-120 and Kinetics-Skeleton. [ABSTRACT FROM AUTHOR]

The skeleton-based action recognition task has recently drawn much attention in computer vision, and graph convolutional networks (GCNs) have shown great advantages in this task. However, in shallow layers of GCNs, the sharing of convolution kernels in regular graph convolution (RGC) may result in the underutilization of some convolution kernels due to the particularity of the skeleton data. To capture richer locally discriminative action features, we propose the exclusive kernel graph convolution (EKGC) for the shallow layers of GCNs, which assigns the vertexes exclusive convolution kernels for graph convolution. At the same time, following the mixed graph convolution strategy, RGC is retained in the deep layers of the network, and the mixed graph convolutional network (MGCN) is proposed to utilize EKGC and RGC reasonably in all layers of the network. In addition, as a new form of skeleton information in GCN-based methods, angle information is explored in our work. Compared with joint data and bone data, angle data have better invariance against rotation and scaling, and they expand the relationships between skeletons and enrich the data representation of action. Finally, our final model, the multi-stream MGCN, is validated on two challenging datasets, achieving the state-of-the-art performance on the NTU-RGBD dataset and making competitive performance on a kinetics-skeleton dataset. [ABSTRACT FROM AUTHOR]