Back to Search Start Over

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition.

Authors :
Tang, Yansong
Wang, Zian
Lu, Jiwen
Feng, Jianjiang
Zhou, Jie
Source :
IEEE Transactions on Circuits & Systems for Video Technology. Oct2019, Vol. 29 Issue 10, p3001-3015. 15p.
Publication Year :
2019

Abstract

In this paper, we investigate the problem of RGB-D egocentric action recognition. Unlike conventional human action videos that are passively recorded by static cameras, egocentric videos are self-generated from wearable sensors that are more flexible and provide the close-ups with the visual attention of the wearers when they act. Moreover, RGB-D videos contain the spatial appearance and temporal information in the RGB modality and reflect the 3D structure of the scenes in the depth modality. To adequately learn the nonlinear structure of heterogeneous representations from different modalities and exploit their complementary characteristics, we develop a multi-stream deep neural networks (MDNN) method, which aims to preserve the distinctive property for each modality and simultaneously explore their sharable information in a unified deep architecture. Specifically, we deploy a Cauchy estimator to maximize the correlations of the sharable components and enforce the orthogonality constraints on the distinctive components to guarantee their high independencies. Since the egocentric action recognition is usually sensitive to hand poses, we extend our MDNN by integrating with the hand cues to enhance the recognition accuracy. Extensive experimental results on a newly collected data set and two additional benchmarks are presented to demonstrate the effectiveness of our proposed method for RGB-D egocentric action recognition. [ABSTRACT FROM AUTHOR]

Details

Language :
English
ISSN :
10518215
Volume :
29
Issue :
10
Database :
Academic Search Index
Journal :
IEEE Transactions on Circuits & Systems for Video Technology
Publication Type :
Academic Journal
Accession number :
138960723
Full Text :
https://doi.org/10.1109/TCSVT.2018.2875441