Search

Unlike vision modalities, body-worn sensors or passive sensing can avoid the failure of action understanding in vision related challenges, e.g. occlusion and appearance variation. However, a standard large-scale dataset does not exist, in which different types of modalities across vision and sensors are integrated. To address the disadvantage of vision-based modalities and push towards multi/cross modal action understanding, this paper introduces a new large-scale dataset recorded from 20 distinct subjects with seven different types of modalities: RGB videos, keypoints, acceleration, gyroscope, orientation, Wi-Fi and pressure signal. The dataset consists of more than 36k video clips for 37 action classes covering a wide range of daily life activities such as desktop-related and check-in-based ones in four different distinct scenarios. On the basis of our dataset, we propose a novel multi modality distillation model with attention mechanism to realize an adaptive knowledge transfer from sensor-based modalities to vision-based modalities. The proposed model significantly improves performance of action recognition compared to models trained with only RGB information. The experimental results confirm the effectiveness of our model on cross-subject, -view, -scene and -session evaluation criteria. We believe that this new large-scale multimodal dataset will contribute the community of multimodal based action understanding.

Recently, instance segmentation models have been developed to a great promising accuracy on public benchmarks. However, these models are too heavy to be applied for real applications due to their low inference speed. In this paper, we propose a faster instance segmentation model utilizing a teacher-student learning framework that transfers the knowledge obtained by a well-trained teacher model to a lightweight student model. In addition to the conventional strategy of knowledge distillation in classification or semantic segmentation networks which are both single-task networks, we investigate a hierarchical distillation (H-Dis) framework for structure information distillation on multi-task learning based instance segmentation. H-Dis consists of two distillation schemes: representation distillation that distills pair-wise quantized feature maps shared by multi-heads, and semantic distillation that makes sure to distill each head information in an instance level. In particular, we present channel-wise distillation for the segmentation head to achieve instance-level mask knowledge transfer. To evaluate our approach, we carry out experiments with different settings of distillation methods on different datasets Pascal VOC and Cityscapes. Our experiments prove that our approach is effective for accelerating instance segmentation models with less accuracy drop under limited computing resources.

3D ball tracking is of great significance to ping-pong game analysis, which can be utilized to applications such as TV content and tactic analysis. To achieve a high success rate in ping-pong ball tracking, the main problems are the lack of unique features and the complexity of background, which make it difficult to distinguish the ball from similar noises. This paper proposes a ball-like observation model and a multi-peak distribution estimation to improve accuracy. For the balllike observation model, we utilize gradient feature from the edge of upper semicircle to construct a histogram, besides, ball-size likelihood is proposed to deal with the situation when noises are different in size with the ball. The multi-peak distribution estimation aims at obtaining a precise ball position in case the partidles' weight distribution has multiple peaks. Experiments are based on ping-pong videos recorded in an official match from 4 perspectives, which in total have 122 hit cases with 2 pairs of players. The tracking success rate finally reaches 99.33%.