Temporal Relation Guided Knowledge Distillation for Continuous Sign Language Recognition

Previous researches in continuous sign language recognition mainly focus on the RGB modality and achieve remarkable performance on real-world and laboratory datasets,but they usually require high computation intensity.On the other hand,the skeleton is a modality with small input data and fast computation speed,but poor at the real-world datasets.This paper proposes a cross-modal knowledge distillation method named temporally related knowledge distillation(TRKD) to alleviate the contradiction between RGB and skeleton modality in performance and calculation speed.TRKD utilizes the RGB modality network as a teacher to guide the skeleton modality network for fast and accurate implementation.We notice that the teacher’s understanding of sign language context is worth learning by student.It proposes to employ the graph convolutional network(GCN) to learn and align the temporally related features of teacher networks and student networks to achieve this goal.Moreover,since the supervised information from the teacher network is not available for traditional loss functions due to the learnable parameters of GCN in the teacher network,we introduce contrastive learning to provide self-supervised information.Multiple ablation experiments on the Phoenix-2014 dataset demonstrate the effectiveness of the proposed method.