Search

In recent years, deep learning has already been applied to English lip-reading. However, Chinese lip-reading starts late and lacks relevant dataset, and the recognition accuracy is not ideal. Therefore, this paper proposes a new hybrid neural network model to establish a Chinese lip-reading system. In this paper, we integrate the attention mechanism into both CNN and RNN. Specifically, we add the convolutional block attention module (CBAM) to the ResNet50 neural network, which enhances its ability to capture the small differences among the mouth patterns of similarly pronounced words in Chinese, improving the performance of feature extraction in the convolution process. We also add the time attention mechanism to the GRU neural network, which helps to extract the features among consecutive lip motion images. Considering the effects of the moments before and after on the current moment in the lip-reading process, we assign more weights to the key frames, which makes the features more representative. We further validate our model through experiments on our self-built dataset. Our experiments show that using convolutional block attention module (CBAM) in the Chinese lip-reading model can accurately recognize Chinese numbers 0–9 and some frequently used Chinese words. Compared with other lip-reading systems, our system has better performance and higher recognition accuracy. [ABSTRACT FROM AUTHOR]

Traditional automatic lip-reading systems generally consist of two stages: feature extraction and recognition, while the handcrafted features are empirical and cannot learn the relevance of lip movement sequence sufficiently. Recently, deep learning approaches have attracted increasing attention, especially the significant improvements of convolution neural network (CNN) applied to image classification and long short-term memory (LSTM) used in speech recognition, video processing and text analysis. In this paper, we propose a hybrid neural network architecture, which integrates CNN and bidirectional LSTM (BiLSTM) for lip reading. First, we extract key frames from each isolated video clip and use five key points to locate mouth region. Then, features are extracted from raw mouth images using an eight-layer CNN. The extracted features have the characteristics of stronger robustness and fault-tolerant capability. Finally, we use BiLSTM to capture the correlation of sequential information among frame features in two directions and the softmax function to predict final recognition result. The proposed method is capable of extracting local features through convolution operations and finding hidden correlation in temporal information from lip image sequences. The evaluation results of lip-reading recognition experiments demonstrate that our proposed method outperforms conventional approaches such as active contour model (ACM) and hidden Markov model (HMM). [ABSTRACT FROM AUTHOR]

• A method is proposed to analyze the radio spectral bias from fre- quency perspective, based on a multi-spectral attention mechanism with DCT for learning-based frequency components selection, termed as FLANs. This attention mechanism is the general case of conventional global aver- age pooling (GAP) and leverage identical structures of the popular neural networks. • We propose a simple yet effective learning-based dynamic frequency selection module in FLANs. The module consists of multiple learning-based switches to identify the trivial frequency components based on the DCT coefficients for static removal during inference. • To the best of our knowledge, this is the first work that compromises the merits of hand-crafted feature mechanism and deep learning in frequency domain for AMC. Extensive experiments have been conducted to validate the superiority of FLANs for automatic modulation classification over a wide variety of state-of-the-art methods on RADIOML 2018.01A dataset. Deep neural networks have been recently applied in automatic modulation classification task and achieved remarkable success. However, Existing neural networks mainly focus on the purely data-driven architecture design, and fail to explore the hand-crafted feature mechanisms which are particularly significant for radio signal presentation in wireless communication. Inspired by digital signal processing theories, we propose frequency learning attention networks (FLANs) to analyze the radio spectral bias from frequency perspective, based on a multi-spectral attention mechanism for learning-based frequency components selection. FLANs are the general case of classical global average pooling and leverage identical structures of the popular neural networks. Extensive experiments have been conducted to validate the superiority of FLANs for automatic modulation classification over a wide variety of state-of-the-art methods on RADIOML 2018.01A dataset. [ABSTRACT FROM AUTHOR]