Search

As the monitor probes are used more and more widely these days, the task of detecting abnormal behaviors in surveillance videos has gained widespread attention. The generalization ability and parameter overhead of the model affect how accurate the detection result is. To deal with the poor generalization ability and high parameter overhead of the model in existing anomaly detection methods, we propose a three-dimensional multi-branch convolutional fusion network, named "Branch-Fusion Net". The network is designed with a multi-branch structure not only to significantly reduce parameter overhead but also to improve the generalization ability by understanding the input feature map from different perspectives. To ignore useless features during the model training, we propose a simple yet effective Channel Spatial Attention Module (CSAM), which sequentially focuses attention on key channels and spatial feature regions to suppress useless features and enhance important features. We combine the Branch-Fusion Net and the CSAM as a local feature extraction network and use the Bi-Directional Gated Recurrent Unit (Bi-GRU) to extract global feature information. The experiments are validated on a self-built Crimes-mini dataset, and the accuracy of anomaly detection in surveillance videos reaches 93.55% on the test set. The result shows that the model proposed in the paper significantly improves the accuracy of anomaly detection in surveillance videos with low parameter overhead. [ABSTRACT FROM AUTHOR]

Convolutional neural network (CNN)-based autonomous driving object detection algorithms have excellent detection results on conventional datasets, but the detector performance can be severely degraded in low-light foggy weather environments. Existing methods have difficulty in achieving a balance between low-light image enhancement and object detection. To alleviate this problem, this paper proposes a foggy traffic environment object detection framework, IDOD-YOLOV7. This network is based on joint optimal learning of image defogging module IDOD (AOD + SAIP) and YOLOV7 detection modules. Specifically, for low-light foggy images, we propose to improve the image quality by joint optimization of image defogging (AOD) and image enhancement (SAIP), where the parameters of the SAIP module are predicted by a miniature CNN network and the AOD module performs image defogging by optimizing the atmospheric scattering model. The experimental results show that the IDOD module not only improves the image defogging quality for low-light fog images but also achieves better results in objective evaluation indexes such as PSNR and SSIM. The IDOD and YOLOV7 learn jointly in an end-to-end manner so that object detection can be performed while image enhancement is executed in a weakly supervised manner. Finally, a low-light fogged traffic image dataset (FTOD) was built by physical fogging in order to solve the domain transfer problem. The training of IDOD-YOLOV7 network by a real dataset (FTOD) improves the robustness of the model. We performed various experiments to visually and quantitatively compare our method with several state-of-the-art methods to demonstrate its superiority over the others. The IDOD-YOLOV7 algorithm not only suppresses the artifacts of low-light fog images and improves the visual effect of images but also improves the perception of autonomous driving in low-light foggy environments. [ABSTRACT FROM AUTHOR]

Automatic visual lipreading, an efficient and convenient way of human–machine interaction, recognizes the content of the conversation from dynamic visual features of the speakers. Automatic lipreading based on acoustic speech alone can effectively prevent interference in complex environment, particularly under noisy conditions. In this paper, we propose novel visual extraction and HMM classification methods for automatic lipreading system, which reduce the dimension by using locally sensitive discriminant analysis algorithm and quantitative cluster by K-means algorithm. A model-based hybrid feature extraction method is proposed by optimizing the weight matrix of the LSDA algorithm. The effectiveness of the suggested approach is demonstrated by preliminary experiments on the English video database. Experimental results demonstrate that the proposed optimized algorithm can increase recognition rate up to 97%, which is 18% higher than the original algorithm. [ABSTRACT FROM AUTHOR]

In autonomous driving, lidar has become the main vehicle sensor due to its advantages such as long-range measurement and high accuracy. However, the collected point cloud data is sparse and unevenly distributed, and it lacks characterization capabilities when facing objects with missing or similar shapes, so that the detection accuracy is low while detecting long-distance small targets with similar shapes and a small number of point clouds. In order to improve the detection accuracy of small targets represented by point clouds, this paper adopts a method that fuses point cloud and RGB image to construct a 3D object detection network architecture based on two-stage complementary fusion. In the first stage of the fusion, we use the FPS method to select some points from the raw point cloud data as key points. Then, we voxelize the raw point cloud and use the 3D sparse convolutional neural network to extract multi-scale points cloud features, which would fuse features of different scales with key points. In the second stage of fusion, a 2D object detector is used to obtain the 2D bounding box and category information of the target in the image and take the camera as the origin to extend along the direction of the 2D bounding box to form a frustum; then the point cloud and target category information within the frustum are fused into the key points. This paper uses key points as a bridge to effectively combine the semantic information of the image such as texture and color with the point cloud features. The experimental results show that the network proposed in this paper has excellent performance on small objects. [ABSTRACT FROM AUTHOR]

In recent years, deep learning has already been applied to English lip-reading. However, Chinese lip-reading starts late and lacks relevant dataset, and the recognition accuracy is not ideal. Therefore, this paper proposes a new hybrid neural network model to establish a Chinese lip-reading system. In this paper, we integrate the attention mechanism into both CNN and RNN. Specifically, we add the convolutional block attention module (CBAM) to the ResNet50 neural network, which enhances its ability to capture the small differences among the mouth patterns of similarly pronounced words in Chinese, improving the performance of feature extraction in the convolution process. We also add the time attention mechanism to the GRU neural network, which helps to extract the features among consecutive lip motion images. Considering the effects of the moments before and after on the current moment in the lip-reading process, we assign more weights to the key frames, which makes the features more representative. We further validate our model through experiments on our self-built dataset. Our experiments show that using convolutional block attention module (CBAM) in the Chinese lip-reading model can accurately recognize Chinese numbers 0–9 and some frequently used Chinese words. Compared with other lip-reading systems, our system has better performance and higher recognition accuracy. [ABSTRACT FROM AUTHOR]

The current field of autonomous driving has achieved superior object detection performance in good weather conditions. However, the environment sensing capability of autonomous vehicles is severely affected in rainfall traffic environments. Although deep-learning-based image derain algorithms have made significant progress, integrating them with high-level vision tasks, such as object detection, remains challenging due to the significant differences between the derain and object detection algorithms. Additionally, the accuracy of object detection in real rain traffic environments is significantly reduced due to the domain transfer problem between the training dataset and the actual rain environment. To address this domain-shifting problem, we propose an adaptive rain image enhancement object detection network for autonomous driving in adverse weather conditions (ARODNet). This network architecture consists of an image adaptive enhancement module, an image derain module, and an object detection module. The baseline detection module (CBAM-YOLOv7) is built by incorporating the YOLOv7 object detection network into a feed-forward convolutional neural network, and it includes an attention module (CBAM). We propose a domain adaptive rain image enhancement module, DRIP, for low-quality images acquired under heavy rainfall conditions. DRIP enhances low-quality images on rainy days by adaptively learning multiple preprocessing weights. To remove the effects of rain patterns and fog clouds on image detection, we introduce DRIP-enhanced images into the depth estimation derain module (DeRain) to prevent rain and fog from obscuring the objects to be detected. Finally, the multistage joint training strategy is adopted to improve the training efficiency, and the object detection is performed while the image is derained. The efficacy of the ARODNet network for object detection in rainy weather traffic environments has been demonstrated through numerous quantitative and qualitative studies. [ABSTRACT FROM AUTHOR]

Traditional automatic lip-reading systems generally consist of two stages: feature extraction and recognition, while the handcrafted features are empirical and cannot learn the relevance of lip movement sequence sufficiently. Recently, deep learning approaches have attracted increasing attention, especially the significant improvements of convolution neural network (CNN) applied to image classification and long short-term memory (LSTM) used in speech recognition, video processing and text analysis. In this paper, we propose a hybrid neural network architecture, which integrates CNN and bidirectional LSTM (BiLSTM) for lip reading. First, we extract key frames from each isolated video clip and use five key points to locate mouth region. Then, features are extracted from raw mouth images using an eight-layer CNN. The extracted features have the characteristics of stronger robustness and fault-tolerant capability. Finally, we use BiLSTM to capture the correlation of sequential information among frame features in two directions and the softmax function to predict final recognition result. The proposed method is capable of extracting local features through convolution operations and finding hidden correlation in temporal information from lip image sequences. The evaluation results of lip-reading recognition experiments demonstrate that our proposed method outperforms conventional approaches such as active contour model (ACM) and hidden Markov model (HMM). [ABSTRACT FROM AUTHOR]

• Lip reading is a novel and intelligent way of human-computer interaction. • Quad-tree Markov random field based unsupervised lip segmentation is proposed. • Multi-scale characteristics of wavelet transform is applied to MRF segmentation. • Better lip segmentation accuracy is achieved compared to other MRF-based methods. In this paper, we propose a new extended method to implement lip segmentation. Based on a quad-tree structure (QTS) Markov random field (MRF) model under unsupervised situation, the proposed method achieves the lip segmentation in wavelet domain. Firstly, we set up a multi-layer hierarchical model, in which each pixel of every layer corresponds to the four nodes in quad-tree structure. Then the probability of a branch node can be calculated by using the probability of the previous branch node through the tree structure easily. Subsequently, a Markov random field derived from the model is obtained so that the segmentation problem is formulated as a labeling optimization problem in the framework of the maximum a posteriori Markov random field (MAP-MRF). Assuming that the pre-assigned cluster of data segments may overestimate the underlying fact, and leads to over-segmentation, we propose a variable-weight segmentation approach to improve the robustness of the segmentation. The experimental results show that this method has better segmentation accuracy than traditional methods. [ABSTRACT FROM AUTHOR]

Lip segmentation is one of critical steps in a lip-reading system, because it closely relates to the accuracy of system recognition. In this paper, we aim to improve the accuracy of lip segmentation. A novel color space is proposed which consists of the component in the CIE-LUV space and the sum of 2 and 3 components of the image after discrete Hartley transform (DHT). We select a rhombus as the initial contour as its shape is approximate to a closed lip shape relatively. These notions are achieved based on the method of the Active contour model. The active contour model (ACM) is performed by the Chan-Vese model, and the result of each component is gained separately. Finally, the ultimate results are obtained by merging the result of each component together. Through experiments we can get a conclusion that this method can get more accurate and smoother lip contour. Meanwhile, the proposed method is more efficient compared with the classic ACM because it avoids some problems in the classic active contour model, like the radius of the initial contour needs to be set manually according to the size of images. [ABSTRACT FROM AUTHOR]

• A method is proposed to analyze the radio spectral bias from fre- quency perspective, based on a multi-spectral attention mechanism with DCT for learning-based frequency components selection, termed as FLANs. This attention mechanism is the general case of conventional global aver- age pooling (GAP) and leverage identical structures of the popular neural networks. • We propose a simple yet effective learning-based dynamic frequency selection module in FLANs. The module consists of multiple learning-based switches to identify the trivial frequency components based on the DCT coefficients for static removal during inference. • To the best of our knowledge, this is the first work that compromises the merits of hand-crafted feature mechanism and deep learning in frequency domain for AMC. Extensive experiments have been conducted to validate the superiority of FLANs for automatic modulation classification over a wide variety of state-of-the-art methods on RADIOML 2018.01A dataset. Deep neural networks have been recently applied in automatic modulation classification task and achieved remarkable success. However, Existing neural networks mainly focus on the purely data-driven architecture design, and fail to explore the hand-crafted feature mechanisms which are particularly significant for radio signal presentation in wireless communication. Inspired by digital signal processing theories, we propose frequency learning attention networks (FLANs) to analyze the radio spectral bias from frequency perspective, based on a multi-spectral attention mechanism for learning-based frequency components selection. FLANs are the general case of classical global average pooling and leverage identical structures of the popular neural networks. Extensive experiments have been conducted to validate the superiority of FLANs for automatic modulation classification over a wide variety of state-of-the-art methods on RADIOML 2018.01A dataset. [ABSTRACT FROM AUTHOR]

As a significant component of the Human Computer Interface (HCI), automatic lip reading is designed for the purpose of understanding the content of speech by interpreting the movements of the lips. Although performance of automatic lip reading system is easily affected by challenging conditions such as noise, illumination and low resolution, enormous advancements in the relevant fields accompanied with enhancement in computer capability have improved the robustness of the system, making it more adaptable to the real environment. In this paper, we study the field and gives a detailed discussion on the actuality and the developing level of automatic lip reading. We emphatically introduce the feature extraction and recognition model algorithms. We also compare and analyze the various visual speech databases for their characteristics and functions in speech recognition systems. In addition, we describe the challenges and offer our insights into future research direction of automatic lip reading. [ABSTRACT FROM AUTHOR]

Lip-reading is one of important approaches for human-computer interaction (HCI). Its development would have a large range of applications, especially in augmented reality. Lip segmentation is the first and foremost step in the lip-reading system. Conventional method of region-based active contour model adopts the global information of image and is unable to perform well. In this paper, from a localized perspective, we introduce the methodology of localized active contour model (LACM) and, meanwhile, propose the method that using LACM to perform the lip segmentation with the initial contour automatically generated. The scope for active contour model is reduced to the local region that reduces the disturbances of unrelated factors. The experimental results demonstrate the method adopts this model would dramatically improve the robustness for lip segmentation. On this basis, we analyze the influence of initial contours and local radiuses, study the efficiencies under different initial contours and compare it with the conventional active contour model which adopts the global information. [ABSTRACT FROM AUTHOR]

Image quality is one of the most crucial influence factors when conducting biometric image-based human identification. However, it is not considered in most existing multi-biometric fusion algorithms. In this paper, a quality assessment-based dynamic weighted fusion algorithm is proposed, which applies a method of using image quality scores, and the scores are evaluated by integrating various image quality metrics to assess the quality of the feature matching process. According to the classification of the dual-modality feature matching quality, a dynamic weighted fusion strategy is proposed to increase the weight of biometric traits with better quality and weaken the impact of low-quality biometric traits on the recognition results, to achieve the adaptive optimization fusion of the biometric score level. Finally, the fused scores are used to make decisions. Experimental results reveal that the proposed algorithm is more robust and has better achievement than unimodal biometrics and traditional fusion algorithms. [ABSTRACT FROM AUTHOR]

With the rapid development of artificial intelligence and the increasing popularity of smart devices, human-computer interaction technology has become a multimedia and multimode technology from being computer-focused to people-centered. Among all ways of human-computer interactions, using language to interact with machines is the most convenient and efficient one. However, the performance of audio speech recognition systems is not satisfied in a noisy environment. Thus, more and more researchers focus their works on visual lip reading technology. By extracting lip movement features of speakers rather than audio features, visual lip reading systems can get superior results when noises and interferences exist. Lip segmentation plays an important role in a visual lip reading system, since the segmentation result is crucial to the final recognition accuracy. In this paper, we propose a localized active contour model-based method using two initial contours in a combined color space. We apply illumination equalization to original RGB images to decrease the interference of uneven illumination. A combined color space consists of the U component in CIE-LUV color space and the sum of C2 and C3 components of the image after discrete Hartley transform. We select a rhombus as the initial contour of a closed mouth, because it has a similar shape to a closed lip. For an open mouth, we utilize a combined semi-ellipse as the initial contours of both outer and inner lip boundaries. After attaining the results of each color component separately, we merge them together to obtain the final segmentation result. From the experiment, we can conclude that this method can get better segmentation results compared with the method using a circle as the initial contour to segment gray images and images in combined color space, especially for open mouth. An extremely obvious advantage of this method is the results of open mouth excluding internal information of mouth such as teeth, black holes, and tongue, because of the introduction of the inner initial contour. [ABSTRACT FROM AUTHOR]

Labanotation is a widely-used notation system for recording human dance movements. Automatically generating Labanotation scores from motion capture data can save significant manual effort and help the preservation of old folk dances in protecting intangible cultural heritages. Existing Labanotation generation methods have limited ability to capture the flexible limb movements as well as the rich periodic, symmetric, or repeated dance steps. In this paper, we present a novel LabanFormer model including a Multi-Scale Graph Attention network (MS-GAT) and a transformer model with Gated Recurrent Positional Encoding (GRPE) to achieve more effective Labanotation generation. First, the proposed MS-GAT can capture flexible limb movements by learning feature correlations between every two joints and aggregating features of neighboring joints over multiple scales. Second, we propose a new GRPE-based transformer to learn global temporal dependencies in the output feature sequences of MS-GAT. The novel GRPE module can encode position information with learnable parameters while handling various sequence lengths. As such, the periodic, symmetric, or repeated steps in dances can be accurately captured. Finally, the corresponding Laban symbols are generated by the decoder of the GRPE-based transformer. Extensive experiments on two real-world datasets show that the proposed LabanFormer model obtains remarkable performance compared with state-of-the-art approaches on the automatic Labanotation generation task. [ABSTRACT FROM AUTHOR]

To address the issue of high miss rates for distant small objects and the diminished system detection performance due to the influence of hazy when autonomous vehicles operate on mountain highways. We propose a framework for small object vehicle detection in hazy traffic environments (SHTDet). This framework aims to enhance small object detection for autonomous driving under hazy conditions on mountainous motorways. Specifically, to restore the clarity of hazy images, we designed an image enhancement (IE), and its parameters are predicted by a convolutional neural network [filter parameter estimation (FPE)]. In addition, to enhance the detection accuracy of small objects, we introduce a cascaded sparse query (CSQ) mechanism, which effectively utilizes high-resolution features while maintaining fast detection speed. We jointly optimize the IE and the detection network (CSQ-FCOS) in an end-to-end manner, ensuring that FPE module can learn a suitable IE. Our proposed SHTDet method is adept at adaptively handling sunny and hazy conditions. Extensive experiments demonstrate the efficacy of the SHTDet method in detecting small objects on hazy sections of mountain highways. [ABSTRACT FROM AUTHOR]