Search

Abstract Self‐attention has been successfully leveraged for long‐range feature‐wise similarities in deep learning super‐resolution (SR) methods. However, most of the SR methods only explore the features on the original scale, but do not take full advantage of self‐similarities features on different scales especially in generative adversarial networks (GAN). In this paper, self‐similarity generative adversarial networks (SSGAN) are proposed as the SR framework. The framework establishes the multi‐scale feature correlation by adding two modules to the generative network: downscale attention block (DAB) and upscale attention block (UAB). Specifically, DAB is designed to restore the repetitive details from the corresponding downsampled image, which achieves multi‐scale feature restoration through self‐similarity. And UAB improves the baseline up‐sampling operations and captures low‐resolution to high‐resolution feature mapping, which enhances the cross‐scale repetitive features to reconstruct the high‐resolution image. Experimental results demonstrate that the proposed SSGAN achieve better visual performance especially in the similar pattern details.

Abstract Infrared target detection has strong anti‐interference ability, long working distance and can work day and night. So it is widely used in military security and transportation fields, and infrared road object detection is critical in traffic checkpoints and autonomous driving. However, the target scale in infrared images changes greatly, small targets are difficult to detect, the poor image quality and low signal‐to‐noise ratio are still huge challenges in infrared target detection. This paper proposes an improved infrared target detection model I‐CenterNet based on the anchor‐free model CenterNet. The EfficientNetV2 with the channel attention mechanism is used instead of the traditional structure as the backbone network to enhance feature extraction. In order to reduce the noise of the input infrared image, Dilated‐Residual U‐net (DRUNet) is used. Meanwhile, feature pyramid and Sub‐Pixel are combined for multi‐scale feature fusion. Data enhancement is implemented to improve model performance. The experimental results show that the average detection accuracy of this model on the Flir infrared data set is 87.9%, and the average detection speed reaches 14.2 frames/s.

Abstract Road geometry perception indicates road direction and drivable area for Lane Keeping System and Lane Departure Warning System in Advanced Driver Assistance Systems or routing and decision‐making in Autonomous Driving. High‐definition map with accurate positioning or lane information is generally required. However, these two types of information are not always available because of positioning failure or lane mark degradation. In this paper, a road geometry estimation method is proposed without requirement of accurate positioning and lane information. Due to the fact the lane guides the front vehicle, the potential correlation between the relative azimuth of front vehicle and the direction of road is explored. The stable and observable information of front vehicles are exploited, such as the license plate location. A radar‐camera fusion system using image processing, and a nonlinear system is then built with Unscented Kalman Filter to estimate the road curvature. Simulation results with Carla demonstrate that the method enhances the road geometry perception by 77.1%, compared with the baseline method. The proposed method can provide reliable road geometry information for autonomous driving when accurate positioning and lane information are unavailable, so that it renders more robust and safer perception for future autonomous driving.

Abstract With the fast development of deep learning models, hierarchical convolutional neural networks have achieved great success in image denoising tasks. To further boost the performance of image denoising, a novel non‐local hierarchical network (NHNet) is proposed. Unlike existing U‐Net‐based hierarchical methods, which mainly focus on downsampling operations, NHNet adopts an initial resolution path and a high resolution path. Specifically, the high‐resolution features are obtained through upsampling, where the non‐local mechanism is adopted to capture the self‐similarity properties, which contribute to a better denoising performance. Cross connections and channel attention layers are added between the two paths to integrate features in different resolutions. Compared with other U‐Net‐based hierarchical networks, NHNet requires fewer parameters. Experiments show that NHNet achieves state‐of‐the‐art performance in Gaussian denoising tasks and gets competitive results when dealing with real image denoising.

Abstract Many countries actively promote the benefits of sports; in particular, water sports are popular among the younger generation. However, despite the mature development of equipment in diving, it requires relevant training to avoid decompression sickness. Sometimes when divers encounter emergencies and they may fail to alert the coach or other companions for rescues. Currently, some divers wear emergency equipment when conducting the exercise; yet, the cost is relatively high, and the operation is complicated that people tend to forget the process when facing emergencies. This article aims to develop a wearable device that has a safety alarm function based on the technique of machine learning. The features of the suggested device are as follows: (1) cost‐effective detectors for monitoring divers' conditions; (2) a combination of an Automatic Identification System (AIS) with a Global Positioning System (GPS) to send a safety alarm with the location of the diver; (3) utilize Bluetooth communication to detect if a diver left the safety range set by the coach; (4) the machine learning technique judges the health status of the diver; (5) the wearable device connects with swim goggles to deliver danger alarms, which enables divers to notice dangerous situations from the lights on the goggles. The approach suggested in this article primarily utilizes wearable devices to ensure divers' safety. The key feature of this device can prevent divers from sweeping away by currents or swimming into risky areas; meanwhile, the device can detect the risk of decompression sickness. The research executed an experiment to verify the design, and the results have proven the feasibility of the study. Additionally, with the cost‐effective detectors installed on the device, the presented equipment has the potential to make it universal and increase the safety of divers.

Abstract Affective video content analysis is an active topic in the field of affective computing. In general, affective video content can be depicted by feature vectors of multiple modalities, so it is important to effectively fuse information. In this work, a novel framework is designed to fuse information from multiple stages in a unified manner. In particular, a unified fusion layer is devised to combine output tensors from multiple stages of the proposed neural network. With the unified fusion layer, a bidirectional residual recurrent fusion block is devised to model the information of each modality. Moreover, the proposed method achieves state‐of‐the‐art performances on two challenging datasets, i.e. the accuracy value on the VideoEmotion dataset is 55.8%, and the MSE values on the two domains of EIMT16 are 0.464 and 0.176 respectively. The code of UMFN is available at: https://github.com/yunyi9/UMFN.

Abstract In practical datasets used for supervised learning, the uneven distribution of the amounts of data between classes is known as the class imbalance problem, and can reduce the performance of basic classifiers. The class imbalance problem arises in various areas such as medical diagnosis, spam filtering, and fraud detection. Bankruptcy prediction is of particular research interest due to the current economic upheaval. In a bankruptcy prediction dataset, there are two classes containing bankrupt and normal companies, and this problem can therefore be solved with binary classification methods. Several advanced models for handling the class imbalance problem in bankruptcy prediction have been developed to improve the predictability performance on a Korean bankruptcy (KB) dataset. To give an overview of imbalanced learning methods for bankruptcy prediction, this study first reviews several state‐of‐the‐art approaches for handling this problem in bankruptcy prediction, including an oversampling‐based framework, a cost‐sensitive method (the CBoost algorithm), a combination of resampling techniques and a cost‐sensitive framework, and an ensemble‐based model (the XGBS algorithm). We also conduct empirical experiments to evaluate the methods surveyed here in terms of two performance metrics, the area under the receiver operating characteristic (ROC) curve and the geometric mean. The results show that the ensemble‐based model outperforms other methods in terms of bankruptcy prediction on the KB dataset.

Abstract It is a challenge work to estimate the 3D human motion from image sequence. There are some problems, such as unsatisfactory estimation error, ambiguous matching and transient occlusion. Although the prior information of learning large‐scale samples exists, these problems are still difficult to be solved. How to extract the feature of the high‐dimensional (HD) sample of 3D human motion and find the desired one will become the key to solve these problems above. Some dimension reduction methods can extract the sample features and build the low‐dimensional (LD) space to view their LD features, but how to search the relevant valid and desired LD samples remains the bottleneck problem, which can be used to reconstruct the 3D human motions denoted by the corresponding high‐dimensional samples. Thus, a new method called dynamic manifold Boltzmann optimization (DMBO) is proposed to estimate the 3D human motion from multi‐view images. DMBO can find the best matching 3D human motion model by the help of the self‐supervised learning from Gaussian incremental dimension reduction model (GIDRM). DMBO can avoid the local optimum during searching and solve the problems above, so that the generation of the accurate 3D human motion corresponding to multi‐view images can be achieved.

Abstract Images captured through a visual sensory system are degraded in a foggy scene, which negatively influences recognition, tracking, and detection of targets. Efficient tools are needed to detect, pre‐process, and enhance foggy scenes. Machine learning (ML) has a significant role in image defogging domain for tackling adverse issues. Unfortunately, regardless of contributions that were made by ML, little attention has been attributed to this topic. This paper summarizes the role of ML methods and relevant aspects in the image defogging research area. Also, the basic terms and concepts are highlighted in image defogging topic. Feature extraction approaches with a summary of advantages and disadvantages are described. ML algorithms are also summarized that have been used for applications related to image defogging, that is, image denoising, image quality assessment, image segmentation, and foggy image classification. Open datasets are also discussed. Finally, the existing problems of the image defogging domain in general and, specifically related to ML which need to be further studied are discussed. To the best knowledge, this the first review paper which sheds a light on the role of ML and relevant aspects in the image defogging domain.

Abstract In traffic systems, traffic forecasting is a critical issue, which has attracted much interest from researchers. It is a challenging task due to the complex spatial‐temporal patterns of traffic data. Previous works focus on designing complex graph‐based neural networks to model spatial‐temporal dependencies from data. By using graphs to represent road networks, these works capture spatial patterns with graph convolutions. However, graphs cannot fully represent spatial relations from road networks. It limits the performance of graph‐based methods. In this paper, we propose a directed hypergraph neural network architecture, Directed Hypergraph Attention Network(DHAT), for traffic forecasting. Unlike previous works, DHAT introduces a directed hypergraph to represent road networks. Compared with graphs, directed hypergraphs could represent spatial information from graphs and outperform them in modeling complex directed relations among multiple nodes. Based on the directed hypergraph, a directed hypergraph convolution is proposed to exploit spatial relations among traffic series. By combining the proposed convolution and attention mechanisms, DHAT can effectively achieve promising predictions for traffic forecasting. To evaluate the performance of DHAT, we have conducted extensive experiments on four real‐world traffic datasets. Compared with other baselines, experimental results show that DHAT reduces Mean Absolute Error by 0.03–0.64 on these datasets.

Abstract Facial target detection is an important task in computer vision. Because heterogeneous face detection shows broad prospects, it has attracted extensive attention from the academic community. In recent years, with the rise of deep learning and its applications in computer vision, face detection technology has made great strides. This paper uses multi‐task cascaded convolutional neural network (MTCNN) for heterogeneous face feature detection. This algorithm makes full use of the advantages of image pyramid, boundary regression, fully convolutional attention networks and non‐maximum suppression. The main idea of this paper is to use candidate frame plus classifier for fast and efficient face detection. Specifically, the candidate window is generated by the proposal network (P‐Net), and the high‐precision candidate window is filtered and selected by the reduced network (R‐Net), and the final bounding box and facial key points are generated by the output network (O‐Net). In order to prove the effectiveness of this method in visible light, near‐infrared and sketch face recognition scenes, it was verified in the datasets of CUFS, CUFSF and CASIA NIR‐VIS 2.0. Experiments show that this method is effective for face images in heterogeneous face and is better than the latest algorithms.

Abstract Liver cancer poses a great threat to people's health. Accurate liver segmentation is crucial to the diagnosis of liver cancer. In recent years, great achievements have been made in liver segmentation using a series of improved networks developed based on U‐Net. U‐Net uses a skip connection splices the feature map in the encoder and decoder. However, this method ignores the difference between the two feature maps, which limits the ability of the network to extract liver structures of different sizes. This study proposes a new network structure, the pyramid convolutional attention fusion network (PCAF‐Net), for 2D liver segmentation. The PCAF mechanism was introduced to fuse the feature maps of different receptive field paths after attention mechanism processing to enhance the semantic expression ability of feature maps in skip connection and to improve the segmentation accuracy. The MICCAI 2017 LiTS dataset and CHAOS were used to validate the proposed method. The results of multi‐index evaluation showed that the segmentation performance of the proposed network was superior to those of other networks. PCAF‐Net achieved accurate liver segmentation, providing a reference for artificial intelligenceassisted clinical diagnosis of liver cancer.

Abstract This paper proposes a novel image classification architecture named ensemble cross‐stage partial attention network based on the backbone network DarkNet53 of Yolov3 to improve the feature extraction capability and the interpretability of image classification. This network has multiple advantages, including light model parameters, fast classification speed, and high classification accuracy for small objects and complex images. Local network architectures of different cross‐phases are added in the proposed network structure to reduce the calculation. Furthermore, channel and hybrid domain attention modules, which, respectively, fuse the branch feature with the extracted channel and spatial attention features, are designed for feature extraction of images. Experimental results confirm the improved performance of the proposed approach on the CIFAR‐100, ImageNet, and UCMerced datasets. In addition, experiments on the MSCOCO dataset suggest the application of the proposed method to object detection with satisfactory accuracy.

Abstract The detection of road infrared targets is essential for autonomous driving. Different from RGB images, the acquisition of infrared images is unaffected by visible light. However, the signal‐to‐noise ratio still presents significant challenges. This study demonstrates an improved infrared target detection model for road infrared target detection. An advanced EfficientNet is incorporated to replace the conventional structure and enhance feature extraction. A Dilated‐Residual U‐Net is also introduced to reduce the noise of infrared images. Meanwhile, the k‐means algorithm and data enhancement are implemented to improve the detection performance. The experimental results show that the mean average precision of the proposed model is observed to be 0.89 for the infrared road dataset with an average detection speed of 10.65 s−1.

Abstract Deraining has been attracting a lot of attention from researchers, and various methods have been proposed, especially deep‐networks are widely adopted in recent years. Their structures and learning become more and more complicated and diverse, making it difficult to analyze the contributions and improvements. In this paper, a comprehensive review for current rain removal methods is first provided to show their contributions. Specifically, they are reviewed in terms of handing rain streaks and rain mist. Second, besides evaluating their rain removal ability, they are also evaluated in terms of their impact on subsequent stereo‐matching task. To this end, a new deraining dataset is first prepared, called Rain‐Kitti2012 and Rain‐Kitti2015. They are created by adding rain part to clean image‐pairs in Kitti2012 and Kitti2015. By then, nine state‐of‐the‐art deraining methods are evaluated with full‐reference and no‐reference image quality assessment metrics. Furthermore, the blurriness and distortion types introduced during deraining are measured. Finally, three learning‐based stereo matching methods are compared, and they take the outputs of deraining methods as inputs. It is further discussed how derained images influence the accuracy of stereo matching, which can provide some insight for jointly handling rain removal and stereo matching. 1: A comprehensive review for the current rain removal methods is provided. They are categorized into rain‐streak‐oriented and rain‐mist‐oriented approaches in terms of degradation type, and are categorized into model‐driven and data‐driven approaches in terms of methodology. 2: A new image deraining dataset is introduced, which is the first dataset that can be used to perform stereo‐matching‐driven evaluation for deraining methods. The dataset is created by adding rain part to clean images in KITTI2012 and KITTI2015. 3: We evaluate 9 deep learning based deraining methods with full‐reference and no‐ reference metrics. In addition, the types of distortions produced by these methods are discussed and measured quantitatively. And, the impact of 9 deraining methods on the subsequent stereo matching task is evaluated, which can provide some insight on how to design stereo matching task‐driven deraining methods.

Abstract To address the existing problems of capsule networks in deep feature extraction and spatial‐spectral feature fusion of hyperspectral images, this paper proposes a hyperspectral image classification method that combines a deep residual 3D capsule network and Markov random field. Based on this method, the deep spatial‐spectral features of hyperspectral images are extracted using the deep residual 3D convolutional structure, the vector capsules of the features are obtained by the initial capsule layer and mapped into probability capsules via the 3D dynamic routing mechanism to construct the classification probability map, and the spatial structure of the classification results is regularised by the Markov random field to further improve the classification accuracy and performance of the images. Two sets of benchmark hyperspectral images, namely Indian Pines and Pavia University data sets, were used to conduct comparative experiments and ablation study. The experimental results showed that, compared with the conventional convolutional neural network and existing capsule network models, the proposed method not only improves the classification accuracy of the images but also partly eliminates the category noise and affords a more regular classification probability map.

Abstract Existing sketch‐based image processing methods include sketch recognition, sketch synthesis and sketch‐based image retrieval. For sketch creation, a meaningful task is proposed namely disentangled and controllable sketch creation (DCSC) based on disentangling the structure and color enhancement. Specifically, as the first subtask, sketch structure enhancement (SSE) is used to enhance a non‐professional sketch (NPS) and obtain a professional sketch (PS), which is a process denoted as NPS2PS. A data set named SketchMan is first provided, consisting of NPSs and PSs with various postures in different scenes. SSE is trained as a conditional image‐to‐image translation problem, and there are three models: direct sketch‐to‐sketch (SS), grayscale guided SS and contour guided SS. Multiple IOU metrics are proposed based on Corner Point Map (CPM), Straight Line Map (SLM) and Segmented Area Map (SAM). As the second subtask, sketch color enhancement (SCE) is trained as a two‐stage framework containing a topology enhancement network (TE‐Net) that maps a sketch to the corresponding grayscale domain and a color injection network (CI‐Net) that injects the global color feature to the AdaIN residual blocks to perform adaptive sketch colorization. The TE‐Net and CI‐Net disentangle the topological and color features to perform more controllable and diverse SCE results. Experimental results demonstrate that our proposed methods are effective to address the challenging and meaningful DCSC task compared with other state‐of‐the‐art methods.

Abstract Visual object tracking has been a fundamental topic in computer vision and many convolutional neural network (CNN) based trackers proposed in recent years have achieved state‐of‐the‐art performance, multi‐domain convolutional neural network (MDNet) is one of the most representative CNN‐based algorithms with high performance. In order to significantly improve the robustness and accuracy of the MDNet algorithm, a multi‐domain convolutional neural network based on spatial self‐attention information learning (SAMDNet) is proposed in this study. The authors use the spatial self‐attention module for the spatial information learning of the model. The spatial self‐attention module selectively aggregates the feature at each position by a weighted sum of the features at all positions. Under the control of self‐learning parameters in this module, spatial attention information can be flexibly acquired. The authors also propose a novel interval loss term to solve the problem of different classes with the same semantics in the training data. Finally, an anomaly detection module is carefully designed for relocation after the algorithm completely lost the target. Extensive experiments on the object tracking benchmark (OTB) and the visual object tracking challenge (VOT) benchmarks show that the proposed tracker outperforming most of the state‐of‐art trackers.

Abstract With the performance increase of convolutional neural network (CNN), the disadvantages of CNN's high storage and high power consumption are followed. Among the methods mentioned in various literature, filter pruning is a crucial method for constructing lightweight networks. However, the current filter pruning method is still challenged by complicated processes and training inefficiency. This paper proposes an effective filter pruning method, which uses the saliency of the feature map (SFM), i.e. information entropy, as a theoretical guide for whether the filter is essential. The pruning principle use here is that the filter with a weak saliency feature map in the early stage will not significantly improve the final accuracy. Thus, one can efficiently prune the non‐salient feature map with a smaller information entropy and the corresponding filter. Besides, an over‐parameterized convolution method is employed to improve the pruned model's accuracy without increasing parameter at inference time. Experimental results show that without introducing any additional constraints, the effectiveness of this method in FLOPs and parameters reduction with similar accuracy has advanced the state‐of‐the‐art. For example, on CIFAR‐10, the pruned VGG‐16 achieves only a small loss of 0.39% in Top‐1 accuracy with a factor of 83.3% parameters, and 66.7% FLOPs reductions. On ImageNet‐100, the pruned ResNet‐50 achieves only a small accuracy degradation of 0.76% in Top‐1 accuracy with a factor of 61.19% parameters, and 62.98% FLOPs reductions.

Abstract Lipreading is a visual recognition of speech by using lip movement, which aims to recognise phrases and sentences spoken by a talking face without the audio. However, the existed models for lipreading suffer from slow training speed and insufficient performance. To accelerate the training speed of the model for lipreading, a batch group training algorithm is proposed, which groups all the data of different frames. In addition, a 3D‐MouthNet‐BLSTM‐CTC architecture for lipreading is proposed to improve model performance. It bases on a 3D convolutional neural network, MouthNet, two Bi‐LSTMs, and a CTC objective function. Experiment results in Oulu‐VS2 and self‐built dataset show that 96.2% accuracy rate is achieved on the Oulu‐VS2 dataset, and 93.8% accuracy rate is achieved on the GRID dataset. This article is about lipreading research. It mainly uses deep learning methods to study lip‐reading. A new network architecture and tests on public data sets are proposed to achieve the best results.

Abstract This paper introduce an efficient object detection network named Trident‐You Only Look Once (YOLO), which is designed for mobile devices with limited computing power. The new architecture is improved based on YOLO v4‐tiny. The authors redesign the network structure and propose a trident feature pyramid network (Trident‐FPN), which can improve the precision and recall of lightweight object detection. Specifically, Trident‐FPN increases the computational complexity by only a small amount of floating point operations per second (FLOPs) and obtains a multi‐scale feature map of the model, which significantly lightweight object detection performance. To enlarge the receptive field of the network with the fewest FLOPs, this paper redesign the receptive field block (RFB) and spatial pyramid pooling (SPP) layer and propose tinier cross‐stage partial RFBs and smaller cross‐stage partial SPPs. This paper present extensive experiments, and Trident‐YOLO shows strong performance compared to that of other popular models on the PASCAL VOC and MS COCO. On the MS COCO and PASCAL VOC 2007 test sets, the mean average precision (mAP) of Trident‐YOLO improved by 4.5% and 5.0%, respectively. Trident‐YOLO also reduce the network size by more than 54.4% compared to YOLO v4‐tiny. With a 23.7% FLOP reduction, the FPS is improved by 1.9 on an Nvidia Jetson Xavier NX.

Abstract Medical image generation from diagnostic report has important research significance for medical‐aided diagnosis. This research can improve the diagnosis speed and accuracy of doctors and effectively save the storage resources of hospitals. The research has made significant progress in the field of natural images, but rarely used in medical images. Medical images have higher requirements for image quality. In this paper, a method based on attention mechanism and content preservation loss to improve image quality is proposed. The model consists of three stages, and each stage generates feature map of different scale combined with attention features as the input of the next stage to optimise semantic consistency between image and text. The last stage will generate an image, whose size is 256 × 256. And the content preservation loss can optimise the similarity between the generated image and the real image by low‐level and high‐level features to meet the high requirements of medical image for texture details. The content preservation loss consists of MSE loss, VGG loss and TV loss. The experiments on two datasets prove that the method can achieve excellent results.

Abstract Radiation therapy is the standard treatment for early stage Nasopharyngeal cancer (NPC). Thus, accurate delineation of target volumes at risk in NPC is important. While manual delineation is time‐consuming and labour‐intensive process and also leads to significant inter‐ and intra‐practitioner variability. Thus, computer‐aided segmentation algorithm is required. However, segmentation task is not trivial due to large variations (e.g., shape and size) of nasopharynx structure across subjects. Moreover, extreme foreground and background class imbalance in NPC segmentation remains challenge. In this paper, we propose a threedimensional densely connected convolutional neural network with multi‐scale feature pyramids for NPC segmentation. We adapt the densely connected convolutional block into a new structure via adding feature pyramids. The concatenated pyramid feature carries multi‐scale and hierarchical semantic information which is effective for segmenting different size of tumors and perceiving hierarchical context information. To address the foreground and background imbalance problem, we propose an enhanced version of focal loss. It prevents the large number of negative voxels far from boundaries from overwhelming the segmentation algorithm. We validated the proposed method on 120 clinical subjects. Experimental results demonstrate that our approach out‐performed state‐of‐the‐art methods and human experts.

Abstract In this paper, a new Multi‐Focus Image Fusion (MFIF) method based on multi‐source joint layering and Convolutional Sparse Representation (CSR) is proposed. Based on the characteristics of multi‐focus source images, a multi‐source joint layering regularization model was designed to divide the sources into a common base‐layer and respective focus detail‐layers. This strategy can overcome the defects caused by source layering separately effectively. In detail‐layer fusion, CSR was employed to extract and global features. It can avoid detail blur and high computational cost caused by image blocking in the conventional sparse representation model. The proposed detail‐layer fusion rule combined the CSR coefficient maps pairwise with the window based select‐max rule. In the experiments, the optimal layering parameter was selected by experiments at first, and then five recently proposed specific MFIF or general image fusion algorithms were contrasted with the proposed method by plenty of subjective and objective experimental comparisons. The experimental results demonstrated the superiority of the authors’ method.

Abstract In the dark scene at night, the face images captured by ordinary visible light (VIS) are generally poor quality and very dim, while the near‐infrared (NIR) can capture high definition and recognizable face images at night. The NIR‐VIS Heterogeneous face recognition has become a hot research field, which helps to build an all‐weather face recognition system. NIR‐VIS HFR is sophisticated because of the large visual difference between NIR images and VIS images. In order to reduce the difficulty of such cross‐modality invariant feature learning, this paper proposes a cross‐modality data gap decomposed by auxiliary modality method (DGD) for NIR‐VIS HFR. First, the brightness component (Y component) of VIS image YCbCr space is used as the auxiliary modality to decompose the cross‐modality data gap. The lightness component retained the structural information of VIS image and was similar to the colour information of NIR modality; in this way, the huge gap between the NIR data and the VIS data is decomposed into two smaller gaps, thus reducing the difficulty of network learning. Second, the data of the three modalities are input into the weight sharing network and training under the combined guidance of cross‐modality gap decomposition loss and intra‐modality gap loss; in this way, the modality invariant features can be learned faster and better. Extensive experiments were conducted on two commonly used datasets CASIA NIR‐VIS 2.0 and Oulu‐CASIA NIR‐VIS to evaluate DGD method. Experimental results indicate DGD method has competitive performance compared with the latest methods.

Abstract With the success of convolutional neural network (CNN) in the field of image processing and image recognition, an improved fully convolutional neural network (FCNN) based on CNN is proposed. Although FCNN has better performance than CNN, the number of parameters of FCNN has also increased significantly compared to CNN. Consequently, this paper proposes a multi‐feature fusion decision convolutional neural network (MFFD‐CNN), which use two kinds down‐sample methods in the pooling layer, and chooses the concatenate or add operation as the way that consolidates the feature of the convolutional layer with a stride of 2 and the max‐pooling layer to execute fusion operations. Meanwhile replaces the traditional softmax loss with an additive margin Softmax (AM‐Softmax) loss. The structure, which is based on the multi‐target classification experiment results of the MSTAR data set, shows can not only obtain an average correct recognition rate 1.2% higher than FCNN, but also better stability without augmenting the training samples of MSTAR data.

Abstract Deep learning techniques and deep networks have recently been extensively studied and widely applied to single image super‐resolution (SR). Among them, channel attention has garnered the most focus owing to its significant boost in the presentational power of a convolutional neural network. However, the original channel attention neglects the critical importance of the positional information, thus imposing performance limitations. Here, a novel perspective, namely, a coordinate attention mechanism, is explored to alleviate the aforementioned problem, and accordingly result in an enhanced SR performance. Specifically, a deep residual coordinate attention SR network (COSR) is proposed, which mainly incorporates the presented coordinate attention blocks into a deep nested residual structure. The coordinate attention captures the positional information by computing the average value vector from the two spatial directions, thus aggregating the features in different coordinates. The nested residual blocks pass low‐frequency information from the top to the end through the skip connection lines, allowing convolution filters to concentrate more on high‐frequency textures and edges, thereby reducing the difficulty of reconstruction. Extensive experiments demonstrate that our proposed COSR achieves a better performance and exceeds many state‐of‐the‐art SR methods in terms of both quantitative metrics and visual quality.

Abstract Several networks exhibiting excellent performance in the field of computer vision still suffer from the challenge of over‐parameterization. Over‐parameterized networks have several parameters that cannot be trained, owing to the poor backpropagation process caused by a small L1 norm, and these parameters suppress the potential for network performance improvement. This study proposes a kernel reactivation method to improve network performance by reusing invalid kernels that cannot be utilized for training because of the small L1 norm. The results indicate that the accuracy of Cifar‐10 in ResNet‐110 is improved by 0.94% compared to the baseline, and the top‐1 and top‐5 accuracies of Tiny‐ImageNet in ResNet‐50 are improved by 1.87% and 1.03% compared to the baseline, respectively.

Abstract Traditional deep convolutional networks have shown that both RGB and depth are complementary for video action recognition. However, it is difficult to enhance the action recognition accuracy because of the limitation of the single convolutional networks to extract the underlying relationship and complementary features between these two kinds of modalities. The authors proposed a novel two stream convolutional networks for multi‐modality action recognition by joint optimisation learning to extract global features from RGB and depth sequences. Specifically, a non‐local multi‐modality compensation block is introduced to learn the semantic fusion features for the recognition performance. Experimental results on two multi‐modality human action datasets, including NTU RGB+D 120 and PKU‐MMD dataset, verify the effectiveness of our proposed recognition framework and demonstrate that the proposed non‐local multi‐modality compensation block can learn complementary features and enhance the recognition accuracy.

Abstract Person re‐identification (Re‐ID) is in significant demand for intelligent security and single or multiple‐target tracking. However, there are issues in the person Re‐ID tasks, such as sharp decline in cross‐data sets detection accuracy, poor generalization and cross‐domain ability of the model. This work mainly studies the generalization and adaptation of cross‐domain person Re‐ID models. Different from most existing methods for cross‐domain Re‐ID tasks, the authors use diversified spatial semantic feature in pixel‐level learning in the target domain to improve the generality and adaptability of the model. In the case that no information of the target domain is used during the model training, the trained model is directly tested on the data set of the target domain. It has proven effective to add the attention cascade module into the backbone network combining with the part‐level branch. The authors conducted extensive experiments based on the three data sets of Market‐1501, DukeMTMC‐ReID and MSMT17, resulting in both single‐domain and cross‐domain tests with an average improvement of Rank1 and mAP values of about 10% compared with Baseline through the authors' proposed method named Part‐Level Attention Network.

Abstract Crowd counting is still a practical and challenging problem owing to scale variations and information loss. Most existing methods based on the straightforward fusion of different features from a deep neural network seem to eliminate this limitation. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. Unlike previous works, a multi‐level features extraction network with gating mechanism for crowd counting is proposed. Specifically, a multi‐channel gated unit to adaptively extract features in different levels of the network is proposed, which can avoid interference from confusing information. To fully aggregate features via multi‐level fusion, multi‐level features extraction scheme is presented. The multi‐level features extraction network learns to fuse features from multiple levels and reduce false predictions. Extensive experiments and evaluations clearly illustrate that the proposed approach achieves state‐of‐the‐art counting performance against other methods on four mainstream crowd counting benchmarks.

Abstract Motion information has been widely exploited for group activity recognition in sports video. However, in order to model and extract the various motion information between the adjacent frames, existing algorithms only use the coarse video‐level labels as supervision cues. This may lead to the ambiguity of extracted features and the omission of changing rules of motion patterns that are also important sports video recognition. In this paper, a latent label mining strategy for group activity recognition in basketball videos is proposed. The authors' novel strategy allows them to obtain the latent labels set for marking different frames in an unsupervised way, and build the frame‐level and video‐level representations with two separate levels of supervision signal. Firstly, the latent labels of motion patterns are digged using the unsupervised hierarchical clustering technique. The generated latent labels are then taken as the frame‐level supervision signal to train a deep CNN for the frame‐level features extraction. Lastly, the frame‐level features are fed into an LSTM network to build the spatio‐temporal representation for group activity recognition. Experimental results on the public NCAA dataset demonstrate that the proposed algorithm achieves state‐of‐the‐art performance.

Abstract The crowd behaviour understanding and density estimation are some of the fast‐growing fields in video surveillance. There are many techniques (detection and regression) that are used as the method of crowd analysis and estimation. In the present approach, SVR (support vector regression) is used as the basic analysis technique and the novel key‐point matching with SURF (speedup robust feature) is used as the feature extractor for moving objects in the video. The traditional linear regression methods used mainly key‐point as one of the statistical features instead of matching with consecutive frames, but we used the magnitude of the optical flow for foreground object extraction instead of inter‐frame difference. The combination of the optical flow of foreground objects and key‐point matching generates new features apart from conventional features such as areas and corners. In this new approach, key‐point pairing with linear regression is tested with the PETS2009 dataset, and performance is compared with the existing approaches.

Abstract Since the era of we‐media, live video industry has shown an explosive growth trend. For large‐scale live video streaming, especially those containing crowd events that may cause great social impact, how to identify and supervise the crowd activity in live video streaming effectively is of great value to push the healthy development of live video industry. The existing crowd activity recognition mainly uses visual information, rarely fully exploiting and utilizing the correlation or external knowledge between crowd content. Therefore, a crowd activity recognition method in live video streaming is proposed by 3D‐ResNet and regional graph convolution network (ReGCN). (1) After extracting deep spatiotemporal features from live video streaming with 3D‐ResNet, the region proposals are generated by region proposal network. (2) A weakly supervised ReGCN is constructed by making region proposals as graph nodes and their correlations as edges. (3) Crowd activity in live video streaming is recognised by combining the output of ReGCN, the deep spatiotemporal features and the crowd motion intensity as external knowledge. Four experiments are conducted on the public collective activity extended dataset and a real‐world dataset BJUT‐CAD. The competitive results demonstrate that our method can effectively recognise crowd activity in live video streaming.

Abstract The objective of crowd counting is to learn a counter that can estimate the number of people in a single image. So far, most of the proposed work evaluates the crowd density by fitting the constructed density map corresponding to the sample. The performance of those algorithms depends on a large amount of carefully prepared data. However, a significant problem with crowd data sets is the difficulty of labeling. To address such a situation, utilizing object counting data in few‐shot scenes is considered and an efficient algorithm to extract the meta‐information is proposed, thus improving the accuracy and convergence rate of the crowd counting tasks. Specifically, the counting network is trained with only object counting tasks constructed on different domains during the meta‐training phase. Then, the meta‐counter is testing on crowd counting tasks in the meta‐testing stage. Experimentally, it is demonstrated that the above way improves the converge rate and accuracy of crowd counting tasks on three crowd counting datasets when meta‐training on ten‐type object counting tasks.

Abstract In the last years, subspace‐based multi‐view face recognition has attracted increasing attention and many related methods have been proposed. However, the most existing methods ignore the specific local structure of different views. This drawback can cause these methods' discriminating ability to degrade when many noisy samples exist in data. To tackle this problem, a multi‐view low‐rank representation method is proposed, which exploits both intrinsic relationships and specific local structures of different views simultaneously. It is achieved by hierarchical Bayesian methods that constrain the low‐rank representation of each view so that it matches a linear combination of an intrinsic representation matrix and a specific representation matrix to obtain common and specific characteristics of different views. The intrinsic representation matrix holds the consensus information between views, and the specific representation matrices indicate the diversity among views. Furthermore, the model injects a clustering structure into the low‐rank representation. This approach allows for adaptive adjustment of the clustering structure while pursuing the optimization of the low‐rank representation. Hence, the model can well capture both the relationship between data and the clustering structure explicitly. Extensive experiments on several datasets demonstrated the effectiveness of the proposed method compared to similar state‐of‐the‐art methods in classification and clustering.

Abstract Crowd understanding and analysis have received increasing attention for couples of decades, and development of human behaviour recognition strongly supports the application of crowd understanding and analysis. Human behaviour recognition usually seeks to automatically analyse ongoing movements and actions in different camera views by using various machine learning methodologies in unknown video clips or image sequences. Compared to other data modalities such as documents and images, processing video data demands much higher computational and storage resources. The idea of using middle level semantic concepts to represent human actions from videos is explored and it is argued that these semantic attributes enable the construction of more descriptive methods for human action recognition. The mid‐level attributes, initialized by a cluster processing, are built upon low level features and fully utilize the discrepancies in different action classes, which can capture the importance of each attribute for each action class. In this way, the representation is constructed to be semantically rich and capable of highly discriminative performance even paired with simple linear classifiers. The method is verified on three challenging datasets (KTH, UCF50 and HMDB51), and the experimental results demonstrate that our method achieves better results than the baseline methods on human action recognition.

Abstract In traffic accidents, motorcycle accidents are the main cause of casualties, especially in developing countries. The main cause of fatal injuries in motorcycle accidents is that motorcycle riders or passengers do not wear helmets. In this paper, an automatic helmet detection of motorcyclists method based on deep learning is presented. The method consists of two steps. The first step uses the improved YOLOv5 detector to detect motorcycles (including motorcyclists) from video surveillance. The second step takes the motorcycles detected in the previous step as input and continues to use the improved YOLOv5 detector to detect whether the motorcyclists wear helmets. The improvement of the YOLOv5 detector includes the fusion of triplet attention and the use of soft‐NMS instead of NMS. A new motorcycle helmet dataset (HFUT‐MH) is being proposed, which is larger and more comprehensive than the existing dataset derived from multiple traffic monitoring in Chinese cities. Finally, the proposed method is verified by experiments and compared with other state‐of‐the‐art methods. Our method achieves mAP of 97.7%, F1‐score of 92.7% and frames per second (FPS) of 63, which outperforms other state‐of‐the‐art detection methods.

Abstract Cross modal retrieval can retrieve images through a text query and vice versa. In recent years, cross modal retrieval has attracted extensive attention. The purpose of most now available cross modal retrieval methods is to find a common subspace and maximize the different modal correlation. To generate specific representations consistent with cross modal tasks, this paper proposes a novel cross modal retrieval framework, which integrates feature learning and latent space embedding. In detail, we proposed a deep CNN and a shallow CNN to extract the feature of the samples. The deep CNN is used to extract the representation of images, and the shallow CNN uses a multi‐dimensional kernel to extract multi‐level semantic representation of text. Meanwhile, we enhance the semantic manifold by constructing cross modal ranking and within‐modal discriminant loss to improve the division of semantic representation. Moreover, the most representative samples are selected by using online sampling strategy, so that the approach can be implemented on a large‐scale data. This approach not only increases the discriminative ability among different categories, but also maximizes the relativity between different modalities. Experiments on three real word datasets show that the proposed method is superior to the popular methods.

Abstract Human detection in crowded scenes is one of the research components of crowd safety problem analysis, such as emergency warning and security monitoring platforms. Although the existing anchor‐free methods have fast inference speed, they are not suitable for object detection in crowded scenes due to the model's inability to predict the well‐fined object detection bounding boxes. This work proposes an end‐to‐end anchor‐free network, Multi‐dimensional Weighted Cross‐Attention Network (MANet), which can perform real‐time human detection in crowded scenes. Specifically, the Double‐flow Weighted Feature Cascade Module (DW‐FCM) is used in the extractor to highlight the contribution of features at different levels. The Triplet Cross Attention Module (TCAM) is used in the detector head to enhance the association dependence of multi‐dimension features, further strengthening human boundary features' discrimination ability at a fine‐grained level. Moreover, the strategy of Adaptively Opposite Thrust Mapping (AOTM) ground‐truth annotation is proposed to achieve bias correction of erroneous mappings and reduce the iterations of useless learning of the network. These strategies effectively alleviate the defect that the existing anchor‐free network cannot correctly distinguish and locate the individual human in crowded scenes. Compared with the anchor‐based detection method, there is no need to set anchor parameters manually, and the detection speed can satisfy the real‐time application. Finally, through extensive comparative experiments on CrowdHuman and WIDER FACE datasets, the results demonstrate that the improved strategy achieves the state‐of‐the‐art result in the anchor‐free methods.

Abstract Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi‐scale feature pyramid network (MFP‐Net) for dense crowd counting is proposed in this paper. The proposed MFP‐Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention‐aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP‐Net is evaluated on five publicly available datasets, and experiments show that the MFP‐Net not only provides better crowd counting results than comparative models, but also requires fewer parameters.

Abstract Anomaly detection has attracted considerable search attention. However, existing anomaly detection databases encounter two major problems. Firstly, they are limited in scale. Secondly, training sets contain only video‐level labels indicating the existence of an abnormal event during the full video while lacking annotations of precise time durations. To tackle these problems, we contribute a new Large‐scale Anomaly Detection (LAD) database as the benchmark for anomaly detection in video sequences, which is featured in two aspects. 1) It contains 2000 video sequences including normal and abnormal video clips with 14 anomaly categories including crash, fire, violence etc. with large scene varieties, making it the largest anomaly analysis database to date. 2) It provides the annotation data, including video‐level labels (abnormal/normal video, anomaly type) and frame‐level labels (abnormal/normal video frame) to facilitate anomaly detection. Leveraging the above benefits from the LAD database, we further formulate anomaly detection as a fully supervised learning problem and propose a multi‐task deep neural network to solve it. We firstly obtain the local spatiotemporal contextual feature by using an Inflated 3D convolutional (I3D) network. Then we construct a recurrent convolutional neural network fed the local spatiotemporal contextual feature to extract the spatiotemporal contextual feature. With the global spatiotemporal contextual feature, the anomaly type and score can be computed simultaneously by a multi‐task neural network. Experimental results show that the proposed method outperforms the state‐of‐the‐art anomaly detection methods on our database and other public databases of anomaly detection. Supplementary materials are available at http://sim.jxufe.cn/JDMKL/ymfang/anomaly‐detection.html.

Abstract Crowd counting aims to count the number of people in crowded scenes, which is important to the security systems, traffic control and so on. The existing methods typically using local features cannot properly handle the perspective distortion and the varying scales in congested scene images, and henceforth perform wrong people counting. To alleviate this issue, this study proposes a multi‐scale residual feature‐aware network (MSR‐FAN) that combines multi‐scale features using multiple receptive field sizes and learns the feature‐aware information on each image. The MSR‐FAN is trained end‐to‐end to generate high‐quality density map and evaluate the crowd number. The method consists of three parts. To handle the perspective changes problem, the first part, the direction‐based feature‐enhanced network, is designed to encode the perspective information in four directions based on the initial image feature. The second part, the proposed multi‐scale residual block module, gets the global information to handle the represent the regional feature better. This module explores features of different scales as well as reinforce the global feature. The third part, the feature‐aware block, is designed to extract the feature hidden in the different channels. Experiment results based on benchmark datasets show that the proposed approach outperforms the existing state‐of‐the‐art methods.

Abstract Detecting human behaviours in images of crowded classroom scenes is a challenging task, due to the large variations of humans in scale and pose perspective. In this paper, two modules are proposed to tackle these two variations. First, an attention‐based RoI (region‐of‐interest) extractor is designed to handle scale variation. Feature fusion and attention mechanism are used to improve the RoI feature with more local and global information. Second, a transformation‐based detection head is introduced to handle perspective variation. The spatial transformation is adopted to extract consistent representation under various perspectives. Moreover, since there is a lack of proper datasets for human behaviour detection in classroom scenes, a new dataset is created, namely CLBD. The experiments on the proposed dataset demonstrate that the modules obtain significant improvements of performance over the state‐of‐the‐art detectors.

Abstract The purpose of 3D human pose estimation is to estimate the 3D coordinates of key points of the human body directly from images. Although multi‐view based methods have better performance and higher precision of coordinate estimation than a single‐view based, they need to know the camera parameters. In order to effectively avoid the restriction of this constraint and improve the generalizability of the model, a dual‐view single‐person 3D pose estimation method without camera parameters is proposed. This method first uses the 2D pose estimation network HR‐net to estimate the 2D joint point coordinates from two images with different views, and then inputs them into the 3D regression network to generate the final 3D joint point coordinates. In order to make the 3D regression network fully learn the spatial structure relationship of the human body and the transformation projection relationship between different views, a self‐supervised training method is designed based on a 3D human pose orthogonal projection model to generate the virtual views. In the pose estimation experiments on the Human3.6 dataset, this method achieves a significantly improved estimation error of 34.5 mm. Furthermore, an action recognition based on the human poses extracted by the proposed method is conducted, and an accuracy of 83.19% is obtained.

Abstract In the area of pedestrian trajectory prediction, the hybrid structures of temporal feature extractor or spatial feature extractor have paved the way for the precise prediction model, and they are in larger and larger scale. Learning of specific feature encoding model not only influenced by the structure of the network, but also by the learning manners such as supervised learning and unsupervised learning. Previous works concentrated on more comprehensive encoders and more delicate designs of feature extractors. However, the mutual influence factors from the neighbour pedestrians associate with the distance to the centre pedestrian seldomly noticed. Most of the existed feature extractors in prediction models trained in the way of supervised learning other than unsupervised manners caused the problem that the extracted features are always handcrafted without the natural distinction of obscure situations. The graph contrastive accelerating encoder is proposed, which accelerates the pedestrian trajectory prediction training process of the state of the art method of spatio‐temporal graph transformer networks. Employing the unsupervised contrastive learning process and the graph of neighbours representing distance affection of nearest and farthest pedestrian to the centre pedestrian, the graph contrastive accelerating encoder significantly shrinked the training time. Holding the final performance on to state of the art level, the proposed method let the lowest pedestrian trajectory prediction error show up in the obviously earlier training steps.

Abstract In the field of target detecting, lots of progress have been made in recent years. Owing to the progress of multiple frames time series data, or video satellites, target detecting from space‐borne satellite videos has been available. However, detecting a slightly moving target from space‐borne videos is still a difficult task, because of the low resolution and illumination variation influence. This paper considers target detecting from time series data as multi‐label problem as there are several different kinds of background objects and targets of interest. To some extent the background of time series data is comparative invariant, using background analysis method to extract the target from the background is promising. This paper proposes a novel target detecting algorithm based on multi‐label learning and Gaussian background description model aiming at extracting slowly moving target. To further enhance performances, multi‐frame fusion and post processing method was utilized to catch the slight difference due to movement. Experimental results on real world datasets indicate that the proposed method outperforms some state‐of‐the‐art algorithms.

Abstract In this paper, a deep learning method for video‐based action recognition is proposed. On the one hand, boundary compensation on the basis of a deep neural network is performed to achieve action proposal. Boundary compensation considering non‐maximum suppression according to sliding window priority is applied to remove redundant windows. To accurately detect boundaries, a boundary compensation network is established with multiple networks to process different numbers of segments. On the other hand, action recognition based on the resultant action proposals is performed. To further utilise boundary compensation, three methods are introduced for key frame selection. Optical flow and RGB features are combined via a channel fusion to realise feature representation. A two‐stream network with a spatiotemporal structure is adopted for action recognition. The proposed method is evaluated on three public datasets. The experimental results demonstrate that the proposed method achieves a superior performance to that of state‐of‐the‐art methods.

Abstract As one of the hot topics in the field of computer vision research, face recognition technology has received significant attention due to its potentiality for a wide range of applications in government as well as commercial purposes. In practical applications, although several existing face recognition methods have achieved good performances in specific scenes, they easily suffer from a sharp decline in recognition rate if affected by different conditions of light, expression, posture and occlusion. Among many factors, influences of complex illuminations on face recognition are particularly significant. To further improve the performance of the existing local binary pattern (LBP) operator, neighbourhood weighted average LBP (NWALBP) is first proposed for fully considering the strong correlations between pixel pairs in the neighbourhood, which extends the traditional LBP uni‐layer neighbourhood template window to the bi‐layer neighbourhood template window and calculates the weighted average of bi‐layer neighbourhood pixels in each direction. Then, inspired by center symmetric LBP (CS‐LBP), centre symmetric NWALBP (CS‐NWALBP) is further proposed, which can effectively reduce computation complexity by only comparing the weighted average values of the neighbourhood pixels that are symmetric about the centre pixel. Finally, by combining the merit of histogram of oriented gradient (HOG), a feature fusion algorithm named CS‐NWALBP+HOG is suggested. Several experiments have eventually demonstrated that our proposed algorithms have more robust performance under complex illumination conditions if compared with many other latest algorithms.

Abstract Cross‐domain action recognition is a challenging vision task due to the domain shift and the absence of labeled data in the target domain. With only labelled source domain and unlabelled target domain data during training, some existing methods rely on an adversarial framework to align the features from different domains to a common latent space. However, the existing adversarial‐based approaches have a major limitation of only attempting to perform the alignment from a holistic view, ignoring the underlying coherence of class structure across domains. A class structure‐aware adversarial loss (CSCAL) is presented to address this issue. The CSCAL incorporates the category information into the adversarial learning branch to capture the fine‐grained alignment of each class, effectively avoiding the false mixup of samples from different categories in the embedding space. Experiments on HMDB51, UCF101 and Olympic Sports datasets show significant improvement compared to the baseline. Code and trained model can be found at https://github.com/bregmangh/CSCAL.