Search

Abstract The goal of super-resolution (SR) is to recover a high-resolution (HR) image from its corresponding low-resolution (LR) image. It is an ill-posed problem. Most recent methods are based on external training data. They can reconstruct pleasing HR results, especially when the input patch has a similar counterpart within the training dataset. Other methods are driven by self-similarity and are called internal methods. They can produce visually plausible HR images when the input images contain abundant regular structures. In this paper, we propose a hybrid method for image SR that exploits the complementary advantages of external and internal SR methods. Each input LR patch is first super-resolved using convolutional neural network (CNN) for external SR and self-similarity for internal SR. Then, we calculate the perceptual similarity between the feature representations from the pre-trained VGG network to learn an adaptive weight. Finally, our algorithm automatically selects the optimal method on the basis of the calculated adaptive weight. The experimental results of our visual and quantitative evaluations verify the effectiveness of the proposed method, by comparing it with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Abstract Collective activity recognition, which analyses the behavior of groups of people in videos, is an important goal of video surveillance systems. In this paper, we focus on collective activity recognition problem and propose a new multi-stream convolutional neural network architecture that utilizes information extracted from multiple regions. The proposed method is the first work that uses a multi-stream network and multiple regions in this problem. Various strategies to fuse multiple spatial and temporal streams are explored. We evaluate the proposed method on two benchmark datasets, the Collective Activity Dataset and the Volleyball Dataset. Our experimental results show that the proposed method improves collective activity recognition performance when compared to the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Abstract The rapid development of social media services has spawned abundant user generated contents (UGC), such as Sina Weibo, which is one of the biggest Chinese microblogging platforms. In order to enhance the quality and popularity of the posted weibo (the microblog), Weibo users usually embed some social information and images or micro-videos, namely the multimodal weibo, and we assume that there is a close correlation among the multimodal weibo data, especially between the visual data (image/micro-video) and its corresponding text, for a multimodal weibo of high quality. Hence, we try to evaluate the quality of multimodal weibo via analyzing the correlation in the multimodal weibo. This paper constructs the classification model based on back propagation (BP) neural network with genetic algorithm (GA), to automatically identify the correlation within the multimodal weibo, and investigates three kinds of features from multimodal weibo to uncover their contributions to the correlation. The experimental results verify the superiority of the GA-BP based classification model over the traditional BP neural network. [ABSTRACT FROM AUTHOR]

Abstract How to correctly recognize and extract named entities, such as disease names, medical measurements and therapies, from online medical diagnosis data remains challenging. For the one hand, conventional natural language processing methods cannot be directly applied in the field of online medical diagnosis (OMD). Although existing supervised or unsupervised learning algorithms have offered strategies for OMD on open websites, such methods might extensively rely on specific knowledge sources or manually designed features. For the other hand, due to the large size of the data and the sophistication of the data structure, it is difficult to establish a robust NLP model for recognition and extraction of clinical named entities in paragraphs. Therefore, in the paper, we try to establish a new deep neural network (DNN), combine the bi-directional long short-term memory (Bi-LSTM) and the conditional random field (CRF), and utilize online medical diagnosis data for recognition and extraction of clinical named entities. Different from existing artificial neural networks (ANN), the proposed neural network well functions even without manual rules or features. The word representations input into the DNN is connected by character-based representations and continuous bags of word clusters (CBOWC) in an embedding way. We test the new DNN on different online medical diagnosis datasets obtained from a scalable web crawler, and compare it with the long short-term memory (LSTM) neural network linguistic model, the convolutional neural network (CNN) model and the most advanced Bi-LSTM-CRF. The fundamentals for the method selection is that they have been suggested to generate acceptable results. According to the comparison analyses, the DNN is proved to be a reliable tool and improves every benchmark performance of OMD. [ABSTRACT FROM AUTHOR]

Abstract Prediction of visual saliency in images and video is needed for video understanding, search and retrieval, coding, watermarking and other applications. The majority of prediction models are founded only on "bottom-up" features. Nevertheless, the "top-down" component of human visual attention becomes prevalent as human observers explore the visual scene. Visual saliency which is always a mix of bottom-up and top-down cues can be predicted on the basis of seen data. In this paper, a model of prediction of visual saliency in video on the basis of Deep convolutional neural networks (CNNs) is proposed. A Deep CNN architecture is designed. Various input channels for a CNN architecture are studied: using the known sensitivity of human visual system to residual motion, pixel colour values are completed with residual motion map. The latter is a normalized energy of residual motion in video frames with regard to the estimated global affine motion model. The experiments show that the choice of the input features for the Deep CNN depends on visual task: for highly dynamic content, the proposed model with residual motion is more efficient and gives decent results with relatively shallow Deep architecture. [ABSTRACT FROM AUTHOR]

Highlights • The first to embed features in a hypersphere manifold for person re-identification. • A new simple but efficient Sphere Loss and SphereReID network. • New balanced sampling strategy and warming-up strategy to boost the training. • We achieve state-of-the-art performance on four popular ReID datasets. • Our source code will be publicly released. Abstract Many current successful Person Re-Identification (ReID) methods train a model with the softmax loss function to classify images of different persons and obtain the feature vectors at the same time. However, the underlying feature embedding space is ignored. In this paper, we use a modified softmax function, termed Sphere Softmax, to solve the classification problem and learn a hypersphere manifold embedding simultaneously. A balanced sampling strategy is also introduced. Finally, we propose a convolutional neural network called SphereReID adopting Sphere Softmax and training a single model end-to-end with a new warming-up learning rate schedule on four challenging datasets including Market-1501, DukeMTMC-reID, CHHK-03, and CUHK-SYSU. Experimental results demonstrate that this single model outperforms the state-of-the-art methods on all four datasets without fine-tuning or re-ranking. For example, it achieves 94.4% rank-1 accuracy on Market-1501 and 83.9% rank-1 accuracy on DukeMTMC-reID. The code and trained weights of our model will be released. [ABSTRACT FROM AUTHOR]

Abstract Traditional saliency model usually utilize handcrafted image features and various prior knowledge to pop out salient regions from complex surroundings. In this paper, we propose a novel FCN-like deep convolutional neural network for pixel-wise salient object detection. Our deep network automatically learns multi-level feature from different convolutional layers of a pre-trained convolutional neural network. Moreover, deeper side outputs are connected to the shallower ones, which provides a better feature representation and helps shallow side outputs to accurately locate salient regions. In addition, we adopt a weighted-fusion module to combine different side outputs for utilizing multi-scale and multi-level features. Finally, a fully connected CRF model can be optimally incorporated to improve spatial coherence and contour localization in the fused saliency map. Both qualitative and quantitative evaluations on four publicly available datasets demonstrate the robustness and efficiency of our proposed approach against 17 state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Abstract Online payment is becoming popular due to the development of e-commerce. So payment safety is more important. Since there is a fraudulent situation, an anti-fraud system is indispensable. GMM was leveraged in many anti-fraud applications, but it only takes positive sample into account. Convolutional neural network is a strong strategy for learning deep representation of samples. So in this paper, we propose a CNNs architecture to deal with this problem. And the distance metric method can effectively identify whether candidates are the same person. Experimental results show the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Abstract Optical coherence tomography (OCT) can achieve the high-resolution 3D tomography imaging of the retina, which is crucial for the diagnosis of retinal diseases. Currently, the classification of retinal OCT images is mainly conducted by ophthalmologists, which is time consuming and subjective. In this paper, we propose an iterative fusion convolutional neural network (IFCNN) method for the automatic classification of retinal OCT image. In the convolutional neural network (CNN), different convolutional layers contain feature information from different scales. Therefore, the proposed network adopts an iterative fusion strategy, which iteratively combines features in current convolutional layer with those of all previous layers in the CNN network, and thus can jointly utilize the features of different convolutional layers to achieve accurate classification of OCT images. Experimental results on a real retinal OCT dataset and a musculoskeletal radiographs dataset demonstrate the superiority of the proposed method over the traditional CNN and several well-known OCT classification methods. [ABSTRACT FROM AUTHOR]

Abstract With the development of information transmission technology and computer technology, information acquisition mode is mainly converted from character to image nowadays. However, in the process of acquiring and transmitting images, image damage and quality decrease due to various factors. Therefore, how to restore image has become a research hotspot in the field of image processing. This paper establishes an image restoration model based on BP neural network. The simulation results show that the proposed method has made a great improvement compared with the traditional image restoration method. [ABSTRACT FROM AUTHOR]

Abstract Predicting the downside risk of a hedge fund is the foundation of risk measurement. These predictions also provide conditions that can be used for designing and implementing risk prevention measures. Hence, this paper proposes a big data hedge fund downside risk evaluation model based on a multi-objective neural network. First, two evaluation indexes are defined. Then, local search is applied to merge parent and descendant populations. Only those individuals from the Pareto front are optimized. Experimental results suggest that this model and method is feasible and valid. Specifically, the VaR model is unable to estimate the possible extreme risk of a hedge fund. In contrast, the CVaR model can accurately measure the risks under extreme market conditions. However, a combination of VaR and CVaR can help a fund manager avoid extreme risks to a hedge fund. [ABSTRACT FROM AUTHOR]

Abstract Facial expression recognition (FER) plays an important role in the applications of human computer interaction. Given the wide use of convolutional neural networks (CNNs) in automatic video and image classification systems, higher-level features can be automatically learned from hierarchical neural networks with big data. However, learning CNNs require large amount of training data for adequate generalization, while the Scale-invariant feature transform (SIFT) does not need large training samples to generate useful feature. In this paper, we propose a new hybrid feature representation for the recognition of facial expressions from a single image frame that uses a combination of SIFT and deep-learning feature of different level extracted from the CNN model, then adopt the combined features and classify the expression by support vector machines (SVM). The performance of the proposed method has been validated on public CK+ databases. To evaluate the generalization ability of our method, we also performed an experiment on a cross-database environment. Experimental results show that the proposed approach can achieve better classification rates compared with state-of-art CNN methods, which indicate the considerable potential of combining shallow feature with deep-learning feature. [ABSTRACT FROM AUTHOR]

Highlights • Propose a new super-resolution method based on CSC and CNN. • Use CNN to reconstruct the high resolution image for the low-frequency part. • Propose an ACSC to reconstruct high resolution image for high-frequency part. • Integrate and develope the advantages of the CSC method and CNN. Abstract The convolutional sparse coding-based super-resolution (CSC-SR) method has shown its good performance in single image super-resolution. It divides the low-resolution (LR) image into low-frequency part and the high-frequency part, and reconstructs their corresponding high-resolution (HR) image with bicubic interpolation and convolutional sparse coding (CSC) method, respectively. This paper is devoted to improve the performance of CSC-SR method. As convolutional neural network (CNN) can reveal the mapping relation between the LR image and the HR image for the low-frequency part better, we replace the bicubic interpolation with CNN to reconstruct the HR image for the low-frequency part. In addition, we propose an adaptive CSC method to reconstruct the HR image for the high-frequency part. We name our proposed super-resolution method as hybrid adaptive convolutional sparse coding-based super-resolution (HACSC-SR) method. Many comparison experiments illustrate that our proposed HACSC-SR method is superior to CSC-SR, CNN as well as several existing super-resolution methods. [ABSTRACT FROM AUTHOR]

In this paper, we address the challenging problem of spatial and temporal action detection in videos. We first develop an effective approach to localize frame-level action regions through integrating static and kinematic information by the early- and late-fusion detection scheme. With the intention of exploring important temporal connections among the detected action regions, we propose a tracking-by-point-matching algorithm to stitch the discrete action regions into a continuous spatio-temporal action tube. Recurrent 3D convolutional neural network is used to predict action categories and determine temporal boundaries of the generated tubes. We then introduce an action footprint map to refine the candidate tubes based on the action-specific spatial characteristics preserved in the convolutional layers of R3DCNN. In the extensive experiments, our method achieves superior detection results on the three public benchmark datasets: UCFSports, J-HMDB and UCF101. [ABSTRACT FROM AUTHOR]

MR technique is prevalent for doctor to diagnose and assess glioblastomas which are the most lethal form of brain tumors. Although Convolutional Neural Networks (CNN) has been applied in automatic brain tumor segmentation and is proved useful and efficient, traditional one-pathway CNN architecture with convolutional layers and max pooling layers has limited receptive fields representing the local context information. Such mindset in traditional CNN may dismiss useful global context information. In this paper, we design a two-pathway model with average and max pooling layers in different paths. Besides, 1 × 1 kernels are followed input layers to add the non-linearity dimensions of input data. Finally, we combine the CNN architecture with fully connected CRF(FCRF) as a mixture model to introduce the global context information to optimize prediction results. Our experiments proved that the mixture model improved segmentation and labeling accuracy. [ABSTRACT FROM AUTHOR]

Highlights • Inspired by the inter generative mechanism in the brain, we propose to use deep convolutional neural network to extract the high-level semantic information of stereoscopic images. • Considering the binocular visual characteristics, we propose to mimic the interaction process by fusing the high-level semantic features of left and right views. Abstract In this paper, we propose a no-reference (NR) quality assessment method for stereoscopic images by deep convolutional neural network (DCNN). Inspired by the internal generative mechanism (IGM) in the human brain, which shows that the brain first analyzes the perceptual information and then extract effective visual information. Meanwhile, in order to simulate the inner interaction process in the human visual system (HVS) when perceiving the visual quality of stereoscopic images, we construct a two-channel DCNN to evaluate the visual quality of stereoscopic images. First, we design a Siamese Network to extract high-level semantic features of left- and right-view images for simulating the process of information extraction in the brain. Second, to imitate the information interaction process in the HVS, we combine the high-level features of left- and right-view images by convolutional operations. Finally, the information after interactive processing is used to estimate the visual quality of stereoscopic image. Experimental results show that the proposed method can estimate the visual quality of stereoscopic images accurately, which also demonstrate the effectiveness of the proposed two-channel convolutional neural network in simulating the perception mechanism in the HVS. [ABSTRACT FROM AUTHOR]

Highlights • A novel semantic segmentation framework is proposed, in which GAN is adopted to capture the long-range information and reinforce spatial contiguity in the output label maps. • The Wasserstein distance is introduced in our GAN system to stabilize the training of GANs and to promote the performance of semantic segmentation. • Numerous experiments are conducted on five publicly available datasets to demonstrate the effectiveness of the proposed method. Abstract Semantic segmentation plays an important role in a series of high-level computer vision applications. In the state-of-the-art semantic segmentation methods based on fully convolutional neural networks, all label variables are predicted independently from each other, and the restricted field-of-views of the convolutional filters are difficult to capture the long-range information. In this paper, a novel post-processing method based on GAN (Generative Adversarial Network) is explored to reinforce spatial contiguity in the output label maps. With the help of fully connected layers in the discriminator, the GAN can capture the long-range information, and provide an auxiliary higher-order potential loss to the segmentation model, thus the segmentation model has the ability of correcting higher order inconsistencies. Furthermore, the optimization scheme in Wasserstein GAN (WGAN) is adopted to the training process of our model to get better performance and stability. Extensive experiments on public benchmarking database demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Graphical abstract Highlights • A powerful network called Asymmetric Filtering-based Dense Convolutional Neural Network (AF D-CNN) addresses person re-identification problem. • Joint Bayesian and re-ranking techniques are used to obtain the ranking lists which do not need dimensionality reduction. • Extensive experiments conducted on several datasets show that the proposed method achieves the state-of-the-art performance. Abstract Person re-identification aims at matching individuals across multiple camera views under surveillance systems. The major challenges lie in the lack of spatial and temporal cues, which makes it difficult to cope with large variations of lighting conditions, viewing angles, body poses and occlusions. How to extract multimodal features including facial features, physical features, behavioral features, color features, etc is still a fundamental problem in person re-identification. In this paper, we propose a novel Convolutional Neural Network, called Asymmetric Filtering-based Dense Convolutional Neural Network (AF D-CNN) to learn powerful features, which can extract different levels' features and take advantage of identity information. Moreover, instead of using typical metric learning methods, we obtain the ranking lists by merging Joint Bayesian and re-ranking techniques which do not need dimensionality reduction. Finally, extensive experiments show that our proposed architecture performs well on four popular benchmark datasets (CUHK01, CUHK03, Market-1501, DukeMTMC-reID). [ABSTRACT FROM AUTHOR]

Highlights • Deep multi-scale intensity and depth feature fusion scheme increases model capacity. • Modified edge preserving loss function can protect the edge information in depth map. • Theoretical time complexity of the proposed DIG-Net is only 50% of IG-Net. Abstract In this paper, we propose an deep intensity guidance based compression artifacts reduction model (denoted as DIG-Net) for depth map. The proposed DIG-Net model can learn an end-to-end mapping from the color image and distorted depth map to the uncompressed depth map. To eliminate undesired artifacts such as discontinuities around object boundary, the proposed model is with three branches, which extracts the high frequency information from color image and depth maps as priors. Based on the modified edge preserving loss function, the deep multi-scale guidance information are learned and fused in the model to make the edge of depth map sharper. Experimental results show the effectiveness and superiority of our proposed model compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Abstract Pedestrian tracking technique is now widely used in many intelligent systems, such as video surveillance, security regions. But many methods suffer from illumination, human posture or human appendant. With the development of Convolutional Neural Networks (CNNs), deep feature can be learned. In this paper, training images will be divided into subregions to reduce the influence of human appendant, such as bags. The remain regions are almost fixed regions. Then these fixed regions will be fed into our CNNs for learning deep features. In order to copy with different sizes of training images, an arbitrarily-sized pooling layer is developed in our CNN architecture. Then, these deeply-learned feature vector can be used in pedestrian recognition. In our work, optical flow is used for pedestrian tracking. Experimental results show our proposed method can achieve pedestrian tracking effectively. [ABSTRACT FROM AUTHOR]

Highlights • An edge detection network architecture based on deep convolutional network is presented. • A novel loss function is proposed to balance positive and negative samples. • The fast learning model is discovered (only 5.4k iteration) for edge detection. • The better generalization ability is verified at different dataset. Abstract In this paper, we propose an edge detector based on feature re-extraction (FRE) of a deep convolutional neural network to effectively utilize features extracted from each stage, and design a new loss function. The proposed detector is mainly composed of three modules: backbone, side-output, and feature fusion. The backbone module provides preliminary feature extraction; the side-output module makes network architecture more robustly map features from different stages of the backbone network to edge-pixel space by applying residual learning, and the feature fusion module generates the edge map. Generalization ability on the same distribution is verified using the BSDS500 dataset, achieving optimal dataset scale (ODS) F-score = 0.804. Cross-distribution generalization ability is verified on the NYUDv2 dataset, achieving ODS F-score = 0.701. In addition, we find that freezing backbone network can significantly speed up training process, without much overall accuracy loss (ODS F-score of 0.791 after 5.4k iterations). [ABSTRACT FROM AUTHOR]

Abstract A large mount of data is indispensable in deep learning. The learning results can be different because of the noise or contaminated tags. So in this paper, a controller design method is proposed to reduce the influence due to noise or damaged label. Our method is based on backstepping control method and observer. In our work, an adaptive function is designed to eliminate the influence of the unmodelable part of the system because of the contaminated tags. For the noise, the observer is used to accurately estimated and effectively compensated. Experimental results show the effectiveness of our method. Our modified system has good performance and can accurately response the input training data in the case of the unmodelable part of the system and the external noise. [ABSTRACT FROM AUTHOR]

Highlights • Natural scene camera trap object segmentation. • Near real-time human/animal detection on CPU machines. • Region proposals extraction through block-wise background modeling. • Spatial image patch verification to reduce the number of false positive regions. In this paper, we couple effective dynamic background modeling with fast deep learning classification to develop an accurate scheme for human-animal detection from camera-trap images with cluttered moving objects. We introduce a new block-wise background model, named as Minimum Feature Difference (MFD), to model the variation of the background of the camera-trap sequences and generate the foreground object proposals. We then develop a region proposals verification to reduce the number of false alarms. Finally, we perform complexity-accuracy analysis of DCNN to construct a fast deep learning classification scheme to classify these region proposals into three categories: human , animals , and background patches. The optimized DCNN is able to maintain high level of accuracy while reducing the computational complexity by 14 times, which allows near real-time implementation of the proposed method on CPU machines. Our experimental results demonstrate that the proposed method outperforms existing methods on our and Alexander von Humboldt Institute camera-trap datasets in both foreground segmentation and object detection. [ABSTRACT FROM AUTHOR]

The large variations in the angle of different camera views and illumination can change the appearance of a lot of people, which makes human re identification is still a challenging problem. Therefore, the development of robust feature descriptors and the design of discriminative distance metrics to measure similarity between pedestrian images are two key aspects of human re identification. In this paper, we propose a method to improve the performance of the re identification using depth learning and multiple metric ensembles. First, we use a variety of data sets to train the general convolutional neural network (CNN), which is used to extract the features of the training and test set after deep level. Deep architecture makes it possible for people to learn more abstract and internal features that are robust to changes in viewpoint and illumination. Then, we utilize the deep features of the training set to learn a specific distance metric and combine it with the cosine distance metric. Multi metric sets can be used to measure the similarity between different images. Finally, a large number of experiments show that our method can effectively improve the recognition performance compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In recent years, with the raise of the neural network and deep learning, significant progress has been achieved in the field of image recognition. Convolutional Neural Network (CNN) has been widely used in multiple image recognition tasks, but the recognition accuracy still has a lot of room for improvement. In this paper, we proposed a hybrid model CNN-GRNN to improve recognition accuracy. The model uses CNN to extract multilayer image representation and it uses General Regression Neural Network (GRNN) to classify image using the extracted feature. The CNN-GRNN model replace Back propagation (BP) neural network inside CNN with GRNN to improve generalization and robustness of CNN. Furthermore, we validate our model on the Oxford-IIIT Pet Dataset database and the Keck Gesture Dataset, the experiment result indicate that our model is superior to Gray Level Co-occurrency (GLCM),HU invariant moments, CNN and CNN_SVM on small sample dataset. Our model has favorable real-time characteristic at the same time. [ABSTRACT FROM AUTHOR]

Large scale scene classification based on image is an important problem in computer vision. In this paper, we propose a method to fuse the local features of scene images into a geometric feature that can reflect both the geometric features of the scene image and the color intensity distribution. First, each scene image is segmented into a set of individually connected regions according to their color intensity distribution. A region adjacency graph is constructed to encode the geometric properties and color intensity of scene images. Later, a 5 tier CNN architecture was constructed to study regional features. Then, a thinning process is carried out to obtain a discriminant and compact template set from the training rag. These templates are used to extract graphlets finished bag (r-bogs) images represented by each scene. Finally, the strategy of boosting development is to classify the extracted r-bogs scenes. Experimental results on different datasets demonstrate the effectiveness and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Highlights • An elaborately designed neural network is proposed for facial expression recognition. • A joint loss is developed to effectively learn high discriminability features. • Two kinds of weights are introduced to balance the joint loss for better optimization. • Experimental results show the superior performance of our proposed method. In this paper, we propose a novel expression-targeted feature learning (ETFL) method for effective facial expression recognition, which takes advantage of multi-task learning for discriminative feature learning. Specifically, the common features are firstly extracted from the lower layers of CNN. Then, based on the common features, the expression-specific features (ESF) are respectively learned for each facial expression via multi-task learning. In order to enhance the discriminability of ESF, we develop a joint loss (the combination of the center loss and a novel inter-class loss) to explicitly reduce intra-class variations while enlarging inter-class differences. Furthermore, we introduce the sample-sensitive weights and the soft-expression weights to balance the joint loss for better performance. Finally, all ESFs are combined for final classification. ETFL effectively exploits the relationship among all facial expressions, which leads to superiority feature discriminability. Experiments on public facial expression databases demonstrate the effectiveness of ETFL compared with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • We integrate a hybrid of CNN and hand-crafted features to learn a correlation filter. • We incorporate a re-detector to re-initialise the tracker in case of tracking failure. • We include a GM-PHD filter to temporally filter detection proposals from online SVM. • We learn a scale correlation filter to estimate the scale of the detected target. Tracking a target of interest in both sparse and crowded environments is a challenging problem, not yet successfully addressed in the literature. In this paper, we propose a new long-term visual tracking algorithm, learning discriminative correlation filters and using an online classifier, to track a target of interest in both sparse and crowded video sequences. First, we learn a translation correlation filter using a multi-layer hybrid of convolutional neural networks (CNN) and traditional hand-crafted features. Second, we include a re-detection module for overcoming tracking failures due to long-term occlusions using online SVM and Gaussian mixture probability hypothesis density (GM-PHD) filter. Finally, we learn a scale correlation filter for estimating the scale of a target by constructing a target pyramid around the estimated or re-detected position using the HOG features. We carry out extensive experiments on both sparse and dense data sets which show that our method significantly outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • The hand-crafted features and the deep features were combined into a deep network. • The importance of different body parts were revealed by the gait descriptor. • The body parts were described in more details after updating data with weights. Gait recognition is an important issue currently. In this paper, we propose to combine deep features and hand-crafted representations into a globally trainable deep model. Specifically, a set of deep feature vectors are firstly extracted by a pre-trained CNN model from the input sequences. Then, a kernel function with respect to the fully connected vector is trained as the guiding weight of the respective receptive fields of the input sequences. Therefore, the hand-crafted features are extracted based on the guiding weight. Finally, the hand-crafted features and the deep features are combined into a unified deep network to complete classification. The optimized gait descriptor, termed as deep convolutional location weight descriptor (DLWD), is capable of effectively revealing the importance of different body parts to gait recognition accuracy. Experiments on two gait data sets (i.e., CASIA-B, OU-ISIR) show that our method outperforms the other existing methods for gait recognition. [ABSTRACT FROM AUTHOR]

Graphical abstract Highlights • A feature visualization for CNNs method is exploited to improve co-segmentation. • Visualization is used as an auxiliary information to discriminate salient regions. • The visualization information is used as an extra energy term in the cost function. • Region occlusion sensitivity is introduced for feature visualization. • An adaptive strategy is designed to detect correct foreground/background regions. This paper addresses the co-segmentation problem using feature visualization for CNNs. Visualization is exploited as an auxiliary information to discriminate salient image regions (dubbed as “heat-regions”) from non-salient ones. Region occlusion sensitivity is proposed for feature visualization. The co-segmentation problem is formulated via a convex quadratic optimization which is initialized by the heat-regions. The information obtained through the visualization is considered as an extra energy term in the cost function. The results of the visualization demonstrate that there exist some heat-regions which are not productive in the co-segmentation. To detect helpful regions among them, an adaptive strategy in the form of an iterative algorithm is proposed according to the consistency among all images. Comparison experiments conducted on two benchmark datasets, iCoseg and MSRC, illustrate the superior performance of the proposed approach over state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have dominated the field of computer vision for nearly a decade. However, due to their limited receptive field, CNNs fail to model the global context. On the other hand, transformers, an attention-based architecture, can model the global context easily. Despite this, there are limited studies that investigate the effectiveness of transformers in crowd counting. In addition, the majority of the existing crowd-counting methods are based on the regression of density maps which requires point-level annotation of each person present in the scene. This annotation task is laborious and also error-prone. This has led to an increased focus on weakly-supervised crowd-counting methods, which require only count-level annotations. In this paper, we propose a weakly-supervised method for crowd counting using a pyramid vision transformer. We have conducted extensive evaluations to validate the effectiveness of the proposed method. Our method achieves state-of-the-art performance. More importantly, it shows remarkable generalizability. • A pyramid vision transformer-based weakly-supervised crowd counting method. • An MLP based multiscale feature aggregation and regression module. • New benchmarks for cross-dataset performance in crowd counting. • A novel end-to-end attention-based crowd-counting method employing smooth- L 1 loss. [ABSTRACT FROM AUTHOR]

With the continuous development of deep learning, neural networks have made great progress in license plate recognition (LPR). Nevertheless, there is still room to improve the performance of license plate recognition for low-resolution and relatively blurry images in remote surveillance scenarios. When it is difficult to enhance the recognition algorithm, we choose super-resolution (SR) to improve the quality of license plate images and thereby provide clearer input for the subsequent recognition stage. In this paper, we propose an automatic super-resolution license plate recognition (SRLPR) network which consists of four parts separately: license plate detection, character detection, single character super-resolution, and recognition. In the training stage, firstly, LP detection model needs to be trained alone and then its detection results will be used to successively train the three subsequent modules. During the test phase, for each input image, the network can get its LP number automatically. We also collect an applicable and challenging LPR dataset called SRLP, which is collected from real remote traffic surveillance. The experimental results demonstrate that our method achieves comprehensive quality of SR images and higher recognition accuracy compared with state-of-the-art methods. The SRLP dataset and the code for training and testing SRLPR network are available at https://pan.baidu.com/s/1vnhRa-c-dBj6jlfBZV5w4g. • An automatic Super-Resolution License Plate Recognition (SRLPR) method is proposed. • DBAGAN is proposed for single character super-resolution in the SRLPR framework. • A challenging SRLP dataset is collected for LPR in remote surveillance. • Experimental results demonstrate the effectiveness of our SRLPR. [ABSTRACT FROM AUTHOR]

Neural network pruning techniques can be effective in accelerating neural network models, making it possible to deploy them on edge devices. In this paper, we propose to prune neural networks using data variance. Unlike other existing methods, this method is somewhat robust and does not invalidate our criteria depending on the number of data batches and the number of training sessions. We also propose a pruning compensation technique. This technique fuses the pruned convolutional information into the remaining convolutional kernel close to it. This fusion operation can effectively help retain the pruned information. We evaluate the proposed method on a number of standard datasets and compare it with several current state-of-the-art methods. Our method always achieves better performance. For example, on Tiny ImageNet, our method can prune 54.2% FLOPs of ResNet50 while obtaining a 0.22% accuracy improvement. • A data-driven network pruning method with high robustness is proposed. • The proposed data-driven pruning method consumes less resources and is highly efficient. • A pruning compensation technique is proposed to reduce the negative impact of pruning. • The effectiveness of the proposed method is validated on several datasets. [ABSTRACT FROM AUTHOR]

The outstanding performance of convolutional neural networks (CNNs) shown in single image super-resolution (SISR) strongly depends on network's depth, which hampers its application in low-power computing devices. In this paper, a lightweight and efficient network (LESR) is proposed for SISR by constructing the shallow feature extraction block (SFBlock), the cascaded sparse mask blocks (SMBlocks) and the feature fusion block (FFBlock). The SFBlock efficiently extracts global informative features from the original low resolution image using sparse self-attention, SMBlock skips the redundant computation in extracted features, and more meaningful information is distilled for the sequential reconstruction block by the FFBlock. In addition, a recently proposed activation function called ACON-C is used to replace the ReLU function to ease the training difficulty. Extensive experiments show that our proposed network performs better than most advanced lightweight SISR algorithms with comparable parameters and less FLOPs on benchmark database for × 2 / 3 / 4 SISR. • A lightweight and efficient network for single image super-resolution is proposed. • It adopts sparse self-attention to extract global information from shallow layers. • It is spare mask blocks skips the redundant computation. • A new active function ACON-C is adopted to enhance nonlinear mapping ability. [ABSTRACT FROM AUTHOR]

• Rethinking the PASCAL-VOC and MS-COCO dataset for small object detection. • There are different labeling errors in these two benchmarks by visual analysis. • Building several specific datasets to analyze small object detection. • Data quality is another factor that affects the performance of small object detection. The data and the algorithm are critical to deep learning-based small object detectors. In this paper, we rethink the PASCAL-VOC and MS-COCO dataset for small object detection. By visual analysis of the original annotations, we find that there are different labeling errors in these two datasets. To solve these problems, we build specific datasets, including SDOD, Mini6K, Mini2022 and Mini6KClean. The experimental results of several typical algorithms (e.g. SSD, YOLOv5, Faster RCNN and Deformable DETR) on the datasets show that data labeling errors (such as missing labels, category label errors, inappropriate labels) are another factor that affects the detection performance of small objects. [ABSTRACT FROM AUTHOR]

In this paper, we propose an efficient and straightforward approach, video you only look once (VideoYOLO), to capture the overall temporal dynamics from an entire video in a single process for action recognition. It remains an open question for action recognition on how to deal with the temporal dimension in videos. Existing methods subdivide a whole video into either individual frames or short clips and consequently have to process these fractions multiple times. A post process is then used to aggregate the partial dynamic cues to implicitly infer the whole temporal information. On the contrary, in VideoYOLO, we first generate a proxy video by selecting a subset of frames to roughly reserve the overall temporal dynamics presented in the original video. A 3D convolutional neural network (3D-CNN) is employed to learn the overall temporal characteristics from the proxy video and predict action category in a single process. Our proposed method is extremely fast. VideoYOLO-32 is able to process 36 videos per second that is 10 times and 7 times faster than prior 2D-CNN (Two-stream (Simonyan and Zisserman, 2014)) and 3D-CNN (C3D (Tran et al., 2015)) based models, respectively, while still achieves superior or comparable classification accuracies on the benchmark datasets, UCF101 and HMDB51. [ABSTRACT FROM AUTHOR]

Vehicle counting in crowded urban setting plays a significant role in public security area. Most existing works on vehicle counting focused on video sequence. Though these techniques has achieved significant progress, it has a significant disadvantage: only moving vehicle could be counted. It is not realistic that vehicles are often stopped in most crowded cases, e.g. carpark and traffic-light intersections. To deal with this issue, in this paper, we propose a novel multi-channel and multi-task convolutional neural networks (CNN) to count vehicles from still images. More specially, we present a novel algorithm to produce illumination invariance image and combine it with original gray image as input channels, which could handle more details. And we deem vehicle counting as a local consistency deep regression problem, using a local label supervised deep CNN model to fit it. Moreover, we utilize surveillance camera view classification as a related task to improve the performance of vehicle counting task and the two tasks are trained end-to-end jointly. To evaluate the proposed model, we collect a real-work dataset for research and extensive experimental results show that the proposed method performs better than existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Weakly supervised semantic segmentation (WSSS) with only image-level supervision is a challenging task. Most existing methods exploit Class Activation Maps (CAM) to generate pixel-level pseudo labels for supervised training. However, due to the local receptive field of Convolution Neural Networks (CNN), CAM applied to CNNs often suffers from partial activation — highlighting the most discriminative part instead of the entire object area. In order to capture both local features and global representations, the Conformer has been proposed to combine a visual transformer branch with a CNN branch. In this paper, we propose TransCAM, a Conformer-based solution to WSSS that explicitly leverages the attention weights from the transformer branch of the Conformer to refine the CAM generated from the CNN branch. TransCAM is motivated by our observation that attention weights from shallow transformer blocks are able to capture low-level spatial feature similarities while attention weights from deep transformer blocks capture high-level semantic context. Despite its simplicity, TransCAM achieves competitive performance of 69.3% and 69.6% on the respective PASCAL VOC 2012 validation and test sets, showing the effectiveness of transformer attention-based refinement of CAM for WSSS. • We propose a transformer-based solution for Weakly Supervised Semantic Segmentation. • We utilize the attention weights from the transformer to refine the CAM. • We find different blocks' attention weights capture distinct feature affinities. • Our method is simple yet effective, showing competitive results on PASCAL VOC 2012. [ABSTRACT FROM AUTHOR]

To solve the problem of low sign language recognition rate under the condition of small samples, a simple and effective static gesture recognition method based on an attention mechanism is proposed. The method proposed in this paper can enhance the features of both the details and the subject of the gesture image. The input of the proposed method depends on the intermediate feature map generated by the original network. Also, the proposed convolutional model is a lightweight general module, which can be seamlessly integrated into any CNN(Convolutional Neural Network) architecture and achieve significant performance gains with minimal overhead. Experiments on two different datasets show that the proposed method is effective and can improve the accuracy of sign language recognition of the benchmark model, making its performance better than the existing methods. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach for low-light image enhancement. We propose a deep simultaneous estimation network (DSE-Net), which simultaneously estimates the reflectance and illumination for low-light image enhancement. The proposed network contains three modules: image decomposition, illumination adjustment, and image refinement module. The DSE-Net uses a novel branched encoder–decoder based image decomposition module for simultaneous estimation. The proposed decomposition module uses a separate decoder to estimate illumination and reflectance. DSE-Net improves the estimated illumination using the illumination adjustment module and feeds it to the proposed refinement module. The image refinement module aims to produce sharp and natural-looking output. Extensive experiments conducted on a range of low-light images demonstrate the efficacy of the proposed model and show its supremacy over various state-of-the-art alternatives. [ABSTRACT FROM AUTHOR]

• AMSFF-Net has ability to pay more attention to the informative features at different multi-stream resolution levels using pixel attention mechanism. • AMSFF-Net has ability to capture semantic information and sharp textural details from the extracted features and retain high-quality image from coarse-to-fine using mixed-convolution attention mechanism. • A sharp image can be recovered by the good kernel estimation. Mixed convolution attention mechanism highlights the task-relevant features and retain the dehazed images with sharp texture details and more vibrant colors. • Deep semantic loss highlights the boundary information and the spatial correlation in the deep features. In this paper, an end-to-end convolutional neural network is proposed to recover haze-free image named as Attention-Based Multi-Stream Feature Fusion Network (AMSFF-Net). The encoder-decoder network structure is used to construct the network. An encoder generates features at three resolution levels. The multi-stream features are extracted using residual dense blocks and fused by feature fusion blocks. AMSFF-Net has ability to pay more attention to informative features at different resolution levels using pixel attention mechanism. A sharp image can be recovered by the good kernel estimation. Further, AMSFF-Net has ability to capture semantic and sharp textural details from the extracted features and retain high-quality image from coarse-to-fine using mixed-convolution attention mechanism at decoder. The skip connections decrease the loss of image details from the larger receptive fields. Moreover, deep semantic loss function emphasizes more semantic information in deep features. Experimental findings prove that the proposed method outperforms in synthetic and real-world images. [ABSTRACT FROM AUTHOR]

In recent years, the research method of depth estimation of target images using Convolutional Neural Networks (CNN) has been widely recognized in the fields of artificial intelligence, scene understanding and three-dimensional (3D) reconstruction. The fusion of semantic segmentation information and depth estimation will further improve the quality of acquired depth images. However, how to deeply combine image semantic information with image depth information and use image edge information more accurately to improve the accuracy of depth image is still an urgent problem to be solved. For this purpose, we propose a novel depth estimation model based on semantic segmentation to estimate the depth of monocular images in this paper. Firstly, a shared parameter model of semantic segmentation information and depth estimation information is built, and the semantic segmentation information is used to guide depth acquisition in an auxiliary way. Then, through the multi-scale feature fusion module, the feature information contained in the neural network on different layers is fused, and the local feature information and global feature information are effectively used to generate high-resolution feature maps, so as to achieve the goal of improving the quality of depth image by optimizing the semantic segmentation model. The experimental results show that the model can fully extract and combine the image feature information, which improves the quality of monocular depth vision estimation. Compared with other advanced models, our model has certain advantages. [ABSTRACT FROM AUTHOR]

In the action recognition, a proper frame sampling method can not only reduce redundant video information, but also improve the accuracy of action recognition. In this paper, an action density based non-isometric frame sampling method, namely NFS, is proposed to discard the redundant video information and sample the rational frames in videos for neural networks to achieve great accuracy on human action recognition, in which action density is introduced in our method to indicate the intensity of actions in videos. Particularly, the action density determination mechanism, focused-clips division mechanism, and reinforcement learning based frame sampling (RLFS) mechanism are proposed in NFS method. Via the evaluations with various neural networks and datasets, our results show that the proposed NFS method can achieve great effectiveness in frame sampling and can assist in achieving better accuracy on action recognition in comparison with existing methods. [ABSTRACT FROM AUTHOR]

Image denoising requires both spatial details and global contextualized information to recover a clean version from the deteriorative one. Previous deep convolution networks usually focus on modeling the local feature and stacked convolution blocks to expand the receptive field, which can catch the long-distance dependencies. However, contrary to the expectation, the extracted local feature incapacity recovers the global details by traditional convolution while the stacked blocks hinder the information flow. To tackle these issues, we introduce the M atrix Factorization D enoising Module (MD) to model the interrelationship between the global context aggregating process and the reconstructed process to attain the context details. Besides, we redesign a new basic block to ease the information flow and maintain the network performance. In addition, we conceive the F eature Fu sion Module (FFU) to fuse the information from the different sources. Inspired by the multi-stage progressive restoration architecture, we adopt two-stage convolution branches progressively reconstructing the denoised image. In this paper, we propose an original and efficient neural convolution network dubbed MFU. Experimental results on various image denoising datasets: SIDD, DND, and synthetic Gaussian noise datasets show that our MFU can produce comparable visual quality and accuracy results with state-of-the-art methods. • We propose MFU, an effective and efficient end-to-end multi-stage progressive convolution network with a new basic block for image denoising. • We represent the global structured information as the low-rank matrix and use the MD module to learn the contextually-enriched and globally structured feature. • Designing a module (FFU) fuses the multi-source information to promote the network reconstruct the image. [ABSTRACT FROM AUTHOR]

Video compression is essential for uploading videos to online platforms which usually have bandwidth limitations. However, the compression reduces the visual quality. To overcome this problem, the visual quality of the low bitrate compressed videos for various standards, including H.264 and HEVC in decoders, needs to be improved. Accordingly, this paper proposes a novel method for improving video quality based on 3D convolutional neural networks (CNNs). This method is totally compatible with the encoders of video compression standards, i.e., H.264, VVC, and HEVC, and can be implemented easily. In particular, the proposed neural network model receives five frames of the low bitrate compressed video as input and subsequently predicts the compression error of frames using the first and fifth frames. Finally, it reconstructs an improved version of the frame with high quality. The CNN is an Additive (3D) model that can predict the eliminated inter-frame redundancies resulting from compression. Our goal is to increase the peak signal to noise ratio (PSNR) and structural index similarity (SSIM) of the luminance (Y) and chrominance (U, V) frames in the video. Additive 3D-CNN achieves an average of 12.4%, 9.9% and 5% BD-rate increases for LP, LB and RA for the Y component. The results indicate that the new proposed algorithm outperforms the previous methods in terms of PSNR, SSIM, and BD-rate. [ABSTRACT FROM AUTHOR]

Recently, Convolutional Neural Networks (CNNs) have achieved great success in Single Image Super-Resolution (SISR). In particular, the recursive networks are now widely used. However, existing recursion-based SISR networks can only make use of multi-scale features in a layer-wise manner. In this paper, a Deep Recursive Multi-Scale Feature Fusion Network (DRMSFFN) is proposed to address this issue. Specifically, we propose a Recursive Multi-Scale Feature Fusion Block (RMSFFB) to make full use of multi-scale features. Besides, a Progressive Feature Fusion (PFF) technique is proposed to take advantage of the hierarchical features from the RMSFFB in a global manner. At the reconstruction stage, we use a deconvolutional layer to upscale the feature maps to the desired size. Extensive experimental results on benchmark datasets demonstrate the superiority of the proposed DRMSFFN in comparison with the state-of-the-art methods in both quantitative and qualitative evaluations. [ABSTRACT FROM AUTHOR]

With the great success of deep neural networks, combining deep learning with traditional dictionary learning has become a hot issue. However, the performance of these methods is still limited for several reasons. First, some existing methods update dictionary learning and classifier as two independent modules, which limits the classification performance. Second, the non-attention dictionary is learned to represent all images, reducing the model representation flexibility. In this paper, we design a novel end-to-end model named Multi-layer Attention Dictionary Pair Learning Network (MADPL-net), which integrates the learning schemes of the convolutional neural network, deep encoder learning, and attention dictionary pair learning (ADicL) into a unified framework. The encoder layer contains the ADicL block, which selects more image-attentive atoms in the dictionary pair block via the softmax function to ensure MADPL-net classification capability. In addition, ADicL schema can yield discriminative dictionary atoms and feature maps with high inter-class separation and high intra-class compactness. To improve the sparse representation learning performance, MADPL-net adds l 1 − norm constraint of the analysis dictionary to the cross-entropy loss function. Extensive experiments show that MADPL-net can achieve excellent performance over other state-of-the-arts. • A powerful architecture, called the multi-layer attention dictionary pair learning network (MADPL-net), is proposed for image classification. • The MADPL-net integrates convolutional neural network, deep encoder learning, and attention dictionary pair learning into a unified framework. • The MADPL-net applies the attentional dictionary learning block to select more image-attentive atoms via the soft-max function. • The MADPL-net adds the l 1 − norm constraint of analysis dictionary to the crossentropy loss function. • The MADPL-net can ensure discriminative dictionary atoms and feature maps with high inter-class separation and high intra-class compactness. [ABSTRACT FROM AUTHOR]

• Channel attention model captures important channels for each feature map position. • Multi-scale features combined from multiple layers handle different person sizes. • Adaptive loss improves the accuracy of crowd counting for sparse crowd samples. Crowd counting algorithms have recently incorporated attention mechanisms into convolutional neural networks (CNNs) to achieve significant progress. The channel attention model (CAM), as a popular attention mechanism, calculates a set of probability weights to select important channel-wise feature responses. However, most CAMs roughly assign a weight to the entire channel-wise map, which makes useful and useless information being treat indiscriminately, thereby limiting the representational capacity of networks. In this paper, we propose a multi-scale and spatial position-based channel attention network (MS-SPCANet), which integrates spatial position-based channel attention models (SPCAMs) with multiple scales into a CNN. SPCAM assigns different channel attention weights to different positions of channel-wise maps to capture more informative features. Furthermore, an adaptive loss, which uses adaptive coefficients to combine density map loss and headcount loss, is constructed to improve network performance in sparse crowd scenes. Experimental results on four public datasets verify the superiority of the scheme. [ABSTRACT FROM AUTHOR]

As a framework of optical machine learning, all-optical diffractive neural network (D 2 NN) has delivered an ideal outcome of feature detection and target classification, currently raising high interest in the optics and photonics community. In this paper, we applied an improved D 2 NN architecture to the field of gesture recognition, which features more complicated contour than the common MNIST handwriting recognition in the previous literature. The proposed network structure incorporates the wavelet-like phase modulation pattern technique and the highway network on the basis of all-optical neural network. Through modulating the phase of incident light, the wavelet-like pattern can substantially reduce the parameters in the network layer. In addition, a highway network is employed to address the vanishing gradient phenomenon in the training process. In the experiment, we numerically achieved blind testing accuracy of 95.6% for identifying ten different gestures, and the number of parameters is only 3% of the regular D 2 NN. Reliability test and analysis show that the proposed method is a high-efficiency solution with low-parameters expecting for implementation of various machine learning tasks. • All-optical diffractive deep neural network (D2NN) for gesture recognition. • The wavelet-like phase modulation pattern for D2NN to reduce the parameters. • The highway network structure to address the vanishing gradient phenomenon. [ABSTRACT FROM AUTHOR]

Due to the copyright issues often involved in the recapture of LCD screen content, recaptured screen image identification has received lots of concerns in image source forensics. This paper analyzes the characteristics of convolutional neural network (CNN) and vision transformer (ViT) in extracting features and proposes a cascaded network structure that combines local-feature and global-feature extraction modules to detect the recaptured screen image from original images with or without demoiréing operation. We first extract the local features of the input images with five convolutional layers and feed the local features into the ViT to enhance the local perception capability of the ViT module, and further extract the global features of the input images. Through thorough experiments, our method achieves a detection accuracy rate of 0.9691 in our generated dataset and 0.9940 in the existing mixture dataset, both showing the best performance among the compared methods. [Display omitted] • First method to identify recaptured screen images with demoiréing operation. • CNN and ViT are set as the local-feature and global-feature extraction modules, respectively. • The attention mechanism and small receptive fields are used to extract global and small-scale moiré features. • Achieve a detection accuracy rate of 0.9691 in our generated demoiréing dataset. [ABSTRACT FROM AUTHOR]