Search

Effectively evaluating the perceptual quality of dehazed images remains an under-explored research issue. In this paper, we propose a no-reference complex-valued convolutional neural network (CV-CNN) model to conduct automatic dehazed image quality evaluation. Specifically, a novel CV-CNN is employed that exploits the advantages of complex-valued representations, achieving better generalization capability on perceptual feature learning than real-valued ones. To learn more discriminative features to analyze the perceptual quality of dehazed images, we design a dual-stream CV-CNN architecture. The dual-stream model comprises a distortion-sensitive stream that operates on the dehazed RGB image, and a haze-aware stream on a novel dark channel difference image. The distortion-sensitive stream accounts for perceptual distortion artifacts, while the haze-aware stream addresses the possible presence of residual haze. Experimental results on three publicly available dehazed image quality assessment (DQA) databases demonstrate the effectiveness and generalization of our proposed CV-CNN DQA model as compared to state-of-the-art no-reference image quality assessment algorithms. [ABSTRACT FROM AUTHOR]

Smoke detection in video is a challenging task because of the irregular shape of smoke, its complex motion state, which is affected by temperature, wind and other external factors, and background disturbances. Pixel-based foreground modeling method is a crucial step in many smoke detection systems and can be applied to efficiently focus on a certain object or a specific region to detect movements or anomalies. In video analysis, it is a natural idea to move the focus from the pixel-level foreground to the feature-level foreground. In this paper, the feature foreground is generated by the middle layer of a convolutional neural network (CNN) to guide the temporal modeling process for smoke objects. A novel temporal module called the Feature Foreground Module (FFM) is proposed to boost learning of a smoke temporal representation. Consider the problem of smoke analysis in video, we present a novel unifying approach, named an enhanced feature foreground network (EFFNet), that performs both smoke source prediction and detection. Efficient branch networks are designed in EFFNet, to predict the source mask and bounding boxes of smoke plumes in video. To the best of our knowledge, this is the first paper to study the source of smoke using deep learning methods. Finally, experiments on a realistic smoke dataset and a public dataset show that EFFNet method performs much better than do previous state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Channel attention mechanisms have been commonly applied in many visual tasks for effective performance improvement. It is able to reinforce the informative channels as well as to suppress the useless channels. Recently, different channel attention modules have been proposed and implemented in various ways. Generally speaking, they are mainly based on convolution and pooling operations. In this paper, we propose Gaussian process embedded channel attention (GPCA) module and further interpret the channel attention schemes in a probabilistic way. The GPCA module intends to model the correlations among the channels, which are assumed to be captured by beta distributed variables. As the beta distribution cannot be integrated into the end-to-end training of convolutional neural networks (CNNs) with a mathematically tractable solution, we utilize an approximation of the beta distribution to solve this problem. To specify, we adapt a Sigmoid-Gaussian approximation, in which the Gaussian distributed variables are transferred into the interval [0,1]. The Gaussian process is then utilized to model the correlations among different channels. In this case, a mathematically tractable solution is derived. The GPCA module can be efficiently implemented and integrated into the end-to-end training of the CNNs. Experimental results demonstrate the promising performance of the proposed GPCA module. Codes are available at https://github.com/PRIS-CV/GPCA. [ABSTRACT FROM AUTHOR]

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA. [ABSTRACT FROM AUTHOR]

Image captioning has made significant achievement through developing feature extractor and model architecture. Recently, the image region features extracted by object detector prevail in most existing models. However, region features are criticized for the lacking of background and full contextual information. This problem can be remedied by providing some complementary visual information from patch features. In this paper, we propose a Double-Stream Position Learning Transformer Network (DSPLTN) which exploits the advantages of region features and patch features. Specifically, the region-stream encoder utilizes a Transformer encoder with Relative Position Learning (RPL) module to enhance the representations of region features through modeling the relationships between regions and positions respectively. As for the patch-stream encoder, we introduce convolutional neural network into the vanilla Transformer encoder and propose a novel Convolutional Position Learning (CPL) module to encode the position relationships between patches. CPL improves the ability of relationship modeling by combining the position and visual content of patches. Incorporating CPL into the Transformer encoder can synthesize the benefits of convolution in local relation modeling and self-attention in global feature fusion, thereby compensating for the information loss caused by the flattening operation of 2D feature maps to 1D patches. Furthermore, an Adaptive Fusion Attention (AFA) mechanism is proposed to balance the contribution of enhanced region and patch features. Extensive experiments on MSCOCO demonstrate the effectiveness of the double-stream encoder and CPL, and show the superior performance of DSPLTN. [ABSTRACT FROM AUTHOR]

Spiking Neural Networks (SNN) and the field of Neuromorphic Engineering has brought about a paradigm shift in how to approach Machine Learning (ML) and Computer Vision (CV) problem. This paradigm shift comes from the adaption of event-based sensing and processing. An event-based vision sensor allows for sparse and asynchronous events to be produced that are dynamically related to the scene. Allowing not only the spatial information but a high-fidelity of temporal information to be captured. Meanwhile avoiding the extra overhead and redundancy of conventional high frame rate approaches. However, with this change in paradigm, many techniques from traditional CV and ML are not applicable to these event-based spatial-temporal visual streams. As such a limited number of recognition, detection and segmentation approaches exist. In this paper, we present a novel approach that can perform instance segmentation using just the weights of a Spike Time Dependent Plasticity trained Spiking Convolutional Neural Network that was trained for object recognition. This exploits the spatial and temporal aspects of the SpikeSEG network's internal feature representations adding this new discriminative capability. We highlight the new capability by successfully transforming a single class unsupervised network for face detection into a multi-person face recognition and instance segmentation network. [ABSTRACT FROM AUTHOR]

We present a spherical harmonics-based convolutional neural network (CNN) for cortical parcellation, which we call SPHARM-Net. Recent advances in CNNs offer cortical parcellation on a fine-grained triangle mesh of the cortex. Yet, most CNNs designed for cortical parcellation employ spatial convolution that depends on extensive data augmentation and allows only predefined neighborhoods of specific spherical tessellation. On the other hand, a rotation-equivariant convolutional filter avoids data augmentation, and rotational equivariance can be achieved in spectral convolution independent of a neighborhood definition. Nevertheless, the limited resources of a modern machine enable only a finite set of spectral components that might lose geometric details. In this paper, we propose (1) a constrained spherical convolutional filter that supports an infinite set of spectral components and (2) an end-to-end framework without data augmentation. The proposed filter encodes all the spectral components without the full expansion of spherical harmonics. We show that rotational equivariance drastically reduces the training time while achieving accurate cortical parcellation. Furthermore, the proposed convolution is fully composed of matrix transformations, which offers efficient and fast spectral processing. In the experiments, we validate SPHARM-Net on two public datasets with manual labels: Mindboggle-101 (N=101) and NAMIC (N=39). The experimental results show that the proposed method outperforms the state-of-the-art methods on both datasets even with fewer learnable parameters without rigid alignment and data augmentation. Our code is publicly available at https://github.com/Shape-Lab/SPHARM-Net. [ABSTRACT FROM AUTHOR]

This paper investigates the principles of embedding learning to tackle the challenging semi-supervised video object segmentation. Unlike previous practices that focus on exploring the embedding learning of foreground object (s), we consider background should be equally treated. Thus, we propose a Collaborative video object segmentation by Foreground-Background Integration (CFBI) approach. CFBI separates the feature embedding into the foreground object region and its corresponding background region, implicitly promoting them to be more contrastive and improving the segmentation results accordingly. Moreover, CFBI performs both pixel-level matching processes and instance-level attention mechanisms between the reference and the predicted sequence, making CFBI robust to various object scales. Based on CFBI, we introduce a multi-scale matching structure and propose an Atrous Matching strategy, resulting in a more robust and efficient framework, CFBI+. We conduct extensive experiments on two popular benchmarks, i.e., DAVIS and YouTube-VOS. Without applying any simulated data for pre-training, our CFBI+ achieves the performance ($\mathcal {J}$ J & $\mathcal {F}$ F ) of 82.9 and 82.8 percent, outperforming all the other state-of-the-art methods. Code: https://github.com/z-x-yang/CFBI. [ABSTRACT FROM AUTHOR]

This paper generalizes the Attention in Attention (AiA) mechanism, in P. Fang et al., 2019 by employing explicit mapping in reproducing kernel Hilbert spaces to generate attention values of the input feature map. The AiA mechanism models the capacity of building inter-dependencies among the local and global features by the interaction of inner and outer attention modules. Besides a vanilla AiA module, termed linear attention with AiA, two non-linear counterparts, namely, second-order polynomial attention and Gaussian attention, are also proposed to utilize the non-linear properties of the input features explicitly, via the second-order polynomial kernel and Gaussian kernel approximation. The deep convolutional neural network, equipped with the proposed AiA blocks, is referred to as Attention in Attention Network (AiA-Net). The AiA-Net learns to extract a discriminative pedestrian representation, which combines complementary person appearance and corresponding part features. Extensive ablation studies verify the effectiveness of the AiA mechanism and the use of non-linear features hidden in the feature map for attention design. Furthermore, our approach outperforms current state-of-the-art by a considerable margin across a number of benchmarks. In addition, state-of-the-art performance is also achieved in the video person retrieval task with the assistance of the proposed AiA blocks. [ABSTRACT FROM AUTHOR]

We observed that remarkable and impressive performance on image-based human pose estimation have been achieved by deep Convolutional Neural Networks (CNN). Nevertheless, directly applying these image-based models on videos is not only computionally intensive, but also may cause jitter and loss. The main reason is that the image-based models purely focus on the local features of individual frames and totally ignore the temporal information among adjacent frames. Some existing methods are proposed to address the temporal coherency issue. However, these methods need to be designed carefully and cannot be combined with existing image-based methods. In this paper, we propose a simple yet effective module to refine the estimated pose by exploiting the temporal coherency among the heatmaps of adjacent frames, which can be easily inserted into image-based networks as a plug-in. We show that the temporal coherency issue among the heatmap frames could be re-formulated as a graph path selection optimization problem. Moreover, to speed up the refinement process, we propose a hierarchical graph optimization to achieve the refinement from coarse to fine. Experimental results on two large-scale video pose estimation benchmarks show that our module can improve the performance with little speed loss when combined with image-based methods as an efficient plug-in. [ABSTRACT FROM AUTHOR]

Crowd counting has important applications in the environments of smart cities, such as intelligent surveillance. In this paper, we propose a novel convolutional neural network (CNN) framework for crowd counting with mixed ground-truth, called top- $k$ relation-based network (TKRNet). Specifically, the estimated density maps generated in a coarse-to-fine manner are treated as coarse locations for crowds so as to assist our TKRNet to regress the scattered point-annotated ground truth. Moreover, an adaptive top- $k$ relation module (ATRM) is proposed to enhance feature representations by leveraging the top- $k$ dependencies between the pixels with an adaptive filtering mechanism. Specifically, we first compute the similarity between two pixels so as to select the top- $k$ relations for each position. Then, a weight normalization operation with an adaptive filtering mechanism is proposed to make the ATRM adaptively eliminate the influence from the low correlation positions in the top- $k$ relations. Finally, a weight attention mechanism is introduced to make the ATRM pay more attention to the positions with high weights in the top- $k$ relations. Extensive experimental results demonstrate the effectiveness of our proposed TKRNet on several public datasets in comparison to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Text clustering is a critical step in text data analysis and has been extensively studied by the text mining community. Most existing text clustering algorithms are based on the bag-of-words model, which faces the high-dimensional and sparsity problems and ignores text structural and sequence information. Deep learning-based models such as convolutional neural networks and recurrent neural networks regard texts as sequences but lack supervised signals and explainable results. In this paper, we propose a deep feature-based text clustering (DFTC) framework that incorporates pretrained text encoders into text clustering tasks. This model, which is based on sequence representations, breaks the dependency on supervision. The experimental results show that our model outperforms classic text clustering algorithms and the state-of-the-art pretrained language model, i.e., BERT, on almost all the considered datasets. In addition, the explanation of the clustering results is significant for understanding the principles of the deep learning approach. Our proposed clustering framework includes an explanation module that can help users understand the meaning and quality of the clustering results. [ABSTRACT FROM AUTHOR]

Dynamic vision sensors (event cameras) have recently been introduced to solve a number of different vision tasks such as object recognition, activities recognition, tracking, etc. Compared with the traditional RGB sensors, the event cameras have many unique advantages such as ultra low resources consumption, high temporal resolution and much larger dynamic range. However, these cameras only produce noisy and asynchronous events of intensity changes, i.e., event-streams rather than frames, where conventional computer vision algorithms can't be directly applied. In our opinion the key challenge for improving the performance of event cameras in vision tasks is finding the appropriate representations of the event-streams so that cutting-edge learning approaches can be applied to fully uncover the spatio-temporal information contained in the event-streams. In this paper, we focus on the event-based human gait identification task and investigate the possible representations of the event-streams when deep neural networks are applied as the classifier. We propose new event-based gait recognition approaches basing on two different representations of the event-stream, i.e., graph and image-like representations, and use graph-based convolutional network (GCN) and convolutional neural networks (CNN) respectively to recognize gait from the event-streams. The two approaches are termed as EV-Gait-3DGraph and EV-Gait-IMG. To evaluate the performance of the proposed approaches, we collect two event-based gait datasets, one from real-world experiments and the other by converting the publicly available RGB gait recognition benchmark CASIA-B. Extensive experiments show that EV-Gait-3DGraph achieves significantly higher recognition accuracy than other competing methods when sufficient training samples are available. However, EV-Gait-IMG converges more quickly than graph-based approaches while training and shows good accuracy with only few number of training samples (less than ten). So image-like presentation is preferable when the amount of training data is limited. [ABSTRACT FROM AUTHOR]

Recent years have witnessed a big leap in automatic visual saliency detection attributed to advances in deep learning, especially Convolutional Neural Networks (CNNs). However, inferring the saliency of each image part separately, as was adopted by most CNNs methods, inevitably leads to an incomplete segmentation of the salient object. In this paper, we describe how to use the property of part-object relations endowed by the Capsule Network (CapsNet) to solve the problems that fundamentally hinge on relational inference for visual saliency detection. Concretely, we put in place a two-stream strategy, termed Two-Stream Part-Object RelaTional Network (TSPORTNet), to implement CapsNet, aiming to reduce both the network complexity and the possible redundancy during capsule routing. Additionally, taking into account the correlations of capsule types from the preceding training images, a correlation-aware capsule routing algorithm is developed for more accurate capsule assignments at the training stage, which also speeds up the training dramatically. By exploring part-object relationships, TSPORTNet produces a capsule wholeness map, which in turn aids multi-level features in generating the final saliency map. Experimental results on five widely-used benchmarks show that our framework consistently achieves state-of-the-art performance. The code can be found on https://github.com/liuyi1989/TSPORTNet. [ABSTRACT FROM AUTHOR]

With the help of convolutional neural networks (CNNs), deep learning-based methods have achieved remarkable performance in face super-resolution (FSR) task. Despite their success, most of the existing methods neglect non-local correlations of face images, leaving much room for improvement. In this paper, we introduce a novel end-to-end trainable attention-driven graph neural network (AD-GNN) for more discriminative feature extraction and feature relation modeling. This is achieved by two major components. The first component is a cross-scale dynamic graph (CDG) block. The CDG block considers cross-scale relationships of patches in distant areas and employs two dynamic graphs to construct enhanced features. The second component is a series of channel attention and spatial dynamic graph (CASDG) blocks. A CASDG block has a channel-wise attention unit and a spatial-aware dynamic graph (SDG) unit. The SDG unit extracts informative features by exploring spatial non-local self-similarity information of the patches using dynamic graph convolution. Using these two components, facial details can be effectively reconstructed with the help of information supplemented by similar but spatially remote patches and structural information of faces. Extensive experiments on two public benchmarks demonstrate the superiority of AD-GNN over the state-of-the-art FSR methods. [ABSTRACT FROM AUTHOR]

The aesthetic quality of an image is defined as the measure or appreciation of the beauty of an image. Aesthetics is inherently a subjective property but there are certain factors that influence it such as, the semantic content of the image, the attributes describing the artistic aspect, the photographic setup used for the shot, etc. In this paper we propose a method for the automatic prediction of the aesthetics of an image that is based on the analysis of the semantic content, the artistic style and the composition of the image. The proposed network includes: a pre-trained network for semantic features extraction (the Backbone); a Multi Layer Perceptron (MLP) network that relies on the Backbone features for the prediction of image attributes (the AttributeNet); a self-adaptive Hypernetwork that exploits the attributes prior encoded into the embedding generated by the AttributeNet to predict the parameters of the target network dedicated to aesthetic estimation (the AestheticNet). Given an image, the proposed multi-network is able to predict: style and composition attributes, and aesthetic score distribution. Results on three benchmark datasets demonstrate the effectiveness of the proposed method, while the ablation study gives a better understanding of the proposed network. [ABSTRACT FROM AUTHOR]

In this paper, we focus on X-ray images (X-radiographs) of paintings with concealed sub-surface designs (e.g., deriving from reuse of the painting support or revision of a composition by the artist), which therefore include contributions from both the surface painting and the concealed features. In particular, we propose a self-supervised deep learning-based image separation approach that can be applied to the X-ray images from such paintings to separate them into two hypothetical X-ray images. One of these reconstructed images is related to the X-ray image of the concealed painting, while the second one contains only information related to the X-ray image of the visible painting. The proposed separation network consists of two components: the analysis and the synthesis sub-networks. The analysis sub-network is based on learned coupled iterative shrinkage thresholding algorithms (LCISTA) designed using algorithm unrolling techniques, and the synthesis sub-network consists of several linear mappings. The learning algorithm operates in a totally self-supervised fashion without requiring a sample set that contains both the mixed X-ray images and the separated ones. The proposed method is demonstrated on a real painting with concealed content, Do na Isabel de Porcel by Francisco de Goya, to show its effectiveness. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of single image de-raining, that is, the task of recovering clean and rain-free background scenes from a single image obscured by a rainy artifact. Although recent advances adopt real-world time-lapse data to overcome the need for paired rain-clean images, they are limited to fully exploit the time-lapse data. The main cause is that, in terms of network architectures, they could not capture long-term rain streak information in the time-lapse data during training owing to the lack of memory components. To address this problem, we propose a novel network architecture combining the time-lapse data and, the memory network that explicitly helps to capture long-term rain streak information. Our network comprises the encoder-decoder networks and a memory network. The features extracted from the encoder are read and updated in the memory network that contains several memory items to store rain streak-aware feature representations. With the read/update operation, the memory network retrieves relevant memory items in terms of the queries, enabling the memory items to represent the various rain streaks included in the time-lapse data. To boost the discriminative power of memory features, we also present a novel background selective whitening (BSW) loss for capturing only rain streak information in the memory network by erasing the background information. Experimental results on standard benchmarks demonstrate the effectiveness and superiority of our approach. [ABSTRACT FROM AUTHOR]

Recently, many arbitrary-oriented object detection (AOOD) methods have been proposed and attracted widespread attention in many fields. However, most of them are based on anchor-boxes or standard Gaussian heatmaps. Such label assignment strategy may not only fail to reflect the shape and direction characteristics of arbitrary-oriented objects, but also have high parameter-tuning efforts. In this paper, a novel AOOD method called General Gaussian Heatmap Label Assignment (GGHL) is proposed. Specifically, an anchor-free object-adaptation label assignment (OLA) strategy is presented to define the positive candidates based on two-dimensional (2D) oriented Gaussian heatmaps, which reflect the shape and direction features of arbitrary-oriented objects. Based on OLA, an oriented-bounding-box (OBB) representation component (ORC) is developed to indicate OBBs and adjust the Gaussian center prior weights to fit the characteristics of different objects adaptively through neural network learning. Moreover, a joint-optimization loss (JOL) with area normalization and dynamic confidence weighting is designed to refine the misalign optimal results of different subtasks. Extensive experiments on public datasets demonstrate that the proposed GGHL improves the AOOD performance with low parameter-tuning and time costs. Furthermore, it is generally applicable to most AOOD methods to improve their performance including lightweight models on embedded platforms. [ABSTRACT FROM AUTHOR]

Multi-class object detection in remote sensing images plays an important role in many applications but remains a challenging task because of scale imbalance and arbitrary orientations of the objects with extreme aspect ratios. In this paper, the Asymmetric Feature Pyramid Network (AFPN), Dynamic Feature Alignment (DFA) module, and Area-IoU regression loss are proposed on the basis of a one-stage cascaded detection method for the detection of multi-class objects with arbitrary orientations in remote sensing images. The designed asymmetric convolutional block is embedded into the AFPN for handling objects with extreme aspect ratios and improving the space representation with ignorable increases in calculation. The DFA module is proposed to dynamically align mismatched features, which are caused by the deviation between predefined anchors and arbitrarily oriented predicted boxes. The refined Area-IoU regression loss, which reconciles two new regression loss functions, the area-guided regression loss and IoU-guided regression loss, is proposed to simultaneously solve the scale imbalance problem and angle sensitivity problem. Experiments on three publicly available datasets, DOTA, HRSC2016, and ICDAR2015, show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Few-shot learning suffers from the scarcity of labeled training data. Regarding local descriptors of an image as representations for the image could greatly augment existing labeled training data. Existing local descriptor based few-shot learning methods have taken advantage of this fact but ignore that the semantics exhibited by local descriptors may not be relevant to the image semantic. In this paper, we deal with this issue from a new perspective of imposing semantic consistency of local descriptors of an image. Our proposed method consists of three modules. The first one is a local descriptor extractor module, which can extract a large number of local descriptors in a single forward pass. The second one is a local descriptor compensator module, which compensates the local descriptors with the image-level representation, in order to align the semantics between local descriptors and the image semantic. The third one is a local descriptor based contrastive loss function, which supervises the learning of the whole pipeline, with the aim of making the semantics carried by the local descriptors of an image relevant and consistent with the image semantic. Theoretical analysis demonstrates the generalization ability of our proposed method. Comprehensive experiments conducted on benchmark datasets indicate that our proposed method achieves the semantic consistency of local descriptors and the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Dynamic scene deblurring is a challenging problem as it is difficult to be modeled mathematically. Benefiting from the deep convolutional neural networks, this problem has been significantly advanced by the end-to-end network architectures. However, the success of these methods is mainly due to simply stacking network layers. In addition, the methods based on the end-to-end network architectures usually estimate latent images in a regression way which does not preserve the structural details. In this paper, we propose an exemplar-based method to solve dynamic scene deblurring problem. To explore the properties of the exemplars, we propose a siamese encoder network and a shallow encoder network to respectively extract input features and exemplar features and then develop a rank module to explore useful features for better blur removing, where the rank modules are applied to the last three layers of encoder, respectively. The proposed method can be further extended to the way of multi-scale, which enables to recover more texture from the exemplar. Extensive experiments show that our method achieves significant improvements in both quantitative and qualitative evaluations. [ABSTRACT FROM AUTHOR]

Over the past few years, Convolutional Neural Networks (CNNs) have achieved remarkable advancement for the tasks of one-shot image classification. However, the lack of effective attention modeling has limited its performance. In this paper, we propose a Two-branch (Content-aware and Position-aware) Attention (CPA) Network via an Efficient Semantic Coupling module for attention modeling. Specifically, we harness content-aware attention to model the characteristic features (e.g., color, shape, texture) as well as position-aware attention to model the spatial position weights. In addition, we exploit support images to improve the learning of attention for the query images. Similarly, we also use query images to enhance the attention model of the support set. Furthermore, we design a local-global optimizing framework that further improves the recognition accuracy. The extensive experiments on four common datasets (miniImageNet, tieredImageNet, CUB-200-2011, CIFAR-FS) with three popular networks (DPGN, RelationNet and IFSL) demonstrate that our devised CPA module equipped with local-global Two-stream framework (CPAT) can achieve state-of-the-art performance, with a significant improvement in accuracy of 3.16% on CUB-200-2011 in particular. [ABSTRACT FROM AUTHOR]

As two practical and important image processing tasks, color demosaicking (CDM) and super-resolution (SR) have been studied for decades. However, most literature studies these two tasks independently, ignoring the potential benefits of a joint solution. In this paper, aiming at efficient and effective joint demosaicking and super-resolution (JDSR), a well-designed two-stage convolutional neural network (CNN) architecture is proposed. For the first stage, by making use of the sampling-pattern information, a pattern-aware feature extraction (PFE) module extracts features directly from the Bayer-sampled low-resolution (LR) image, while keeping the resolution of the extracted features the same as the input. For the second stage, a dual-branch feature refinement (DFR) module effectively decomposes the features into two components with different spatial frequencies, on which different learning strategies are applied. On each branch of the DFR module, the feature refinement unit, namely, densely-connected dual-path enhancement blocks (DDEB), establishes a sophisticated nonlinear mapping from the LR space to the high-resolution (HR) space. To achieve strong representational power, two paths of transformations and the channel attention mechanism are adopted in DDEB. Extensive experiments demonstrate that the proposed method is superior to the sequential combination of state-of-the-art (SOTA) CDM and SR methods. Moreover, with much smaller model size, our approach also surpasses other SOTA JDSR methods. [ABSTRACT FROM AUTHOR]

Temporal action detection is a challenging task in video understanding, due to the complexity of the background and rich action content impacting high-quality temporal proposals generation in untrimmed videos. Capsule networks can avoid some limitations of the invariance caused by pooling and inability from convolutional neural networks, which can better understand the temporal relations for temporal action detection. However, because of the extremely computationally expensive procedure, capsule network is difficult to be applied to the task of temporal action detection. To address this issue, this paper proposes a novel U-shaped capsule network framework with a k-Nearest Neighbor (k-NN) mechanism of 3D convolutional dynamic routing, which we named U-BlockConvCaps. Furthermore, we build a Capsules Boundary Network (CapsBoundNet) based on U-BlockConvCaps for dense temporal action proposal generation. Specifically, the first module is one 1D convolutional layer for fusing the two-stream with RGB and optical flow video features. The sampling module further processes the fused features to generate the 2D start-end action proposal feature maps. Then, the multi-scale U-Block convolutional capsule module with 3D convolutional dynamic routing is used to process the proposal feature map. Finally, the feature maps generated from the CapsBoundNet are used to predict starting, ending, action classification, and action regression score maps, which help to capture the boundary and intersection over union features. Our work innovatively improves the dynamic routing algorithm of capsule networks and extends the use of capsule networks to the temporal action detection task for the first time in the literature. The experimental results on benchmarks THUMOS14 show that the performance of CapsBoundNet is obviously beyond the state-of-the-art methods, e.g., the mAP@tIoU = 0.3, 0.4, 0.5 on THUMOS14 are improved from 63.6% to 70.0%, 57.8% to 63.1%, 51.3% to 52.9%, respectively. We also got competitive results on the action detection dataset of ActivityNet1.3. [ABSTRACT FROM AUTHOR]

To address the problem of automatic identification of fine-grained fracture types, in this paper, we propose a novel framework using 3D convolutional neural network (CNN) to learn fracture features from voxelized bone models which are obtained by establishing isomorphic mapping from fractured bones to a voxelized template. The network, which is named FractureNet, consists of four discriminators forming a multi-stage hierarchy. Each discriminator includes multiple sub-classifiers. These sub-classifiers are chained by two kinds of feature chains (feature map chain and classification feature chain) in the form of a full m-ary tree to perform multi-stage classification tasks. The features learned and classification results obtained at previous stages serve as prior knowledge for current learning and classification. All sub-classifiers are jointly learned in an end-to-end network via a multi-stage loss function integrating losses of the four discriminators. To make our FractureNet more robust and accurate, a data augmentation strategy termed r-combination with constraints is further proposed on the basis of an adjacency relation and a continuity relation between voxels to create a large-scale fracture dataset of voxel models. Extensive experiments show that the proposed method can recognize various fracture types in patients accurately and effectively, and enables significant improvements over the state-of-the-arts on a variety of fracture recognition tasks. Moreover, ancillary experiments on the CIFAR-10 and the PadChest datasets at large scales further support the superior performance of the proposed FractureNet. [ABSTRACT FROM AUTHOR]

In this paper, we introduce the idea of blockwise classification to count objects. The current mainstream method for counting objects is to regress the density map or to regress the redundant count map via a deep convolutional neural network (CNN). However, these methods suffer from two critical issues: inaccurately generated regression targets and serious sample imbalances. First, the ground truth density map is generated by convolving the dot map using a Gaussian kernel. Because an inappropriate kernel can cover the background or uncover objects, this approach introduces a form of noise, and therefore results in ambiguities when training the networks. Second, inhomogeneously distributed objects often exist in images, which gives rise to a data collection bias. This leads to a long-tailed distribution of region counts, which is a typical characteristic that occurs with imbalanced samples; therefore, underestimations in high-density regions and overestimations in low-density regions are common. In this paper, we address these two issues within one framework—blockwise count level classification. The intuition behind this idea is that while it may not be possible to provide an exact count of pixels or patches, it is possible to provide a count of a region that falls within a certain interval with high confidence. Our method classifies the count levels of each block produced by nonlinearly quantizing the continuous counts, thus transforming the imbalance of sample patch counts into a class imbalance of count levels. Consequently, an information-entropy-inspired loss can be applied to alleviate this issue. Through ablative studies, we analyze the impact of imbalanced data, Gaussian kernel sizes, quantization errors, and the effectiveness of each module in our method. Without bells and whistles, our method outperforms or performs competitively with other state-of-the-art approaches on seven object-counting benchmarks, including four crowd-counting datasets from ShanghaiTech, WorldExpo’10, UCF-QNRF and UCF_CC_50, one vehicle-counting dataset (TRANCOS), one maize-tassel-counting dataset (MTC), and one challenging sonar fish-counting dataset that we constructed. The results suggest that our framework provides a strong and improved baseline for object counting. [ABSTRACT FROM AUTHOR]

Recently, substantial research effort has focused on how to apply CNNs or RNNs to better capture temporal patterns in videos, so as to improve the accuracy of video classification. In this paper, we investigate the potential of a purely attention based local feature integration. Accounting for the characteristics of such features in video classification, we first propose Basic Attention Clusters (BAC), which concatenates the output of multiple attention units applied in parallel, and introduce a shifting operation to capture more diverse signals. Experiments show that BAC can achieve excellent results on multiple datasets. However, BAC treats all feature channels as an indivisible whole, which is suboptimal for achieving a finer-grained local feature integration over the channel dimension. Additionally, it treats the entire local feature sequence as an unordered set, thus ignoring the sequential relationships. To improve over BAC, we further propose the channel pyramid attention schema by splitting features into sub-features at multiple scales for coarse-to-fine sub-feature interaction modeling, and propose the temporal pyramid attention schema by dividing the feature sequences into ordered sub-sequences of multiple lengths to account for the sequential order. Our final model pyramid×pyramid attention clusters (PPAC) combines both channel pyramid attention and temporal pyramid attention to focus on the most important sub-features, while also preserving the temporal information of the video. We demonstrate the effectiveness of PPAC on seven real-world video classification datasets. Our model achieves competitive results across all of these, showing that our proposed framework can consistently outperform the existing local feature integration methods across a range of different scenarios. [ABSTRACT FROM AUTHOR]

Video deblurring, which aims at restoring the sharp video from blurry video, is drawing increasing attention in the field of computer vision. In this paper, a method called Multi-Attention Convolutional Neural Network (MACNN) consisting of the temporal-spatial attention module, the frame channel attention module, and the feature extraction-reconstruction module is proposed. First, we use the temporal-spatial attention module and the frame channel attention module to capture features with temporal and spatial information existing across neighboring frames. Then, these captured features are fused and reconstructed to restore the sharp frame. Last but not least, we train MACNN together with a content loss and a perceptual loss in an end-to-end manner to recover realistic video details. Both quantitative and qualitative evaluation results on standard benchmarks demonstrate the proposed MACNN is superior to the state-of-the-art methods in terms of accuracy, efficiency, and visual effect. [ABSTRACT FROM AUTHOR]

Facial expression recognition is important for human–computer interaction and other applications. Several facial expression datasets have been published in recent decades and have enabled improvements in algorithms for classifying emotions. However, recognition of realistic expressions in real-world conditions is still challenging because of uncontrolled conditions, such as lighting, brightness, pose, and occlusion. In this paper, we propose a light attention embedding network based on the spatial attention mechanism (LAENet-SA), which can focus on locations in an image that are relevant to emotion. LAENet-SA allows a small number of attention modules to be embedded and can be constructed from typical convolutional neural networks. The performance of LAENet-SA on facial expression recognition has been validated on three facial expression datasets, including a lab-controlled dataset and two in-the-wild datasets. Experimental results show that LAENet-SA improved the performance on each dataset, compared with state-of-the-art methods, and achieved better generalization when tested on facial images with occlusion. [ABSTRACT FROM AUTHOR]

Multi-task learning in Convolutional Neural Networks (CNNs) has led to remarkable success in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of task-specific features at single network layers. However, they generally learn an unchanged operation of feature combination after training, regardless of the characteristic changes of task-specific features across different inputs. In this paper, we propose a novel Adaptive Feature Aggregation (AFA) layer for multi-task CNNs, in which a dynamic aggregation mechanism is designed to allow each task adaptively determines the degree to which the knowledge sharing or preserving between tasks is needed based on the characteristics of inputs. We introduce two types of aggregation modules to the AFA layer, which realize the adaptive feature aggregation by capturing the feature dependencies of different tasks along the channel and spatial axes, respectively. The AFA layer is a plug-and-play component with low parameter and computation overheads, and can be trained end-to-end along with backbone networks. For both pixel-level and image-level tasks, we empirically show that our approach substantially outperforms the previous state-of-the-art methods of multi-task CNNs. The code and models are available at https://github.com/zhenshen-mla/AFANet. [ABSTRACT FROM AUTHOR]

Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequency detail information from source images. Then the algorithm unrolling is implemented where each iteration is mapped to a CNN layer and each optimization model is transformed into a trainable neural network. Compared with the general network architectures, the proposed framework combines the model-based prior information and is designed more reasonably. After the unrolling operation, our model contains two decomposers (encoders) and an additional reconstructor (decoder). In the training phase, this network is trained to reconstruct the input image. While in the test phase, the base (or detail) decomposed feature maps of infrared/visible images are merged respectively by an extra fusion layer, and then the decoder outputs the fusion image. Qualitative and quantitative comparisons demonstrate the superiority of our model, which can robustly generate fusion images containing highlight targets and legible details, exceeding the state-of-the-art methods. Furthermore, our network has fewer weights and faster speed. [ABSTRACT FROM AUTHOR]

Generating features representing the textures and structures of an image is very important characteristic of a super resolution network. Morphological operations are the nonlinear mathematical operations that can process signals focusing on their structures and textures. In this paper, we propose a novel residual block to generate and process morphological features and fuse them with the conventional spatial features, in order to produce a very rich and highly representational set of residual feature maps. The proposed residual block is then used in a deep convolutional neural network for the task of image super resolution. It is shown that the capability of the proposed block in generating and using the morphological features can significantly improve the super resolution performance of a deep network. The super resolution network employing the proposed residual block is shown to outperform the state-of-the-art low-complexity image super resolution networks on various benchmark datasets. [ABSTRACT FROM AUTHOR]

Convolutional neural networks are built upon simple but useful convolution modules. The traditional convolution has a limitation on feature extraction and object localization due to its fixed scale and geometric structure. Besides, the loss of spatial information also restricts the networks’ performance and depth. To overcome these limitations, this paper proposes a novel anisotropic convolution by adding a scale factor and a shape factor into the traditional convolution. The anisotropic convolution augments the receptive fields flexibly and dynamically depending on the valid sizes of objects. In addition, the anisotropic convolution is a generalized convolution. The traditional convolution, dilated convolution and deformable convolution can be viewed as its special cases. Furthermore, in order to improve the training efficiency and avoid falling into a local optimum, this paper introduces a simplified implementation of the anisotropic convolution. The anisotropic convolution can be applied to arbitrary convolutional networks and the enhanced networks are called ACNs (anisotropic convolutional networks). Experimental results show that ACNs achieve better performance than many state-of-the-art methods and the baseline networks in tasks of image classification and object localization, especially in classification task of tiny images. [ABSTRACT FROM AUTHOR]

Most convolutional neural network (CNN)-based cloud detection methods are built upon the supervised learning framework that requires a large number of pixel-level labels. However, it is expensive and time-consuming to manually annotate pixelwise labels for massive remote sensing images. To reduce the labeling cost, we propose an unsupervised domain adaptation (UDA) approach to generalize the model trained on labeled images of source satellite to unlabeled images of the target satellite. To effectively address the domain shift problem on cross-satellite images, we develop a novel UDA method based on grouped features alignment (GFA) and entropy minimization (EM) to extract domain-invariant representations to improve the cloud detection accuracy of cross-satellite images. The proposed UDA method is evaluated on “Landsat- $8~\rightarrow $ ZY-3” and “GF- $1\rightarrow $ ZY-3” domain adaptation tasks. Experimental results demonstrate the effectiveness of our method against existing state-of-the-art UDA approaches. The code of this paper has been made available online (https://github.com/nkszjx/grouped-features-alignment). [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (CNNs) have emerged as a new paradigm for Mammogram diagnosis. Contemporary CNN-based computer-aided-diagnosis systems (CADs) for breast cancer directly extract latent features from input mammogram image and ignore the importance of morphological features. In this paper, we introduce a novel end-to-end deep learning framework for mammogram image processing, which computes mass segmentation and simultaneously predicts diagnosis results. Specifically, our method is constructed in a dual-path architecture that solves the mapping in a dual-problem manner, with an additional consideration of important shape and boundary knowledge. One path, called the Locality Preserving Learner (LPL), is devoted to hierarchically extracting and exploiting intrinsic features of the input. Whereas the other path, called the Conditional Graph Learner (CGL), focuses on generating geometrical features via modeling pixel-wise image to mask correlations. By integrating the two learners, both the cancer semantics and cancer representations are well learned, and the component learning paths in return complement each other, contributing an improvement to the mass segmentation and cancer classification problem at the same time. In addition, by integrating an automatic detection set-up, the DualCoreNet achieves fully automatic breast cancer diagnosis practically. Experimental results show that in benchmark DDSM dataset, DualCoreNet has outperformed other related works in both segmentation and classification tasks, achieving 92.27% DI coefficient and 0.85 AUC score. In another benchmark INbreast dataset, DualCoreNet achieves the best mammography segmentation (93.69% DI coefficient) and competitive classification performance (0.93 AUC score). [ABSTRACT FROM AUTHOR]

Liver lesion segmentation is an essential process to assist doctors in hepatocellular carcinoma diagnosis and treatment planning. Multi-modal positron emission tomography and computed tomography (PET-CT) scans are widely utilized due to their complementary feature information for this purpose. However, current methods ignore the interaction of information across the two modalities during feature extraction, omit the co-learning of the feature maps of different resolutions, and do not ensure that shallow and deep features complement each others sufficiently. In this paper, our proposed model can achieve feature interaction across multi-modal channels by sharing the down-sampling blocks between two encoding branches to eliminate misleading features. Furthermore, we combine feature maps of different resolutions to derive spatially varying fusion maps and enhance the lesions information. In addition, we introduce a similarity loss function for consistency constraint in case that predictions of separated refactoring branches for the same regions vary a lot. We evaluate our model for liver tumor segmentation using a PET-CT scans dataset, compare our method with the baseline techniques for multi-modal (multi-branches, multi-channels and cascaded networks) and then demonstrate that our method has a significantly higher accuracy (${p} < 0.05$) than the baseline models. [ABSTRACT FROM AUTHOR]

This paper introduces an explanatory graph representation to reveal object parts encoded inside convolutional layers of a CNN. Given a pre-trained CNN, each filter1 in a conv-layer usually represents a mixture of object parts. We develop a simple yet effective method to learn an explanatory graph, which automatically disentangles object parts from each filter without any part annotations. Specifically, given the feature map of a filter, we mine neural activations from the feature map, which correspond to different object parts. The explanatory graph is constructed to organize each mined part as a graph node. Each edge connects two nodes, whose corresponding object parts usually co-activate and keep a stable spatial relationship. Experiments show that each graph node consistently represented the same object part through different images, which boosted the transferability of CNN features. The explanatory graph transferred features of object parts to the task of part localization, and our method significantly outperformed other approaches. [ABSTRACT FROM AUTHOR]

In this paper, we tackle the problem of single image dehazing with a convolutional neural network. Within this network, we develop a multi-level fusion module to utilize both low-level and high-level features. The low-level features help to recover finer details, and the high-level features discover abstract semantics. They are complementary in the restoring of clear images. Moreover, a Residual Mixed-convolution Attention Module (RMAM) with an attention block is proposed to guide the network to focus on important features in the learning process. In this RMAM, group convolution, depth-wise convolution, and point-wise convolution are mixed, and thus it is much faster than its counterparts. With these two modules, we thus have an end-to-end network without explicitly estimating the atmospheric light intensity and the transmission map in the classical atmosphere scattering model. Both qualitative and quantitative experimental studies are carried out on public datasets including RESIDE, DCPDN-TestA, and the real-world dataset. The extensive results demonstrate both the effectiveness and efficiency of the proposed solution to single image dehazing. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., common and unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion. [ABSTRACT FROM AUTHOR]

This paper proposes a generic method to learn interpretable convolutional filters in a deep convolutional neural network (CNN) for object classification, where each interpretable filter encodes features of a specific object part. Our method does not require additional annotations of object parts or textures for supervision. Instead, we use the same training data as traditional CNNs. Our method automatically assigns each interpretable filter in a high conv-layer with an object part of a certain category during the learning process. Such explicit knowledge representations in conv-layers of the CNN help people clarify the logic encoded in the CNN, i.e., answering what patterns the CNN extracts from an input image and uses for prediction. We have tested our method using different benchmark CNNs with various architectures to demonstrate the broad applicability of our method. Experiments have shown that our interpretable filters are much more semantically meaningful than traditional filters. [ABSTRACT FROM AUTHOR]

Reducing radiation dose in cardiac catheter-based X-ray procedures increases safety but also image noise and artifacts. Excessive noise and artifacts can compromise vital image information, which can affect clinical decision-making. Developing more effective X-ray denoising methodologies will be beneficial to both patients and healthcare professionals by allowing imaging at lower radiation dose without compromising image information. This paper proposes a framework based on a convolutional neural network (CNN), namely Ultra-Dense Denoising Network (UDDN), for low-dose X-ray image denoising. To promote feature extraction, we designed a novel residual block which establishes a solid correlation among multiple-path neural units via abundant cross connections in its representation enhancement section. Experiments on synthetic additive noise X-ray data show that the UDDN achieves statistically significant higher peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) than other comparative methods. We enhanced the clinical adaptability of our framework by training using normally-distributed noise and tested on clinical data taken from procedures at St. Thomas’ hospital in London. The performance was assessed by using local SNR and by clinical voting using ten cardiologists. The results show that the UDDN outperforms the other comparative methods and is a promising solution to this challenging but clinically impactful task. [ABSTRACT FROM AUTHOR]

Good performance and high efficiency are both critical for estimating human pose in practice. Recent state-of-the-art methods have greatly boosted the pose detection accuracy through deep convolutional neural networks, however, the strong performance is typically achieved without high efficiency. In this paper, we design a novel network architecture for human pose estimation, which aims to strike a fine balance between speed and accuracy. Two essential tasks for successful pose estimation, preserving spatial location and extracting semantic information, are handled separately in the proposed architecture. Semantic knowledge of joint type is obtained through deep and wide sub-networks with low-resolution input, and high-resolution features indicating joint location are processed by shallow and narrow sub-networks. Because accurate semantic analysis mainly asks for adequate depth and width of the network and precise spatial information mostly requests preserving high-resolution features, good results can be produced by fusing the outputs of the sub-networks. Moreover, the computational cost can be considerably reduced comparing with existing networks, since the main part of the proposed network only deals with low-resolution features. We refer to the architecture as “parallel pyramid” network (PPNet), as features of different resolutions are processed at different levels of the hierarchical model. The superiority of our network is empirically demonstrated on two benchmark datasets: the MPII Human Pose dataset and the COCO keypoint detection dataset. PPNet outcompetes all recent methods by using less computation and memory to achieve better human pose estimation results. [ABSTRACT FROM AUTHOR]

Deep neural networks have demonstrated remarkable reconstruction for single-image super-resolution (SISR). However, most existing CNN-based SISR methods directly learn the relation between low-resolution (LR) and high-resolution (HR) images, neglecting to explore the recurrence of internal patches, hence hindering the representational power of CNNs. In this paper, we propose a novel single image Super-Resolution network based on Graph ATtention network (SRGAT) to make full use of the internal patch-recurrence in a natural image. The proposed model employs a feature mapping block with a recurrent structure to refine low-level representations with high-level information. Especifically, the feature mapping block contains a parallel graph similarity branch and a content branch, where the graph similarity branch aims at exploiting the similarity and symmetry across different image patches in low-resolution feature space and provides additional priors for the content branch to enhance texture details. Specifically, we consider the internal patch-recurrence of an image by constructing a graph network on image feature patches. In this way, the information from neighboring patches can be interacted using graph attention network (GAT) to help it recover additional textures, which complements the textures learned from the content branch. Extensive quantitative and qualitative evaluations on five benchmark datasets demonstrate that the proposed algorithm performs favorably against the state-of-the-art super-resolution methods. [ABSTRACT FROM AUTHOR]

The automatic detection and identification of fish from underwater videos is of great significance for fishery resource assessment and ecological environment monitoring. However, due to the poor quality of underwater images and unconstrained fish movement, traditional hand-designed feature extraction methods or convolutional neural network (CNN)-based object detection algorithms cannot meet the detection requirements in real underwater scenes. Therefore, to realize fish recognition and localization in a complex underwater environment, this paper proposes a novel composite fish detection framework based on a composite backbone and an enhanced path aggregation network called Composited FishNet. By improving the residual network (ResNet), a new composite backbone network (CBresnet) is designed to learn the scene change information (source domain style), which is caused by the differences in the image brightness, fish orientation, seabed structure, aquatic plant movement, fish species shape and texture differences. Thus, the interference of underwater environmental information on the object characteristics is reduced, and the output of the main network to the object information is strengthened. In addition, to better integrate the high and low feature information output from CBresnet, the enhanced path aggregation network (EPANet) is also designed to solve the insufficient utilization of semantic information caused by linear upsampling. The experimental results show that the average precision (AP)0.5:0.95, AP50 and average recall (AR)max=10 of the proposed Composited FishNet are 75.2%, 92.8% and 81.1%, respectively. The composite backbone network enhances the characteristic information output of the detected object and improves the utilization of characteristic information. This method can be used for fish detection and identification in complex underwater environments such as oceans and aquaculture. [ABSTRACT FROM AUTHOR]

Single Image Super-Resolution (SISR) is essential for many computer vision tasks. In some real-world applications, such as object recognition and image classification, the captured image size can be arbitrary while the required image size is fixed, which necessitates SISR with arbitrary scaling factors. It is a challenging problem to take a single model to accomplish the SISR task under arbitrary scaling factors. To solve that problem, this paper proposes a bilateral upsampling network which consists of a bilateral upsampling filter and a depthwise feature upsampling convolutional layer. The bilateral upsampling filter is made up of two upsampling filters, including a spatial upsampling filter and a range upsampling filter. With the introduction of the range upsampling filter, the weights of the bilateral upsampling filter can be adaptively learned under different scaling factors and different pixel values. The output of the bilateral upsampling filter is then provided to the depthwise feature upsampling convolutional layer, which upsamples the low-resolution (LR) feature map to the high-resolution (HR) feature space depthwisely and well recovers the structural information of the HR feature map. The depthwise feature upsampling convolutional layer can not only efficiently reduce the computational cost of the weight prediction of the bilateral upsampling filter, but also accurately recover the textual details of the HR feature map. Experiments on benchmark datasets demonstrate that the proposed bilateral upsampling network can achieve better performance than some state-of-the-art SISR methods. [ABSTRACT FROM AUTHOR]

Gait recognition aims to recognize persons’ identities by walking styles. Gait recognition has unique advantages due to its characteristics of non-contact and long-distance compared with face and fingerprint recognition. Cross-view gait recognition is a challenge task because view variance may produce large impact on gait silhouettes. The development of deep learning has promoted cross-view gait recognition performances to a higher level. However, performances of existing deep learning-based cross-view gait recognition methods are limited by lack of gait samples under different views. In this paper, we take a Multi-view Gait Generative Adversarial Network (MvGGAN) to generate fake gait samples to extend existing gait datasets, which provides adequate gait samples for deep learning-based cross-view gait recognition methods. The proposed MvGGAN method trains a single generator for all view pairs involved in single or multiple datasets. Moreover, we perform domain alignment based on projected maximum mean discrepancy to reduce the influence of distribution divergence caused by sample generation. The experimental results on CASIA-B and OUMVLP dataset demonstrate that fake gait samples generated by the proposed MvGGAN method can improve performances of existing state-of-the-art cross-view gait recognition methods obviously on both single-dataset and cross-dataset evaluation settings. [ABSTRACT FROM AUTHOR]

Shadow removal can significantly improve the image visual quality and has many applications in computer vision. Deep learning methods based on CNNs have become the most effective approach for shadow removal by training on either paired data, where both the shadow and underlying shadow-free versions of an image are known, or unpaired data, where shadow and shadow-free training images are totally different with no correspondence. In practice, CNN training on unpaired data is more preferred given the easiness of training data collection. In this paper, we present a new Lightness-Guided Shadow Removal Network (LG-ShadowNet) for shadow removal by training on unpaired data. In this method, we first train a CNN module to compensate for the lightness and then train a second CNN module with the guidance of lightness information from the first CNN module for final shadow removal. We also introduce a loss function to further utilise the colour prior of existing data. Extensive experiments on widely used ISTD, adjusted ISTD and USR datasets demonstrate that the proposed method outperforms the state-of-the-art methods with training on unpaired data. [ABSTRACT FROM AUTHOR]

Image inpainting is a challenging computer vision task that aims to fill in missing regions of corrupted images with realistic contents. With the development of convolutional neural networks, many deep learning models have been proposed to solve image inpainting issues by learning information from a large amount of data. In particular, existing algorithms usually follow an encoding and decoding network architecture in which some operations with standard schemes are employed, such as static convolution, which only considers pixels with fixed grids, and the monotonous normalization style (e.g., batch normalization). However, these techniques are not well-suited for the image inpainting task because the random corrupted regions in the input images tend to mislead the inpainting process and generate unreasonable content. In this paper, we propose a novel dynamic selection network (DSNet) to solve this problem in image inpainting tasks. The principal idea of the proposed DSNet is to distinguish the corrupted region from the valid ones throughout the entire network architecture, which may help make full use of the information in the known area. Specifically, the proposed DSNet has two novel dynamic selection modules, namely, the validness migratable convolution (VMC) and regional composite normalization (RCN) modules, which share a dynamic selection mechanism that helps utilize valid pixels better. By replacing vanilla convolution with the VMC module, spatial sampling locations are dynamically selected in the convolution phase, resulting in a more flexible feature extraction process. Besides, the RCN module not only combines several normalization methods but also normalizes the feature regions selectively. Therefore, the proposed DSNet can illustrate realistic and fine-detailed images by adaptively selecting features and normalization styles. Experimental results on three public datasets show that our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

Pedestrian detection methods have been significantly improved with the development of deep convolutional neural networks. Nevertheless, detecting ismall-scaled pedestrians and occluded pedestrians remains a challenging problem. In this paper, we propose a pedestrian detection method with a couple-network to simultaneously address these two issues. One of the sub-networks, the gated multi-layer feature extraction sub-network, aims to adaptively generate discriminative features for pedestrian candidates in order to robustly detect pedestrians with large variations on scale. The second sub-network targets on handling the occlusion problem of pedestrian detection by using deformable regional region of interest (RoI)-pooling. We investigate two different gate units for the gated sub-network, namely, the channel-wise gate unit and the spatio-wise gate unit, which can enhance the representation ability of the regional convolutional features among the channel dimensions or across the spatial domain, repetitively. Ablation studies have validated the effectiveness of both the proposed gated multi-layer feature extraction sub-network and the deformable occlusion handling sub-network. With the coupled framework, our proposed pedestrian detector achieves promising results on both two pedestrian datasets, especially on detecting small or occluded pedestrians. On the CityPersons dataset, the proposed detector achieves the lowest missing rates (i.e. 40.78% and 34.60%) on detecting small and occluded pedestrians, surpassing the second best comparison method by 6.0% and 5.87%, respectively. [ABSTRACT FROM AUTHOR]