Search

This paper presents a co-salient object detection method to find common salient regions in a set of images. We utilize deep saliency networks to transfer co-saliency prior knowledge and better capture high-level semantic information. The resulting initial co-saliency maps are enhanced by seed propagation steps over an integrated graph. The deep saliency networks are trained in a supervised manner to avoid weakly supervised online learning and exploit them not only to extract high-level features but also to produce both intra- and inter-image saliency maps. Through a refinement step, the initial co-saliency maps can uniformly highlight co-salient regions and locate accurate object boundaries. To handle input image groups inconsistent in size, we propose to pool multi-regional descriptors including both within-segment and within-group information. In addition, the integrated multilayer graph is constructed to find the regions that the previous steps may not detect by seed propagation with low-level descriptors. In this paper, we utilize the useful complementary components of high- and low-level information and several learning-based steps. Our experiments have demonstrated that the proposed approach outperforms comparable co-saliency detection methods on widely used public databases and can also be directly applied to co-segmentation tasks. [ABSTRACT FROM AUTHOR]

Domain adaptation manages to build an effective target classifier or regression model for unlabeled target data by utilizing the well-labeled source data but lying different distributions. Intuitively, to address domain shift problem, it is crucial to learn domain invariant features across domains, and most existing approaches have concentrated on it. However, they often do not directly constrain the learned features to be class discriminative for both source and target data, which is of vital importance for the final classification. Therefore, in this paper, we put forward a novel feature learning method for domain adaptation to construct both domain invariant and class discriminative representations, referred to as DICD. Specifically, DICD is to learn a latent feature space with important data properties preserved, which reduces the domain difference by jointly matching the marginal and class-conditional distributions of both domains, and simultaneously maximizes the inter-class dispersion and minimizes the intra-class scatter as much as possible. Experiments in this paper have demonstrated that the class discriminative properties will dramatically alleviate the cross-domain distribution inconsistency, which further boosts the classification performance. Moreover, we show that exploring both domain invariance and class discriminativeness of the learned representations can be integrated into one optimization framework, and the optimal solution can be derived effectively by solving a generalized eigen-decomposition problem. Comprehensive experiments on several visual cross-domain classification tasks verify that DICD can outperform the competitors significantly. [ABSTRACT FROM AUTHOR]

Kinship recognition is a prominent research aiming to find if kinship relation exists between two different individuals. In general, child closely resembles his/her parents more than others based on facial similarities. These similarities are due to genetically inherited facial features that a child shares with his/her parents. Most existing researches in kinship recognition focus on full facial images to find these kinship similarities. This paper first presents kinship recognition for similar full facial images using proposed Global-based dual-tree complex wavelet transform (G-DTCWT). We then present novel patch-based kinship recognition methods based on dual-tree complex wavelet transform (DT-CWT): Local Patch-based DT-CWT (LP-DTCWT) and Selective Patch-Based DT-CWT (SP-DTCWT). LP-DTCWT extracts coefficients for smaller facial patches for kinship recognition. SP-DTCWT is an extension to LP-DTCWT and extracts coefficients only for representative patches with similarity scores above a normalized cumulative threshold. This threshold is computed by a novel patch selection process. These representative patches contribute more similarities in parent/child image pairs and improve kinship accuracy. Proposed methods are extensively evaluated on different publicly available kinship datasets to validate kinship accuracy. Experimental results showcase efficacy of proposed methods on all kinship datasets. SP-DTCWT achieves competitive accuracy to state-of-the-art methods. Mean kinship accuracy of SP-DTCWT is 95.85% on baseline KinFaceW-I and 95.30% on KinFaceW-II datasets. Further, SP-DTCWT achieves the state-of-the-art accuracy of 80.49% on the largest kinship dataset, Families In the Wild (FIW). [ABSTRACT FROM AUTHOR]

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue task involves multiple rounds of dialogues which cover a broad range of visual content that could be related to any objects, relationships or high-level semantics. Thus one of the key challenges in Visual Dialogue task is to learn a more comprehensive and semantic-rich image representation that can adaptively attend to the visual content referred by variant questions. In this paper, we first propose a novel scheme to depict an image from both visual and semantic views. Specifically, the visual view aims to capture the appearance-level information in an image, including objects and their visual relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Furthermore, on top of such dual-view image representations, we propose a Dual Encoding Visual Dialogue (DualVD) module, which is able to adaptively select question-relevant information from the visual and semantic views in a hierarchical mode. To demonstrate the effectiveness of DualVD, we propose two novel visual dialogue models by applying it to the Late Fusion framework and Memory Network framework. The proposed models achieve state-of-the-art results on three benchmark datasets. A critical advantage of the DualVD module lies in its interpretability. We can analyze which modality (visual or semantic) has more contribution in answering the current question by explicitly visualizing the gate values. It gives us insights in understanding of information selection mode in the Visual Dialogue task. The code is available at https://github.com/JXZe/Learning_DualVD. [ABSTRACT FROM AUTHOR]

Salient object detection has undergone a very rapid development with the blooming of Deep Neural Network (DNN), which is usually taken as an important preprocessing procedure in various computer vision tasks. However, the down-sampling operations, such as pooling and striding, always make the final predictions blurred at edges, which has seriously degenerated the performance of salient object detection. In this paper, we propose a simple yet effective approach, i.e., Hierarchical and Interactive Refinement Network (HIRN), to preserve the edge structures in detecting salient objects. In particular, a novel multi-stage and dual-path network structure is designed to estimate the salient edges and regions from the low-level and high-level feature maps, respectively. As a result, the predicted regions will become more accurate by enhancing the weak responses at edges, while the predicted edges will become more semantic by suppressing the false positives in background. Once the salient maps of edges and regions are obtained at the output layers, a novel edge-guided inference algorithm is introduced to further filter the resulting regions along the predicted edges. Extensive experiments on several benchmark datasets have been conducted, in which the results show that our method significantly outperforms a variety of state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Hashing has been an important and effective technology in image retrieval due to its computational efficiency and fast search speed. The traditional hashing methods usually learn hash functions to obtain binary codes by exploiting hand-crafted features, which cannot optimally represent the information of the sample. Recently, deep learning methods can achieve better performance, since deep learning architectures can learn more effective image representation features. However, these methods only use semantic features to generate hash codes by shallow projection but ignore texture details. In this paper, we proposed a novel hashing method, namely hierarchical recurrent neural hashing (HRNH), to exploit hierarchical recurrent neural network to generate effective hash codes. There are three contributions of this paper. First, a deep hashing method is proposed to extensively exploit both spatial details and semantic information, in which, we leverage hierarchical convolutional features to construct image pyramid representation. Second, our proposed deep network can exploit directly convolutional feature maps as input to preserve the spatial structure of convolutional feature maps. Finally, we propose a new loss function that considers the quantization error of binarizing the continuous embeddings into the discrete binary codes, and simultaneously maintains the semantic similarity and balanceable property of hash codes. Experimental results on four widely used data sets demonstrate that the proposed HRNH can achieve superior performance over other state-of-the-art hashing methods. [ABSTRACT FROM PUBLISHER]

The task of feature selection is to find the most representative features from the original high-dimensional data. Because of the absence of the information of class labels, selecting the appropriate features in unsupervised learning scenarios is much harder than that in supervised scenarios. In this paper, we investigate the potential of locally linear embedding (LLE), which is a popular manifold learning method, in feature selection task. It is straightforward to apply the idea of LLE to the graph-preserving feature selection framework. However, we find that this straightforward application suffers from some problems. For example, it fails when the elements in the feature are all equal; it does not enjoy the property of scaling invariance and cannot capture the change of the graph efficiently. To solve these problems, we propose a new filter-based feature selection method based on LLE in this paper, which is named as LLE score. The proposed criterion measures the difference between the local structure of each feature and that of the original data. Our experiments of classification task on two face image data sets, an object image data set, and a handwriting digits data set show that LLE score outperforms state-of-the-art methods, including data variance, Laplacian score, and sparsity score. [ABSTRACT FROM PUBLISHER]

Mobile devices usually mount a depth sensor to resolve ill-posed problems, like salient object detection on cluttered background. The main barrier of exploring RGBD data is to handle the information from two different modalities. To cope with this problem, in this paper, we propose a boundary-aware cross-modal fusion network for RGBD salient object detection. In particular, to enhance the fusion of color and depth features, we present a cross-modal feature sampling module to balance the contribution of the RGB and depth features based on the statistics of their channel values. In addition, in our multi-scale dense fusion network architecture, we not only incorporate edge-sensitive losses to preserve the boundary of the detected salient region, but also refine its structure by merging the estimated saliency maps of different scales. We accomplish the multi-scale saliency map merging using two alternative methods which produce refined saliency maps via per-pixel weighted combination and an encoder-decoder network. Extensive experimental evaluations demonstrate that our proposed framework can achieve the state-of-the-art performance on several public RGBD-based datasets. [ABSTRACT FROM AUTHOR]

A new multi-scale deep learning (MDL) framework is proposed and exploited for conducting image interpolation in this paper. The core of the framework is a seeding network that needs to be designed for the targeted task. For image interpolation, a novel attention-aware inception network (AIN) is developed as the seeding network; it has two key stages: 1) feature extraction based on the low-resolution input image; and 2) feature-to-image mapping to enlarge image’s size or resolution. Note that the designed seeding network, AIN, needs to be trained with a matched training dataset at each scale. For that, multi-scale image patches are generated using our proposed pyramid cut, which outperforms the conventional image pyramid method by completely avoiding aliasing issue. After training, the trained AINs are then combined for processing the input image in the testing stage. Extensive experimental simulation results obtained from seven image datasets (comprising 359 images in total) have clearly shown that the proposed MAIN consistently delivers highly accurate interpolated images. [ABSTRACT FROM AUTHOR]

Various weather conditions, such as rain, haze, or snow, can degrade visual quality in images/videos, which may significantly degrade the performance of related applications. In this paper, a novel framework based on sequential dual attention deep network is proposed for removing rain streaks (deraining) in a single image, called by SSDRNet (Sequential dual attention-based Single image DeRaining deep Network). Since the inherent correlation among rain steaks within an image should be stronger than that between the rain streaks and the background (non-rain) pixels, a two-stage learning strategy is implemented to better capture the distribution of rain streaks within a rainy image. The two-stage deep neural network primarily involves three blocks: residual dense blocks (RDBs), sequential dual attention blocks (SDABs), and multi-scale feature aggregation modules (MAMs), which are all delicately and specifically designed for rain removal. The two-stage strategy successfully learns very fine details of the rain steaks of the image and then clearly removes them. Extensive experimental results have shown that the proposed deep framework achieves the best performance on qualitative and quantitative metrics compared with state-of-the-art methods. The corresponding code and the trained model of the proposed SSDRNet have been available online at https://github.com/fityanul/SDAN-for-Rain-Removal. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel method for Multispectral document image binarization (MSdB) through the Non-negative Matrix Factorization (NMF) approach. We propose a three-step MSdB-NMF framework: i) NMF-based feature extraction algorithm by introducing a new optimization problem; ii) post-processing method iii); apply any existing gray/RGB binarization scheme. In the first step, we extract $N$ features out of $B$ spectral bands ($N < B$) and their corresponding coefficient matrix. We introduce a novel objective formulation that considers the robustness (related to the noise and various types of degradations) and sparseness (related to the ratio of text pixels versus the background). We employ the multiplicative updating rules to solve the proposed minimization problem and prove the convergence of the proposed feature extraction algorithm. In the next step, we select an appropriate feature vector, equivalently the corresponding coefficient vector. We propose to select it either visually or automatically via a post-processing method, which uses the benchmark binarization methods as baseline. In the last step, we apply some existing binarization methods such as Sauvola and Howe over the selected coefficient vector. Our proposed binarization framework is applicable for any kind of MS or hyperspectral (HS) document image without considering any prior knowledge such as the side information about the spectral bands of MS/HS document image. We evaluate our proposed binarization framework over two MS document image datasets. The experimental results confirm that our proposed framework outperforms several state-of-the-art binarization schemes including the winner of the contest in MS-TEx-2015. [ABSTRACT FROM AUTHOR]

Recent progress in vision-based fire detection is driven by convolutional neural networks. However, the existing methods fail to achieve a good tradeoff among accuracy, model size, and speed. In this paper, we propose an accurate fire detection method that achieves a better balance in the abovementioned aspects. Specifically, a multiscale feature extraction mechanism is employed to capture richer spatial details, which can enhance the discriminative ability of fire-like objects. Then, the implicit deep supervision mechanism is utilized to enhance the interaction among information flows through dense skip connections. Finally, a channel attention mechanism is employed to selectively emphasize the contribution between different feature maps. Experimental results demonstrate that our method achieves 95.3% accuracy, which outperforms the suboptimal method by 2.5%. Moreover, the speed and model size of our method are 3.76% faster on the GPU and 63.64% smaller than the suboptimal method, respectively. [ABSTRACT FROM AUTHOR]

Crowd counting is a challenging problem due to the diverse crowd distribution and background interference. In this paper, we propose a new approach for head size estimation to reduce the impact of different crowd scale and background noise. Different from just using local information of distance between human heads, the global information of the people distribution in the whole image is also under consideration. We obey the order of far- to near-region (small to large) to spread head size, and ensure that the propagation is uninterrupted by inserting dummy head points. The estimated head size is further exploited, such as dividing the crowd into parts of different densities and generating a high-fidelity head mask. On the other hand, we design three different head mask usage mechanisms and the corresponding head masks to analyze where and which mask could lead to better background filtering. Based on the learned masks, two competitive models are proposed which can perform robust crowd estimation against background noise and diverse crowd scale. We evaluate the proposed method on three public crowd counting datasets of ShanghaiTech, UCFQNRF and UCFCC_50. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art crowd counting approaches. [ABSTRACT FROM AUTHOR]

Three-dimensional multi-modal data are used to represent 3D objects in the real world in different ways. Features separately extracted from multimodality data are often poorly correlated. Recent solutions leveraging the attention mechanism to learn a joint-network for the fusion of multimodality features have weak generalization capability. In this paper, we propose a hamming embedding sensitivity network to address the problem of effectively fusing multimodality features. The proposed network called HamNet is the first end-to-end framework with the capacity to theoretically integrate data from all modalities with a unified architecture for 3D shape representation, which can be used for 3D shape retrieval and recognition. HamNet uses the feature concealment module to achieve effective deep feature fusion. The basic idea of the concealment module is to re-weight the features from each modality at an early stage with the hamming embedding of these modalities. The hamming embedding also provides an effective solution for fast retrieval tasks on a large scale dataset. We have evaluated the proposed method on the large-scale ModelNet40 dataset for the tasks of 3D shape classification, single modality and cross-modality retrieval. Comprehensive experiments and comparisons with state-of-the-art methods demonstrate that the proposed approach can achieve superior performance. [ABSTRACT FROM AUTHOR]

Recognition of facial expressions across various actors, contexts, and recording conditions in real-world videos involves identifying local facial movements. Hence, it is important to discover the formation of expressions from local representations captured from different parts of the face. So in this paper, we propose a dynamic kernel-based representation for facial expressions that assimilates facial movements captured using local spatio-temporal representations in a large universal Gaussian mixture model (uGMM). These dynamic kernels are used to preserve local similarities while handling global context changes for the same expression by utilizing the statistics of uGMM. We demonstrate the efficacy of dynamic kernel representation using three different dynamic kernels, namely, explicit mapping based, probability-based, and matching-based, on three standard facial expression datasets, namely, MMI, AFEW, and BP4D. Our evaluations show that probability-based kernels are the most discriminative among the dynamic kernels. However, in terms of computational complexity, intermediate matching kernels are more efficient as compared to the other two representations. [ABSTRACT FROM AUTHOR]

Salient object detection aims at locating the most conspicuous objects in natural images, which usually acts as a very important pre-processing procedure in many computer vision tasks. In this paper, we propose a simple yet effective Hierarchical U-shape Attention Network (HUAN) to learn a robust mapping function for salient object detection. Firstly, a novel attention mechanism is formulated to improve the well-known U-shape network, in which the memory consumption can be extensively reduced and the mask quality can be significantly improved by the resulting U-shape Attention Network (UAN). Secondly, a novel hierarchical structure is constructed to well bridge the low-level and high-level feature representations between different UANs, in which both the intra-network and inter-network connections are considered to explore the salient patterns from a local to global view. Thirdly, a novel Mask Fusion Network (MFN) is designed to fuse the intermediate prediction results, so as to generate a salient mask which is in higher-quality than any of those inputs. Our HUAN can be trained together with any backbone network in an end-to-end manner, and high-quality masks can be finally learned to represent the salient objects. Extensive experimental results on several benchmark datasets show that our method significantly outperforms most of the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Data reconstruction, which aims at preserving statistical properties of the data during the reconstruction has become a new criterion for feature selection. Although feature selection could benefit from the perspective of data reconstruction, it is unable to exploit other crucial information, namely, graph structure and pairwise constraints. To address previously mentioned deficiency, we propose a novel feature selection approach in this paper, known as unsupervised feature selection via data reconstruction and side information. More specifically, the proposed method takes advantage of the prior knowledge regarding pairwise constraints (side information), the minimization of data reconstruction error, and the graph embedding simultaneously, such that pivotal features are selected with preserving data manifold structure. To obtain the robust solution, a robust loss function is applied to the feature selection problem, which interpolates between $\ell _{1}$ -norm and $\ell _{2}$ -norm. Eventually, extensive experiments are conducted to demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Learning robust and discriminative representations is essential for 3D object retrieval. In this paper, we present an improved Multi-view Convolutional Neural Network (MVCNN) for view-based 3D object representation learning. Our technical contributions are divided into two aspects. First, we propose to employ Group-view Similarity Learning (GSL) over the multi-view representations before the aggregation operation (i.e., max-pooling in MVCNN). We assume that the similarity information among the view groups of different 3D objects can provide an important cue but has been neglected more or less by previous methods. To enhance it, we add a branch to the original MVCNN architecture and learn to maintain such group-view similarity relationships. Second, we utilize an end-to-end metric learning loss function to improve the representation learning process. In particular, we propose an improved Triplet-Center Loss (TCL) named Adaptive Margin based Triplet-Center Loss (AMTCL). The original TCL assumes a fixed and common margin to control the relative distance relationship between a sample to its corresponding class center and to the nearest negative center. Though TCL has demonstrated its great capacity on the 3D object retrieval task, however, when considering the distinguishability between samples of one class and samples of another class, we assume that it would be more appropriate that the margin takes different values based on the distinguishability of samples of different classes. Therefore we propose to adaptively and dynamically adjust the margin hyperparameter based on the normalized confusion matrix which is obtained on the training set during the training process. Extensive experiments on several public 3D shape benchmarks show that our method, GSL + AMTCL, can learn more suitable representations for 3D object retrieval, obtaining superior performance against state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, a novel full-reference image quality assessment (IQA) method for evaluating the quality of the distorted light field (LF) image against its reference LF image is proposed, called the log-Gabor feature-based light field coherence (LGF-LFC). Based on the fact that to compare two LF images, it essentially boils down to measure how coherent of these two LF images, we attempt to measure the degree of their LF coherence (LFC). To pursue this goal, the salient features from the reference and distorted LF images under comparison need to be extracted. By considering that the Gabor feature has the ability to well characterize the human visual system (HVS) perception, and the special characteristics of the LF images, the multi-scale and single-scale Gabor feature extraction schemes are developed to extract the multi-scale log-Gabor features from the sub-aperture images (SAIs) and the single-scale log-Gabor feature from the epi-polar images (EPIs), respectively. Note that the former can reflect the image details (via the SAIs), while the latter indicates the viewing consistency (via the EPI’s depth information). The similarity measurements are subsequently conducted on the comparison of their SAIs and that of their EPIs separately, followed by combining them together for arriving at the final score. Extensive simulation results have clearly demonstrated that the proposed LGF-LFC is more consistent with the perception of the HVS on the quality evaluation of the LF images than multiple classical and state-of-the-art IQA methods. [ABSTRACT FROM AUTHOR]

Low-light image enhancement is a challenging task that has attracted considerable attention. Pictures taken in low-light conditions often have bad visual quality. To address the problem, we regard the low-light enhancement as a residual learning problem that is to estimate the residual between low- and normal-light images. In this paper, we propose a novel Deep Lightening Network (DLN) that benefits from the recent development of Convolutional Neural Networks (CNNs). The proposed DLN consists of several Lightening Back-Projection (LBP) blocks. The LBPs perform lightening and darkening processes iteratively to learn the residual for normal-light estimations. To effectively utilize the local and global features, we also propose a Feature Aggregation (FA) block that adaptively fuses the results of different LBPs. We evaluate the proposed method on different datasets. Numerical results show that our proposed DLN approach outperforms other methods under both objective and subjective metrics. [ABSTRACT FROM AUTHOR]

The deep convolutional neural network (CNN) has achieved great success in image recognition. Many image quality assessment (IQA) methods directly use recognition-oriented CNN for quality prediction. However, the properties of IQA task is different from image recognition task. Image recognition should be sensitive to visual content and robust to distortion, while IQA should be sensitive to both distortion and visual content. In this paper, an IQA-oriented CNN method is developed for blind IQA (BIQA), which can efficiently represent the quality degradation. CNN is large-data driven, while the sizes of existing IQA databases are too small for CNN optimization. Thus, a large IQA dataset is firstly established, which includes more than one million distorted images (each image is assigned with a quality score as its substitute of Mean Opinion Score (MOS), abbreviated as pseudo-MOS). Next, inspired by the hierarchical perception mechanism (from local structure to global semantics) in human visual system, a novel IQA-orientated CNN method is designed, in which the hierarchical degradation is considered. Finally, by jointly optimizing the multilevel feature extraction, hierarchical degradation concatenation (HDC) and quality prediction in an end-to-end framework, the Cascaded CNN with HDC (named as CaHDC) is introduced. Experiments on the benchmark IQA databases demonstrate the superiority of CaHDC compared with existing BIQA methods. Meanwhile, the CaHDC (with about 0.73M parameters) is lightweight comparing to other CNN-based BIQA models, which can be easily realized in the microprocessing system. The dataset and source code of the proposed method are available at https://web.xidian.edu.cn/wjj/paper.html. [ABSTRACT FROM AUTHOR]

In this paper, we propose to learn recurrent 3D attention (A3D) for video-based person re-identification. Attention model plays a key role in both spatial and temporal domains for video representation. Most existing methods apply spatial attention model to extract feature from a single image and aggregate image features with attentive temporal pooling or RNN. However, the inherent consistencies and correlations between spatial and temporal clues are not leveraged. Our A3D method aims to utilize the joint constraints of temporal and spatial attentions to enhance the robustness of attention model. Towards this goal, we treat the pedestrian video as a unified 3D bin where the temporal domain is denoted as an additional dimension. Then we develop an attention agent to iteratively select the locations of the salient spatial-temporal parts in the 3D bin. In addition, we formulate our sequential 3D attention learning as a Markov Decision Process and train the representation network and attention detector with the policy gradient method in an end-to-end manner. We evaluate the proposed method on three challenging datasets including iLIDS-VID, PRID-2011 and the large-scale MARS dataset, and consistently improve the performance in comparison with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Generating text to have a consistent style with only a few observed highly-stylized text samples is a difficult task for image processing. The text style involving the typography, i.e., font, stroke, color, decoration, effects, etc., should be considered for transfer. In this paper, we propose a novel approach to stylize target text by decoding weighted deep features from only a few referenced samples. The deep features, including content and style features of each referenced text, are extracted from a Convolutional Neural Network (CNN) that is optimized for character recognition. Then, we calculate the similarity scores of the target text and the referenced samples by measuring the distance along the corresponding channels from the content features of the CNN when considering only the content, and assign them as the weights for aggregating the deep features. To enforce the stylized text to be realistic, a discriminative network with adversarial loss is employed. We demonstrate the effectiveness of our network by conducting experiments on three different datasets which have various styles, fonts, languages, etc. Additionally, the coefficients for character style transfer, including the character content, the effect of similarity matrix, the number of referenced characters, the similarity between characters, and performance evaluation by a new protocol are analyzed for better understanding our proposed framework. [ABSTRACT FROM AUTHOR]

Learning discriminative, view-invariant and multi-scale representations of object appearance with different semantic levels is of paramount importance for person Re-Identification (ReID). Recently, the community has focused on learning deep Re-ID models to capture a single holistic representation. To improve the achieved results, additional visual attributes and object part-driven models have been considered, inevitably introducing additional human annotation labor or computational efforts. In this paper, we argue that pyramid-inspired methods capturing multi-scale information may overcome such requirements. Precisely, multi-scale pooled regions representing visual information of an object are integrated within a novel deep architecture factorizing them into discriminative features at multiple semantic levels. These are exploited through an attention mechanism later considered in an identification-similarity multi-task loss, trained by means of a curriculum learning strategy. Extensive results on three person ReID benchmarks demonstrate that better performance than existing methods are achieved. Code is available at https://github.com/iN1k1. [ABSTRACT FROM AUTHOR]

In this paper, we present an end-to-end architecture for multi-exposure image fusion based on generative adversarial networks, termed as MEF-GAN. In our architecture, a generator network and a discriminator network are trained simultaneously to form an adversarial relationship. The generator is trained to generate a real-like fused image based on the given source images which is expected to fool the discriminator. Correspondingly, the discriminator is trained to distinguish the generated fused images from the ground truth. The adversarial relationship makes the fused image not limited to the restriction of the content loss. Therefore, the fused images are closer to the ground truth in terms of probability distribution, which can compensate for the insufficiency of single content loss. Moreover, aiming at the problem that the luminance of multi-exposure images varies greatly with spatial location, the self-attention mechanism is employed in our architecture to allow for attention-driven and long-range dependency. Thus, local distortion, confusing results, or inappropriate representation can be corrected in the fused image. Qualitative and quantitative experiments are performed on publicly available datasets, where the results demonstrate that MEF-GAN outperforms the state-of-the-art, in terms of both visual effect and objective evaluation metrics. Our code is publicly available at https://github.com/jiayi-ma/MEF-GAN. [ABSTRACT FROM AUTHOR]

Existing successful person re-identification (Re-ID) models often employ the part-level representation to extract the fine-grained information, but commonly use the loss that is particularly designed for global features, ignoring the relationship between semantic parts. In this paper, we present a novel triplet loss that emphasizes the salient parts and also takes the consideration of alignment. This loss is based on the crossing-bing matching metric that also known as Wasserstein Distance. It measures how much effort it would take to move the embeddings of local features to align two distributions, such that it is able to find an optimal transport matrix to re-weight the distance of different local parts. The distributions in support of local parts is produced via a new attention mechanism, which is calculated by the inner product between high-level global feature and local features, representing the importance of different semantic parts w.r.t. identification. We show that the obtained optimal transport matrix can not only distinguish the relevant and misleading parts, and hence assign different weights to them, but also rectify the original distance according to the learned distributions, resulting in an elegant solution for the mis-alignment issue. Besides, the proposed method is easily implemented in most Re-ID learning system with end-to-end training style, and can obviously improve their performance. Extensive experiments and comparisons with recent Re-ID methods manifest the competitive performance of our method. [ABSTRACT FROM AUTHOR]

Video prediction refers to predicting and generating future video frames given a set of consecutive frames. Conventional video prediction methods usually criticize the discrepancy between the ground-truth and predictions frame by frame. As the prediction error accumulates recursively, these methods would easily become out of control and are often confined to the short-term horizon. In this paper, we introduce a retrospection process to rectify the prediction errors beyond criticizing the future prediction. The introduced retrospection process is designed to look back what have been learned from the past and rectify the prediction deficiencies. To this end, we build a retrospection network to reconstruct the past frames given the currently predicted frames. A retrospection loss is introduced to push the retrospection frames being consistent with the observed frames, so that the prediction error is alleviated. On the other hand, an auxiliary route is built by reversing the flow of time and executing a similar retrospection. These two routes interact with each other to boost the performance of retrospection network and enhance the understanding of dynamics across frames, especially for the long-term horizon. An adversarial loss is employed to generate more realistic results in both prediction and retrospection process. In addition, the proposed method can be used to extend many state-of-the-art video prediction methods. Extensive experiments on the natural video dataset demonstrate the advantage of introducing the retrospection process for long-term video prediction. [ABSTRACT FROM AUTHOR]

Polarimetric synthetic aperture radar (PolSAR) image segmentation is currently of great importance in image processing for remote sensing applications. However, it is a challenging task due to two main reasons. Firstly, the label information is difficult to acquire due to high annotation costs. Secondly, the speckle effect embedded in the PolSAR imaging process remarkably degrades the segmentation performance. To address these two issues, we present a contextual PolSAR image semantic segmentation method in this paper. With a newly defined channel-wise consistent feature set as input, the three-dimensional discrete wavelet transform (3D-DWT) technique is employed to extract discriminative multi-scale features that are robust to speckle noise. Then Markov random field (MRF) is further applied to enforce label smoothness spatially during segmentation. By simultaneously utilizing 3D-DWT features and MRF priors for the first time, contextual information is fully integrated during the segmentation to ensure accurate and smooth segmentation. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on three real benchmark PolSAR image data sets. Experimental results indicate that the proposed method achieves promising segmentation accuracy and preferable spatial consistency using a minimal number of labeled pixels. [ABSTRACT FROM AUTHOR]

Accurate depth estimation from images is a fundamental problem in computer vision. In this paper, we propose an unsupervised learning based method to predict high-quality depth map from multiple images. A novel multi-view constrained DenseDepthNet is designed for this task. Our DenseDepthNet can effectively leverage both the low-level and high-level features of input images and generate appealing results, especially with sharp details. We employ the public datasets KITTI and Cityscapes for training in an end-to-end unsupervised fashion. A novel depth consistency loss based on multi-view geometry constraint is also applied to the corresponding points across pairwise images, which helps to improve the quality of predicted depth maps significantly. We conduct comprehensive evaluations on our DenseDepthNet and our depth consistency loss function. Experiments validate that our method outperforms the state-of-the-art unsupervised methods and produce comparable results with supervised methods. [ABSTRACT FROM AUTHOR]

Traditional fine-grained image recognition is required to distinguish different subordinate categories (e.g., birds species) based on the visual cues beneath raw images. Due to both small inter-class variations and large intra-class variations, it is desirable to capture the subtle differences between these sub-categories, which is crucial but challenging for fine-grained recognition. Recently, language modality aggregation has been proved as a successful technique to improve visual recognition in the experience. In this paper, we introduce an end-to-end trainable Progressive Mask Attention (PMA) model for fine-grained recognition by leveraging both visual and language modalities. Our Bi-Modal PMA model can not only stage-by-stage capture the most discriminative part in the visual modality by our mask-based fashion, but also explore the out-of-visual-domain knowledge from the language modality in an interactional alignment paradigm. Specifically, at each stage, a self-attention module is proposed to attend to the key patch from images or text descriptions. Besides, a query-relational module is designed to seize the key words/phrases of texts and further bridge the connection between two modalities. Later, the learned representations of bi-modality from multiple stages are aggregated as the final features for recognition. Our Bi-Modal PMA model only needs raw images and raw text descriptions, without requiring bounding boxes/part annotations in images or key word annotations in texts. By conducting comprehensive experiments on fine-grained benchmark datasets, we demonstrate that the proposed method achieves superior performance over the competing baselines, on either vision and language bi-modality or single visual modality. [ABSTRACT FROM AUTHOR]

Light field imaging, which captures spatial-angular information of light incident on image sensors, enables many interesting applications such as image refocusing and augmented reality. However, due to the limited sensor resolution, a trade-off exists between the spatial and angular resolutions. To increase the angular resolution, view synthesis techniques have been adopted to generate new views from existing views. However, traditional learning-based view synthesis mainly considers the image quality of each view of the light field and neglects the quality of the refocused images. In this paper, we propose a new loss function called refocused image error (RIE) to address the issue. The main idea is that the image quality of the synthesized light field should be optimized in the refocused image domain because it is where the light field is viewed. We analyze the behavior of RIE in the spectral domain and test the performance of our approach against previous approaches on both real (INRIA) and software-rendered (HCI) light field datasets using objective assessment metrics such as MSE, MAE, PSNR, SSIM, and GMSD. Experimental results show that the light field generated by our method results in better refocused images than previous methods. [ABSTRACT FROM AUTHOR]

Both structural and contextual information is essential and widely used in image analysis. However, current multi-view stereo (MVS) approaches usually use a single common pre-trained model as pixel descriptor to extract features, which mix structural and contextual information together and thus increase the difficulty of matching correspondence. In this paper, we propose FADE (feature aggregation for depth estimation), which treats spatial and context information separately and focuses on aggregating features for efficient learning of the MVS problem. Spatial information includes image details such as edges and corners, whereas context information comprises object features such as shapes and traits. To aggregate these multi-level features, we use an attention mechanism to select important features for matching. We then build a plane sweep volume by using a homography backward warping method to generate match candidates. Furthermore, we propose a novel cost volume regularization network aims to minimize the noise in the matching candidates. Finally, we take advantage of 3D stacked hourglass and regression to produces high-quality depth maps. With these well-aggregated features, FADE can efficiently perform dense depth reconstruction, achieving state-of-the-art performance in terms of accuracy and requiring the least amount of model parameters. [ABSTRACT FROM AUTHOR]

Crowd counting is challenging due to unconstrained imaging factors, e.g., background clutters, non-uniform distribution of people, large scale and perspective variations. Dealing with these problems using deep neural networks requires rich prior knowledge and multi-scale contextual representations. In this paper, we propose a Cross-stage Refinement Network (CRNet) that can refine predicted density maps progressively based on hierarchical multi-level density priors. In particular, CRNet is composed of several fully convolutional networks. They are stacked together recursively with the previous output as the next input, and each of them serves to utilize previous density output to gradually correct prediction errors of crowd areas and refine the predicted density maps at different stages. Cross-stage multi-level density priors are further exploited in our recurrent framework by the cross-stage skip layers based on ConvLSTM. To cope with different challenges of unconstrained crowd scenes, we explore different crowd-specific data augmentation methods to mimic real-world scenarios and enrich crowd feature representations from different aspects. Extensive experiments show the proposed method achieves superior performances against state-of-the-art methods on four widely-used challenging benchmarks in terms of counting accuracy and density map quality. Code and models are available at this https://github.com/lytgftyf/Crowd-Counting-via-Cross-stage-Refinement-Networks. [ABSTRACT FROM AUTHOR]

Inevitable compression artifacts in multi-view video (MVV) can clearly degrade the quality of experience in many interaction-oriented 3D visual applications. Under the framework of asymmetric coding, low-quality images can be enhanced with high-quality images from the neighboring viewpoints considering the similarity among different views. However, compression artifacts and warping error cause different cross-view quality gaps for various sequences, and thus the contribution of cross-view priors can hardly be located and considered in previous works. In this paper, we propose a multi-view graph neural network (MV-GNN) to reduce compression artifacts in multi-view compressed images. We dedicate to design a fusion mechanism which can exploit contributions from neighboring viewpoints and meanwhile suppress the misleading information. In our method, a GNN-based fusion mechanism is designed to fuse the cross-view information under the aggregation and update mechanism of GNN. Experiments show that 1.672 dB and 0.0242 average gains on PSNR and SSIM metrics can be obtained, respectively. For the subjective evaluations, blocking effect in the compressed images are clearly suppressed and the damaged object boundary are better recovered. The experimental results demonstrate that our MV-GNN outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Data-driven rain streak removal methods, most of which rely on synthesized paired data, usually come across the generalization problem when being applied in real scenarios. In this paper, we propose a novel deep-learning based rain streak removal method injected with self-supervision to obtain the capacity of removing more varied-scale rain streaks in practical applications. To this end, in this work, efforts are made from two perspectives. First, considering that rain streak removal is highly correlated with texture characteristics, we create a fractal band learning (FBL) network based on frequency band recovery. It integrates commonly seen band feature operations as neural forms and effectively improves the capacity to capture discriminative features for deraining. Second, to further improve the generalization ability of FBL to remove rain streaks of varied scales, we incorporate scale-robust self-supervision to regularize the network training. The constraint forces the extracted features of an input rain image at different scales to be equivalent after rescaling operations. Therefore, our method can offer similar responses based on solely image content without the interference of scale change and is capable to remove varied-scale rain streaks. Extensive experiments in quantitative and qualitative evaluations demonstrate the superiority of our method for rain streak removal, especially for the real cases where very large rain streaks exist, and prove the effectiveness of each component. [ABSTRACT FROM AUTHOR]

In recent years, deep neural networks have achieved excellent performance in many fields of artificial intelligence. The requirements for the interpretability and robustness of neural networks are also increasing. In this paper, we propose to understand the functional mechanism of neural networks by extracting critical subnetworks. Specifically, we denote the critical subnetworks as a group of important channels across layers such that if they were suppressed to zeros, the final test performance would deteriorate severely. This novel perspective can not only reveal the layerwise semantic behavior within the model but also present more accurate visual explanations appearing in the data through attribution methods. Moreover, we propose two adversarial example detection methods based on the properties of sample-specific and class-specific subnetworks, which provides the possibility for increasing the model robustness. [ABSTRACT FROM AUTHOR]

Facial expression recognition, face synthesis, and face alignment are three coherently related tasks and can be solved in a joint framework. To achieve this goal, in this paper, we propose a novel end-to-end deep learning model by exploiting the expression code, geometry code and generated data jointly for simultaneous pose-invariant facial expression recognition, face image synthesis, and face alignment. The proposed deep model enjoys several merits. First, to the best of our knowledge, this is the first work to address these three tasks jointly in a unified deep model to complement and enhance each other. Second, the proposed model can effectively disentangle the global and local identity representation from different expression and geometry codes. As a result, it can automatically generate facial images with different expressions under arbitrary geometry codes. Third, these three tasks can further boost their performance for each other via our model. Extensive experimental results on three standard benchmarks demonstrate that the proposed deep model performs favorably against state-of-the-art methods on the three tasks. [ABSTRACT FROM AUTHOR]

Different from many other attributes, facial expression can change in a continuous way, and therefore, a slight semantic change of input should also lead to the output fluctuation limited in a small scale. This consistency is important. However, current Facial Expression Recognition (FER) datasets may have the extreme imbalance problem, as well as the lack of data and the excessive amounts of noise, hindering this consistency and leading to a performance decreasing when testing. In this paper, we not only consider the prediction accuracy on sample points, but also take the neighborhood smoothness of them into consideration, focusing on the stability of the output with respect to slight semantic perturbations of the input. A novel method is proposed to formulate semantic perturbation and select unreliable samples during training, reducing the bad effect of them. Experiments show the effectiveness of the proposed method and state-of-the-art results are reported, getting closer to an upper limit than the state-of-the-art methods by a factor of 30% in AffectNet, the largest in-the-wild FER database by now. [ABSTRACT FROM AUTHOR]

In this paper, we tackle no-reference image quality assessment (NR-IQA), which aims to predict the perceptual quality of a distorted image without referencing its pristine-quality counterpart. Inspired by the free-energy principle, we assume that, while perceiving a distorted image, the human visual system (HVS) tends to predict the pristine image then estimates the perceptual quality based on the distorted-restored pair. Furthermore, the perceptual quality depends heavily on the way how human beings attend to distorted images, namely, the cooperation of foveal vision and the eye movement mechanism. Inspired by these properties of the HVS, given the distorted-restored pair, we implement an attention-driven NR-IQA method with reinforcement learning (RL). The model learns a policy to attend to several regions parallelly. The observations of the fixation regions are aggregated in a weighted average way, which is inspired by the robust averaging strategy. For policy learning, the rewards are derived from two tasks—classifying the distortion type and estimating the perceptual score. The goal of policy learning is to maximize the expectation of the accumulated rewards. Extensive experiments on LIVE, TID2008, TID2013 and CSIQ demonstrate the superiority of our methods. [ABSTRACT FROM AUTHOR]

Existing saliency models typically incorporate contexts holistically. However, for each pixel, usually only part of its context region contributes to saliency prediction, while other parts are likely either noise or distractions. In this paper, we propose a novel pixel-wise contextual attention network (PiCANet) to selectively attend to informative context locations at each pixel. The proposed PiCANet generates an attention map over the contextual region of each pixel and construct attentive contextual features via selectively incorporating the features of useful context locations. We present three formulations of the PiCANet via embedding the pixel-wise contextual attention mechanism into the pooling and convolution operations with attending to global or local contexts. All the three models are fully differentiable and can be integrated with convolutional neural networks with joint training. In this work, we introduce the proposed PiCANets into a U-Net model for salient object detection. The generated global and local attention maps can learn to incorporate global contrast and regional smoothness, which help localize and highlight salient objects more accurately and uniformly. Experimental results show that the proposed PiCANets perform effectively for saliency detection against the state-of-the-art methods. Furthermore, we demonstrate the effectiveness and generalization ability of the PiCANets on semantic segmentation and object detection with improved performance. [ABSTRACT FROM AUTHOR]

A key for person re-identification is achieving consistent local details for discriminative representation across variable environments. Current stripe-based feature learning approaches have delivered impressive accuracy, but do not make a proper trade-off between diversity, locality, and robustness, which easily suffers from part semantic inconsistency for the conflict between rigid partition and misalignment. This paper proposes a receptive multi-granularity learning approach to facilitate stripe-based feature learning. This approach performs local partition on the intermediate representations to operate receptive region ranges, rather than current approaches on input images or output features, thus can enhance the representation of locality while remaining proper local association. Toward this end, the local partitions are adaptively pooled by using significance-balanced activations for uniform stripes. Random shifting augmentation is further introduced for a higher variance of person appearing regions within bounding boxes to ease misalignment. By two-branch network architecture, different scales of discriminative identity representation can be learned. In this way, our model can provide a more comprehensive and efficient feature representation without larger model storage costs. Extensive experiments on intra-dataset and cross-dataset evaluations demonstrate the effectiveness of the proposed approach. Especially, our approach achieves a state-of-the-art accuracy of 96.2%@Rank-1 or 90.0%@mAP on the challenging Market-1501 benchmark. [ABSTRACT FROM AUTHOR]

Current non-Lambertian photometric stereo methods generally require a large number of images to ensure accurate surface normal estimation. To achieve accurate surface normal recovery under a sparse set of lights, this paper proposes a non-Lambertian photometric stereo network based on a derived inverse reflectance model with collocated light. The model is deduced using monotonicity of isotropic reflectance and the univariate property of collocated light to decouple the surface normal from the reflectance function. Thus, the surface normal can be estimated by three steps, i.e., model fitting, shadow rejection, and normal estimation. We leverage a supervised deep learning technique to enhance the shadow rejection ability and the flexibility of the inverse reflectance model. Shadows are handled through max-pooling. Information from a neighborhood image patch is utilized to improve the flexibility to various reflectances. Experiments using both synthetic and real images demonstrate that the proposed method achieves state-of-the-art accuracy in surface normal estimation. [ABSTRACT FROM AUTHOR]

Principal skewness analysis (PSA) has been introduced for feature extraction in hyperspectral imagery. As a third-order generalization of principal component analysis (PCA), its solution of searching for the local maximum skewness direction is transformed into the problem of calculating the eigenpairs (the eigenvalues and the corresponding eigenvectors) of a coskewness tensor. By combining a fixed-point method with an orthogonal constraint, the new eigenpairs are prevented from converging to the same previously determined maxima. However, in general, the eigenvectors of the supersymmetric tensor are not inherently orthogonal, which implies that the results obtained by the search strategy used in PSA may unavoidably deviate from the actual eigenpairs. In this paper, we propose a new nonorthogonal search strategy to solve this problem and the new algorithm is named nonorthogonal principal skewness analysis (NPSA). The contribution of NPSA lies in the finding that the search space of the eigenvector to be determined can be enlarged by using the orthogonal complement of the Kronecker product of the previous eigenvector with itself, instead of its orthogonal complement space. We also give a detailed theoretical proof on why we can obtain the more accurate eigenpairs through the new search strategy by comparison with PSA. In addition, after some algebraic derivations, the complexity of the presented algorithm is also greatly reduced. Experiments with both simulated data and real multi/hyperspectral imagery demonstrate its validity in feature extraction. [ABSTRACT FROM AUTHOR]

In this paper, we devise a saliency prediction model for stereoscopic videos that learns to explore saliency inspired by the component-based interactions including spatial, temporal, as well as depth cues. The model first takes advantage of specific structure of 3D residual network (3D-ResNet) to model the saliency driven by spatio-temporal coherence from consecutive frames. Subsequently, the saliency inferred by implicit-depth is automatically derived based on the displacement correlation between left and right views by leveraging a deep convolutional network (ConvNet). Finally, a component-wise refinement network is devised to produce final saliency maps over time by aggregating saliency distributions obtained from multiple components. In order to further facilitate research towards stereoscopic video saliency, we create a new dataset including 175 stereoscopic video sequences with diverse content, as well as their dense eye fixation annotations. Extensive experiments support that our proposed model can achieve superior performance compared to the state-of-the-art methods on all publicly available eye fixation datasets. [ABSTRACT FROM AUTHOR]

People in crowd scenes always exhibit consistent behaviors and form collective motions. The analysis of collective motion has motivated a surge of interest in computer vision. Nevertheless, the effort is hampered by the complex nature of collective motions. Considering the fact that collective motions are formed by individuals, this paper proposes a new framework for both quantifying and detecting collective motion by investigating the spatio-temporal behavior of individuals. The main contributions of this work are threefold: 1) an intention-aware model is built to fully capture the intrinsic dynamics of individuals; 2) a structure-based collectiveness measurement is developed to accurately quantify the collective properties of crowds; 3) a multi-stage clustering strategy is formulated to detect both the local and global behavior consistency in crowd scenes. Experiments on real world data sets show that our method is able to handle crowds with various structures and time-varying dynamics. Especially, the proposed method shows nearly 10% improvement over the competitors in terms of NMI, Purity and RI. Its applicability is illustrated in the context of anomaly detection and semantic scene segmentation. [ABSTRACT FROM AUTHOR]

Despite great success has been achieved in activity analysis, it still has many challenges. Most existing works in activity recognition pay more attention to designing efficient architecture or video sampling strategy. However, due to the property of fine-grained action and long term structure in video, activity recognition is expected to reason temporal relation between video sequences. In this paper, we propose an efficient temporal reasoning graph (TRG) to simultaneously capture the appearance features and temporal relation between video sequences at multiple time scales. Specifically, we construct learnable temporal relation graphs to explore temporal relation on the multi-scale range. Additionally, to facilitate multi-scale temporal relation extraction, we design a multi-head temporal adjacent matrix to represent multi-kinds of temporal relations. Eventually, a multi-head temporal relation aggregator is proposed to extract the semantic meaning of those features convolving through the graphs. Extensive experiments are performed on widely-used large-scale datasets, such as Something-Something, Charades and Jester, and the results show that our model can achieve state-of-the-art performance. Further analysis shows that temporal relation reasoning with our TRG can extract discriminative features for activity recognition. [ABSTRACT FROM AUTHOR]

Image clustering is more challenging than image classification. Without supervised information, current deep learning methods are difficult to be directly applied to image clustering problems. Image clustering needs to deal with three main problems: 1) the curse of dimensionality caused by high-dimensional image data; 2) extracting the effective image features; 3) combining feature extraction, dimensionality reduction and clustering. In this paper, we propose a new clustering framework called Deep Embedded Dimensionality Reduction Clustering (DERC) via Probability-Based Triplet Loss, which effectively solves the above issues. To the best of our knowledge, the DERC is the first framework that effectively combines image embedding, dimensionality reduction, and clustering into the image clustering process. We also propose to incorporate a novel probability-based triplet loss measure to retrain the DERC network as a unified framework. By integrating the reconstruction loss and the probability-based triplet loss, we can improve the image clustering accuracy. Extensive experiments show that our proposed methods outperform state-of-the-art methods on many commonly used datasets. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks have achieved excellent successes for object recognition in still images. However, the improvement of Convolutional Neural Networks over the traditional methods for recognizing actions in videos is not so significant, because the raw videos usually have much more redundant or irrelevant information than still images. In this paper, we propose a Spatial-Temporal Attentive Convolutional Neural Network (STA-CNN) which selects the discriminative temporal segments and focuses on the informative spatial regions automatically. The STA-CNN model incorporates a Temporal Attention Mechanism and a Spatial Attention Mechanism into a unified convolutional network to recognize actions in videos. The novel Temporal Attention Mechanism automatically mines the discriminative temporal segments from long and noisy videos. The Spatial Attention Mechanism firstly exploits the instantaneous motion information in optical flow features to locate the motion salient regions and it is then trained by an auxiliary classification loss with a Global Average Pooling layer to focus on the discriminative non-motion regions in the video frame. The STA-CNN model achieves the state-of-the-art performance on two of the most challenging datasets, UCF-101 (95.8%) and HMDB-51 (71.5%). [ABSTRACT FROM AUTHOR]

Robust spatiotemporal representations of natural videos have several applications including quality assessment, action recognition, object tracking etc. In this paper, we propose a video representation that is based on a parameterized statistical model for the spatiotemporal statistics of mean subtracted and contrast normalized (MSCN) coefficients of natural videos. Specifically, we propose an asymmetric generalized Gaussian distribution (AGGD) to model the statistics of MSCN coefficients of natural videos and their spatiotemporal Gabor bandpass filtered outputs. We then demonstrate that the AGGD model parameters serve as good representative features for distortion discrimination. Based on this observation, we propose a supervised learning approach using support vector regression (SVR) to address the no-reference video quality assessment (NRVQA) problem. The performance of the proposed algorithm is evaluated on publicly available video quality assessment (VQA) datasets with both traditional and in-capture/authentic distortions. We show that the proposed algorithm delivers competitive performance on traditional (synthetic) distortions and acceptable performance on authentic distortions. The code for our algorithm will be released at https://www.iith.ac.in/lfovia/downloads.html. [ABSTRACT FROM AUTHOR]

Image registration is a required step in many practical applications that involve the acquisition of multiple related images. In this paper, we propose a methodology to deal with both the geometric and intensity transformations in the image registration problem. The main idea is to modify an accurate and fast elastic registration algorithm (Local All-Pass—LAP) so that it returns a parametric displacement field, and to estimate the intensity changes by fitting another parametric expression. Although we demonstrate the methodology using a low-order parametric model, our approach is highly flexible and easily allows substantially richer parametrisations, while requiring only limited extra computation cost. In addition, we propose two novel quantitative criteria to evaluate the accuracy of the alignment of two images (“salience correlation”) and the number of degrees of freedom (“parsimony”) of a displacement field, respectively. Experimental results on both synthetic and real images demonstrate the high accuracy and computational efficiency of our methodology. Furthermore, we demonstrate that the resulting displacement fields are more parsimonious than the ones obtained in other state-of-the-art image registration approaches. [ABSTRACT FROM AUTHOR]