Search

The visual quality of an image mainly relies on its content and its distortions. However, the adaptability between their contributions to the image quality has not be well investigated yet. Besides, albeit of many promising efforts, lacking sufficient labeled data still hinders the robust representation of quality-related information. In this work, we first design a self-supervised architecture, named collaborative autoencoder (COAE), to separately represent the content and the distortion information, and then develop a Self-Adaptive Weighting based quAlity predictoR (SAWAR) to balance the individual representations of the content and the distortions in the prediction of image quality. Specifically, the COAE is trained with large-scale unlabeled data, consisting of a content autoencoder (CAE) and a distortion autoencoder (DAE) that work collaboratively and individually. While the CAE is a standard autoencoder for the content representation, the design of the DAE is unique. We introduce the CAE-encoded content representation as an extra input to the decoder of the DAE to learn to reconstruct distorted images, thus effectively forcing it to extract the distortion representation. The SAWAR, whose parameter number is much smaller than that of the COAE, is trained with labeled data in existing IQA datasets. It takes advantage of the interaction between the image content and the distortions to adaptively balance their contributions. Extensive experiments show that the COAE effectively extracts quality-related representations and the SAWAR achieves the state-of-the-art performance.

Video super-resolution (SR) has important real world applications such as enhancing viewing experiences of legacy low-resolution videos on high resolution display devices. However, there are no visual quality assessment (VQA) models specifically designed for evaluating SR videos while such models are crucially important both for advancing video SR algorithms and for viewing quality assurance. This paper addresses this gap. We start by contributing the first video super-resolution quality assessment database (VSR-QAD) which contains 2,260 SR videos annotated with mean opinion score (MOS) labels collected through an approximately 400 man-hours psychovisual experiment by a total of 190 subjects. We then build on the new VSR-QAD and develop the first VQA model specifically designed for evaluating SR videos. The model features a two-stream convolutional neural network architecture and a two-stage training algorithm designed for extracting spatial and temporal features characterizing the quality of SR videos. We present experimental results and data analysis to demonstrate the high data quality of VSR-QAD and the effectiveness of the new VQA model for measuring the visual quality of SR videos. The new database and the code of the proposed model will be available online at https://github.com/key1cdc/VSRQAD.

Displaying standard dynamic range (SDR) videos on high dynamic range (HDR) devices requires inverse tone mapping (ITM). However, such mapping can introduce banding artifacts. This paper presents a banding removal method for inversely tone mapped HDR videos based on deep convolutional neural networks (DCNNs) and adaptive filtering. Three banding relevant feature maps are first extracted and then fed to two DCNNs, a ShapeNet and a PositionNet. The PositionNet learns a soft mask indicating the locations where banding is likely to have occurred and filtering is required while the ShapeNet predicts the filter shapes appropriate for different locations. An advantage of the method is that the adaptive filters can be jointly optimized with a learning-based ITM algorithm for creating high-quality HDR videos. Experimental results show that our method outperforms state-of-the-art algorithms qualitatively and quantitatively.

Existing class imbalance learning paradigms focus on lifting the importance of minority instance, aiming to improve the model in terms of certain evaluation metrics (e.g., AUC and $F_{1}$F1-measure). One drawback of these methods is that they lack enough transparency, hence, cannot be fully trusted in vital domains. To this end, this paper deal with the class imbalance learning task with counterfactual instances. Given an instance and a classifier, a counterfactual is a fake instance which, while having smallest distance to the original instance, is classified as a different class by the classifier. Therefore, the most important features for a classifier can be identified by inspecting the difference between an instance and its counterfactual. To utilize counterfactuals, a novel Explainable Generative Adversarial Network (EXGAN) is proposed. EXGAN has a unique “two generators versus multiple discriminators” architecture where the generators are used to generate effective counterfactuals and discriminators are trained for the class imbalance learning task. In addition to the architecture, an innovative ensemble loss function ensuring each discriminator complementing each other is designed to overcome the class imbalance issue. Extensive experiments prove that the counterfactuals generated by EXGAN can be used to produce effective local explanation and provide significant better class imbalance learning ability than existing competitors.

Anomaly detection has wide applications in machine intelligence but is still a difficult unsolved problem. Major challenges include the rarity of labeled anomalies and it is a class highly imbalanced problem. Traditional unsupervised anomaly detectors are suboptimal while supervised models can easily make biased predictions towards normal data. In this paper, we present a new supervised anomaly detector through introducing the novel Ensemble Active Learning Generative Adversarial Network (EAL-GAN). EAL-GAN is a conditional GAN having a unique one generator versus multiple discriminators architecture where anomaly detection is implemented by an auxiliary classifier of the discriminator. In addition to using the conditional GAN to generate class balanced supplementary training data, an innovative ensemble learning loss function ensuring each discriminator makes up for the deficiencies of the others is designed to overcome the class imbalanced problem, and an active learning algorithm is introduced to significantly reduce the cost of labeling real-world data. We present extensive experimental results to demonstrate that the new anomaly detector consistently outperforms a variety of SOTA methods by significant margins.

We exploit the computational capability of deep convolutional neural network (CNN) architecture and the natural interpretability of the co-occurrence matrix (CM) to learn deep co-occurrence features (DCOFs). The DCOFs represent the statistics of the co-occurrences of pixels thus overcoming the black box nature of traditional deep representation learning while at the same time solving the inherent computational difficulty of CM. We propose a parametric co-occurrence matrix (PCM) model to approximate the CM with multivariate Gaussian functions, and have developed three approaches to decomposing the PCM model into linear and nonlinear operations such that the model can be easily implemented using standard CNN operations and to learn the DCOFs of arbitrary shapes. The CNN implementation of the PCM model, termed PCMCNN, can be used as a standard plugin module of a deep learning system and adaptively learns the DCOFs for downstream applications. We demonstrate the broad applicability of the DCOFs and their effectiveness in fine-grained image classification tasks such as texture classification and GAN (generative adversarial network) image detection. The introduction of the PCMCNN module makes it much more compact and efficient than conventional implementations of deep learning models, achieving comparable classification performances to state of the art methods on a variety of benchmarking datasets with models that are more than 30 folds smaller and 11 times less complex. The small model size for learning the DCOFs makes the new method particularly effective for few shot classification of large number of texture categories where the small number of training samples can easily cause traditional deep learning models to overfit. This work shows the potential benefits of combining the principles of traditional handcrafted features and deep representation learning to take advantage of both for advancing state of the art.

Class imbalance is a prevalent phenomenon in various real-world applications and it presents significant challenges to model learning, including deep learning. In this work, we embed ensemble learning into the deep convolutional neural networks (CNNs) to tackle the class-imbalanced learning problem. An ensemble of auxiliary classifiers branching out from various hidden layers of a CNN is trained together with the CNN in an end-to-end manner. To that end, we designed a new loss function that can rectify the bias toward the majority classes by forcing the CNN’s hidden layers and its associated auxiliary classifiers to focus on the samples that have been misclassified by previous layers, thus enabling subsequent layers to develop diverse behavior and fix the errors of previous layers in a batch-wise manner. A unique feature of the new method is that the ensemble of auxiliary classifiers can work together with the main CNN to form a more powerful combined classifier, or can be removed after finished training the CNN and thus only acting the role of assisting class imbalance learning of the CNN to enhance the neural network’s capability in dealing with class-imbalanced data. Comprehensive experiments are conducted on four benchmark data sets of increasing complexity (CIFAR-10, CIFAR-100, iNaturalist, and CelebA) and the results demonstrate significant performance improvements over the state-of-the-art deep imbalance learning methods.

Learning a common space that is simultaneously semantically discriminative and modality invariant stands as the primary challenge in cross-modal retrieval. Conventional approaches usually employ pairwise or triplet data relationships to learn the common space, which can only capture the data similarity locally but would be unable to effectively characterize the global geometry of the common embedding space, and thus would limit the performance of cross-modal retrieval. This paper proposes to integrate the principles of the shared proxyand neighborhood component analysis in order to learn a shared space for different modalities. The objective of this shared space is to minimize the distance between a sample’s representation and its corresponding proxy, while also maximizing the distances between a sample’s representation and the proxies that are not associated with the sample. Our proposed framework, named Cross-mOdal proXy learnIng (COXI), incorporates a cross-modal shared proxy loss, a discriminative loss, and a modality invariant loss to facilitate supervised cross-modal retrieval. Extensive experiments on benchmark datasets clearly shows that COXI outperforms state-of-the-art cross-modal retrieval techniques. Code is available on https://github.com/LigangZheng/COXI.

In this article the authors proposed a fast and fully unsupervised approach for a foreground object co-localization and segmentation of unconstrained videos. This article first computes both the actual edges and motion boundaries of the video frames, and then aligns them by the proposed HOG affinity map approach. Then, by filling the occlusions generated by the aligned edges, the paper obtained more precise masks about the foreground object. With an accumulation process, these masks could be derived as the motion-based likelihood, which is used as a unary term in the proposed graph model. Another unary term is called color-based likelihood, which is computed by the color distribution of foreground and background. Experiment results shows the method is fast and effective to detect and segment foreground objects.

This paper introduces a novel method to efficiently discover landmark images in large image collections. Each cluster is considered a combination of several sub-clusters, which are composed of images taken from different viewpoints of an identical landmark. For each sub-cluster, we find its local centre represented by a group of similar images and define it as the bundling centre (BC). Therefore, we start image discovery by identifying the BCs and accomplish the task by efficiently growing and merging those sub-clusters represented by different BCs. In our proposed method, we use a min-Hash-based method to build a sparse graph to avoid time-consuming, full-scale, exhaustive pairwise image matching. Based on the information provided by the sparse graph, BCs are identified as local dense neighbours sharing high intra-similarity. We have also proposed a weighted voting method to grow these BCs with high accuracy. More importantly, the fixed local centres ensure that each sub-cluster contains identical landmarks and generates results with high precision. In addition, compared to a single representative (iconic) image, the group of similar images obtained by each BC can provide more comprehensive cluster information and, thus, overcome the problem of low recall caused by information lost during visual word quantisation. We present the experimental results of three datasets and show that, without query expansion, our method can boost the landmark image discovery performances of current techniques.

Current subjective image quality assessments have been developed in the laboratory environments, under controlledconditions, and are dependent on the participation of limited numbers of observers. In this research, with the help of Web 2.0 and social media technology, a new method for building a subjective image quality metric has been developed where the observers are the Internet users. A website with a simple user interface that enables Internet users from anywhere at any time to vote for a better quality version of a pair of the same image has been constructed. Users' votes are recorded and used to rank the images according to their perceived visual qualities. We have developed three rank aggregation algorithms to process the recorded pair comparison data, the first uses a naive approach, the second employs a Condorcet method, and the third uses the Dykstra's extension of Bradley-Terry method. The website has been collecting data for about three months and has accumulated over 10,000 votes at the time of writing this paper. Results show that the Internet and its allied technologies such as crowdsourcing offer a promising new paradigm for image and video quality assessment where hundreds of thousands of Internet users can contribute to building more robust image quality metrics. We have made Internet user generated social image quality (SIQ) data of a public image database available online (http://www.hdri.cs.nott.ac.uk/siq/) to provide the image quality research community with a new source of ground truth data. The website continues to collect votes and will include more public image databases and will also be extended to include videos to collect social video quality (SVQ) data. All data will be public available on the website in due course.

High dynamic range (HDR) photography is an emerging technology that has the potential to dramatically enhance the visual quality and realism of digital photos. One of the key technical challenges of HDR photography is displaying HDR photos on conventional devices through tone mapping or dynamic range compression. Although many different tone mapping techniques have been developed in recent years, evaluating tone mapping operators prove to be extremely difficult. Web2.0, social media and crowd-sourcing are emerging Internet technologies which can be harnessed to harvest the brain power of the mass to solve difficult problems in science, engineering and businesses. Paired comparison is used in the scientific study of preferences and attitudes and has been shown to be capable of obtaining an interval-scale ordering of items along a psychometric dimension such as preference or importance. In this paper, we exploit these technologies for evaluating HDR tone mapping algorithms. We have developed a Web2.0 style system that enables Internet users from anywhere to evaluate tone mapped HDR photos at any time. We adopt a simple paired comparison protocol, Internet users are presented a pair of tone mapped images and are simply asked to select the one that they think is better or click a "no difference" button. These user inputs are collected in the web server and analyzed by a rank aggregation algorithm which ranks the tone mapped photos according to the votes they received. We present experimental results which demonstrate that the emerging Internet technologies can be exploited as a new paradigm for evaluating HDR tone mapping algorithms. The advantages of this approach include the potential of collecting large user inputs under a variety of viewing environments rather than limited user participation under controlled laboratory environments thus enabling more robust and reliable quality assessment. We also present data analysis to correlate user generated qualitative indices with quantitative image statistics which may provide useful guidance for developing better tone mapping operators.

The most commonly used image magnification techniques are interpolation based: nearest neighbor, bilinear, and bicubic. The drawbacks of these traditional methods are that images magnified by the simple nearest neighbor method often appear “blocky,” while images magnified by linear and cubic interpolations usually appear “blurry.” In this work, a new technique, which improves the performance of the traditional image magnification methods, is presented. We show how a differential image pyramid is first constructed using traditional interpolation methods, then how a vector quantizer is designed using the pyramidal data. The vector quantizer is a look-up table, termed the interresolution look-up table (IRLUT), which uses the lower resolution image vector as input to find as its output the corresponding higher resolution image vector. The improved image is produced by using the IRLUT's outputs to compensate for the image magnified by the traditional methods. Experimental results which show that images generated by the current method have sharper edges as well as lower reconstruction mean-square errors than those produced by traditional methods are presented.

An adaptive nonlinear scheme for the blocking effect reduction in low bit rate block coded images is presented. A method is developed to extract useful features from the compressed image and uses them as inputs to three-layer feedforward neural networks. The neural networks learn to reduce the coding errors in the block border areas. The new technique has been applied to process JPEG compressed images and results are presented which show improvements in both visual quality and peak-signal to noise ratio (PSNR).