Search

In virtual reality (VR), correct and precise estimations of user’s visual fixations and head movements can enhance the quality of experience by allocating more computation resources for analysing and rendering on the areas of interest. However, there is insufficient research about understanding the visual exploration of users when modeling VR visual attention. To bridge the gap between the saliency prediction for traditional 2D content and omnidirectional content, we construct the visual attention dataset and propose the visual saliency prediction framework for panoramic videos. Around the instantaneous viewing behavior, we propose a traditional method to adapt 2D saliency models and design a CNN-based model to better predict visual saliency. In the proposed traditional model, mechanism of visual attention and viewing behaviors are considered in the computation of edge weights on graphs which are interpreted as Markov chains. The fraction of the visual attention that is diverted to each high-clarity vision (HCV) area is estimated through equilibrium distribution of this chain. We also propose the Graph-Based CNN model. The RGB channel and optical flow form the spatial-temporal units of HCVs, from which node feature vectors are extracted. Graph convolution is used to learn the mutual information between node feature vectors of HCVs and retain geometric information. Then feature vectors are aligned according to geometry structure of equirectangular format, and the feature decoder maps the aligned feature maps to the data distribution. We also construct the dynamic omnidirectional monocular (DOM) saliency dataset with 64 diverse videos evaluated by 28 people. The subjective results show that the instantaneous viewing behavior is important in the VR experience. Extensive experiments are conducted on the dataset and the results demonstrate the effectiveness of the proposed framework. The dataset will be released to facilitate the future studies related to visual saliency prediction for 360-degree contents. [ABSTRACT FROM AUTHOR]

Bit-depth enhancement (BDE) is a challenging task due to stubborn false contour artifacts and disappeared detailed information. Given the mixture of structural distortions and real edges in low bit-depth (LBD) images, both large and small receptive fields (RFs) are critical for BDE tasks. However, even powerful state-of-the-art CNN-based methods can hardly capture sufficient LBD features under multiple RFs. This paper proposes a residual-guided multiscale fusion network (RMFNet) to explore multiscale features in a residual manner. We find that the shuffling operation provides desired multiscale inputs for effectively distinguishing false contours from real edges without any loss of information. Therefore, we shuffle LBD images to multiple scales and then fully extract residual features under different RFs with corresponding subnets. To facilitate interscale guidance from the global context to the local context, we progressively transfer the encoded residual features between adjacent subnets from top to bottom. We further propose a dual-branch depthwise group fusion (DDGF) module to fully capture inter- and inner correlations of multiscale features with fewer parameters. Finally, extensive experiments show that our algorithm achieves excellent performance improvement both quantitatively and qualitatively, verifying its effectiveness. [ABSTRACT FROM AUTHOR]

The emerging vision-and-language navigation (VLN) problem aims at learning to navigate an agent to the target location in unseen photo-realistic environments according to the given language instruction. The main challenges of VLN arise mainly from two aspects: first, the agent needs to attend to the meaningful paragraphs of the language instruction corresponding to the dynamically-varying visual environments; second, during the training process, the agent usually imitate the expert demonstrations, i.e., the shortest-path to the target location specified by associated language instructions. Due to the discrepancy of action selection between training and inference, the agent solely on the basis of imitation learning does not perform well. Existing VLN approaches address this issue by sampling the next action from its predicted probability distribution during the training process. This allows the agent to explore diverse routes from the environments, yielding higher success rates. Nevertheless, without being presented with the golden shortest navigation paths during the training process, the agent may arrive at the target location through an unexpected longer route. To overcome these challenges, we design a cross-modal grounding module, which is composed of two complementary attention mechanisms, to equip the agent with a better ability to track the correspondence between the textual and visual modalities. We then propose to recursively alternate the learning schemes of imitation and exploration to narrow the discrepancy between training and inference. We further exploit the advantages of both these two learning schemes via adversarial learning. Extensive experimental results on the Room-to-Room (R2R) benchmark dataset demonstrate that the proposed learning scheme is generalized and complementary to prior arts. Our method performs well against state-of-the-art approaches in terms of effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

Fractional interpolation has been extensively utilized in a series of video coding standards to generate fractional precision prediction to remove the temporal redundancy in consecutive frames. In the traditional interpolation filter based methods, the fractional samples are interpolated through a linear combination of the neighboring integer samples. This method is simple yet unable to accurately characterize the nonstationary video signals. Recently, convolutional neural network has been utilized in the fractional interpolation and shows superior performance compared with the traditional methods. However, only the reconstruction of the reference frame is used as the infer information source. All the other information contained in the bitstream or generated during the encoding/decoding procedure denoted as the compression prior is not utilized at all. In this paper, we give the first trial to involve some compression priors into the CNN to improve the performance of a in-loop coding tool. Specifically, we propose a Compression Priors assisted Convolutional Neural Network (CPCNN) to further improve the fractional interpolation efficiency. In addition to the reconstructed component, we additionally utilize two other compression priors – the corresponding residual component and col-located high quality component to boost the performance. Specifically, the residual component that indicates the prediction efficiency and contains effective texture information is utilized as a complementary input to the reconstructed one. While the col-located component provides more useful high quality information to help the reconstruction get rid of the quality fluctuation. Furthermore, a special network structure is designed to learn powerful representations of these triple input components. Comprehensive experiments have been conducted to demonstrate the effectiveness of our proposed CPCNN. The experimental results show that compared to HEVC, our proposed CPCNN achieves on average of 5.3%, 2.8% and 1.9% BD-Rate savings under LDP, LDB and RA configurations, respectively. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive region proposal scheme with feature channel regularization to facilitate robust object tracking. We consider tracking as a linear regression problem and an ensemble of correlation filters is trained on-line to distinguish the foreground target from the background. Further, we integrate adaptively learned region proposals into an enhanced two-stream tracking framework based on correlation filters. For the tracking stream, we learn two-stage cascade correlation filters on deep convolutional features to ensure competitive tracking performance. For the detection stream, we employ adaptive region proposals, which are effective in recovering target objects from tracking failures caused by heavy occlusion or out-of-view movement. In contrast to traditional tracking-by-detection methods using random samples or sliding windows, we perform target re-detection over adaptively learned region proposals. Since region proposals naturally take the objectness information into account, we show that the proposed adaptive region proposals can handle the challenging scale estimation problem as well. In addition, we observe the channel redundancy and noisy of feature representation, especially for the convolutional features. Thus, we apply a channel regularization to the correlation filter learning. Extensive experimental validations on OTB, VOT and UAV-123 datasets demonstrate that the proposed method performs favorably against state-of-the-art tracking algorithms. [ABSTRACT FROM AUTHOR]

Being able to accurately predict the common gaze point of a group of persons is of particular interest to precise marketing and automatic group attention assessment. Group gaze estimation faces challenges including small face/head size and outlier observers. To address these challenges, we proposed a novel framework called Multi-stream Group Gaze Learning and Estimation (MUGGLE). The MUGGLE infrastructure includes two inference streams: 1) a holistic stream which utilizes fused attention map as input to a global deep convolutional structure to explore the global geometric configurations and contexts of interesting persons in the scene; and 2) an aggregative stream which robustly aggregates individual gazes via a recurrent structure (e.g., LSTM) to obtain outlier-tolerant estimation. Both streams are seamlessly integrated via a fusion network. Extensive experiments are performed on a fully annotated group gaze image dataset with 8,000+ images and 100,000+ faces (which is publicly releasable). The results demonstrate the effectiveness of the proposed MUGGLE framework in group gaze estimation. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of image contrast enhancement. Most existing relevant technologies often suffer from the drawback of excessive enhancement, thereby introducing noise/artifacts and changing visual attention regions. One frequently used solution is manual parameter tuning, which is, however, impractical for most applications since it is labor intensive and time consuming. In this research, we find that saliency preservation can help produce appropriately enhanced images, i.e., improved contrast without annoying artifacts. We therefore design an automatic contrast enhancement technology with a complete histogram modification framework and an automatic parameter selector. This framework combines the original image, its histogram equalized product, and its visually pleasing version created by a sigmoid transfer function that was developed in our recent work. Then, a visual quality judging criterion is developed based on the concept of saliency preservation, which assists the automatic parameters selection, and finally properly enhanced image can be generated accordingly. We test the proposed scheme on Kodak and Video Quality Experts Group databases, and compare with the classical histogram equalization technique and its variations as well as state-of-the-art contrast enhancement approaches. The experimental results demonstrate that our technique has superior saliency preservation ability and outstanding enhancement effect. [ABSTRACT FROM PUBLISHER]

We develop a framework for estimating the noise level of a natural image using two important statistics: 1) high kurtosis and 2) scale invariance in transform domain. By exploring the said priors of natural image statistics in 2-D discrete cosine transform (DCT) domain, we reveal the limitations of these statistics for images with highly directional edges or large smooth areas. Then, we derive a novel two-step estimation scheme for noise variance: 1) in preliminary estimation, an integration of wavelet and nondirectional DCT transform is used to alleviate the influence of image’s structures and 2) a noise-injection rectification is further devised to deal with the noise-free image contents. A simulation and comparative study demonstrates that this algorithm reliably infers noise variance and its robustness over wide ranges of visual content and noise levels, while outperforming some relevant methods. This paper can significantly improve the performance of existing denoising techniques that require the noise variance as a critical parameter. [ABSTRACT FROM PUBLISHER]

In this paper, a novel spatial error concealment (EC) algorithm is proposed. Under the sequential recovery framework, pixels in missing blocks are successively reconstructed based on adaptive linear predictor. The predictor automatically tunes its order and support shape according to local contexts. The predictor order and support shape are determined using Bayesian information criterion, which is able to strike a balance between the bias and variance of the prediction errors. The flexibility of the order-adaptive predictor is able to recover more important features or structures. A novel scan order based on the uncertainty of each pixel is also proposed to alleviate error propagation problem. Compared with the state-of-the-art EC algorithms, experimental results show that the proposed method gives better reconstruction performance in terms of objective and subjective evaluations. [ABSTRACT FROM PUBLISHER]

Temporal psychovisual modulation (TPVM) is a newly proposed information display paradigm, which can be implemented by nonnegative matrix factorization (NMF) with additional upper bound constraints on the variables. In this paper, we study all the state-of-the-art algorithms in NMF, extend them to incorporate the upper bounds and discuss their potential use in TPVM. By comparing all the NMF algorithms with their extended versions, we find that: 1) the factorization error of the truncated alternating least squares algorithm always fluctuates throughout the iterations, 2) the alternating nonnegative least squares based algorithms may slow down dramatically under the upper bound constraints, and 3) the hierarchical alternating least squares (HALS) algorithm converges the fastest and its final factorization error is often the smallest among all the algorithms. Based on the experimental results of the HALS, we propose a guideline of determining the parameter setting of TPVM, that is, the number of viewers to support and the scaling factor for adjusting the light intensity of the images formed by TPVM. This paper will facilitate the applications of TPVM. [ABSTRACT FROM AUTHOR]

The analysis of user search goals for a query can be very useful in improving search engine relevance and user experience. Although the research on inferring user goals or intents for text search has received much attention, little has been proposed for image search. In this paper, we propose to leverage click session information, which indicates high correlations among the clicked images in a session in user click-through logs, and combine it with the clicked images' visual information for inferring user image-search goals. Since the click session information can serve as past users' implicit guidance for clustering the images, more precise user search goals can be obtained. Two strategies are proposed to combine image visual information with the click session information. Furthermore, a classification risk based approach is also proposed for automatically selecting the optimal number of search goals for a query. Experimental results based on a popular commercial search engine demonstrate the effectiveness of the proposed method. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a single image super-resolution and enhancement algorithm using local fractal analysis. If we treat the pixels of a natural image as a fractal set, the image gradient can then be regarded as a measure of the fractal set. According to the scale invariance (a special case of bi-Lipschitz invariance) feature of fractal dimension, we will be able to estimate the gradient of a high-resolution image from that of a low-resolution one. Moreover, the high-resolution image can be further enhanced by preserving the local fractal length of gradient during the up-sampling process. We show that a regularization term based on the scale invariance of fractal dimension and length can be effective in recovering details of the high-resolution image. Analysis is provided on the relation and difference among the proposed approach and some other state of the art interpolation methods. Experimental results show that the proposed method has superior super-resolution and enhancement results as compared to other competitors. [ABSTRACT FROM AUTHOR]