Search

This paper revisits the vector of locally aggregated descriptors (VLAD), which aggregates the residuals of local descriptors to their cluster centers. Since VLAD usually adopts a small-size codebook, the clusters are coarse and residuals not discriminative. To address this problem, this paper proposes to generate a number of residual codebooks descended from the original clusters. After quantizing local descriptors with these codebooks, we pool the resulting secondary residuals as well as the primary ones to obtain the fine residuals. We show that, with two-step aggregation, the fine-residual VLAD has the same dimension as the original. Experiments on two image search benchmarks confirm the improved discriminative power of our method: we observe consistent superiority to the baseline and competitive performance to the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Traditional Bag-of-Words (BoWs) model is commonly generated with many steps, including local feature extraction, codebook generation and feature quantization, etc. Those steps are relatively independent with each other and are hard to be jointly optimized. Moreover, the dependency on hand-crafted local feature makes BoWs model not effective in conveying high-level semantics. These issues largely hinder the performance of BoWs model in large-scale image applications. To conquer these issues, we propose an End-to-End BoWs (E2BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model takes an image as input, then identifies and separates semantic objects in it, and finally outputs visual words with high semantic discriminative power. Specifically, our model firstly generates Semantic Feature Maps (SFMs) corresponding to different object categories through convolutional layers, then introduces Bag-of-Words Layers (BoWL) to generate visual words from each individual feature map. We also introduce a novel learning algorithm to reinforce the sparsity of the generated E2BoWs model, which further ensures the time and memory efficiency. We evaluate the proposed E2BoWs model on several image search datasets including MNIST, SVHN, CIFAR-10, CIFAR-100, MIRFLICKR-25K and NUS-WIDE. Experimental results show that our method achieves promising accuracy and efficiency compared with recent deep learning based retrieval works. [ABSTRACT FROM AUTHOR]

Highlights • We propose a simple yet effective quantization to embed deep binary codes. • Activations from multiple CNN layers function together through weighted score fusion in the proposed framework. • Handcrafted local descriptor SIFT, as a kind of low level feature, can also be combined in our fusion procedure. • Regularized diffusion process are customized on the ranking list to make the similarity estimation vary smoothly. • Extensive experiments are conducted on four public datasets, and state-of-the-art results are obtained on Holidays and UKBench datasets. Abstract Convolutional Neural Networks (CNNs) have achieved a breakthrough on a large number of image retrieval benchmarks. However, most previous works make use of the CNNs following the image classification strategy, where the last fully connected layer activations of the whole image are occupied as a single holistic feature vector. To improve the representation power of CNNs, this paper proposes a Multi-layer Fusion (MF) approach to aggregate deep activations for image retrieval task. The key insight of our approach is that different layers of a CNN are sensitive to specific patterns, and are complementary with each other for image representation. Specifically, our approach transforms CNN activations to deep binary codes embedded in the inverted index of Bag-of-Words structure for fast retrieval. Those activations are derived from multiple layers of a CNN on local patches, for features from orderless local areas have proved superior to global ones in the low level handcrafted cases. Corresponding weights and diffusion process are thereafter utilized to penalize and re-rank the individual similarity scores of layers. Our method is efficient, which extracts visual features from different layers only once. Furthermore, the proposed MF approach can be easily extended to include SIFT features to enhance the representation power. Extensive experiments on four public retrieval datasets quantitatively evaluate the effectiveness of our contributions, and the proposed algorithm prove to be the new state-of-the-art on the Holidays and UKBench datasets. [ABSTRACT FROM AUTHOR]

The scale-invariant feature transform (SIFT) feature plays a very important role in multimedia content analysis, such as near-duplicate image and video retrieval. However, the storage and query costs of SIFT become unbearable for large-scale databases. In this paper, SIFT features are robustly encoded with temporal information by tracking the SIFT to generate temporal-concentration SIFT (TCSIFT), which highly compresses the quantity of local features to reduce visual redundancy, and keeps the advantages of SIFT as much as possible at the same time. On the basis of TCSIFT, a novel framework for large-scale video copy retrieval is proposed in which the processes of retrieval and validation are implemented at the feature and frame level. Experimental results for two different datasets, i.e., CC_WEB_VIDEO and TRECVID, demonstrate that our method can yield comparable accuracy, compact storage size, and more efficient execution time, as well as adapt to various video transformations. [ABSTRACT FROM AUTHOR]

In this work, we propose an efficient image re-ranking method, without regional information of each indexed feature stored in the inverted file, to re-rank all retrieved images. The motivation of the proposed method is that, there are usually many visual words in the query image that only give votes to irrelevant images. With this observation, we propose to use visual words which are actually useful in finding relevant images to re-rank the retrieved images. To achieve this goal, we first initialize some images similar to the query by maximizing a quadratic function when giving an initial ranking of the retrieved images. The quadratic function is constructed by storing the similarities among a short-list of top ranked images into an affinity matrix, where the similarity between any two images is computed by the proposed graph diffusion. Then, we select a subset of visual words in the query with an alternating optimization strategy: (1) at each iteration, visual words are selected based on the set of similar images that we have found, (2) and in turn, the set of similar images is updated with the set of selected words. These two steps are repeated until convergence. Experimental results on standard benchmark datasets show that the proposed method achieves an order of magnitude speedups over the state-of-the-art spatial based re-ranking techniques, and obtains much better retrieval quality as well. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we present a spatiotemporal co-location video pattern mining approach with application to robust action retrieval in YouTube videos. First, we introduce an attention shift scheme to detect and partition the focused human actions from YouTube videos, which is based upon the visual saliency [13] modeling together with both the face [35] and body [32] detectors. From the segmented spatiotemporal human action regions, we extract 3D-SIFT [17] detector. Then, we quantize all detected interest points from the reference YouTube videos into a vocabulary, based on which assign each individual interest point with a word identity. An APrior based frequent itemset mining scheme is then deployed over the spatiotemporal co-located words to discover co-location video patterns. Finally, we fuse both visual words and patterns and leverage a boosting based feature selection to output the final action descriptors, which incorporates the ranking distortion of the conjunctive queries into the boosting objective. We carried out quantitative evaluations over both KTH human motion benchmark [26], as well as over 60-hour YouTube videos, with comparisons to the state-of-the-arts. [Copyright &y& Elsevier]