Search

Classification of imbalanced multi-class image datasets is a challenging problem in computer vision. Most of the real-world datasets are imbalanced in nature because of the uneven distribution of the samples in each class. The problem with an imbalanced dataset is that the minority class having a smaller number of instance samples is left undetected. Most of the traditional machine learning algorithms can detect the majority class efficiently but lag behind in the efficient detection of the minority class, which ultimately degrades the overall performance of the classification model. In this paper, we have proposed a novel combination of visual codebook generation using deep features with the non-linear Chi2 SVM classifier to tackle the imbalance problem that arises while dealing with multi-class image datasets. The low-level deep features are first extracted by transfer learning using the ResNet-50 pre-trained network, and clustered using k-means. The center of each cluster is a visual word in the codebook. Each image is then translated into a set of features called the Bag-of-Visual-Words (BOVW) derived from the histogram of visual words in the vocabulary. The non-linear Chi2 SVM classifier is found most optimal for classifying the ensuing features, as proved by a detailed empirical analysis. Hence with the right combination of learning tools, we are able to tackle classification of multi-class imbalanced image datasets in an effective manner. This is proved from the higher scores of accuracy, F1-score and AUC metrics in our experiments on two challenging multi-class datasets: Graz-02 and TF-Flowers, as compared to the state-of-the-art methods.

As a widely-used method in the image categorization tasks, the Bag-of-Words (BoW) method still suffers from many limitations such as overlooking spatial information. In this paper, we propose four improvements to the BoW method to consider spatial and semantic information as well as information from multiple views. In particular, our contributions are: (a) encoding spatial information based on a combination of wavelet transform image scaling and a new image partitioning scheme, (b) proposing a spatial-information- and content-aware visual word dictionary generation approach, (c) developing a content-aware feature weighting approach to considers the significance of the features for different semantics, (d) proposing a novel weighting strategy to fuse color information when discriminative shape features are lacking. We call our method Scale-Space Multi-View Bag of Words (SSMV-BoW). We conducted extensive experiments to evaluate our SSMV-BoW and compare it to the state-of-the-art scene categorization methods. For our experiments, we use four publicly available and widely used scene categorization benchmark datasets. Results demonstrate that our SSMV-BoW outperforms the methods using both hand-crafted and deep learning features. In addition, ablation studies show that all four improvements contribute to the performance of our SSMV-BoW.

In this paper, a novel face recognition approach based on strings of successive values (SSV) is presented. In contrast to most of the existing local descriptors which encode only a limited number of pixels included in a mask, the strings extract more discriminative information over the whole face region, by moving from the current pixel to the next one, and to the other next, and so on, according to the variations of their intensities. Therefore, the SSV can be stopped in any place of the face area, which allows us to encode more edge information and texture information than the existing methods. The proposed face recognition scheme requires several steps. Firstly, the images are divided into non-overlapping sub-regions from which the strings are extracted since each pixel produces two different strings. Thereafter, the dictionary of visual words is created to reduce the number of strings obtained from each patch of the image. Therefore, the face image is described only by visual words, because each string is replaced by its nearest dictionary word. As a result, the occurrence of visual words is computed in a histogram as a face descriptor. Finally, the recognition is performed by using the nearest neighbor classifier with the Hellinger distance. The effectiveness of the proposed approach is evaluated on three different databases, and the experimental results show that the recognition performances achieved are competitive or even outperform the literature state of the art methods.

In this paper, we propose a multistage KNN collaborative coding based Bag-of-Feature (MKCC-BoF) method to address SSPP problem, which tries to weaken the semantic gap between facial features and facial identification. First, local descriptors are extracted from the single training face images and a visual dictionary is obtained offline by clustering a large set of descriptors with K-means. Then, we design a multistage KNN collaborative coding scheme to project local features into the semantic space, which is much more efficient than the most commonly used non-negative sparse coding algorithm in face recognition. To describe the spatial information as well as reduce the feature dimension, the encoded features are then pooled on spatial pyramid cells by max-pooling, which generates a histogram of visual words to represent a face image. Finally, a SVM classifier based on linear kernel is trained with the concatenated features from pooling results. Experimental results on three public face databases show that the proposed MKCC-BoF is much superior to those specially designed methods for SSPP problem. Moreover, it also has great robustness to expression, illumination, occlusion and, time variation.

Scene classification methods based on effective feature extraction and coding have obtained promising results in recent years. But the K-nearest neighbor search strategy in Locality-constrained Linear Coding (LLC) increases the complexity of the algorithm due to the exhaustive search. To solve the problem, an improved approximate nearest neighbor search strategy is proposed to improve the computational efficiency of LLC. Considering the mapping relationship between the visual words and features, a collaborative hashing method is incorporated to transform the high dimensional features into binary code form, and the original Euclidean space is transformed into the Hamming space that consists of multi similar features. The similar visual words can be queried quickly. Then the nearest neighbors can be searched efficiently through Hamming distance ranking, which can improve the coding efficiency. The experimental results on standard datasets demonstrate the effectiveness of the proposed approach, and the average classification accuracy can be improved.

Most existing feature descriptors for video have limited representation ability. In order to improve the recognition accuracy of method for detecting the videos that include violent scenes and take advantage of the logical structure of video sequences, a novel feature constructing approach based on three dimensional histograms of gradient orientation (HOG3D), the Bag of Visual Words (BoVW) model, and feature pooling technology is proposed. This approach, combined with kernel extreme learning machine (KELM), can be used to detect violent scene. First, the HOG3D feature is extracted on the block level for video, and then the K-Means clustering algorithm is implemented to generate visual words. Then, the bag of visual words framework is used for the quantization of feature. And the feature pooling technology is operated to generate a feature vector for an entire video segment, and feature vectors of training data and testing data were used separately to train the model and evaluate the performance of the proposed approach. The experimental results showed that the proposed feature descriptor had good representation and generalization abilities. The proposed approach is efficient for violent scene detection, and the accuracy matches the best result on Hockey dataset, and it outperforms state-of-the-art on Movies.

In recent years, human action recognition systems have been increasingly developed to support a wide range of application areas, such as surveillance, behaviour analysis, security, and many others. In particular, data fusion approaches that use depth and colour information (i.e., RGB-D data) seem to be particularly promising for recognizing large classes of human actions with a high level of accuracy. Anyway, existing data fusion approaches are mainly based on feature fusion strategies, which tend to suffer of some limitations, including the difficult of combining different feature types and the management of missing information. To address the two problems just reported, we propose an RGB-D data based human action recognition system supported by a decision fusion strategy. The system, starting from the well-known Joint Directors of Laboratories (JDL) data fusion model, analyses human actions separately for each channel (i.e., depth and colour). The actions are modelled as a sum of visual words by using the traditional Bag-of-Visual-Words (BoVW) model. Subsequently, on each channel, these actions are classified by using a multi-class Support Vector Machine (SVM) classifier. Finally, the classification results are fused by a Naive Bayes Combination (NBC) method. The effectiveness of the proposed system has been proven on the basis of three public datasets: UTKinect-Action3D, CAD-60, and LIRIS Human Activities. Experimental results, compared with key works of the current state-of-the-art, have shown that what we propose can be considered a concrete contribute to the action recognition field.

© 2018, Springer Science+Business Media, LLC, part of Springer Nature. The convolutional neural network (CNN) features can give good description of image content, which usually represent an image with a single feature vector. Although CNN features are more compact than local descriptors, they still cannot efficiently deal with large-scale retrieval due to the linearly incremental cost of computation and storage. To address this issue, we build a simple but effective indexing framework on inverted table, which significantly decreases both search time and memory usage. First, several strategies are fully investigated to adapt inverted table to CNN features for compensating for quantization error. We use multiple assignment for the query and database images to increase the probability that relevant images are assigned to the same visual word obtained via clustering. Embedding codes are also introduced to improve retrieval accuracy by removing false matches. Second, a novel indexing framework that combines inverted table and hashing codes is proposed. This framework is faster than the reformed inverted tables with the introduced strategies. Experiment on several benchmark datasets demonstrates that our method yields faster retrieval speed compared to brute-force search. We also provide fair comparison between popular CNN features.

In this paper, a novel term frequency-inverse document frequency (tf-idf) based method that utilizes deep Convolutional Neural Networks (CNN) for Content Based Image Retrieval (CBIR) is proposed. That is, we treat the learned filters of the convolutional layers of a CNN model as detectors of visual words. Each of these filters has been trained to be activated in different visual patterns. Thus, since the activations of each filter provide information about the degree of presence of the visual pattern that the filter has learned during the training procedure, we consider the activations of these filters as the tf part. Subsequently, we propose three approaches of computing the idf part. Finally, we propose a query expansion technique on top of the formulated descriptors. The proposed approach interconnects the standard tf-idf method with the modern CNN analysis for visual content, providing a very powerful image retrieval technique with improved results as it is highlighted by extensive experiments in four challenging image datasets.

The bag-of-words (BoW) has been widely regarded as the most successful algorithms for content-based image related tasks, such as large scale image retrieval, classification, and object categorization. Large visual words acquired by BoW quantization through large vocabulary or codebooks have been receiving much attention in the past years. However, not only construction of large vocabulary but also the quantization process impose a heavy burden in terms of time and memory complexities. In order to tackle this issue, we propose an efficient hierarchical BoW (HBoW) to achieve large visual words through quantization by a compact vocabulary instead of large vocabulary. Our vocabulary is very compact since it is only composed of two small dictionaries which is learned through segmental sparse decomposition of local features. To generate the BoW with large size, we first divide the local features into two half parts, and use the two small dictionaries to compute their sparse codes. Then, we map the two indices of the maximum elements of the two sparse codes to a large set of visual words based upon the fact that data with similar properties will share the same base weighted with the largest sparse coefficient. To further make similar patches have higher probability of select the same dictionary base to get similar BoW vectors, we propose a novel collaborative dictionary learning method by imposing the similarity regularization factor together with the row sparsity regularization across data instances during group sparse coding. Additionally, based on index combination of top-2 large sparse codes of local descriptors, we propose a soft BoW assignment method so that our proposed HBoW can tolerate different word selection for similar patches. By employing the inverted file structure built through our HBoW, K-nearest neighbors (KNN) can be efficiently retrieved. After incorporation of our fast KNN search into the SVM-KNN classification method, our HBoW can be used for efficient image classification and logo recognition. Experiments on serval well-known datasets show that our approach is effective for large scale image classification and retrieval.

Based on Bag of Visual Words (BoVWs) model, this paper proposes a novel method using an integrated feature to detect sign text in the street view images. BRISK features are first extracted from the street view images for dictionary learning. The Self-Growing and Self-Organized Neural Gas (SGONG) network is then used to cluster adaptively the extracted BRISK descriptors for generating visual words. The histogram of visual words is further calculated to form the appearance feature of the sign text. For eliminating the color differences and further highlighting the histogram similarity of all colors of signs, a color invariant histogram, called CIHS histogram, is presented to represent the color information of the sign text. By integrating the visual words histograms and CIHS histograms, an integrated descriptor, called Appearance and Color (A&C) descriptor, is specifically designed as the input features for cascade-Adaboost classifier. In the multi-scale sliding window text sign detection, integral image is applied to the spatial distribution map of each visual word for avoiding repeated extraction of features. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods and the detectors with the traditional descriptors.

Effective feature representation is crucial to view-based 3D object retrieval (V3OR). Most previous works employed hand-crafted features to represent the views of each object. Although deep learning based methods has shown its excellent performance in many vision tasks, it is hard to get excellent performance for unsupervised 3D object retrieval. In this paper, we propose to combine the off-the-shelf deep model and graph model to retrieve unseen objects. By employing the powerful deep classification models which are trained from millions of images, we obtain significant improvements compared with state of the art methods. We validate the effectiveness of the ready CNN from other domains that can greatly facilitate the representative ability of objects’ views. In addition, we analyze the representative abilities of different fully connected layers for V3OR, and propose to employ multigraph learning to fuse the deep features of different layers. The autoencoder is then explored to improve the retrieval speed to a large extent. Experiments on two popular datasets are carried out to demonstrate the effectiveness of the proposed method.

Texture characteristic is an important attribute of medical images, and has been applied in many medical image applications. This paper proposes a simple approach to employ the texture features of medical images for retrieval. The developed approach first conducts image filtering to medical images using different Gabor and Schmid filters, and then uniformly partitions the filtered images into non-overlapping patches. These operations provide extensive local texture information of medical images. The bag-of-words model is finally used to obtain feature representations of the images. Compared with several existing features, the proposed one is more discriminative and efficient. Experiments on two benchmark medical CT image databases have demonstrated the effectiveness of the proposed approach.

With the development of manufacture, more and more 3D models are generated by users and many differnet factories. 3D model retrieval has been receiving more and more attention in computer vision and the field of data analysis. In this paper, we propose a novel 3D model retrieval algorithm by cross-modal feature mapping (CMFM), which utilize one single image as query information to address 3D model retrieval problem. Specifically, in this paper, we first proposed to leverage 2D image to handle 3d model retrieval problem, which is one new problem in this field. The proposed feature learning method can benefit: 1) avoiding the interference of query image recorded by different visual sensor; 2) handling cross-modal data retrieval by simple computer vision technologies, which can guarantee the performance of retrieval and also control that the retrieval time hold a low level; 3) the low complexity of this method can guarantee that this method can be applied in many fields. Finally, we validate the retrieval method on three popular datasets. Extensive comparison experiments show the superiority of the proposed mehtod. To the best of our knowledge, it is the first method to handle 3D model retreival based on one single 2D image.

To achieve powerful infringement retrieval for reference images in digital publications, a new improved Scale Invariant Feature Transformation (SIFT) algorithm has been proposed in this paper. The retrieval process in the improved algorithm is innovatively divided into two stages to achieve coarse retrieval and fine retrieval respectively. In coarse retrieval, Geographical Statistics (GeoStat) is creatively used to describe the global spatial relationship of key-points in different orientations in an image and then generate a 144-dimensional feature vector to represent each image. In fine retrieval, only partial images, which are highly similar to the query image obtained from results of coarse retrieval, need to be considered. And the indexing and matching process in the improved algorithm is improved by adding a judgment process to improve the matching speed and reduce the mistaken matching rate. Experimental results show that the proposed algorithm has more advantages in retrieval speed and higher retrieval accuracy than the original SIFT algorithm. And the proposed algorithm is also more suitable for infringement retrieval of reference images in digital publications than the original one.

Content-based image retrieval (CBIR) retrieves images from image database based on the visual similarity of query image. For the implementation of CBIR, feature extraction plays a significant role, where colour feature is quite remarkable. But, due to achromatic surfaces or unevenly colored, the role of texture is also important. This paper introduced an efficient and fast CBIR system, which is based on the combination of computationally light weighted colour and texture features viz. chromaticity moment, colour percentile, and local binary pattern. For searching, this paper proposes inverse variance based varying weighted similarity measure (low for high variance feature and high for low variance feature), which reduces the effect of redundancy by assigning the priority to each feature, and effectively retrieves relevant images. In addition, this paper also proposes query image classification and retrieval model by filtering out irrelevant class images using Random Forests (RF) classifier, which recovers the class of a query image based on distinct learning (supervised) of various decision trees. This successful ensemble classification of query images reduces the semantic gap, searching space, and enhances the retrieval performance. Extensive experimental analyses on benchmark databases confirm the usefulness and effectiveness of this work.

In order to improve the retrieval accuracy of image retrieval systems, research focus has been shifted from designing sophisticated low-level feature extraction algorithms to combining image retrieval processing with rich semantics and knowledge-based methods. In this paper, we aim at improving text-based image retrieval for complex natural language queries by using a semantic parser (Knowledge Parser or K-Parser). From text written in natural language, the K-parser extracts a graphical semantic representation of the objects involved, their properties as well as their relations. We analyze both the image textual captions and the natural language queries with the K-parser. As a technical solution, we leverage RDF in two ways: first, we store the parsed image captions as RDF triples; second, we translate image queries into SPARQL queries. When applied to the Flickr8k dataset with a set of 16 custom queries, we notice that the K-parser exhibits some biases that negatively affect the accuracy of the queries. We propose two techniques to address the weaknesses: (1) we introduce a set of rules to transform the output of K-parser and fix some basic, recurrent parsing mistakes that occur on the captions of Flickr8k; (2) we leverage two popular commonsense knowledge databases, ConceptNet and WordNet, to raise the accuracy of queries on broad concepts. Using those two techniques, we can fix most of the initial retrieval errors, and accurately execute our set of 16 queries on the Flickr8k dataset.

We have witnessed 3D shape models abundant in many application fields including 3D CAD/CAM, augmented/mixed reality (AR/MR), and entertainment. Creating 3D shape models from scratch is still very expensive. Efficient and accurate methods for shape retrieval is essential for 3D shape models to be reused. To retrieve similar 3D shape models, one must provide an arbitrary 3D shape as a query. Most of the research on 3D shape retrieval has been conducted with a “whole” shape as a query (aka whole-to-whole shape retrieval), while a “part” shape (aka part-to-whole shape retrieval) is more practically requested as a query especially by mechanical engineering with 3D CAD/CAM applications. A “part” shape is naturally constructed by a 3D range scanner as an input device. In this paper, we focus on the efficient method for part-to-whole shape retrieval where the “part” shape is assumed to be given by a 3D range scanner. Specifically, we propose a Super-Vector coding feature with SURF local features extracted from the View-Normal-Angle image, or the image synthesized by taking account of the angle between the view vector and the surface normal vector, together with the depth-buffered image, for part-to-whole shape retrieval. In addition, we propose a weighted whole-to-whole re-ranking method taking advantage of global information based on the result of part-to-whole shape retrieval. Through experiments we demonstrate that our proposed method outperforms the previous methods with or without re-ranking.

Rapid increase in digital images demands effective and efficient image retrieval systems. In text based image retrieval, images are annotated with keywords based on human perception. A user query is composed of keywords according to his/her requirements. Query keywords are matched with the keywords associated with images, for retrieval. This process has been extended with ontology to resolve semantic heterogeneities. However, crisp annotation and retrieval processes could not produce the desired results because both processes involve human perception. To overcome this problem, we have proposed a retrieval system that makes use of fuzzy ontology for improving retrieval performance. For modeling the semantic description of an image, it is divided into regions in our dataset and then regions are classified into concepts. The concepts are combined into categories. The concepts, categories and images are linked among themselves with fuzzy values in ontology. The retrieved results are ranked based on the relevancy between the keywords of a query and images. For evaluating the performance of the proposed methodology, we have used both the objective and subjective measures. Experimental results show that the proposed system performs better than the existing systems in terms of retrieval performance.

Image content analysis plays a key role in areas such as image classification, clustering, indexing, retrieving, and object and scene recognition. However, although several image content descriptors have been proposed in the literature, their low performance score or high computational cost makes them unsuitable for content-based image retrieval on large datasets. This paper presents an efficient content-based image retrieval approach that uses histogram-based descriptors to represent color, edge, and texture features, and a k-nearest neighbor classifier to retrieve the best matches for query images. The compactness and speed of the proposed descriptors allow their application in heterogeneous photographic collections whilst showing strong image discrimination in the presence of significant content variation. Experimentation was conducted on four different image collections using four distance metrics. The results show that the proposed approach consistently achieves noteworthy mean average precision, recall, and precision measures. It outperforms state-of-the-art approaches based on the MPEG 7 descriptors (SCD, CLD, and EHD), whilst producing comparable results to those achieved by novel SIFT-based and SURF-based approaches that require more complex data manipulation.

Learning-based hashing methods are becoming the mainstream for large scale visual search. They consist of two main components: hash codes learning for training data and hash functions learning for encoding new data points. The performance of a content-based image retrieval system crucially depends on the feature representation, and currently Convolutional Neural Networks (CNNs) has been proved effective for extracting high-level visual features for large scale image retrieval. In this paper, we propose a Multiple Hierarchical Deep Hashing (MHDH) approach for large scale image retrieval. Moreover, MHDH seeks to integrate multiple hierarchical non-linear transformations with hidden neural network layer for hashing code generation. The learned binary codes represent potential concepts that connect to class labels. In addition, extensive experiments on two popular datasets demonstrate the superiority of our MHDH over both supervised and unsupervised hashing methods.

With successful launch of high spatial resolution (HSR) sensors, highly detailed spatial information is provided for remote sensing research. This improvement has allowed researchers to monitor environmental changes on a small spatial scale. However traditional pixel-based classification approaches are not able to interpret high spatial resolution remote sensing imagery effectively. Bag of visual words (BoVW) framework, on the other hand, is becoming one of the most popular approaches to validate the performance of remote sensing image datasets. While pixel-based approaches may not fully describe very high-resolution remote sensing images, BoVW model is narrowing the gap between low-level features and high-level semantic features by generating an intermediate description of image features. This paper presents a comparative study to evaluate the potential of using different coding approaches of BoVW model to solve the land-use scene classification problem. Initially, this work summarizes different configurations of BoVW framework in coding and clustering. Later, we perform an extensive evaluation of BoVW on land-use scene classification and retrieval. Finally we draw several conclusions regarding different coding strategies of BoVW, codebook size and number of training images. The approach is validated on two commonly used datasets in remote sensing, UC Merced a 21-class land-use dataset and RSDataset a 19-class satellite scene dataset.

Multimodal Retrieval provides new paradigms and methods aimed at effectively searching through the enormous volume of data. Multimodal retrieval is a well studied problem often used in image retrieval. Most of the existing works in image retrieval under the pretext of multimodality stress on bridging the semantic gap by using both textual and visual features. In this paper, we use relevance feedback from the user-generated documents associated with the images for expanding textual query and study its effect on both image and text retrieval. We employ a topic decomposition based keyphrase extraction technique to expand the textual queries. Our results articulate the fact that an insightful textual query expansion always improves retrieval performance for both textual or image retrieval. Also, we adopt optimum weight learning scheme to combine the modalities in a privileged way. We perform a comparative study with two well established keyphrase extraction techniques which are used for textual query expansion. A detailed set of experiments on a standard real world dataset is also carried out for the same.

Most effective particular object and image retrieval approaches are based on the bag-of-words (BoW) model, and all state-of-the-art performance mainly involves a query expansion procedure, which is able to significantly improve retrieval results. Nowadays, Convolutional Neural Network(CNN) is widely applied in computer vision field, including image classification, caption, recognition and retrieval, etc. We introduce an extension to query expansion: an automatic method to select good candidate samples for interactive annotation which is used in query expansion using both BoW method and CNN feature. In this work, we address the query expansion framework using active learning, where the main focus is on the sample selection step in the process of query expansion. More specifically, we propose an active sample selection algorithm based on binary relevance classification, based on the assumption that most confusing samples of the classifiers have high probability to contain helpful true positives for query expansion, which significantly improves the retrieval performance. It takes full use of the multimodal information of the shortlist obtained from the basic retrieval to train a binary relevance classifier, which is used to pick up the most confusing samples for human annotation, with top list as unlabeled data and bottom list as fake negatives. And it can achieve a faster and better retrieval than naive top sample selection method. We also fuse BoW vector and CNN prediction in the retrieval system for a better performance. To evaluate the performance of our proposed method, experiments are conducted on Standard Oxford (5K and 105K) and Paris (6K) datasets, and experimental results and comparison with the state-of-the-art methods demonstrate the effectiveness of the proposed method.

Multiple Classifier System has found its applications in many areas such as handwriting recognition, speaker recognition, medical diagnosis, fingerprint recognition, personal identification and others. However, there have been rare attempts to develop content-based image retrieval (CBIR) system that uses multiple classifiers to learn visual similarity. Texture as a primitive visual content is often used in many important applications (viz. Medical image analysis and medical CBIR system). In this paper, a texture image retrieval system is developed that learns the visual similarity in terms of class membership using multiple classifiers. The way proposed approach combines the decisions of multiple classifiers to obtain final class memberships of query for each of the output classes is also a novel concept. A modified distance that is weighted with the membership values obtained through similarity learning is used for ranking. Three different algorithms are proposed for the retrieval of images against a query image displaying the strength of multiple classifier approach, class membership score and their interplay to achieve the objective defined in terms of simplicity, retrieval effectiveness and speed. The proposed methods based on multiple classifiers achieve higher retrieval accuracy with lower standard deviation compared to all the competing methods irrespective of the texture database and feature set used. The multiple classifier retrieval schemes proposed here is tested for texture image retrieval. However, these can be used for any other challenging retrieval problems.

The rapid growth of digital image collections has prompted the need for development of software tools that facilitate efficient searching and retrieval of images from large image databases. Towards this goal, we propose a content-based image retrieval scheme for retrieval of images via their color, texture, and shape features. Using three specialized histograms (i.e. color, wavelet, and edge histograms), we show that a more accurate representation of the underlying distribution of the image features improves the retrieval quality. Furthermore, in an attempt to better represent the user’s information needs, our system provides an interactive search mechanism through the user interface. Users searching through the database can select the visual features and adjust the associated weights according to the aspects they wish to emphasize. The proposed histogram-based scheme has been thoroughly evaluated using two general-purpose image datasets consisting of 1000 and 3000 images, respectively. Experimental results show that this scheme not only improves the effectiveness of the CBIR system, but also improves the efficiency of the overall process.

This article addresses the problem of representation, indexing and retrieval of images through the signature-based bag of visual words (S-BoVW) paradigm, which maps features extracted from image blocks into a set of words without the need of clustering processes. Here, we propose the first ever method based on the S-BoVW paradigm that considers information of texture to generate textual signatures of image blocks. We also propose a strategy that represents image blocks with words which are generated based on both color as well as texture information. The textual representation generated by this strategy allows the application of traditional text retrieval and ranking techniques to compute the similarity between images. We have performed experiments with distinct similarity functions and weighting schemes, comparing the proposed strategy to the well-known cluster-based bag of visual words (C-BoVW) and S-BoVW methods proposed previously. Our results show that the proposed strategy for representing images is a competitive alternative for image retrieval, and overcomes the baselines in many scenarios.

Information retrieval systems are getting more attention in the era of multimedia technologies such as an image, video, audio and text files. The large numbers of images are challenges in computer systems field to store, manage data effectively and efficiently. The shape retrieval feature of different objects in the image also remains a difficult problem due to distinct angle view of different objects in a scene only; few studies have reported solution to the problem of finding relative locations of ROIs. In this paper, we proposed three methods such as1. Geolocation-based image retrieval (GLBIR), 2.Unsupervised feature technique Principal component analysis (PCA) and 3.multiple region-based image retrieval. The first proposed (GLBIR) method identifies geo location an image using visual attention based mechanism and its color layout descriptors. These features are extracted from geo-location of query image from Flickr database. Our proposed model does not fully semantic understanding of image content, uses visual metrics for example; the proximity ,color contrast, size and nearness to image's boundaries to locate viewer's attention. We analyzed results and compared with state of art CBIR Systems and GLBIR Technique. Our second method to refine images exploiting and fusing by unsupervised feature technique using principal component analysis (PCA). The visually similar images clustering together with analyses image retrieval process and remove outliers initially retrieved image set by PCA. To evaluation our proposed approach, we used thousands of images downloaded from Flickr and CIFAR-10 databases using Flickr public API. Finally, we determinately proposed a system for image retrieval based on region. It provides a user interface for availing to designate the watershed ROI within an input image. During the retrieval of images, regions' feature vectors having codes of region homogeneous to a region of input image are utilized for comparison. Standard datasets are used for evaluation of proposed approach. The experiment demonstrates and effectiveness of the proposed method to achieve higher annotation performance increases accuracy and reduces image retrieval time. We evaluated our proposed approach on images dataset from Flickr and CIFAR-10.

During the past few years, the Bag-of-Words (BoW) model based on SIFT features has been one of the most adopted approaches by the Content-Based Image Retrieval (CBIR) systems. However, these CBIR systems have shown some weaknesses and shortcomings especially for large scale image collections. This is due to two main causes: First, information is lost in the quantization step and second, the SIFT features describe only the local gradient. To tackle these issues, we proposed to take advantage of the Hamming Embedding, soft assignment and multiple assignment techniques, on the one hand, and to fuse SIFT and color features at the indexing level in a multi-index structure, on the other. In fact, in this paper, generic and non-parametric image retrieval schemes as well as a novel multi-IDF design based on multi-index structure were proposed. Extensive experiments were conducted on three public datasets (Holidays, Ukbench and MIR Flickr 1 M as distractor). The experimental results are promising and outperform the state-of-the-art CBIR systems. In addition, only 117 bits are needed to represent each key-point which enables us to make our image retrieval schema suitable for large-scale experiments.

Content based image retrieval systems rely heavily on the set of features extracted from images. Effective image representation emerges as a crucial step in such systems. A key challenge in visual content representation is to reduce the so called `semantic gap'. It is the inability of existing methods to describe contents in a human-oriented way. Content representation methods inspired by the human vision system have shown promising results in image retrieval. Considerable work has been carried out during the past two decades for developing methods to extract descriptors inspired by the human vision system and attempt to retrieve visual contents efficiently according to the user needs, thereby reducing the semantic gap. Despite the extensive research being conducted in this area, limitations in current image retrieval systems still exist. This paper presents a descriptor for personalized social image collections which utilizes the local structure patterns in salient edge maps of images at multiple scales. The human visual system at the basic level is sensitive to edges, corners, intersections, and other such intensity variations in images generating local structure patterns. Analyzing these patterns at multiple scales allow the most salient fine-grained and coarse-grained features to be captured. The features are accumulated in a local structure patterns histogram to index images allowing flexible querying of visual contents. The retrieval results show that the proposed descriptor ranks well among similar state-of-the-art methods for large social image collections.

Content based image retrieval systems (CBIR) are used to search images on the basis of their visual content in a huge image database. This approach uses a multi-clustering technique with a multi-searching process. In addition it integrate, Mean SIFT (Scale Invariant Feature Transform) descriptor as local feature and HSV(hue, saturation, value) histogram as global feature. Our proposition aims to a maximum separation of the execution to gain time and keep the performance of each descriptor. Local and global features are combined for more relevant results. Getting several views to relevant results to cover the subjectivity in displaying results. This article, detailed our proposed method with a comparison with the FIRE (Flexible Image Retrieval) Engine and LIRE (Lucene Image Retrieval). The results demonstrate the feasibility and relevance of our proposition.

In this paper we depict an implemented system for medical image retrieval. Our system performs retrieval based on both textual and visual content, separately and combined, using advanced encoding and quantization techniques. The text-based retrieval subsystem uses textual data acquired from an image's corresponding article to generate a suitable representation. Using a vector space model, the generated representations structure is altered to increase performance. Query expansion with pseudo-relevance feedback is applied to fine-tune the results. The content-based retrieval subsystem performs retrieval based on visual features extracted from the images. A Gaussian Mixture Model is constructed from the extracted visual features, in our case - RGB histograms, and is used in encoding the same features into Fisher Vectors. With scalability and speed in mind, we utilized a product quantization technique over the generated vectors, which provides fast response times over large image collections. Product quantization drastically reduces the size of the image representation at almost no cost to accuracy, thus improving the scalability factor of our system. Our system uses modality classification to further improve retrieval results. This subsystem labels the image modality based on their visual content. The images are described using state-of-the-art opponentSIFT visual features. Classification was performed using Support Vector Machines (SVMs). The predictions from the SVMs are used for re-ranking the resulting images based on their modality and the modality of the query. The system was evaluated against the standardized ImageCLEF 2013, 2012 and 2011 medical datasets and it reported state-of-the-art performance for all datasets.

Visual vocabulary representation approach has been successfully applied to many multimedia and vision applications, including visual recognition, image retrieval, and scene modeling/categorization. The idea behind the visual vocabulary representation is that an image can be represented by visual words, a collection of local features of images. In this work, we will develop a new scheme for the construction of visual vocabulary based on the analysis of visual word contents. By considering the content homogeneity of visual words, we design a visual vocabulary which contains macro-sense and micro-sense visual words. The two types of visual words are appropriately further combined to describe an image effectively. We also apply the visual vocabulary to construct image retrieving and categorization systems. The performance evaluation for the two systems indicates that the proposed visual vocabulary achieves promising results.

Sparse coding has received extensive attention in the literature of image classification. Traditional sparse coding strategies tend to approximate local features in terms of a linear combination of basis vectors, without considering feature neighboring relationships. In this scenario, similar instances in the feature space may result in totally different sparse codes. To address this shortcoming, we investigate how to develop new sparse representations which preserve feature similarities. We commence by establishing two modules to improve the discriminative ability of sparse representation. The first module selects discriminative features for each class, and the second module eliminates non-informative visual words. We then explore the distribution of similar features over the dominant basis vectors for each class. We incorporate the feature distribution into the objective function, spanning a class-specific low dimensional subspace for effective sparse coding. Extensive experiments on various image classification tasks validate that the proposed approach consistently outperforms several state-of-the-art methods.

Latent Dirichlet Allocation (LDA) is one popular topic extraction method, which has been applied in many applications such as textual retrieval, user recommendation system and video cluster. In this paper, we apply LDA model for visual topics extraction and utilized the topic distribution visual feature of image to handle 3D object retrieval problem. Different from the traditional LDA model, we add the spatial information of visual feature for document generation. First, we extract SIFT features from each 2D image extracted from 3D object. Then, we structure the visual documents according to the spatial information of 3D model. Finally, LDA model is used to extract the topic model for handling the retrieval problem. We further propose a multi-topic model to improve retrieval performance. Extensive comparison experiments were on the popular ETH, NTU and MV-RED 3D model datasets. The results demonstrate the superiority of the proposed method.

We present a method for the efficient retrieval and browsing of immense amounts of realistic 3D human body motion capture data. The proposed method organizes motion capture data based on statistical K-means (SK---means), democratic decision making, unsupervised learning, and visual key frame extraction, thus achieving intuitive retrieval by browsing thumbnails of semantic key frames. We apply three steps for the efficient retrieval of motion capture data. The first is obtaining the basic type clusters by clustering motion capture data using the novel SK-means algorithm, and after which, immediately performing character matching. The second is learning the retrieval information of users during the retrieval process and updating the successful retrieval rate of each data; the search results are then ranked on the basis of successful retrieval rate by democratic decision making to improve accuracy. The last step is generating thumbnails with semantic generalization, which is conducted by using a novel key frame extraction algorithm based on visualized data analysis. The experiment demonstrates that this method can be utilised for the efficient organization and retrieval of enormous motion capture data.

Content-based image retrieval aims at substituting traditional indexing based on manual annotation by using automatically-extracted visual indexing features. Novel techniques are needed however to efficiently deal with the semantic gap (i.e. the partial match between the low-level features and the visual content). Here, we investigate a query-free retrieval approach first proposed by Ferecatu and Geman. This approach relies solely on an iterative relevance feedback mechanism that drives a heuristic sampling of the collection, and aims to take explicitly into account the semantic gap. Our contributions are related to three complementary aspects. First, we formalize a large-scale approach based on a hierarchical tree-like organization of the images computed off-line. Second, we propose a versatile modulation of the exploration/exploitation trade-off based on the consistency of the system internal states between successive iterations. Third, we elaborate a long-term optimization of the similarity metric based on the user searching session logs accumulated off-line. We implemented a web-application that integrates all our contributions, and distribute it under the AGPL Version 3 free software license. We organized user-based evaluation campaigns using ImageNet dataset, and show empirically that our contributions significantly improve the retrieval performance of the original framework, that they are complementary to each other, and that their overall integration is consistently beneficial.

Diversification of search results allows for better and faster search, gaining knowledge about different perspectives and viewpoints on retrieved information sources. Recently various methods for diversification of image retrieval results have been proposed, mainly using textual information or techniques imported from the natural language processing domain. However, images contain much more information than their textual descriptions and the use of visual features deserves special attention in this context. Visual saliency provides information about parts of the image perceived as most important, which are instinctively targeted by humans when shooting a photo or looking at a picture. For this reason we propose to exploit such information to improve diversification of search results. To this purpose, we introduce a saliency-based method to re-rank the results of a query and we show that it can achieve significantly better performances as compared to the baseline approach. Experimental validation conducted on a number of queries applied to various datasets demonstrates the potential of the use of saliency information for the diversification of image retrieval results.

Along with the rapid development of mobile terminal devices, landmark recognition applications based on mobile devices have been widely researched in recent years. Due to the fast response time requirement of mobile users, an accurate and efficient landmark recognition system is thus urgent for mobile applications. In this paper, we propose a landmark recognition framework by employing a novel discriminative feature selection method and the improved extreme learning machine (ELM) algorithm. The scalable vocabulary tree (SVT) is first used to generate a set of preliminary codewords for landmark images. An efficient codebook learning algorithm derived from the word mutual information and Visual Rank technique is proposed to filter out those unimportant codewords. Then, the selected visual words, as the codebook for image encoding, are used to produce a compact Bag-of-Words (BoW) histogram. The fast ELM algorithm and the ensemble approach using the ELM classifier are utilized for landmark recognition. Experiments on the Nanyang Technological University campus's landmark database and the Fifteen Scene database are conducted to illustrate the advantages of the proposed framework.

In this paper, a new color image retrieval technique is proposed with local features given by Scale Invariant Feature Transform (SIFT) key points that are described in the integer-natured, computationally light Orthogonal Polynomials Transform (OPT) domain. The transform's point spread operators are derived from a generating function, modification to which has been proposed for alleviating computational complexity further. The expressive power of the transform coefficients has been exploited for forming the descriptors of SIFT key points of a given image. The key point descriptors, OPT-SIFT so formed have good expressive power, despite being shorter in length and having a reduced computational complexity. A retrieval technique has been proposed based on OPT-SIFT features. The proposed retrieval technique has been experimented with images from standard databases such COIL-100 and Corel and the results demonstrate the superiority of the proposed descriptors when compared to other descriptors.

Automatic image annotation is a vital and challenging problem in pattern recognition and image understanding areas. The existing models directly extract visual features from segmented image regions. Since segmented image regions may still have multi-objects, the extractive visual features may not effectively describe corresponding regions. In addition, existing models did not consider the visual representations of corresponding keywords, which would lead to appearing plenty of irrelevant annotations in final annotation results, and these annotations did not relate to any part of images considering visual contents. In order to overcome the above problems, an image annotation model based on multi-scale salient region and relevant visual keywords is proposed. In this model, each image is segmented by using multi-scale grid segmentation method and the global contrast based method is used to extract the saliency maps from each image region. Visual features are extracted from each salient region. In addition, each keyword is divided into two categories: abstract words or non-abstract words. Visual seeds of each non-abstract word are established, and then a new method is proposed to extract visual keyword collections by using corresponding seeds. According to the traits of abstract words, an algorithm based on subtraction regions is proposed to extract visual seeds and corresponding visual keyword collections of each abstract word. Adaptive parameter method and a fast solution algorithm are proposed to determine the similarity thresholds of each keyword. Finally, multi-scale visual features and the combinations of the above methods are used to improve the annotation performance. Our model can improve the object descriptions of images and image regions. Experimental results verify the effectiveness of the proposed model.

Traditional BOV (bag-of-visterms) constructing methods ignore different visual words' contributions to image representation. Moreover, the size of visual vocabulary is generally set according to experience. Frequent weighted concept lattice (FWCL) is an interesting version of the concept lattice, and helps realize knowledge extraction in a more efficient way. In this paper, we present a dynamic refinement method for visual vocabulary formation based on frequency weighted concept lattice (FWCL) by using its characteristic of hierarchical data analysis and reduction. The original visual words' weight value is assigned with information entropy, and FWCLs of each semantic category are constructed. According to the extent thresholds, each category of visual vocabulary is built and combined to generate the reduced global visual vocabulary for image representation. By adjusting the extent thresholds, different sizes of reduced visual vocabularies are dynamically extracted from this type of hierarchical structure. Lastly, experiments are carried on the two commonly used datasets, and experimental results show the effectiveness of our method on the scene classification.

This work offers an overview of the state-of-the-art on the emerging area of 3D object retrieval based on partial queries. This research area is associated with several application domains, including face recognition and digital libraries of cultural heritage objects. The existing partial 3D object retrieval methods can be mainly classified as: i) view-based, ii) part-based, iii) bag of visual words (BoVW)-based, and iv) hybrid methods combining these three main paradigms or methods which cannot be straightforwardly classified. Several methodological aspects are identified, including the use of interest points and the exploitation of 2.5D projections, whereas the available evaluation datasets and campaigns are addressed. A thorough discussion follows, identifying advantages and limitations.

Improving query quality and robustness is a hot topic in information and image retrieval field, which has resulted in many interesting works. To address the same problem for deformable non-rigid 3D shape retrieval, two topics are considered in this paper. The first one we discussed is shape representation, which is related to feature extraction and fusion. For feature extraction, we create a global feature to achieve a coarser-scale shape appearance description. Then, to alleviate the drawbacks of retrieval by single feature, we develop a novel fusion method for multiple feature fusion, which turns out to be superior to weighted sum approach with a low complexity. The second topic studied in this paper is to further refine the retrieval results by introducing a new retrieval guidance algorithm based on category prediction. To evaluate the proposed methods, experiments on three popular non-rigid datasets are carried out. The evaluation results suggest that our shape representation method has achieved state-of-the-art performance. Then, by adjusting the retrieval results of existing methods, our retrieval guidance algorithm has promoted the accuracy with nice effects.

Although the object detection and recognition has received growing attention for decades, a robust fire and flame detection method is rarely explored. This paper presents an empirical study, towards a general and solid approach to fast detect fire and flame in videos, with the applications in video surveillance and event retrieval. Our system consists of three cascaded steps: (1) candidate regions proposing by a background model, (2) fire region classifying with color-texture features and a dictionary of visual words, and (3) temporal verifying. The experimental evaluation and analysis are done for each step. We believe that it is a useful service to both academic research and real-world application. In addition, we release the software of the proposed system with the source code, as well as a public benchmark and data set, including 64 video clips covered both indoor and outdoor scenes under different conditions. We achieve an 82% Recall with 93% Precision on the data set, and greatly improve the performance by state-of-the-arts methods., 14 pages, 6 figures

As the amount of digital image continues to grow in usage, users are experiencing increased difficulty in finding specific images in the image collection. This paper proposes a novel image searching scheme that extracts the image feature using combination of ASURF (Advanced Speed-Up Robust Feature) and DCD (Dominant Color Descriptor). The system for mobile image searches runs in real-time on iPhone, and can be easily used to find a natural color image. To evaluate the proposed scheme, we assessed the performance of simulation in term of average precision and F-score on two image database, which is commonly used in the field of image retrieval. The experimental results revealed that the proposed algorithm exhibited a significant improvement of over 14.4 % in retrieval effectiveness, compared to open source OpenSURF. The main contribution of this paper is that the proposed approach achieves high accuracy and stability by using ASURF and DCD in searching for natural image on mobile platform.

Sketch-Based 3D Object Retrieval (SB3DOR) algorithms retrieve 3D models similar to hand-drawn sketch queries. It is one of the most effective modalities to query 3D models by their shape. However, comparison of a sketch, which is a 2D image, with a 3D model is not straightforward. Most of the SB3DOR algorithms compare a sketch image with images of 3D models rendered from multiple viewpoints. However, retrieval accuracies of state-of-the-art SB3DOR algorithms are still not satisfactory, due, in part, to stylistic variation, semantic influence, abstraction, and drawing error found in sketches. To improve retrieval accuracy of SB3DOR systems, we propose a manifold-based similarity metric learning algorithm that relates two kinds of features, that are, of sketches and 3D models. Features in a high dimensional space often lie on a lower dimensional subspace, or manifold. Feature similarity may be computed more accurately on the manifold than in the original high dimensional space. Our Cross-Domain Manifold Ranking (CDMR) algorithm tries to keep the two distinct feature manifolds, one for sketch features and the other for 3D model features, intact. These two manifolds are interlinked by using inter-feature similarity to form a Cross-Domain Manifold (CDM). If available, semantic labels may also be used in forming the CDM. Relevance diffusion is used to compute similarities between a sketch and 3D models in a database. Experimental evaluation showed that the CDMR algorithm produces higher retrieval accuracy than the algorithms we compared against.

In the last few years, extensive effort has been spent to develop better performed 3-D object retrieval methods. View-based methods have attracted a significant amount of attention, not only because their state-of-art performance, but also they merely require some of a 3-D object's 2-D view images. However, most recent approaches only deal with the images' primordial-extracted features and ignore their hidden relationships. Considering these latent characters, a visual-topic-model 3-D object retrieval approach is introduced in this paper. In this framework, dense scale invariant feature transform(dense-SIFT) descriptors are extracted from a set of views of each 3-D object, and all the dense-SIFT descriptors are grouped into bag-of-word features using k-means clustering. Then, the topic distribution of a 3-D object is generated via latent dirichlet allocation (LDA) given its bag-of-word features. Gibbs sampling is applied in the learning and inference processing of LDA. We conduct experiments on the Princeton Shape Benchmark (PSB) and National Taiwan University 3D model database (NTU), and the experimental results demonstrate that the proposed method can achieve better retrieval effectiveness than the state-of-the-art methods under several standard evaluation measures.

Object recognition technology is usually used for recognizing specific objects, such as book covers, landmarks, vehicles, etc. This technology is supported by multi-dimensional local image descriptors in most situations. These descriptors are designed to be robust to the environmental changes, such as illumination change, view angle change, scale change, etc. If there are many target objects in your database, object recognition using large scale local image descriptor database may not be a trivial task, because of the high dimensionality of the local image descriptors. For consistent responses from a large-scale database with a reasonable time delay, we need to have a proper data structure which supports the indexing and querying functionality. A vocabulary tree is a data structure based on local image descriptors, and this data structure is commonly used to cope with massive databases containing local image descriptors. By using a vocabulary tree, a local image descriptor can be mapped to a vocabulary tree's leaf node ID, constructing a visual word for object recognition. Visual words are then effectively exploited by a traditional text retrieval engine. In this study, we built a large-scale object recognition system using a vocabulary tree that had leaf nodes of 1 million Scale-Invariant Feature Transform (SIFT) descriptors, which is the most promising local image descriptor in terms of precision. We implement proposed system using publicly available software so that further enhancements and/or reproducibility would be easily accomplished. We then compared and evaluated the proposed system's performance with the current MPEG CDVS (Compact Descriptors for Visual Search) standard using a database containing two dimensional planar object datasets of three categories with one million distracter images. In addition to these datasets, which are equivalent to those of CDVS, we add a new dataset which are made to mimic realistic occlusion and clutter effects. Experimental results show that our proposed system's performance is comparable to that of the CDVS achieving 90 % precision at 5 s retrieval time. We also find characteristics of vocabulary tree limiting adaptation to a specific application domain.

The technology of medical data production has been rapidly changed over the past few years. Modern computer technology has created the possibility of creating multi-modal medical images. Medical data often contain multi-modal information such as visual information (image) as well as textual information. Both types of information are important for medical retrieval system (MRS). Due to the information limitation at different levels of sources, the application of information fusion becomes a real need in medical application. In this research, an information fusion framework was built to develop the multi-modality medical image retrieval system (IFM3IRS). The framework utilizes two sources of information involving text and visual-based retrieval process. The application is based on sequential order where the result from text-based process will automatically be the input in visual-based process. The main contributions of this paper are the development of a new ranking model called MedHieCon ranking model which applies semantic concepts of modality, anatomy and pathology in text-based process and also the learning approach of medical images using medical concept model in visual-based process. ImageCLEFmed 2010 data collection was used to evaluate IFM3IRS and it shows that our information fusion framework is in top list among other researchers. Although text-based retrieval system has proven to be a better performance in MRS; it is significant to determine the overall performance improvements which include the fusion of text and image.