Search

With advancement of various information processing and storage techniques, the scale of digital music collections has been growing at very fast speed during recent decades. To support high-quality content-based retrieval over such a large volume of music data, how to develop indexing structure with good effectiveness, efficiency and scalability becomes an important research issue. However, existing techniques mainly focus on improving query efficiency. Very few approaches have been proposed to address issues related to scalability and accuracy. In this study, we address the problem via introducing a novel indexing technique called effective music indexing framework (EMIF) to facilitate scalable and accurate music retrieval. It is designed based on a "classification-and-indexing" principle and consists of two main functionality modules: (1) music classification—a novel semantic-sensitive classification to identify an input song's category and (2) indexing module—multiple local indexing structures, one for each semantic category to reduce query response time significantly. In particular, the classification model combining linear discriminative mixture model (LDMM) and advanced score fusion scheme has been applied to estimate category of music accurately. Layered architecture enables EMIF to enjoy superior scalability and efficiency. To evaluate the approach, a set of experimental studies has been carried out using two large music test collections and the results demonstrate various advantages of EMIF over state-of-the-art approaches including efficiency, scalability and effectiveness. [ABSTRACT FROM AUTHOR]

As mobile device penetration increases, it has become pervasive for images to be associated with locations in the form of geotags. Geotags bridge the gap between the physical world and the cyberspace, giving rise to new opportunities to extract further insights into user preferences and behaviors. In this article, we aim to exploit geotagged photos from online photo-sharing sites for the purpose of personalized Point-of-Interest (POI) recommendation. Owing to the fact that most users have only very limited travel experiences, data sparseness poses a formidable challenge to personalized POI recommendation. To alleviate data sparseness, we propose to augment current collaborative filtering algorithms along from multiple perspectives. Specifically, hybrid preference cues comprising user-uploaded and user-favored photos are harvested to study users’ tastes. Moreover, heterogeneous high-order relationship information is jointly captured from user social networks and POI multimodal contents with hypergraph models. We also build upon the matrix factorization algorithm to integrate the disparate sources of preference and relationship information, and apply our approach to directly optimize user preference rankings. Extensive experiments on a large and publicly accessible dataset well verified the potential of our approach for addressing data sparseness and offering quality recommendations to users, especially for those who have only limited travel experiences. [ABSTRACT FROM AUTHOR]

As an emerging technology to support scalable content-based image retrieval (CBIR), hashing has recently received great attention and became a very active research domain. In this study, we propose a novel unsupervised visual hashing approach called semantic-assisted visual hashing (SAVH). Distinguished from semi-supervised and supervised visual hashing, its core idea is to effectively extract the rich semantics latently embedded in auxiliary texts of images to boost the effectiveness of visual hashing without any explicit semantic labels. To achieve the target, a unified unsupervised framework is developed to learn hash codes by simultaneously preserving visual similarities of images, integrating the semantic assistance from auxiliary texts on modeling high-order relationships of inter-images, and characterizing the correlations between images and shared topics. Our performance study on three publicly available image collections: Wiki, MIR Flickr, and NUS-WIDE indicates that SAVH can achieve superior performance over several state-of-the-art techniques. [ABSTRACT FROM PUBLISHER]

Advanced networking technologies and massive online social media have stimulated a booming growth of travel heterogeneous information in recent years. By employing such information, smart travel guide systems, such as landmark ranking systems, have been proposed to offer diverse online travel services. It is essential for a landmark ranking system to structure, analyze, and search the travel heterogeneous information to produce human-expected results. Therefore, currently the most fundamental yet challenging problems can be concluded: 1) how to fuse heterogeneous tourism information and 2) how to model landmark ranking. In this paper, a novel landmark search system is introduced based on a newly designed heterogeneous information fusion scheme and a query-dependent landmark ranking strategy. Different from the existing travel guide systems, the proposed system can effectively combine the heterogeneous information from multimodality media into a landmark reranking list via a user’s query. Experimental results conducted on a large travel information collection illustrate the advantages of the proposed system in terms of both effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

High quality test collections have been becoming more and more important for the technological advancement in geo-referenced image retrieval and analytics. In this paper, we present a large scale test collection to support robust performance evaluation of landmark image search and corresponding construction methodology. Using the approach, we develop a very large scale test collection consisting of three key components: (1) 355,141 images of 128 landmarks in five cities across three continents crawled from Flickr; (2) different kinds of textual features for each image, including surrounding text (e.g. tags), contextual data (e.g. geo-location and upload time), and metadata (e.g. uploader and EXIF); and (3) six types of low-level visual features. In order to support robust and effective performance assessment, a series of baseline experimental studies have been conducted on the search performance over both textual and visual queries. The results demonstrate importance and effectiveness of the test collection. [ABSTRACT FROM AUTHOR]

System performance assessment and comparison are fundamental for large-scale image search engine development. This article documents a set of comprehensive empirical studies to explore the effects of multiple query evidences on large-scale social image search. The search performance based on the social tags, different kinds of visual features and their combinations are systematically studied and analyzed. To quantify the visual query complexity, a novel quantitative metric is proposed and applied to assess the influences of different visual queries based on their complexity levels. Besides, we also study the effects of automatic text query expansion with social tags using a pseudo relevance feedback method on the retrieval performance. Our analysis of experimental results shows a few key research findings: (1) social tag-based retrieval methods can achieve much better results than content-based retrieval methods; (2) a combination of textual and visual features can significantly and consistently improve the search performance; (3) the complexity of image queries has a strong correlation with retrieval results' quality-more complex queries lead to poorer search effectiveness; and (4) query expansion based on social tags frequently causes search topic drift and consequently leads to performance degradation. [ABSTRACT FROM AUTHOR]

Accurate video tagging has been becoming increasingly crucial for online video management and search. This article documents a novel framework called comprehensive video tagger (CVTagger) to facilitate accurate tag-based video annotation. The system applies both multimodal and temporal properties combined with a novel classification framework with hierarchical structure based on multilayer concept model and regression analysis. The advanced architecture enables effective incorporation of both video concept dependency and temporal dynamics. Using a large-scale test collection containing 50,000 YouTube videos, a set of empirical studies have been carried out and experimental results demonstrate various advantages of CVTagger over the state-of-the-art techniques. [ABSTRACT FROM AUTHOR]

The Internet has brought about fundamental changes in the way peoples generate and exchange media information. Over the last decade, unsolicited message images (image spams) have become one of the most serious problems for Internet service providers (ISPs), business firms and general end users. In this paper, we report a novel system called RoBoTs ( Ro bust Bo os T rap based s pam detector) to support accurate and robust image spam filtering. The system is developed based on multiple visual properties extracted from different levels of granularity, aiming to capture more discriminative contents for effective spam image identification. In addition, a resampling based learning framework is developed to effectively integrate random forest and linear discriminative analysis (LDA) to generate comprehensive signature of spam images. It can facilitate more accurate and robust spam classification process with very limited amount of initial training examples. Using three public available test collections, the proposed system is empirically compared with the state-of-the-art techniques. Our results demonstrate its significantly higher performance from different perspectives. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, a novel approach is developed to achieve automatic image collection summarization. The effectiveness of the summary is reflected by its ability to reconstruct the original set or each individual image in the set. We have leveraged the dictionary learning for sparse representation model to construct the summary and to represent the image. Specifically we reformulate the summarization problem into a dictionary learning problem by selecting bases which can be sparsely combined to represent the original image and achieve a minimum global reconstruction error, such as MSE (Mean Square Error). The resulting “Sparse Least Square” problem is NP-hard, thus a simulated annealing algorithm is adopted to learn such dictionary, or image summary, by minimizing the proposed optimization function. A quantitative measurement is defined for assessing the quality of the image summary by investigating both its reconstruction ability and its representativeness of the original image set in large size. We have also compared the performance of our image summarization approach with that of six other baseline summarization tools on multiple image sets (ImageNet, NUS-WIDE-SCENE and Event image set). Our experimental results have shown that the proposed dictionary learning approach can obtain more accurate results as compared with other six baseline summarization algorithms. [Copyright &y& Elsevier]

Over the past decade, there has been explosive growth in the availability of multimedia data, particularly image, video, and music. Because of this, content-based music retrieval has attracted attention from the multimedia database and information retrieval communities. Content-based music retrieval requires us to be able to automatically identify particular characteristics of music data. One such characteristic, useful in a range of applications, is the identification of the singer in a musical piece. Unfortunately, existing approaches to this problem suffer from either low accuracy or poor scalability. In this article, we propose a novel scheme, called Hybrid Singer Identifier (HSI), for efficient automated singer recognition. HSI uses multiple low-level features extracted from both vocal and nonvocal music segments to enhance the identification process; it achieves this via a hybrid architecture that builds profiles of individual singer characteristics based on statistical mixture models. An extensive experimental study on a large music database demonstrates the superiority of our method over state-of-the-art approaches in terms of effectiveness, efficiency, scalability, and robustness. [ABSTRACT FROM AUTHOR]

Abstract: As one of the most important cultural heritages, classical western paintings have always played a special role in human live and been applied for many different purposes. While image classification is the subject of a plethora of related publications, relatively little attention has been paid to automatic categorization of western classical paintings which could be a key technique of modern digital library, museums and art galleries. This paper studies automatic classification on large western painting image collection. We propose a novel framework to support automatic classification on large western painting image collections. With this framework, multiple visual features can be integrated effectively to improve the accuracy of identification process significantly. We also evaluate our method and its competitors based on a large image collection. A careful study on the empirical results indicates the approach enjoys great superiority over the state-of-the-art approaches in different aspects. [Copyright &y& Elsevier]

This article is a comprehensive evaluation of a new framework for indexing image data, called CMVF, which can combine multiple data properties with a hybrid architecture. The goal of this system is to allow straightforward incorporation of multiple visual feature vectors, based on properties such as color, texture and shape, into a single low-dimension vector that is more effective for retrieval than the larger individual feature vectors. Moreover, CMVF is not only constrained to visual properties, but can also incorporate human classification criteria to further strengthen image retrieval process. The controlled study present in this paper concentrates on CMVF's performance on images, examining how the incorporation of extra features into the indexing affects both efficiency and effectiveness. Analysis and empirical evidence suggest that the inclusion of extra visual features can significantly improve system performance. Furthermore, it demonstrated that CMVF's effectiveness is robust against various kinds of common image distortions and initial (random) configuration of neural network. [ABSTRACT FROM AUTHOR]

Automatically identifying characters in movies has attracted researchers’ interest and led to several significant and interesting applications. However, due to the vast variation in character appearance as well as the weakness and ambiguity of available annotation, it is still a challenging problem. In this paper, we investigate this problem with the supervision of actor-character name correspondence provided by the movie cast. Our proposed framework, namely, Cast2Face, is featured by: 1) we restrict the assigned names within the set of character names in the cast; 2) for each character, by using the corresponding actor and movie name as keywords, we retrieve from the Google image search and get a group of face images to form the gallery set; 3) the probe face tracks in the movie are then identified as one of the actors by a robust kernel multitask joint sparse representation and classification method; and 4) the conditional random field model with consideration of the constraints between face tracks is introduced to enhance the final labeling. Finally, the assigned actor name of a face track is then mapped to the character name based on the cast again. Besides face naming, we further apply the proposed method to spotlight the summarization of a particular actor in his/her movies. We conduct extensive experiments and empirical evaluations on several feature-length movies to demonstrate the satisfying performance of our method. [ABSTRACT FROM AUTHOR]

Deep Metric Learning (DML) is a crucial machine learning method in computer vision, and constructing an effective loss function. Average Precision (AP) is a well-known evaluation metric for image retrieval tasks. However, AP's non-differentiable and non-decomposable nature constrains its potential for adoption as an optimization goal. We propose a deep metric learning method with learned parametric metric learning loss and a contextual memory block (LM-Metric) for large-scale image retrieval tasks, which overcome AP's drawbacks and integrate AP within DML loss. We first introduce a parametric pairwise weighting scheme via policy gradient optimization and model the batch-wise inter-sample relationship via a Gated Recurrent Unit (GRU). Furthermore, a conditional Normalizing Flow-based contextual memory feature block to learn a compact single embedding for each image containing the contextual information during retrieval. We perform experiments on retrieval benchmark datasets and improve performance over the state-of-the-arts. • We propose a policy gradient-based meta-learned parametric metric learning scheme (LM-Metric). • We consider the inter-sample relationship during training to solve the non-decomposability problem of AP optimization. • We validate the effectiveness of our method, by applying it to variouis (DML) loss, e.g., Triplet, Margin, and Proxy-anchor losses. [ABSTRACT FROM AUTHOR]

Person re-identification (re-ID) is of great importance to video surveillance systems by estimating the similarity between a pair of cross-camera person shorts. Current methods for estimating such similarity require a large number of labeled samples for supervised training. In this paper, we present a pseudo-pair based self-similarity learning approach for unsupervised person re-ID without human annotations. Unlike conventional unsupervised re-ID methods that use pseudo labels based on global clustering, we construct patch surrogate classes as initial supervision, and propose to assign pseudo labels to images through the pairwise gradient-guided similarity separation. This can cluster images in pseudo pairs, and the pseudos can be updated during training. Based on pseudo pairs, we propose to improve the generalization of similarity function via a novel self-similarity learning:it learns local discriminative features from individual images via intra-similarity, and discovers the patch correspondence across images via inter-similarity. The intra-similarity learning is based on channel attention to detect diverse local features from an image. The inter-similarity learning employs a deformable convolution with a non-local block to align patches for cross-image similarity. Experimental results on several re-ID benchmark datasets demonstrate the superiority of the proposed method over the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Recent techniques based on sparse representation (SR) have demonstrated promising performance in high-level visual recognition, exemplified by the highly accurate face recognition under occlusion and other sparse corruptions. Most research in this area has focused on classification algorithms using raw image pixels, and very few have been proposed to utilize the quantized visual features, such as the popular bag-of-words feature abstraction. In such cases, besides the inherent quantization errors, ambiguity associated with visual word assignment and misdetection of feature points, due to factors such as visual occlusions and noises, constitutes the major cause of dense corruptions of the quantized representation. The dense corruptions can jeopardize the decision process by distorting the patterns of the sparse reconstruction coefficients. In this paper, we aim to eliminate the corruptions and achieve robust image analysis with SR. Toward this goal, we introduce two transfer processes (ambiguity transfer and mis-detection transfer) to account for the two major sources of corruption as discussed. By reasonably assuming the rarity of the two kinds of distortion processes, we augment the original SR-based reconstruction objective with \mmb\ell{\bf 0}-norm regularization on the transfer terms to encourage sparsity and, hence, discourage dense distortion/transfer. Computationally, we relax the nonconvex \mmb\ell{\bf 0}-norm optimization into a convex \mmb\ell{\bf 1}-norm optimization problem, and employ the accelerated proximal gradient method to optimize the convergence provable updating procedure. Extensive experiments on four benchmark datasets, Caltech-101, Caltech-256, Corel-5k, and CMU pose, illumination, and expression, manifest the necessity of removing the quantization corruptions and the various advantages of the proposed framework. [ABSTRACT FROM AUTHOR]

Motion information has been widely exploited for group activity recognition in sports video. However, in order to model and extract the various motion information between the adjacent frames, existing algorithms only use the coarse video‐level labels as supervision cues. This may lead to the ambiguity of extracted features and the omission of changing rules of motion patterns that are also important sports video recognition. In this paper, a latent label mining strategy for group activity recognition in basketball videos is proposed. The authors' novel strategy allows them to obtain the latent labels set for marking different frames in an unsupervised way, and build the frame‐level and video‐level representations with two separate levels of supervision signal. Firstly, the latent labels of motion patterns are digged using the unsupervised hierarchical clustering technique. The generated latent labels are then taken as the frame‐level supervision signal to train a deep CNN for the frame‐level features extraction. Lastly, the frame‐level features are fed into an LSTM network to build the spatio‐temporal representation for group activity recognition. Experimental results on the public NCAA dataset demonstrate that the proposed algorithm achieves state‐of‐the‐art performance. [ABSTRACT FROM AUTHOR]

Effective person re-identification (Re-ID) is often required in real applications. While most exiting approaches either assume the detected pedestrian bounding box well-aligned or utilize limited human structural information (pose, attention, segmentation) to calibrate the misalignment. However, the value of utilizing attributes for pedestrian alignment is still under explored. Furthermore, the hierarchy of attributes in previous works has been largely ignored, appearance feature and attribute feature are often fused in a rigid way. This directly limits the discriminatory and robustness of feature representation. In this paper, we propose a Refined Attribute-aligned Network (RAN), which consists of a coarse-alignment and a fine-alignment module. First, the pre-trained part and attribute predictor are used to generate body parts and candidate attributes. Then the body parts are used for coarse alignment and the attributes are selected by an agent. The agent is optimized with policy gradient algorithm, which can maximize the accumulative reward to increase the probability of proper attribute selection. Finally, for the fine-alignment, the attribute maps and body part features are aggregated by a bilinear-pooling layer to support accurate Re-ID. Extensive experimental results based on multiple datasets including CUHK03, DukeMTMC and Market-1501 demonstrate the superiority of our method over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• A novel fashion product recommendation method using multimodal information. • Integrating both text and image mining techniques. • Facilitating similar and mix-and-match fashion product recommendations. • Leveraging text-based product attributes and image features in the recommendations. Fashion recommendation has attracted much attention given its ready applications to e-commerce. Traditional methods usually recommend clothing products to users on the basis of their textual descriptions. Product images, although covering a large resource of information, are often ignored in the recommendation processes. In this study, we propose a novel fashion product recommendation method based on both text and image mining techniques. Our model facilitates two kinds of fashion recommendation, namely, similar product and mix-and-match, by leveraging text-based product attributes and image features. To suggest similar products, we construct a new similarity measure to compare the image colour and texture descriptors. For mix-and-match recommendation, we firstly adopt convolutional neural network (CNN) to classify fine-grained clothing categories and fine-grained clothing attributes from product images. Algorithm is developed to make mix-and-match recommendations by integrating the image extracted categories and attributes information are with text-based product attributes. Our comprehensive experimental work on a real-life online dataset has demonstrated the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

The vocabulary gap between health seekers and providers has hindered the cross-system operability and the inter-user reusability. To bridge this gap, this paper presents a novel scheme to code the medical records by jointly utilizing local mining and global learning approaches, which are tightly linked and mutually reinforced. Local mining attempts to code the individual medical record by independently extracting the medical concepts from the medical record itself and then mapping them to authenticated terminologies. A corpus-aware terminology vocabulary is naturally constructed as a byproduct, which is used as the terminology space for global learning. Local mining approach, however, may suffer from information loss and lower precision, which are caused by the absence of key medical concepts and the presence of irrelevant medical concepts. Global learning, on the other hand, works towards enhancing the local medical coding via collaboratively discovering missing key terminologies and keeping off the irrelevant terminologies by analyzing the social neighbors. Comprehensive experiments well validate the proposed scheme and each of its component. Practically, this unsupervised scheme holds potential to large-scale data. [ABSTRACT FROM AUTHOR]

Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Chinese poetry has been a favorite literary genre for thousands of years. Chinese ancient poetry is still being read and practiced, and many famous ancient Chinese poets are honored and adorned. Recently, deep learning has been widely adopted for poetry generation. In this paper, we present a new context-aware Chinese poetry generation method based on sequence-to-sequence framework. We generate a new concept called keyword team, which is a combination of all the keywords to capture the context of the Chinese poetry. Then we use the keyword, the keyword team and the previously generated lines to generate the present line in the poetry. We find that, by including keyword teams into the generation of the poetry, it can additionally perceive the keywords of preceding and succeeding lines to generate the present line, which can effectively improve the adhesion among the overall lines. The comprehensive evaluation results show that our proposed model outperforms many of the state-of-the-art poetry generation models. [ABSTRACT FROM AUTHOR]

• We deeply analyze the characteristics of global motions in action recognition scenario and develop a novel independent uni-variate model for global motion representation. It is a simplified version of perspective model. The number of its parameters is same as that of affine model. It makes a good trade-off between robustness and complexity which is more suitable for action recognition applications. • We propose an iterative optimization scheme for global motion estimation. The outlier points with local and global motion will be gradually discarded by an adaptive threshold during each iteration and the estimation process is implemented in a coarse-to-fine manner. Moreover, the local motion field is estimated based on mixed and estimated global motion fields through a spatio-temporal threshold based scheme. • We evaluate our proposed global and local motion estimation scheme on action recognition tasks. The separated local motion is adopted as input instead of original mixed motion field. Extensive experiments on multiple deep neural networks and action recognition datasets demonstrate the effectiveness of our proposed method. Motion information used in the existed video action recognition schemes is mixing of global motion(GM) and local motion(LM). In fact, GM & LM have their respective semantic concepts. Thus, it is promising to decouple GM and LM from the mixed motions. Numerous efforts have been made on the design of global motion models for video encoding, video dejittering, video denoising, and so on. Nevertheless, some of the models are too basic to cover the camera motions in action recognition while others are over-complicated. In this paper, we focus on the characteristic of the action recognition and propose a novel independent univariate GM model. It ignores camera rotation, which appears rarely in action recognition videos, and represents the GM in x and y direction respectively. Furthermore, GM is position invariant because it is from the universal camera motion. Pixels with global motions are subjected to the same parametric model and pixels with mixed motion can be seen as outliers. Motivated by this, we develop an iterative optimization scheme for GM estimation which removes the outlier points step by step and estimates global motions in a coarse-to-fine manner. Finally, the LM is estimated through a Spatio-temporal threshold-based method. Experimental results demonstrate that the proposed GM model makes a better trade-off between the model complexity and the robustness. And the iterative optimization scheme is more effective than the existed algorithms. The compared experiments using four popular action recognition models on UCF-101 (for action recognition) and NCAA (for group activity recognition) demonstrate that local motions are more effective than the mixed motions. [ABSTRACT FROM AUTHOR]