Search

Inferring the complete 3D shape of an object from an RGB image has shown impressive results, however, existing methods rely primarily on recognizing the most similar 3D model from the training set to solve the problem. These methods suffer from poor generalization and may lead to low-quality reconstructions for unseen objects. Nowadays, stereo cameras are pervasive in emerging devices such as dual-lens smartphones and robots, which enables the use of the two-view nature of stereo images to explore the 3D structure and thus improve the reconstruction performance. In this paper, we propose a new deep learning framework for reconstructing the 3D shape of an object from a pair of stereo images, which reasons about the 3D structure of the object by taking bidirectional disparities and feature correspondences between the two views into account. Besides, we present a large-scale synthetic benchmarking dataset, namely StereoShapeNet , containing 1,052,976 pairs of stereo images rendered from ShapeNet along with the corresponding bidirectional depth and disparity maps. Experimental results on the StereoShapeNet benchmark demonstrate that the proposed framework outperforms the state-of-the-art methods. The project page is available at https://haozhexie.com/project/stereo-3d-reconstruction. [ABSTRACT FROM AUTHOR]

Sketch recognition remains a significant challenge due to the limited training data and the substantial intra-class variance of freehand sketches for the same object. Conventional methods for this task often rely on the availability of the temporal order of sketch strokes, additional cues acquired from different modalities and supervised augmentation of sketch datasets with real images, which also limit the applicability and feasibility of these methods in real scenarios. In this paper, we propose a novel sketch-specific data augmentation (SSDA) method that leverages the quantity and quality of the sketches automatically. From the aspect of quantity, we introduce a Bezier pivot based deformation (BPD) strategy to enrich the training data. Towards quality improvement, we present a mean stroke reconstruction (MSR) approach to generate a set of novel types of sketches with smaller intra-class variances. Both of these solutions are unrestricted from any multi-source data and temporal cues of sketches. Furthermore, we show that some recent deep convolutional neural network models that are trained on generic classes of real images can be better choices than most of the elaborate architectures that are designed explicitly for sketch recognition. As SSDA can be integrated with any convolutional neural networks, it has a distinct advantage over the existing methods. Our extensive experimental evaluations demonstrate that the proposed method achieves the state-of-the-art results (84.27%) on the TU-Berlin dataset, outperforming the human performance by a remarkable 11.17% increase. Finally, more experiments show the practical value of our approach for the task of sketch-based image retrieval. [ABSTRACT FROM AUTHOR]

Recognition of general actions has witnessed great success in recent years. However, the existing general action representations cannot work well to recognize fine-grained actions, which usually share high similarities in both appearance and motion pattern. To solve this problem, we introduce the visual attention mechanism into the proposed descriptor, termed Actionness-pooled Deep-convolutional Descriptor (ADD). Instead of pooling features uniformly from the entire video, we aggregate features in sub-regions that are more likely to contain actions according to actionness maps. This endows ADD with the superior capability of capturing the subtle differences between fine-grained actions. We conduct experiments on HIT Dances dataset, one of the few existing datasets for fine-grained action analysis. Quantitative results have demonstrated that ADD remarkably outperforms traditional CNN-based representations. Extensive experiments on two general action benchmarks, JHMDB and UCF101, have additionally proved that combining ADD with end-to-end ConvNet can further boost the recognition performance. Besides, taking advantage of ADD, we reveal the sparsity characteristic existing in actions and point out a potential direction to design more effective action analysis models by extracting both representative and discriminative action patterns. [ABSTRACT FROM AUTHOR]

In recent years, deep hashing methods have been proved to be effective since it employs convolutional neural network to learn features and hashing codes simultaneously. However, these methods are mostly supervised. In real-world applications, it is a time-consuming and overloaded task for annotating a large number of images. In this paper, we propose a novel unsupervised deep hashing method for large-scale image retrieval. Our method, namely unsupervised semantic deep hashing (USDH), uses semantic information preserved in the CNN feature layer to guide the training of network. We enforce four criteria on hashing codes learning based on VGG-19 model: 1) preserving relevant information of feature space in hashing space; 2) minimizing quantization loss between binary-like codes and hashing codes; 3) improving the usage of each bit in hashing codes by using maximum information entropy, and 4) invariant to image rotation. Extensive experiments on CIFAR-10, NUSWIDE have demonstrated that USDH outperforms several state-of-the-art unsupervised hashing methods for image retrieval. We also conduct experiments on Oxford 17 datasets for fine-grained classification to verify its efficiency for other computer vision tasks. [ABSTRACT FROM AUTHOR]

Temporal action proposal generation (TAPG) serves as a promising solution for video analysis. However, the performance of existing methods is still far from satisfactory for real-world applications. We attribute it to a crucial issue, i.e., hard multiple instances. In this paper, we investigate why this is the case. We discover that when processing multiple instances videos, mainstream approaches always recognize multiple instances as one instance due to boundary ambiguity or ignoring insignificant backgrounds between these instances. To address this problem, we propose a M ultiple I nstances F ocused Net work (MIFNet) that improves the quality of action proposals by considering boundary correlations and fusing multi-scale proposals. In particular, we first propose a pure boundary embedding module named Boundary Constraint Module (BCM) for suppressing the generation of hard negatives proposal by evaluating boundary correlation. The BCM introduces a boundary contrastive learning strategy that can pull the positive boundary pairs' representation closer and push the negative pairs' representation away. Then, a Proposal Blending Module (PBM) is proposed, which augments the proposal-level representation by modeling information among multi-scale proposals so that proposals can be complemented with local details as well as global information. The experimental results on the ActivityNet-v1.3 and THUMOS14 benchmarks demonstrate that MIFNet outperforms the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Identification of characters in TV series and movies is an important and challenging problem. Actor identification results are important information for many higher level multimedia analysis tasks, such as semantic indexing and retrieval, interaction analysis and video summarization. Compared with previous works on actor identification that mainly focus on static features based on face identification and costume detection, our work in this paper attempts to use abundant dynamic information contained in videos to improve the performance when the appearances of actors are hard to detect or changes greatly over time. We propose to mine representative actions of each actor, and show the remarkable power of such actions for actor identification. Videos are firstly divided into shots and represented by Bag of Words (BoW) using spatial-temporal features. Integrating the prototype theory with representativeness, we propose two novel methods to rank the shots and obtain the representative actions. Our methods for actor identification combines representative actions with actors’ appearance. We validate the performance of the proposed methods on episodes of the TV series “The Big Bang Theory”. The experimental results show that the representative actions are consistent with human judgements and can greatly improve the matching performance as complementary to existing handcrafted static features for actor identification. [ABSTRACT FROM AUTHOR]

General natural image deblurring methods do not work well for document images. We exploit a two-tone prior to steer the intermediate latent image towards a piece-wise constant image with only two distinct gray levels. This prior is helpful for the process of kernel estimation to overcome undesirable local minima, and it is not too restrictive to deblur text images with complex backgrounds. Our kernel estimation method comprises two stages, where we first employ contrast-enhancing two-tone prior and then use intermediate-value inhibition regularizer. The resulting optimization formulation is solved by half-quadratic splitting and alternating minimization techniques. The experimental results show that the proposed method is capable of achieving accurate results and compares well with the state-of-the-art. [ABSTRACT FROM AUTHOR]

Auto-encoder—a tricky three-layered neural network, known as auto-association before, constructs the “building block” of deep learning, which has been demonstrated to achieve good performance in various domains. In this paper, we try to investigate the dimensionality reduction ability of auto-encoder, and see if it has some kind of good property that might accumulate when being stacked and thus contribute to the success of deep learning. Based on the above idea, this paper starts from auto-encoder and focuses on its ability to reduce the dimensionality, trying to understand the difference between auto-encoder and state-of-the-art dimensionality reduction methods. Experiments are conducted both on the synthesized data for an intuitive understanding of the method, mainly on two and three-dimensional spaces for better visualization, and on some real datasets, including MNIST and Olivetti face datasets. The results show that auto-encoder can indeed learn something different from other methods. Besides, we preliminarily investigate the influence of the number of hidden layer nodes on the performance of auto-encoder and its possible relation with the intrinsic dimensionality of input data. [ABSTRACT FROM AUTHOR]

Along with the rapid development of digital information technology, video surveillance systems have been widely used in numerous public places, such as squares, shopping malls and banks, to monitor crowd in case of anomalous events. Meanwhile, great challenges have been posed to worldwide researchers because the analysis of the exponentially growing crowd activity data is an arduous task. In this paper, we develop a novel unsupervised crowd activity discovery algorithm aiming to automatically explore latent action patterns among crowd activities and partition them into meaningful clusters. Inspired by the computational model of human vision system, we present a spatio-temporal saliency-based representation to simulate visual attention mechanism and encode human-focused components in an activity stream. Combining with feature pooling, we can obtain a more compact and robust activity representation. Based on affinity matrix of activities, N-cut is performed to generate clusters with meaningful activity patterns. We carry out experiments on our HIT-BJUT dataset and the UMN dataset. The experimental results demonstrate that the proposed unsupervised discovery method is fast and capable of automatically mining meaningful activities from large-scale and unbalanced video data with mixed crowd activities. [ABSTRACT FROM AUTHOR]

Abstract: Most previous works on video classification and recommendation were only based on video contents, without considering the affective analysis of viewers. In this paper, we presented a novel method to classify and recommend videos based on affective analysis, mainly on facial expression recognition of viewers, by fusing the spatio–temporal features. For spatial features, we integrate Haar-like features into compositional ones according to the features’ correlation, and train a mid classifier. Then this process is embedded into the improved AdaBoost learning algorithm to obtain spatial features. And for temporal feature fusion, we adopt HDCRFs based on HCRFs by introducing a time dimension variable. The spatial features are embedded into HDCRFs to recognize facial expressions. Experiments on the Cohn–Kanada database show that the proposed method has a promising performance. Then viewers' changing facial expressions are collected frame by frame from the camera when they are watching videos. Finally, we draw affective curves which tell the process of affection changes. Through the curves, we segment each video into affective sections, classify videos into categories, and list recommendation scores. Experimental results on our collected database show that most subjects are satisfied with the classification and recommendation results. [Copyright &y& Elsevier]

Abstract: Handling appearance variations is a very challenging problem for visual tracking. Existing methods usually solve this problem by relying on an effective appearance model with two features: (1) being capable of discriminating the tracked target from its background, (2) being robust to the target''s appearance variations during tracking. Instead of integrating the two requirements into the appearance model, in this paper, we propose a tracking method that deals with these problems separately based on sparse representation in a particle filter framework. Each target candidate defined by a particle is linearly represented by the target and background templates with an additive representation error. Discriminating the target from its background is achieved by activating the target templates or the background templates in the linear system in a competitive manner. The target''s appearance variations are directly modeled as the representation error. An online algorithm is used to learn the basis functions that sparsely span the representation error. The linear system is solved via minimization. The candidate with the smallest reconstruction error using the target templates is selected as the tracking result. We test the proposed approach using four sequences with heavy occlusions, large pose variations, drastic illumination changes and low foreground-background contrast. The proposed approach shows excellent performance in comparison with two latest state-of-the-art trackers. [Copyright &y& Elsevier]

Deep convolutional neural networks have demonstrated breakthrough accuracies for image classification. A series of feature extractors learned from CNN have been used in other computer vision tasks. However, CNN features of different layers aim to encode different-level information. High-layer features care more about semantic information but less detail information, while low-layer features contain more detail information but suffer from the problem of background clutter and semantic ambiguity. We propose to exploit complementary strengths of different layers in a simple but effective way. A mapping function is designed to highlight the effectiveness of low-layer similarity, when measuring fine-grained similarity between query image and its nearest neighbors with similar semantic. Extensive experiments show that our method can achieve competitive performance on popular retrieval benchmarks. Extensive experiments show that the proposed method outperforms the features extracted from single layers and their direct concatenations. Meanwhile, our method achieves competitive performance on popular retrieval benchmarks. [ABSTRACT FROM AUTHOR]

3D depth data, especially dynamic 3D depth data, offer several advantages over traditional intensity videos for expressing objects׳ actions, such as being useful in low light levels, resolving the silhouette ambiguity of actions, and being color and texture invariant. With the wide popularity of somatosensory equipment (Kinect for example), more and more dynamic 3D depth data are shared on the Internet, which results in an urgent need to retrieve these data efficiently and effectively. In this paper, we propose a generalized strategy for dynamic 3D depth data matching and apply this strategy in action retrieval task. Firstly, an improved 3D shape context descriptor (3DSCD) is proposed to extract features of each static depth frame. Then we employ dynamic time warping (DTW) to measure the temporal similarity between two 3D dynamic depth sequences. Experimental results on our collected dataset consisting of 170 dynamic 3D depth video clips show that the proposed 3DSCD has a rich descriptive power on depth data and that the method using 3DSCD and DTW achieves high matching accuracy. Finally, to address the matching efficiency problem, we utilize the bag of word (BoW) model to quantize the 3DSCD of each static depth frame into visual word packages. So the original feature matching problem is simplified into a two-histogram matching problem. The results demonstrate the matching efficiency of our proposed method, while still maintaining high matching accuracy. [ABSTRACT FROM AUTHOR]

Source-free unsupervised domain adaptation (SFUDA) aims to enable the utilization of a pre-trained source model in an unlabeled target domain without access to source data. Self-training is a way to solve SFUDA, where confident target samples are iteratively selected as pseudo-labeled samples to guide target model learning. However, prior heuristic noisy pseudo-label filtering methods all involve introducing extra models, which are sensitive to model assumptions and may introduce additional errors or mislabeling. In this work, we propose a method called Uncertainty-aware Pseudo-label-filtering Adaptation (UPA) to efficiently address this issue in a coarse-to-fine manner. Specially, we first introduce a sample selection module named Adaptive Pseudo-label Selection (APS), which is responsible for filtering noisy pseudo labels. The APS utilizes a simple sample uncertainty estimation method by aggregating knowledge from neighboring samples and confident samples are selected as clean pseudo-labeled. Additionally, we incorporate Class-Aware Contrastive Learning (CACL) to mitigate the memorization of pseudo-label noise by learning robust pair-wise representation supervised by pseudo labels. Through extensive experiments conducted on three widely used benchmarks, we demonstrate that our proposed method achieves competitive performance on par with state-of-the-art SFUDA methods. [ABSTRACT FROM AUTHOR]

Multiple object tracking is a key problem for many computer vision applications such as video surveillance, advanced driver assistance or animation. Most of existing tracking-by-detection methods are mainly based on object appearances and motions. However, the contextual information around the target has not been fully exploited. In this paper, we pay more attention to the contextual information and propose an Exchanging Object Context (EOC) model, which takes full advantage of the context information. Specifically, we implement an efficient and accurate online multiple object tracking algorithm with a novel affinity measure to associate detections. This measure calculates the similarity between targets and detections with the background smoothness after exchanging the contexts between detections and targets, using a novel color histogram descriptor. We refine the bounding boxes by measuring the context changes. Extensive experimental results on two public benchmarks demonstrate the effectiveness of the proposed tracking method with comparisons to several state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

View-based 3D object retrieval techniques have become increasingly important in various fields, and lots of ingenious studies have promoted the development of retrieval performance from different aspects. In this paper, we focus on the 2D projective views that represent the 3D objects and propose a boosting approach by evaluating the discriminative ability of each object’s views. The dissimilarity between views’ semantic and discriminative ability is firstly investigated through classification performance. We then propose a simple, robust and effective measurement to study the view’s discriminative ability. By employing the proposed reverse distance metric, we utilize the discriminative information for many to many view set matching. The proposed algorithm is then employed with various features to boost the multi-model graph learning based retrieval method. We compare our approach with several state of the art methods on ETH-80 dataset and National Taiwan University 3D model dataset. The results demonstrate the effectiveness of our method and its excellent boosting performance. [ABSTRACT FROM AUTHOR]