Search

This paper presents a method to improve the traditional mixture of deformable part models (MDPM) method from the learning perspective. First, an object part configuration learning algorithm based on group sparsity constraint is introduced to automatically discover the object part number, size, and location. The algorithm imposes two additional regularization terms in addition to the standard hinge loss function. The first term focuses on automatic part selection and the second term focuses on automatic part placement. Second, this paper introduces an improved MDPM training framework. The framework applies a learned transformation to normalize the prediction score from each individual deformable part model (DPM) into a pseudo probability such that the partition of the entire object appearance feature space becomes less sensitive to the prior distributions of different DPMs. Finally, the two proposed improvements are combined and formulated under the expectation-maximization framework. We evaluate our method mainly using the PASCAL VOC2007 and VOC2010 detection benchmarks and show that the proposed learning algorithms could increase the detection mean AP score by 2.4% and 0.9%, respectively, on these two data sets when using the proposed part selection method and the training algorithm. We also present further in-depth analysis of the proposed algorithm in the experiments. [ABSTRACT FROM PUBLISHER]

Few-shot learning is an extremely challenging task in computer vision that has attracted increased research attention in recent years. However, most recent methods do not fully use the task’s information, and few of the seen samples result in large intraclass differences among the same classes. In this paper, we propose a novel task encoding with distribution calibration (TEDC) model for few-shot learning, which uses the relationships among the feature distributions to reduce intraclass differences. In the TEDC model, an integrated feature extraction module (IFEM) is proposed, which extracts the multiangle visual features of an image and fuses them to obtain more representative features. To effectively utilize the task information, a novel task encoding module (TEM) is proposed, which obtains the task features by fusing all the seen samples’ information and uses them to adjust all the samples’ features for more generalizable task-specific representations. We also propose a distribution calibration module (DCM) to reduce the bias between the distribution of the support features and the query features in the same class. Extensive experiments show that our proposed TEDC model achieves an excellent performance and outperforms the state-of-the-art methods on three widely used few-shot classification benchmarks, specifically miniImageNet, tieredImageNet and CUB-200-2011. [ABSTRACT FROM AUTHOR]

Accelerating the inference of CNNs is critical to their deployment in real-world applications. Among all pruning approaches, the methods of implementing a sparsity learning framework have shown effectiveness as they learn and prune the models in an end-to-end data-driven manner. However, these works impose the same sparsity regularization on all filters indiscriminately, which can hardly result in an optimal structure-sparse network. In this paper, we propose a Saliency-Adaptive Sparsity Learning (SASL) approach for further optimization. A novel and effective estimation of each filter, i.e., saliency, is designed, which is measured from two aspects: the importance for prediction performance and the consumed computational resources. During sparsity learning, the regularization strength is adjusted according to the saliency, so our optimized format can better preserve the prediction performance while zeroing out more computation-heavy filters. The calculation for saliency introduces minimum overhead to the training process, which means our SASL is very efficient. During the pruning phase, in order to optimize the proposed data-dependent criterion, a hard sample mining strategy is utilized, which shows higher effectiveness and efficiency. Extensive experiments demonstrate the superior performance of our method. Notably, on ILSVRC-2012 dataset, our approach can reduce 49.7% FLOPs of ResNet-50 with very negligible 0.39% top-1 and 0.05% top-5 accuracy degradation. [ABSTRACT FROM AUTHOR]

Recently, angular margin-based methods have become the mainstream approach for unconstrained face recognition with remarkable success. However, robust face recognition still remains a challenge, as the face is subject to variations in pose, age, expression, occlusion, and illumination, especially in unconstrained scenarios. Since the training dataset are always collected in unconstrained scenarios, it is inevitable that there’re significant number of hard examples in the training process. In this paper, we design a hard example selection function to effectively identify hard examples in the training procedure with the supervision of angular margin-based losses. Furthermore, a novel Intra-class Hard Example Mining (IHEM) loss function is proposed, which penalizes the cosine distance between the hard examples and their class centers to enhance the discriminative power of face representations. To ensure high performance for face recognition, we combine the supervision of angular margin-based loss and IHEM loss for model training. Specifically, during the training procedure, the angular margin-based loss guarantees the power of feature discrimination for face recognition, while the IHEM loss further encourages the intra-class compactness of hard example. Extensive results demonstrate the superiority of our approach. [ABSTRACT FROM AUTHOR]

Change/motion detection is a challenging problem in video analysis and surveillance system. Recently, the state-of-the-art methods using the sample-based background model have demonstrated astonishing results with this problem. However, they are ineffective in the dynamic scenes that contain complex motion patterns. In this paper, we introduce a novel data-driven approach that combines the sample-based background model with a feature extractor obtained by training a triplet network. We construct the network by three identical convolutional neural networks, each of which is called a motion feature network. Our network can automatically learn motion patterns from small image patches and transform input images of any size into feature embeddings for high-level representations. The sample-based background model of each pixel is then employed by using the color information and the extracted feature embeddings. We also propose an approach to generate triplet examples from CDNet 2014 for training our network model from scratch. The offline trained network can be used on the fly without re-training on any video sequence before each execution. Therefore, it is feasible for real-time surveillance systems. In this paper, we show that our method outperforms the other state-of-the-art methods on CDNet 2014 and other benchmarks (BMC and Wallflower). [ABSTRACT FROM AUTHOR]

In this paper, the problem of human pose estimation in a 2D still image is addressed. A framework called adaptive holons representation (ADORE) that takes advantage of local and global cues is proposed to improve the pose estimation accuracy. In particular, ADORE is made up of two components: 1) the holons part, independent losses pose nets (ILPNs) is designed to first infer joints location on the global level; and 2) the adaptive part, convolutional local detectors (CLDs) is proposed to subsequently detect the joints in the potential regions generated by ILPN. Pose estimation is formulated as a classification problem toward body joints in ILPN, which consists of two independent loss layers that, respectively, instruct the learning of $x$ and $y$ coordinates of a joint. Experimental results on two challenging benchmark tasks demonstrate that our proposed framework is more efficient than other deep models and has desirable performance. [ABSTRACT FROM AUTHOR]

In order to retain more feature information of local areas on a point cloud, local grouping and subsampling are the necessary data structuring steps in most hierarchical deep learning models. Due to the disorder nature of the points in a point cloud, the significant time cost may be consumed when grouping and subsampling the points, which consequently results in poor scalability. This paper proposes a fast data structuring method called PSNet (Point Structuring Net). PSNet transforms the spatial features of the points and matches them to the features of local areas in a point cloud. PSNet achieves grouping and sampling at the same time while the existing methods process sampling and grouping in two separate steps (such as using FPS plus kNN). PSNet performs feature transformation pointwise while the existing methods uses the spatial relationship among the points as the reference for grouping. Thanks to these features, PSNet has two important advantages: 1) the grouping and sampling results obtained by PSNet is stable and permutation invariant; and 2) PSNet can be easily parallelized. PSNet can replace the data structuring methods in the mainstream point cloud deep learning models in a plug-and-play manner. We have conducted extensive experiments. The results show that PSNet can improve the training and inference speed significantly while maintaining the model accuracy. [ABSTRACT FROM AUTHOR]

Meta-learning offers an effective solution to learn new concepts under scarce supervision through an episodic-training scheme: a series of target-like tasks sampled from base classes are sequentially fed into a meta-learner to extract cross-task knowledge, which can facilitate the quick acquisition of task-specific knowledge of the target task with few samples. Despite its noticeable improvements, the episodic-training strategy samples tasks randomly and uniformly, without considering their hardness and quality, which may not progressively improve the meta-leaner’s generalization. In this paper, we propose Progressive Meta-learning using tasks from easy to hard. First, based on a predefined curriculum, we develop a Curriculum-Based Meta-learning (CubMeta) method. CubMeta is in a stepwise manner, and in each step, we design a BrotherNet module to establish harder tasks and an effective learning scheme for obtaining an ensemble of stronger meta-learners. Then we move a step further to propose an end-to-end Self-Paced Meta-learning (SepMeta) method. The curriculum in SepMeta is effectively integrated as a regularization term into the objective so that the meta-learner can measure the hardness of tasks adaptively, according to what the model has already learned. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed methods. Our code is available at https://github.com/nobody-777. [ABSTRACT FROM AUTHOR]

Researchers have devoted great efforts to image dehazing with prior assumptions in the past decade. Recently developed example-based approaches typically lack elegant models for the hazy process and meanwhile demand synthetic hazy images by manual selection. The priors from observations, and those trained from synthetic images cannot always reflect true structural information of natural images in practice. In this paper, we present a learning model for haze removal by using two-layer Gaussian process regression (GPR). By using training examples, the two-layer GPR establishes a direct relationship from the input image to the depth-dependent transmission, and learns local image priors to further improve the estimation. We also provide a systematic scheme to automatically collect suitable training pairs, which works for both simulated examples and images of natural scenes. Both qualitative and quantitative comparisons on real-world and synthetic hazy images demonstrate the effectiveness of the proposed approach, especially for white or bright objects and heavy haze regions in which traditional methods may fail. [ABSTRACT FROM PUBLISHER]

The use of local features for image representation has become popular in recent years. Local features are often used in the bag-of-visual-words scheme. Although proven effective, this method still has two drawbacks. First, local regions from which local features are extracted are not discriminative enough for visual tasks. Hence, the combination of local features is necessary. Second, the semantic gap between visual features and human perception also hinders the performance. To address these two problems, in this paper, we propose a novel contextual exemplar classifier-based method for image representation and apply it for classification tasks. Each exemplar classifier is trained to separate one training image from the other images of different classes. We partition each image into a number of regions and use the responses of these exemplar classifiers as the image region’s representation. The contextual relationship is then modeled using mixture Dirichlet distributions. A bilayer model is used to predict image classes with L2 constraints. Experimental results on the Natural Scene, Caltech-101/256, Flower-17/102, and SUN-397 data sets show that the proposed method is able to outperform the state-of-the-art local feature-based methods for image classification. [ABSTRACT FROM PUBLISHER]

Temporal and spatial contexts, characterizing target appearance variations and target-background differences, respectively, are crucial for improving the online adaptive ability and instance-level discriminative ability of object tracking. However, most existing trackers focus on either the temporal context or the spatial context during tracking and have not exploited these contexts simultaneously and effectively. In this paper, we propose a Spatial-TEmporal Memory (STEM) network to exploit these contexts jointly for object tracking. Specifically, we develop a key-value structured memory model equipped with a key-value index-based memory reading mechanism to model the spatial and temporal contexts simultaneously. To update the memory with new target states and ensure the diversity of the memory, we introduce a similarity-aware memory update scheme. In addition, we construct an entropy-guided ensemble strategy to fuse the prediction models based on these two contexts, such that these two contexts can be exploited to estimate the target state jointly. Extensive experimental results on eight challenging datasets, including OTB2015, TC128, UAV123, VOT2018, LaSOT, TrackingNet, GOT-10k, and OxUvA, demonstrate that the proposed method performs favorably against state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

A key factor that makes action detection in videos different from general video classification is human-guided clues, especially motion signals. Since not all the pixels in a video are informative for action recognition, the irrelevant and redundant parts can lead to a lot of noise and be burdensome for both feature extraction and classifier training. This encourages the researchers to seek out the design of the attentive model that can dynamically focus computations on the key spatiotemporal volumes. In this paper, we propose a motion-centric attention model for action detection in videos which imitates the human perception of saccade and fixation procedures while detecting actions in a video. Specifically, we first present a strategy to generate motion-centric locations based on the density peak of motion signals, providing reliable candidates around which actions have high possibilities to occur. Then, we introduce an attention model that conducts the saccade and fixation procedures on these candidates to observe local spatiotemporal visual information, preserve internal comprehension, and produce the action proposals on temporal bounds. Afterward, a classifier with several variants is prepared to classify the action proposals and decide which one to fixate and generate the final predictions. We show how to efficiently train our model to produce fast and accurate action detection, by scanning only a small fraction of locations in a video. The extensive experiments on three challenging datasets show promising results with both accuracy and speed. [ABSTRACT FROM AUTHOR]

In this paper, we study the fine-grained categorization problem under the few-shot setting, i.e., each fine-grained class only contains a few labeled examples, termed Fine-Grained Few-Shot classification (FGFS). The core predicament in FGFS is the high intra-class variance yet low inter-class fluctuations in the dataset. In traditional fine-grained classification, the high intra-class variance can be somewhat relieved by conducting the supervised training on the abundant labeled samples. However, with few labeled examples, it is hard for the FGFS model to learn a robust class representation with the significantly higher intra-class variance. Moreover, the inter- and intra-class variance are closely related. The significant intra-class variance in FGFS often aggravates the low inter-class variance issue. To address the above challenges, we propose a Target-Oriented Alignment Network (TOAN) to tackle the FGFS problem from both intra- and inter-class perspective. To reduce the intra-class variance, we propose a target-oriented matching mechanism to reformulate the spatial features of each support image to match the query ones in the embedding space. To enhance the inter-class discrimination, we devise discriminative fine-grained features by integrating local compositional concept representations with the global second-order pooling. We conducted extensive experiments on four public datasets for fine-grained categorization, and the results show the proposed TOAN obtains the state-of-the-art. [ABSTRACT FROM AUTHOR]

In general, an image attribute is a human-nameable visual property that has a semantic connotation. Appropriate modeling of the intrinsic contextual correlations among attributes plays a fundamental role in attribute detection. In this paper, we consider image attribute detection from the perspective of regularized deep learning. In particular, we propose a regularized deep belief network (rDBN) to perform the image attribute detection task, which is composed of two parts: 1) a detection DBN (dDBN) that models the joint distribution of images and their corresponding attributes, which acts as an attribute detector and 2) a contextual restricted Boltzmann machine that explicitly models the correlations among attributes acting as a regularizer that restraints the output detection result given by the dDBN to meet the contextual prior of attributes. Furthermore, we propose an efficient fine-tuning scheme that can further optimize the performance of the dDBN by backpropagation. Experimental results show that the proposed rDBN obtains improvements over the state-of-the-art methods for attribute detection on the benchmark data sets. [ABSTRACT FROM PUBLISHER]

In this paper, we provide a variational Bayesian (VB) treatment of multistream fused hidden Markov models (MFHMMs), and apply it in the context of active learning-based visual workflow recognition (WR). Contrary to training methods yielding point estimates, such as maximum likelihood or maximum a posteriori training, the VB approach provides an estimate of the posterior distribution over the MFHMM parameters. As a result, our approach provides an elegant solution toward the amelioration of the overfitting issues of point estimate-based methods. Additionally, it provides a measure of confidence in the accuracy of the learned model, thus allowing for the easy and cost-effective utilization of active learning in the context of MFHMMs. Two alternative active learning algorithms are considered in this paper: query by committee, which selects unlabeled data that minimize the classification variance, and a maximum information gain method that aims to maximize the alteration in model variance by proper data labeling. We demonstrate the efficacy of the proposed treatment of MFHMMs by examining two challenging WR scenarios, and show that the application of active learning, which is facilitated by our VB approach, allows for a significant reduction of the MFHMM training costs. [ABSTRACT FROM AUTHOR]

A number of visual tracking methods achieve the state-of-the-art performance based on deep learning recently. However, most of these trackers utilize the deep neural network in regression task or classification task separately. In this paper, we propose an adversarial deep tracking framework. The framework is composed of a fully convolutional Siamese neural network (regression network) and a discriminative classification network. Then, we jointly optimize the regression network and the classification network by adversarial learning. In the uniform framework, the regression network and classification network can be trained end-to-end as a whole using large amounts of video training data sets. During the testing phase, the regression network generates a response map which reflects the location and the size of the target within each candidate search patch, and the classification network discriminates which response map is the best in terms of the corresponding template patch and candidate search patch. In addition, we propose an attention visualization algorithm for our tracker, and it reflects the area that attracts the attention of our tracker during tracking. The experimental results on three large-scale visual tracking benchmarks (OTB-100, TC-128, and VOT2016) demonstrate the effectiveness of the proposed tracking algorithm and show that our tracker performs comparably against the state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

Image representation based on sparse coding generalizes the bag of words model. Although it reduces the reconstruction error for local features to achieve the state-of-the-art image classification performance, the large computational cost hinders the application of sparse coding-based image features. In this paper, we propose approximating a sparse code using the output of a simple neural network. The resulting parameter learning model for the neural network automatically incorporates non-negative and shift-invariant constraints, leading to an efficient normalized non-negative sparse coding (N3SC) sparse encoder. Without the use of the traditional iterative process to solve the sparse coding objective, the sparse encoder directly “converts” each local feature into a sparse code. We also introduce a method for training the encoder based on the auto-encoder method. In addition, we formally propose the corresponding sparse coding scheme called N3SC, which enforces both the non-negative constraint and the shift-invariant constraint in addition to the traditional sparse coding criteria. As demonstrated by several experiments, the obtained N3SC encoder requires only 3%–10% of the processing time for image feature extraction compared with the standard sparse coding scheme. At the same time, the features extracted using the exact solutions of the N3SC coding scheme and the N3SC encoder offer superior image classification accuracy compared to the accuracy of many existing sparse coding-based representations. [ABSTRACT FROM AUTHOR]

In comparison to single-model based trackers, the model-ensembled tracking strategy has shown a substantial adaptivity in handling various tracking challenges. As the performance of the tracker has been improved by combining different model outputs linearly, the insufficient consideration of each ensemble member’s contribution still limits the tracking performance to be further enhanced. As the performance of the tracker has been improved by combining different model outputs linearly, the insufficient consideration of each ensemble member’s contribution still limits the tracking performance to be further enhanced. In this paper, a tracking strategy based on multiple instance models regression (MIMRT) is proposed with a unified ensembling scheme. By formulating the tracking initialization with an instance model, the encoding process for an object’s specific detail is performed corresponding to the samples in each frame. The advantage of this operation is to guarantee the model frame-wise discrimination of short-term training, as well as to evaluate the reliability of each instance model by utilizing the long-lifetime samples obtained throughout the whole tracking procedure. To finalize the proposed tracking, all the independent instance models attached to the learned regression coefficients are ensembled with respect to the long-lifetime samples. This also effectively bridges the instance model as a latent variable to investigate a semantic association between the tracking model and the overall samples. A comprehensive experiment has shown that the proposed tracker is able to achieve superior performances compared to the state-of-art tracking approaches on both short-term datasets (e.g., OTB2013, OTB100, VOT2016, UAV123) and long-term dataset (UAV20L). [ABSTRACT FROM AUTHOR]

As one of the indispensable components in convolutional neural network (CNN), loss function assists in updating parameters of CNN models during the training phase. Generally, different loss functions can assist convolutional neural network (CNN) to learn different feature representations, and different feature representations can be treated as different knowledge learned from objects. In this paper we introduce a novel training framework, namely Loss Transferring (LT), to improve the generalization ability of CNN. LT contains multiple training phases, and each training phase uses a different loss function. Under this framework, CNN models can combine different knowledge of objects by transferring the knowledge learned via one loss function to another. LT contains two components, i.e., loss function set and training strategy. In order to build appropriate loss function set, we establish two basic guides. And according to these basic guides, we design a new loss function in the last layer of CNN models (layer before softmax operation), namely Near Classifier Hyper-Plane (N-CHP) loss, which makes the learned object features belonging to the same category have the minimum intra-class distance and be near the classifier hyper-plane. Based on the two loss function set $\{\text {MSE, softmax}\}$ and $\{\text {N-CHP, softmax}\}$ , we setup two specific training methods, $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which can be universally applied to different CNN models with low additional computation cost. Meanwhile, two training strategies, multi-phase strategy 1 and multi-phase strategy 2, are further proposed to improve the training efficiency of LT. Extensive experimental results on shallow, moderate and deep models with four benchmark datasets, including MNIST, SVHN, CIFAR-10 and CIFAR-100, demonstrate that CNN models can bring obvious performance improvements when working with $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which verifies the effectiveness of LT and the proposed two basic guides. [ABSTRACT FROM AUTHOR]

Although deep features have achieved the state-of-the-art performance in action recognition recently, the hand-crafted shallow features still play a critical role in characterizing human actions for taking advantage of visual contents in an intuitive way such as edge features. Therefore, the shallow features can serve as auxiliary visual cues supplementary to deep representations. In this paper, we propose a discriminative subspace learning model (DSLM) to explore the complementary properties between the hand-crafted shallow feature representations and the deep features. As for the RGB action recognition, this is the first work attempting to mine multi-level feature complementaries by the multi-view subspace learning scheme. To sufficiently capture the complementary information among heterogeneous features, we construct the DSLM by integrating the multi-view reconstruction error and classification error into an unified objective function. To be specific, we first use Fisher Vector to encode improved dense trajectories (iDT+FV) for shallow representations and two-stream convolutional neural network models (T-CNN) for generating deep features. Moreover, the presented DSLM algorithm projects multi-level features onto a shared discriminative subspace with the complementary information and discriminating capacity simultaneously incorporated. Finally, the action types of test samples are identified by the margins from the learned compact representations to the decision boundary. The experimental results on three datasets demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

This paper proposes a novel approach for crowd counting in low to high density scenarios in static images. Current approaches cannot handle huge crowd diversity well and thus perform poorly in extreme cases, where the crowd density in different regions of an image is either too low or too high, leading to crowd underestimation or overestimation. The proposed solution is based on the observation that detecting and handling such extreme cases in a specialized way leads to better crowd estimation. Additionally, existing methods find it hard to differentiate between the actual crowd and the cluttered background regions, resulting in further count overestimation. To address these issues, we propose a simple yet effective modular approach, where an input image is first subdivided into fixed-size patches and then fed to a four-way classification module labeling each image patch as low, medium, high-dense or no-crowd. This module also provides a count for each label, which is then analyzed via a specifically devised novel decision module to decide whether the image belongs to any of the two extreme cases (very low or very high density) or a normal case. Images, specified as high- or low-density extreme or a normal case, pass through dedicated zooming or normal patch-making blocks respectively before routing to the regressor in the form of fixed-size patches for crowd estimate. Extensive experimental evaluations demonstrate that the proposed approach outperforms the state-of-the-art methods on four benchmarks under most of the evaluation criteria. [ABSTRACT FROM AUTHOR]

The informativeness and discriminativeness of features collaboratively ensure high-accuracy Facial Expression Recognition (FER) in the wild. Most of existing methods use the single-path deep convolutional neural network with softmax loss for basic FER, while they cannot deal with the challenging situations of the compound FER in the wild, because they fail to learn informative and discriminative features in a targeted manner. To this end, we present an Informative and Discriminative Feature Learning (IDFL) framework that consists of two key components: the Multi-Path Attention Convolutional Neural Network (MPACNN) and Balanced Separate loss (BS loss), for both basic and compound high-accuracy FER in the wild. Specifically, MPACNN leverages different paths to learn diverse features. These features are then adaptively fused into informative ones via an attention module, such that the model can adequately capture detailed information for both basic and compound FER. The BS loss maximizes the inter-class distance of features and minimizes the intra-class one. In this way, the features are discriminative enough for high-accuracy FER in the wild. Particularly, the BS loss is invoked as the objective function of MPACNN, so the model can learn informative and discriminative features at the same time, yielding better performance. Seven databases are utilized to evaluate the proposed method, and the results demonstrate that our method achieves state-of-the-art performance on both basic and compound expressions with good generalization ability. Moreover, our model contains fewer parameters and can be trained faster than other related models. [ABSTRACT FROM AUTHOR]

Pose estimation is a task with wide application prospects in computer vision, which remains a challenging problem. In this paper, a novel pose estimation algorithm is proposed on the basis of pose clustering and body-part candidates recombination. Different from most previous methods with a single pictorial structure (PS) model, we generate mixture PS models based on clusters of the poses to achieve more faithful appearances and spatial relations estimation within each cluster. In addition, to address the problems of individual body-part false detection and double-counting, we extract some of the best estimation results in the optimal clustered model as the candidates of body parts and recombine them by solving a constrained maximization problem. Experiments on a public challenging data set show that our method is more accurate than the state-of-the-art algorithms and performs effectively in tackling the double-counting phenomena. [ABSTRACT FROM PUBLISHER]

Recently, most background modeling approaches represent distributions of background changes by using parametric models such as Gaussian mixture models. Because of significant illumination changes and dynamic moving backgrounds with time, variations of background changes are hard to be modeled by parametric background models. Moreover, how to efficiently and effectively update parameters of parametric models to reflect background changes remains a problem. In this paper, we propose a novel coarse-to-fine detection theory algorithm to extract foreground objects on the basis of nonparametric background and foreground models represented by binary descriptors. We update background and foreground models by a first-in–first-out strategy to maintain the most recent observed background and foreground instances. As shown in the experiments, our method can achieve better foreground extraction results and fewer false alarms of surveillance videos with lighting changes and dynamic backgrounds in both collected and CDnet 2012 benchmark data sets. [ABSTRACT FROM AUTHOR]

Limited annotated data available for the recognition of facial expression and particularly action units makes it hard to train a deep network which can learn disentangled invariant features. However, a supervised linear model is undemanding in terms of training data. In this paper, we propose an elegant linear model to untangle facial actions from expressive face videos which contain a mixture of linearly-representable attributes. Previous attempts require an explicit decoupling of identity and expression which is practically inexact. Instead, we exploit the low-rank property across frames to implicitly subtract the intrinsic neutral face, which are modeled jointly with sparse representation only on the residual expression components. On CK+, our one-shot C-HiSLR on raw-face pixel-intensities performs far more competitive than conventional shape+SVM models with landmark detection and two-stepped SRC of the same type yet applied on manually prepared expression components. It is also comparable with the piecewise linear model DCS and temporal models, such as CRF and Bayes nets. We apply it to action unit (AU) recognition on MPI-VDB achieving a decent performance. As expression is a mixture of AUs, the result gives hopes of approximating an expression using a piecewise linear model. [ABSTRACT FROM AUTHOR]

This paper presents a robust joint discriminative appearance model-based tracking method using online random forests and mid-level feature (superpixels). To achieve superpixel-wise discriminative ability, we propose a joint appearance model that consists of two random forest-based models, i.e., the background-target discriminative model (BTM) and the distractor-target discriminative model (DTM). More specifically, the BTM effectively learns discriminative information between the target object and the background. In contrast, the DTM is used to suppress distracting superpixels, which significantly improves the tracker’s robustness and alleviates the drifting problem. A novel online random forest regression algorithm is proposed to build the two models. The BTM and DTM are linearly combined into a joint model to compute a confidence map. Tracking results are estimated using the confidence map, in which the position and scale of the target are estimated orderly. Furthermore, we design a model updating strategy to adapt the appearance changes over time by discarding degraded trees of the BTM and DTM and initializing new trees as replacements. We test the proposed tracking method on two large tracking benchmarks, the CVPR2013 tracking benchmark and VOT2014 tracking challenge. Experimental results show that the tracker runs at real-time speed and achieves favorable tracking performance compared with the state-of-the-art methods. The results also suggest that the DTM improves tracking performance significantly and plays an important role in robust tracking. [ABSTRACT FROM AUTHOR]

Human action recognition and annotation is an active research topic in computer vision. How to model various actions, varying with time resolution, visual appearance, and others, is a challenging task. In this paper, we propose a boosted exemplar learning (BEL) approach to model various actions in a weakly supervised manner, i.e., only action bag-level labels are provided but action instance level ones are not. The proposed BEL method can be summarized as three steps. First, for each action category, amount of class-specific candidate exemplars are learned through an optimization formulation considering their discrimination and co-occurrence. Second, each action bag is described as a set of similarities between its instances and candidate exemplars. Instead of simply using a heuristic distance measure, the similarities are decided by the exemplar-based classifiers through the multiple instance learning, in which a positive (or negative) video or image set is deemed as a positive (or negative) action bag and those frames similar to the given exemplar in Euclidean Space as action instances. Third, we formulate the selection of the most discriminative exemplars into a boosted feature selection framework and simultaneously obtain an action bag-based detector. Experimental results on two publicly available datasets: the KTH dataset and Weizmann dataset, demonstrate the validity and effectiveness of the proposed approach for action recognition. We also apply BEL to learn representations of actions by using images collected from the Web and use this knowledge to automatically annotate action in YouTube videos. Results are very impressive, which proves that the proposed algorithm is also practical in unconstraint environments. [ABSTRACT FROM AUTHOR]

Although face recognition has made dramatic improvements in recent years, there are still many challenges in real-world applications such as face recognition for the elderly and children, for the surveillance scenes and for Near infrared vs. Visible light (NIR-VIS) heterogeneous scene, etc. Due to the existence of these challenges, there are usually domain gaps between training (source domain) and test (target domain). A common way to improve the performance on the target domain is fine-tuning the base model trained on source domain using target data. However, it will severely degrade performance on the source domain. Another way which jointly trains models using both source and target data, suffers from the heavy computations and large data storage, especially when we continue to encounter new domains. In response to these problems, we introduce a new challenging task: Single Exemplar Domain Incremental Learning (SE-DIL), which utilizes the target domain data and just one exemplar per identity from source domain data to quickly improve the performance on the target domain while keeping the performance on the source domain. To deal with SE-DIL, we propose our Fast Adapting without Forgetting (FAwF) method with three components: margin-based exemplar selection, prototype-based class extension and hard&soft knowledge distillation. Through FAwF, we can well maintain the source domain performance with only one sample per source domain class, greatly reducing the fine-tuning time-cost and data storage. Besides, we collected a large-scale children face dataset KidsFace with $12 {K}$ identities for studying the SE-DIL in face recognition. Extensive analysis and experiments on our KidsFace-Test protocol and other challenging face test sets show that our method performs better than the state-of-the-art methods on both target and source domain. [ABSTRACT FROM AUTHOR]

Visual saliency provides important information for understanding scenes in many computer vision tasks. The existing video saliency algorithms mainly focus on predicting spatial and temporal saliency maps. However, these maps are simply fused without considering the complex dynamic scenes in videos. To overcome this drawback, we propose a deep convolutional fusion framework for video saliency prediction. The proposed model, which is based on coupled fully convolutional networks (FCNs), effectively encodes the spatiotemporal information by integrating spatial and temporal features. We demonstrate that this information is helpful for accurately fusing the spatial and temporal saliency maps according to changes in video scenes. In particular, we gradually design three different deep fusion architectures to investigate how to better utilize the spatiotemporal information. Moreover, we propose a reasonable sampling strategy for selecting suitable training sets for the coupled FCNs. Through extensive experiments, we demonstrate that our model outperforms the state-of-the-art algorithms on four public video saliency data sets. [ABSTRACT FROM AUTHOR]

Multiperson activity recognition in videos is a challenging task, due to the complexity of interactions among multiple persons. In this paper, a new statistical model, named coupled observation decomposed hidden Markov model (CODHMM), is presented to model multiperson activities in videos. A human activity that involves multiple persons is analyzed in two levels: the individual level that describes each individual's motion details and the interaction level that expresses the shared information among multiple persons. The two levels are modeled by two hidden Markov chains that are interdependent and interact with each other. The observation in each chain at each time slice is decomposed into subobservations according to the number of features and the number of persons. For each activity to be recognized, a CODHMM is built and model parameters are learnt by a generalized expectation maximization (EM) algorithm. Given an input video that contains an unknown activity, maximum likelihood algorithms are developed to classify it into one of the learnt activity categories. Experimental results show that the CODHMM can successfully classify human activities involving multiple persons with high accuracy and low computations. [ABSTRACT FROM AUTHOR]

Realistic human action recognition in videos has been a useful yet challenging task. Video shots of same actions may present huge intra-class variations in terms of visual appearance, kinetic patterns, video shooting, and editing styles. Heterogeneous feature representations of videos pose another challenge on how to effectively handle the redundancy, complementariness and disagreement in these features. This paper proposes a localized multiple kernel learning (L-MKL) algorithm to tackle the issues above. L-MKL integrates the localized classifier ensemble learning and multiple kernel learning in a unified framework to leverage the strengths of both. The basis of L-MKL is to build multiple kernel classifiers on diverse features at subspace localities of heterogeneous representations. L-MKL integrates the discriminability of complementary features locally and enables localized MKL classifiers to deliver better performance in its own region of expertise. Specifically, L-MKL develops a locality gating model to partition the input space of heterogeneous representations to a set of localities of simpler data structure. Each locality then learns its localized optimal combination of Mercer kernels of heterogeneous features. Finally, the gating model coordinates the localized multiple kernel classifiers globally to perform action recognition. Experiments on two datasets show that the proposed approach delivers promising performance. [ABSTRACT FROM AUTHOR]

The human vision system usually has a specifically activated area of neurons when recognizing a category of images. Inspired by this visual mechanism, we propose a layerwise class-aware convolutional neural network architecture to explicitly discover category-tailored neurons on intermediate hidden layers to improve the network learning ability. Instead of directly selecting activated neurons for different categories, we inversely suppress those neurons of intermediate layers irrelevant with the given target class to produce a class-specific subnetwork, which implicitly enhances the discriminability of hidden layer features due to the increase of the inter-class discrepancy on them. Together with the classifier of the top layer, we jointly learn this network by formulating the suppressor of hidden layers as a penalty term in the objective function. To address class-specific neuron suppression in each hidden layer, we also introduce a statistic method based on mutual information to dynamically and automatically update the suppressed neurons during the network training. Extensive experiments demonstrate that the proposed model is superior to the state-of-the-art models. [ABSTRACT FROM AUTHOR]

Crowds arise in a variety of situations, such as public concerts and sporting matches. In typical conditions, the crowd moves in an orderly manner, but panic situations may lead to catastrophic results. We propose a computer vision method to identify motion pattern changes in human crowds that can be related to an unusual event. The proposed approach can identify global changes, by evaluating 2D motion histograms in time, and also local effects, by identifying clusters that present similar spatial locations and velocity vectors. The method is tested both on publicly available data sets involving crowded scenarios and on synthetic data produced by a crowd simulation algorithm, which allows the creation of controlled environments with known motion patterns that are particularly suitable for multicamera scenarios. [ABSTRACT FROM PUBLISHER]

Deep convolutional neural networks (CNNs) have achieved tremendous successes but tend to suffer from high computation costs mainly due to heavy over-parameterization, resulting in the difficulty of directly applying them to the ever-growing application demands based on low-end edge devices with strong power restriction and real-time inference requirement. Recently, there has much research attention devoted to compressing the network via pruning to address this issue. Most of the existing methods rely on some hand-designed pruning rules, which suffer from several limitations. Firstly, manually designed rules are only applicable to limited application scenarios, which can hardly generalize well in a broader scope. And these rules are typically designed based on human experience and via trial and error, and thus highly subjective. Then, channels of different layers in a network may have diverse distributions, which means the same pruning rule is not appropriate for each layer. To address these limitations, we propose a novel channel pruning scheme, in which the task-irrelevant channels are removed in a task-driven manner. Specifically, an adaptively differentiable search module is proposed to find the best pruning rule automatically for different layers in CNNs under sparsity constraints. Besides, we employed knowledge distillation to alleviate the excessive performance loss. Once the training process is finished, a compact network will be obtained by removing channels based on layer-wise pruning rules. We have evaluated the proposed method on some well-known benchmark datasets including CIFAR, MNIST, and ImageNet in comparison to several state-of-the-art pruning methods. Experimental results demonstrate the superiority of our method over the compared ones in terms of both parameters and FLOPs reduction. [ABSTRACT FROM AUTHOR]

Visual object tracking can be considered as an online procedure to adaptively measure the foreground object similarity itself. However, many previous works usually adopt a fixed metric or offline metric learning to evaluate this dynamic process; even with some online metric learning (OML) trackers, their models often suffer from overfitting issues. To overcome these deficiencies, we propose a self-supervised tracking method that incorporates adaptive metric learning and semisupervised learning into a unified framework. For similarity measurement, we design a new OML model via low-rank constraint to handle overfitting. In particular, we employ the max norm instead of the trace norm used in our previous work. This not only maintains the low-rank property to overcome overfitting, but also reduces the computational complexity from O(n^3) to O(n^2) , such that the new model is more suitable for object tracking. Moreover, by associating the information from stored training templates with unlabeled testing samples, a bilinear graph is defined accordingly to propagate the label of each sample. High-confidence samples are then collected for self-training the model and updating the templates concurrently to handle large scale. Experiments on various benchmark data sets and comparisons to several state-of-the-art methods demonstrate the effectiveness and efficiency of our algorithm. [ABSTRACT FROM PUBLISHER]

Mesh denoising is a critical technology in geometry processing that aims to recover high-fidelity 3D mesh models of objects from noise-corrupted versions. In this work, we propose a learning-based mesh normal denoising scheme, called NormalNet, which employs deep networks to find the correlation between the volumetric representation and denoised face normal. Overall, NormalNet follows the iterative framework of filtering-based mesh denoising. During each iteration, firstly, a local partition normalization strategy is applied to split the local structure around each face into dense voxels, in which both the structure and face normal information can be preserved during this transformation. Benefiting from the thorough information preservation, we can use simple residual networks, which employ the volumetric representation as the input and produce the learned denoised face normal, to achieve satisfactory results. Finally, the vertex positions are updated according to the denoised normals. Besides introducing normalization into mesh denoising, our main contributions include a classification-based training faces selection strategy for balancing the training set and a mismatched-faces rejection strategy for removing the mismatched faces between noisy mesh and ground truth. Compared to state-of-the-art works, NormalNet can effectively remove noise while preserving the original features and avoiding pseudo-features. [ABSTRACT FROM AUTHOR]

Cross Age Face Recognition (CAFR) is a challenging task in the field of face recognition. There still exist some limitations in mainstream CAFR methods. On one hand, some methods need synthesizing multiple groups of features and fusing the results under different ages, where the similarity score in some age groups may affect final classification results. On the other hand, many other methods takes a facial image as a linear combination of identity information and age information, which treats age factor as a value independent of identity information, but may be inconsistent with the aging pattern of many individuals. And these methods require both age labels and identity labels in training, which is limited by the scale of existing CAFR datasets. To address the above limitations, this work proposes the Parallel Multi-path Age Distinguish Network (PMADN) model. Specifically, our model consists of two cascading networks, an Age Distinguish Mapping Network (ADMN) and a Cross-Age Feature Recombination Network (CFRN). Firstly, the face features are mapped into different age groups by parallel multi-path full connected layers in ADMN, which can better extract the identity features in a small age span. Secondly, CFRN nonlinearly recombines the mapped features to extract the age robust features that are beneficial to identity classification, which can avoid the simple linear combination of identity factor and age factor in the existing methods. What’s more, our algorithm is combined with transfer learning and only uses the age label and a pre-trained ordinary face recognition network for training, which can make use of a larger aging face dataset for training. Extensive CAFR experiments performed on the benchmark MORPH Album2, CACD-VS and Cross Age LFW databases demonstrate the effectiveness and superiority of our method. [ABSTRACT FROM AUTHOR]

Recurrent neural network (RNN) has become popular for human motion prediction thanks to its ability to capture temporal dependencies. However, it has limited capacity in modeling the complex spatial relationship in the human skeletal structure. In this work, we present a novel diffusion convolutional recurrent predictor for spatial and temporal movement forecasting, with multi-step random walks traversing bidirectionally along an adaptive graph to model interdependency among body joints. In the temporal domain, existing methods rely on a single forward predictor with the produced motion deflecting to the drift route, which leads to error accumulations over time. We propose to supplement the forward predictor with a forward discriminator to alleviate such motion drift in the long term under adversarial training. The solution is further enhanced by a backward predictor and a backward discriminator to effectively reduce the error, such that the system can also look into the past to improve the prediction at early frames. The two-way spatial diffusion convolutions and two-way temporal predictors together form a quadruple network. Furthermore, we train our framework by modeling the velocity from observed motion dynamics instead of static poses to predict future movements that effectively reduces the discontinuity problem at early prediction. Our method outperforms the state of the arts on both 3D and 2D datasets, including the Human3.6M, CMU Motion Capture and Penn Action datasets. The results also show that our method correctly predicts both high-dynamic and low-dynamic moving trends with less motion drift. [ABSTRACT FROM AUTHOR]

The state-of-the-art objects discovery approaches can be categorized into two: deep learning methods based on a convolutional neural network with region proposals, and conventional machine learning methods based on topic models, low-rank matrix factorization, or image processing. The deep learning methods for objects discovery are based on sacrificing computational complexity to achieve precision, and the training time can be long without GPU platforms, whereas the conventional methods usually lack detection accuracy. To effectively address the problems of the training complexity and the detection speed, we present a new objects discovery approach by proposing a two-stage (training and verification) method. In the training stage, a topic model with words co-occurrence prior is proposed on the basis of Latent Dirichlet Allocation (LDA) model, in which the co-occurrence information among the features is sufficiently utilized. In the verification stage, we propose an anchor-box polishing algorithm that fine-tunes the detection results corresponding to the pre-trained topic model from some conventional algorithms with fast detection time. The experiments on various datasets demonstrate that the proposed approach can improve the detection performance in terms of efficiency and computing costs. It is also robust to objects different in colors, lightings, scales, and so on. Interestingly, the proposed method can be combined with many fast but inaccurate detection algorithms, in which it enhances the model’s flexibility. [ABSTRACT FROM AUTHOR]