Search

Multi-object tracking (MOT), which is crucial for computer vision and video processing, has immense potential for improvement. Traditional tracking-by-detection approaches include feature-based object re-identification methods that use trained features, but these methods suffer from a lack of suitable training data. In training datasets used for MOT, every object in a video sequence must have its own location and ID. However, assigning IDs to each object in every sequence is considerably labor-intensive, and hence current MOT datasets are unsuitable for training re-identification networks. To resolve this issue, this paper proposes a novel self-supervised learning method using several short videos that contain no human-added labels, based on the idea that each video is a set of temporally corresponding image frames. We then describe how to improve tracking performance using a re-identification network trained in a self-supervised manner. In addition, ablation studies were conducted in order to define the optimal parameters, such as number of clips, data augmentation, and appropriate matching algorithms. The proposed approach achieved competitive performance compared with current best-practice methods including supervised methods, achieving MOT accuracy = 62.0% and ID F1-score = 62.7% on the MOT17 benchmark. [ABSTRACT FROM AUTHOR]

This paper proposes a deep learning method for low-light image enhancement, which exploits the generation capability of Neural Networks (NNs) while requiring no training samples except the input image itself. Based on the Retinex decomposition model, the reflectance and illumination of a low-light image are parameterized by two untrained NNs. The ambiguity between the two layers is resolved by the discrepancy between the two NNs in terms of architecture and capacity, while the complex noise with spatially-varying characteristics is handled by an illumination-adaptive self-supervised denoising module. The enhancement is done by jointly optimizing the Retinex decomposition and the illumination adjustment. Extensive experiments show that the proposed method not only outperforms existing non-learning-based and unsupervised-learning-based methods, but also competes favorably with some supervised-learning-based methods in extreme low-light conditions. [ABSTRACT FROM AUTHOR]

Thanks to the powerful learning capability, deep neural networks (DNNs) have acquired broad applications in single image reflection removal. The DNN-based algorithms relax the constraints of specific priors and learn to generate visually pleasant background layers from massive training data. However, most of them employ a single network structure to recover both the semantic information and local details of the background, which may lead to obvious reflection residue or even failure. To mitigate this deficiency, in this work, we propose a Multi-stage Curvature-guided De-Reflection Network (MCDRNet), which combines multiple network architectures in a unified framework to progressively reconstruct the background layer and refine the fine-grained details. Our framework consists of three stages, where the encoder-decoders are exploited in the first two stages to recover the semantic components of background layers with lower scales and a variant ResNet is applied in the last stage to refine the background details with the original input resolution. In the first two stages, to introduce the structural guidance for the reflection removal, we cascade another decoder branch to restore the curvature map of the background. In addition, at the end of the first two stages, instead of directly passing the intermediate estimates to the next stage, we propose a Non-local Attention Module (NAM) to augment and transmit the features from decoders. Extensive experimental results on several public datasets demonstrate that the proposed MCDRNet outperforms the state-of-the-art methods quantitatively and generates visually better reflection removal results. The source code and pre-trained models are available at https://github.com/NamecantbeNULL/MCDRNet. [ABSTRACT FROM AUTHOR]

Although hand pose estimation has achieved a great success in recent years, there are still challenges with RGB-based estimation tasks, the most significant of which is the absence of labeled training data. At present, the synthetic dataset has plenty of images with accurate annotation, but the difference from real-world datasets affects generalization. Therefore, a transfer learning strategy, which tries to transfer knowledge from a labeled source domain to an unlabeled target domain, is a frequent solution. Existing methods such as mean-teacher, Cyclegan, and MCD will train models with the help of some easily accessible domains such as synthetic data. However, these methods are not guaranteed to operate well in real-world settings due to the domain shift. In this paper, we design a new unsupervised domain adaptation method named Multi-branch Adversarial Regressors (MarsDA) in hand pose estimation, where it could be better for feature migration. Specifically, we first generate pseudo-labels for unlabeled target domain data. Then, the new adversarial training loss between multiple regression branches we designed for hand pose estimation is introduced to narrow the domain gap. In this way, our model can reduce the noise of pseudo labels caused by the domain gap and improve the accuracy of pseudo labels. We evaluate our method on two publicly available real-world datasets, H3D and STB. Experimental results show that our method outperforms existing methods by a large margin. [ABSTRACT FROM AUTHOR]

Given a face image, most of previous works in gaze estimation infer the gaze via a well-trained model with supervised training. However, the distribution of test data may be very different compared to that of training data since samples might be corrupted in real-world scenarios (e.g., taking a photo in strong light). This will lead to a gap between source domain (i.e., training data) and target domain (i.e., test data). In this paper, we first introduce self-supervised learning into our method for addressing challenging situations in gaze estimation. Moreover, existing appearance-based gaze estimation methods focus on directing towards the development of powerful regressors, which mainly utilize face and eye images simultaneously or face (eye) images only. However, the problem of inter cues between face and eye features has been largely overlooked. To this end, we propose a novel Modulation-based Adaptive Network (MANet) for gaze estimation, which uses high-level knowledge to filter the distractive information and bridges the intrinsic relationship between face and eye features. Further, we combine self-supervised learning and MANet to learn to adapt to challenging cases, such as abnormal lighting conditions and poor-quality images, by minimizing a self-supervised loss and a supervised loss jointly. The experimental results on several datasets demonstrate the effectiveness of our proposed approach with a real-time speed of 900 fps on a PC with an NVIDIA Titan RTX GPU. [ABSTRACT FROM AUTHOR]

As a recent noticeable topic, domain generalization (DG) aims to first learn a generic model on multiple source domains and then directly generalize to an arbitrary unseen target domain without any additional adaption. In previous DG models, by generating virtual data to supplement observed source domains, the data augmentation based methods have shown its effectiveness. To simulate the possible unseen domains, most of them enrich the diversity of original data via image-level style transformation. However, we argue that the potential styles are hard to be exhaustively illustrated and fully augmented due to the limited referred styles, leading the diversity could not be always guaranteed. Unlike image-level augmentation, we in this paper develop a simple yet effective feature-based style randomization module to achieve feature-level augmentation, which can produce random styles via integrating random noise into the original style. Compared with existing image-level augmentation, our feature-level augmentation favors a more goal-oriented and sample-diverse way. Furthermore, to sufficiently explore the efficacy of the proposed module, we design a novel progressive training strategy to enable all parameters of the network to be fully trained. Extensive experiments on three standard benchmark datasets, i.e., PACS, VLCS and Office-Home, highlight the superiority of our method compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Background matting is a recently developed image matting approach, with applications to image and video editing. It refers to estimating both the alpha matte and foreground from a pair of images with and without foreground objects. Recent work has applied deep learning to background matting, with very promising performance achieved. However, existing deep models are supervised which require a large dataset with ground truth alpha mattes for training. To avoid the cost of data collection and possible bias in training data, this paper proposes a dataset-free unsupervised deep learning-based approach for background matting. Observing that the local smoothness of alpha matte can be well characterized by the untrained network prior called deep matte prior, we model the foreground and alpha matte using the priors encoded by two generative convolutional neural networks. To avoid possible overfitting during unsupervised learning, a two-stage learning scheme is developed which contains projection-based training and Bayesian post refinement. An alpha-matte-driven initialization scheme is also developed for performance boost. Even without calling external training data, the proposed approach provides competitive performance to recent supervised learning-based methods in the experiments. [ABSTRACT FROM AUTHOR]

Facial Expression Recognition (FER) is a long-standing but challenging research problem in computer vision. Existing approaches mainly focus on single-label emotional prediction, which cannot handle the complex multi-label FER task because of the coupling behavior of multiple emotions on a single facial image. To this end, in this paper, we propose a novel Self-supervised Exclusive-Inclusive Interactive Learning (SEIIL) method to facilitate discriminative multi-label FER in the wild, which can effectively handle the coupled multiple sentiments with limited unconstrained training data. Specifically, we construct an emotion disentangling module to capture the inclusive and exclusive characteristics of facial expressions, which can decouple the compound numerous emotions on an image. Moreover, an adaptively-weighted ensemble technique is conceived to aggregate category-level latent exclusive embeddings, and then a conditional adversarial interactive learning module is designed to fully leverage the complementary between the inclusive and formulated latent representations. Furthermore, to tackle the insufficient data for training, we introduce a self-supervised learning strategy to augment the amount and diversity of facial images, which can endow the model with advanced generalization ability. Under this strategy, the proposed two modules can be concurrently utilized in our SEIIL to jointly handle the coupled emotions and alleviate the overfitting problem. Extensive experimental results on six databases illustrate the superb performance of our method against state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

A computer-aided diagnosis (CAD) system that characterizes nodules in medical images can help radiologists determine its malignancy. Preparing large volumes of labeled data for CAD systems, however, requires advanced medical knowledge. This makes it extremely difficult to develop such systems, despite their growing demand. In this paper, we propose a new training method to build an image classifier for characterization of nodules utilizing pseudo-labels, i.e., image labels automatically retrieved from radiology reports. A radiology report is a type of record in which radiologists present a summary of lesion characteristics and diagnosis. Labeling radiology reports is much easier than labeling radiology images, and can be done without high expertise. Using several thousand labeled reports, we constructed a hierarchical attention network-based text classifier to assign pseudo-labels of the characteristics of pulmonary nodules with high accuracy (macro F1-score of 0.941). Experimental results show that the image classifier trained with the pseudo-labels can achieve almost the same performance as the one trained with the labels annotated by radiologists: AUC 0.848 for the model trained with the pseudo-labels on 3,000 computed tomography (CT) images and 0.847 for the model trained with the manual labels on 800 CT images. [ABSTRACT FROM AUTHOR]

Feature selection aims at choosing a subset of features to represent the original feature space. In practice, however, it is hard to achieve desirable performance due to limited training data. To alleviate this issue, we propose a novel problem named feature selection with multi-source transfer where the privileged information from another data source or modality– only available during the training phase, is exploited to improve the performance of feature selection. To be exact, we propose a novel objective function that formulates the privileged information into feature selection. Moreover, an efficient optimization algorithm is introduced to solve the proposed problem of high dimension. Extensive experimental results demonstrate that the proposed algorithm significantly outperforms several popular algorithms, especially when the training data size and the selected feature size are small. [ABSTRACT FROM AUTHOR]

Although deep face recognition benefits significantly from large-scale training data, a current bottleneck is the labelling cost. A feasible solution to this problem is semi-supervised learning, exploiting a small portion of labelled data and large amounts of unlabelled data. The major challenge, however, is the accumulated label errors through auto-labelling, compromising the training. In this paper, we present an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. Specifically, we introduce a multi-agent method, named GroupNet (GN), to endow our solution with the ability to identify the wrongly-labelled samples and preserve the clean samples. We show that GN alone achieves the leading accuracy in traditional supervised face recognition even when the noisy labels take over 50% of the training data. Further, we develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN. It starts with a small amount of labelled data and consequently conducts high-confidence labelling on a large amount of unlabelled data to boost further training. The more data is labelled by NRoLL, the higher confidence is with the label in the dataset. To evaluate the competitiveness of our method, we run NRoLL with a rough condition that only one-fifth of the labelled MSCeleb is available and the rest is used as unlabelled data. On a wide range of benchmarks, our method compares favorably against the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Mesh denoising is a critical technology in geometry processing that aims to recover high-fidelity 3D mesh models of objects from noise-corrupted versions. In this work, we propose a learning-based mesh normal denoising scheme, called NormalNet, which employs deep networks to find the correlation between the volumetric representation and denoised face normal. Overall, NormalNet follows the iterative framework of filtering-based mesh denoising. During each iteration, firstly, a local partition normalization strategy is applied to split the local structure around each face into dense voxels, in which both the structure and face normal information can be preserved during this transformation. Benefiting from the thorough information preservation, we can use simple residual networks, which employ the volumetric representation as the input and produce the learned denoised face normal, to achieve satisfactory results. Finally, the vertex positions are updated according to the denoised normals. Besides introducing normalization into mesh denoising, our main contributions include a classification-based training faces selection strategy for balancing the training set and a mismatched-faces rejection strategy for removing the mismatched faces between noisy mesh and ground truth. Compared to state-of-the-art works, NormalNet can effectively remove noise while preserving the original features and avoiding pseudo-features. [ABSTRACT FROM AUTHOR]

Cross-modal hashing is for the purpose of compressing heterogeneous multi-modal data into compact binary codes for the cross-modal retrieval, where accuracy and efficiency are two primary issues. To achieve high accuracy and efficiency, we put forward a novel method named Fast Unmediated Hashing (FUH) for cross-modal retrieval. For this method, motivated by the fact that label vector is a natural binary representation of samples for retrieval, we directly learn the cross-modal hash codes from semantic labels without any intermediate representation. This will capture more relations among different modalities, and reduce the number of variables. However, directly learning hash codes from labels would weaken the discrimination of hash codes. To address this issue, double supervision involving label information and pairwise similarity is proposed to enhance the discrimination. In addition, to decrease the training time, we present a strategy to bypass the similarity matrix-related operation in each iteration of optimization, thus some other related terms can also be computed offline to lower training complexity. Compared to several state-of-the-art techniques on three public datasets, the experimental results have manifested the superiority of FUH concerning efficiency and accuracy. [ABSTRACT FROM AUTHOR]

Given that retrieval and storage have compelling efficiency, similarity-preserving hashing has been extensively employed to approximate nearest neighbor search in large-scale image retrieval. Hash codes that are extremely compact not only can further lower the storage cost, but also accelerate the retrieval speed. However, existing methods perform poorly in retrieval based on an extremely short-length hash code, which attributes to the weak ability of classification and poor distribution of hash bit. To tackle this issue, in this study, we propose a novel reinforced short-length hashing (RSLH). In particular, this proposed method applies the mutual reconstruction between the hash representation and semantic label to retain the semantic information. Furthermore, to enhance the accuracy of hash representation, a pairwise similarity matrix is designed to make a balance between accuracy and training expenditure on memory. Besides, we integrate a parameter boosting strategy to strengthen the precision with the consideration of bit balance and uncorrelation constraints. Extensive experiments on three large-scale image benchmarks demonstrate the superior performance of RSLH under various short-length hashing scenarios. [ABSTRACT FROM AUTHOR]

Linear Regression (LR) is a popular and effective technique in pattern recognition area, which aims to find a transform matrix between source data and target data (usually label matrix). However, a binary zero-one label matrix may be too strict and inappropriate for regression. Besides, directly projecting source data to target data by one transform matrix may lose some intrinsic data information. To address these issues, this paper proposes a novel Pairwise Relations oriented Discriminative Regression (PRDR) method. In PRDR, the source data is regressed into a latent space instead of label space. To supervise the discriminative projection learning, the pairwise relations in source data space and label space are exploited in the latent space simultaneously. The pairwise label relations are transferred into the latent subspace by solving a distance-distance difference minimization problem, and the intraclass instance relations are also preserved in latent space. These two constraints ensure the pairwise similarity of data points after transformation which is beneficial for classification. By further enlarging the margins between true and false classes, PRDR is extended to a robust version, i.e., R-PRDR. An efficient algorithm is presented to solve the PRDR model. Extensive experiments on several popular image datasets demonstrate the effectiveness and efficiency of the proposed method compared with some state-of-the-art regression approaches. [ABSTRACT FROM AUTHOR]

Group re-identification (G-ReID) is an important yet less-studied task. Its challenges not only lie in appearance changes of individuals, but also involve group layout and membership changes. To address these issues, the key task of G-ReID is to learn group representations robust to such changes. Nevertheless, unlike ReID tasks, there still lacks comprehensive publicly available G-ReID datasets, making it difficult to learn effective representations using deep learning models. In this article, we propose a Domain-Transferred Single and Couple Representation Learning Network (DotSCN). Its merits are two aspects: 1) Owing to the lack of labelled training samples for G-ReID, existing G-ReID methods mainly rely on unsatisfactory hand-crafted features. To gain the power of deep learning models in representation learning, we first treat a group as a collection of multiple individuals and propose transferring the representation of individuals learned from an existing labeled ReID dataset to a target G-ReID domain without a suitable training dataset. 2) Taking into account the neighborhood relationship in a group, we further propose learning a novel couple representation between two group members, that achieves better discriminative power in G-ReID tasks. In addition, we propose a weight learning method to adaptively fuse the domain-transferred individual and couple representations based on an L-shape prior. Extensive experimental results demonstrate the effectiveness of our approach that significantly outperforms state-of-the-art methods by 11.7% CMC-1 on the Road Group dataset and by 39.0% CMC-1 on the DukeMCMT dataset. [ABSTRACT FROM AUTHOR]

Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model overfitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis. [ABSTRACT FROM AUTHOR]

Current multi-domain image-to-image translation models assume a fixed set of domains and that all the data are always available during training. However, over time, we may want to include additional domains to our model. Existing methods either require re-training the whole model with data from all domains or require training several additional modules to accommodate new domains. To address these limitations, we present IncrementalGAN, a multi-domain image-to-image translation model that can incrementally learn new domains using only a single generator. Our approach first decouples the domain label representation from the generator to allow it to be re-used for new domains without any architectural modification. Next, we introduce a distillation loss that prevents the model from forgetting previously learned domains. Our model compares favorably against several state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

Recent video anomaly detection methods focus on reconstructing or predicting frames. Under this umbrella, the long-standing inter-class data-imbalance problem resorts to the imbalance between foreground and stationary background objects in video anomaly detection and this has been less investigated by existing solutions. Naively optimizing the reconstructing loss yields a biased optimization towards background reconstruction rather than the objects of interest in the foreground. To solve this, we proposed a simple yet effective solution, termed attention-driven loss to alleviate the foreground-background imbalance problem in anomaly detection. Specifically, we compute a single mask map that summarizes the frame evolution of moving foreground regions and suppresses the background in the training video clips. After that, we construct an attention map through the combination of the mask map and background to give different weights to the foreground and background region respectively. The proposed attention-driven loss is independent of backbone networks and can be easily augmented in most existing anomaly detection models. Augmented with attention-driven loss, the model is able to achieve AUC 86.0% on Avenue, 83.9% on Ped1, 96% on Ped2 datasets. Extensive experimental results and ablation studies further validate the effectiveness of our model. [ABSTRACT FROM AUTHOR]

Existing cross-media retrieval methods are usually conducted under the supervised setting, which need lots of annotated training data. Generally, it is extremely labor-consuming to annotate cross-media data. So unsupervised cross-media retrieval is highly demanded, which is very challenging because it has to handle heterogeneous distributions across different media types without any annotated information. To address the above challenge, this paper proposes Domain Adaptation with Scene Graph (DASG) approach, which transfers knowledge from the source domain to improve cross-media retrieval in the target domain. Our DASG approach takes Visual Genome as the source domain, which contains image knowledge in the form of scene graph. The main contributions of this paper are as follows: First, we propose to address unsupervised cross-media retrieval by domain adaptation. Instead of using the labor-consuming annotated information of cross-media data in the training stage, our DASG approach learns cross-media correlation knowledge from Visual Genome, and then transfers the knowledge to cross-media retrieval through media alignment and distribution alignment. Second, our DASG approach utilizes fine-grained information via scene graph representation to enhance generalization capability across domains. The generated scene graph representation builds (subject $\rightarrow $ relationship $\rightarrow $ object) triplets by exploiting objects and relationships within image and text, which makes the cross-media correlation more precise and promotes unsupervised cross-media retrieval. Third, we exploit the related tasks including object and relationship detection for learning more discriminative features across domains. Leveraging the semantic information of objects and relationships improves cross-media correlation learning for retrieval. Experiments on two widely-used cross-media retrieval datasets, namely Flickr-30K and MS-COCO, show the effectiveness of our DASG approach. [ABSTRACT FROM AUTHOR]

Despite the increasing need of analyzing human poses on the street and in the wild, multi-person 3D pose estimation using monocular static or moving camera in real-world scenarios remains a challenge, either requiring large-scale training data or high computation complexity due to the high degrees of freedom in 3D human poses. We propose a novel scheme to effectively track and hierarchically estimate 3D human poses in natural videos in an efficient fashion. Without the need of using labelled 3D training data, we formulate torso estimation as a Perspective-N-Point (PNP) problem, and limb pose estimation as an optimization problem, and hierarchically structure the high dimensional poses to efficiently address the challenge. Experiments show good performance and high efficiency of multi-person 3D pose estimation on real-world videos, including street scenarios and various human daily activities from fixed and moving cameras, resulting in great new opportunities to understand and predict human behaviors. [ABSTRACT FROM AUTHOR]

As we move toward large-scale object detection, it is unrealistic to expect annotated training data, in the form of bounding box annotations around objects, for all object classes at sufficient scale; therefore, the methods capable of unseen object detection are required. We propose a novel zero-shot method based on training an end-to-end model that fuses semantic attribute prediction with visual features to propose object bounding boxes for seen and unseen classes. While we utilize semantic features during training, our method is agnostic to semantic information for unseen classes at test-time. Our method retains the efficiency and effectiveness of YOLOv2 for objects seen during training, while improving its performance for novel and unseen objects. The ability of the state-of-the-art detection methods to learn discriminative object features to reject background proposals also limits their performance for unseen objects. We posit that, to detect unseen objects, we must incorporate semantic information into the visual domain so that the learned visual features reflect this information and lead to improved recall rates for unseen objects. We test our method on PASCAL VOC and MS COCO dataset and observed significant improvements on the average precision of unseen classes. [ABSTRACT FROM AUTHOR]

Due to the shortcomings of the weakly supervised and fully supervised object detection (i.e., unsatisfactory performance and expensive annotations, respectively), leveraging partially labeled images in a cost-effective way to train an object detector has attracted much attention. In this paper, we formulate this challenging task as a missing bounding-boxes’ object detection problem. Specifically, we develop a pseudo ground truth mining procedure to automatically find the missing bounding boxes for the unlabeled instances, called pseudo ground truths here, in the training data, and then combine the mined pseudo ground truths and the labeled annotations to train a fully supervised object detector. Furthermore, we propose an incremental learning framework to gradually incorporate the results of the trained fully supervised detector to improve the performance of the missing bounding-boxes’ object detection. More importantly, we find an effective way to label the massive images with limited labors and funds, which is crucial when building a large-scale weakly/webly labeled dataset for object detection. The extensive experiments on the PASCAL VOC and COCO benchmarks demonstrate that our proposed method can narrow the gap between the fully supervised and weakly supervised object detectors, and outperform the previous state-of-the-art weakly supervised detectors by a large margin (more than 3% mAP absolutely) when the missing rate equals 0.9. Moreover, our proposed method with 30% missing bounding-box annotations can achieve comparable performance to some fully supervised detectors. [ABSTRACT FROM AUTHOR]

People counting in extremely dense crowds is a challenging problem due to severe occlusions, few pixels per head, cluttered environments, and skewed camera perspectives. In this paper, we present a novel algorithm for people counting in highly dense crowd images. Our approach relies on the fact that the head is the most visible part of an individual in a dense crowd. As such, a head detector can be used to estimate the spatially varying head size, which is the key feature used in our head counting procedure. We leverage the state-of-the art convolutional neural network for the sparse head detection in a dense crowd. After sub-dividing the image into rectangular patches, we first use an speeded-up robust features-based support vector machine binary classifier to label each patch as crowd/not-crowd and eliminate all not-crowd patches. Regression is then performed on each crowd patch to estimate average head size. The number of individuals in each patch is estimated by dividing the patch area with the estimated head size. For the crowd patches where no heads are detected, the counts are estimated based on distance-based weighted averaging over the counts from neighboring patches. Finally, the individual patch counts are summed up to obtain the total count. We evaluate our approach on three publicly available datasets for extremely dense crowds: UCF_CC_50, ShanghaiTech, and AHU-Crowd. Our approach gives comparable results on these challenging datasets to other state of the art algorithms but, unlike other algorithms, our proposed method does not require the laborious task of obtaining labeled training data of dense crowd images. [ABSTRACT FROM AUTHOR]

In many practical engineering applications, the number of actions that have been finished should be known, particularly for an untrimmed video sequence that includes an event with a series of actions, it is important to know the number of actions that have been finished. In this paper, we termed this process as visual event progress estimation (EPE). However, the research related to this problem is few in the research community. To solve this problem, a visual human action analysis-based framework, namely one-shot simultaneously action detection and identification (SADI)-EPE, is presented in this paper. The visual EPE is modeled as an online one-shot learning-based problem; sliding window and attention-based bag of key poses formulate our framework. Unlike most of the action analysis methods relying on a number of training data of some predefined classes, our method can realize SADI for any event if one sample of the event is given, which makes it feasible for practical applications. At the same time, not only SADI but also the progress estimation of the event can be realized by our algorithm. In terms of methodology, the key pose is defined by an invariant pose descriptor from skeletal data and silhouette data. Moreover, in order to extract representative and discriminative poses from one training sample, we present a new bidirectional $k$ NN-based attention weighted key pose selection method, which can filter the unrelated actions and model different importance of various key poses. In addition, an attention-based multi-modal fusion scheme, which addresses the difficulty of high-dimensional features and few training samples, is proposed to augment the performance of our algorithm. Finally, we propose an evaluation criterion for the estimation problem. Extensive results demonstrated the efficacy of our proposed framework. [ABSTRACT FROM AUTHOR]

Limited annotated data available for the recognition of facial expression and particularly action units makes it hard to train a deep network which can learn disentangled invariant features. However, a supervised linear model is undemanding in terms of training data. In this paper, we propose an elegant linear model to untangle facial actions from expressive face videos which contain a mixture of linearly-representable attributes. Previous attempts require an explicit decoupling of identity and expression which is practically inexact. Instead, we exploit the low-rank property across frames to implicitly subtract the intrinsic neutral face, which are modeled jointly with sparse representation only on the residual expression components. On CK+, our one-shot C-HiSLR on raw-face pixel-intensities performs far more competitive than conventional shape+SVM models with landmark detection and two-stepped SRC of the same type yet applied on manually prepared expression components. It is also comparable with the piecewise linear model DCS and temporal models, such as CRF and Bayes nets. We apply it to action unit (AU) recognition on MPI-VDB achieving a decent performance. As expression is a mixture of AUs, the result gives hopes of approximating an expression using a piecewise linear model. [ABSTRACT FROM AUTHOR]

The methodology for finding the same individual in a network of cameras must deal with significant changes in appearance caused by variations in illumination, viewing angle, and a person’s pose. Re-identification requires solving two fundamental problems: 1) determining a distance measure between features extracted from different cameras that cope with illumination changes (metric learning) and 2) ensuring that matched features refer to the same body part (correspondence). Most metric learning approaches focus on finding a robust distance measure between bounding box images, neglecting the alignment aspects. In this paper, we propose to learn appearance measures for patches that are combined using deformable models. Learning metrics for patches avoids strong dimensionality reduction, thus keeping more information. Additionally, we allow patches to change their locations, directly addressing the correspondence problem. As patches from different locations may share the same metric, our method effectively multiplies the amount of training data and allows patch metrics to be learned on the smaller amounts of labeled images. Different metric learning approaches (KISSME, XQDA, and LSSL) together with different deformable models (spring constraints and one-to-one matching constraints) are investigated and compared. For describing patches, we propose to learn a deep feature representation with convolutional neural networks, thus obtaining highly effective features for re-identification. We demonstrate that our approach significantly outperforms state-of-the-art methods on multiple data sets. [ABSTRACT FROM AUTHOR]

An action is something which is done by an agent. Most action recognition researchers merely focus on the actions to be recognized, and ignore the differences of agents. Philosophers and behaviorists discover that actions are common among many species, but are performed in different ways and with different levels of sophistication. In this paper, in order to bridge action recognition tasks between different agents, we introduce a new problem, cross-agent action recognition, i.e., recognizing action for one particular agent (target) while training from other agents (source). We model this problem under three different scenarios: single source and single target, multiple sources and single target, and multiple sources and multiple targets. To this end, corresponding methods based on transfer learning are proposed to address these problems. We further design three different strategies to model the situation when a partial labeled data is provided for the target. Experimental results show that the performances of the transfer method are generally better than those of the comparative method without transfer learning, especially when we have multiple sources. Particularly, the transfer method outperforms the others significantly when the source is a human adult. In addition, cross-agent method significantly improves the results when partially labeled data is provided for the target. These demonstrate that for action recognition, knowledge can be transferred across different agents. A straightforward application of this finding is to use human action (training data is abundant) data to enhance animal action recognition. [ABSTRACT FROM AUTHOR]

The hashing method maps similar data of various types to binary hashcodes with smaller hamming distance, and it has received broad attention due to its low-storage cost and fast retrieval speed. However, the existing limitations make the present algorithms difficult to deal with for large-scale data sets: 1) discrete constraints are involved in the learning of the hash function and 2) pairwise or triplet similarity is adopted to generate efficient hashcodes, resulting in both time and space complexity greater than $O(n^{2})$. To address these issues, we propose a novel discrete supervised hash learning framework that can be scalable to large-scale data sets of various types. First, the discrete learning procedure is decomposed into a binary classifier learning scheme and binary codes learning scheme, which makes the learning procedure more efficient. Second, by adopting the asymmetric low-rank matrix factorization, we propose the fast clustering-based batch coordinate descent method, such that the time and space complexity are reduced to $O(n)$. The proposed framework also provides a flexible paradigm to incorporate with arbitrary hash function, including deep neural networks and kernel methods, as well as any types of data to hash, including images and videos. Experiments on large-scale data sets demonstrate that the proposed method is superior or comparable with the state-of-the-art hashing algorithms. [ABSTRACT FROM AUTHOR]

A better understanding of human interactions in videos can be achieved by simultaneously considering the coarse interactions between people, the action of each individual, and the activity of all people as a whole. We divide the recognition task into two stages. The first stage discriminates interactions and noninteractions, actions and activities based on local image information, while during the second stage, actions and activities are recognized in a global manner based on the local recognition results. A conditional random field (CRF) is designed to model human interactions in the spatio-temporal space. Different from most existing global models which cover either action or activity variables only, our model covers them both by considering the interactions between different types of variables. The graph structure of the CRF is predicted by a model learned from training data, which is different from traditional graph construction methods that typically rely on human heuristics. We learn the parameters of the CRF via structured support vector machine. We propose an efficient inference algorithm to tackle the estimation of labels in long videos containing many people. Our model admits both semantic-level understanding of human interactions in videos and competitive action and activity recognition performance. [ABSTRACT FROM PUBLISHER]

This paper proposes an ensemble of invariant features (EIFs), which can properly handle the variations of color difference and human poses/viewpoints for matching pedestrian images observed in different cameras with nonoverlapping field of views. Our proposed method is a direct reidentification (re-id) method, which requires no prior domain learning based on prelabeled corresponding training data. The novel features consist of the holistic and region-based features. The holistic features are extracted by using a publicly available pretrained deep convolutional neural network used in generic object classification. In contrast, the region-based features are extracted based on our proposed two-way Gaussian mixture model fitting, which overcomes the self-occlusion and pose variations. To make a better generalization during recognizing identities without additional learning, the ensemble scheme aggregates all the feature distances using the similarity normalization. The proposed framework achieves robustness against partial occlusion, pose, and viewpoint changes. Moreover, the evaluation results show that our method outperforms the state-of-the-art direct re-id methods on the challenging benchmark viewpoint invariant pedestrian recognition and 3D people surveillance data sets. [ABSTRACT FROM PUBLISHER]

Face sketch synthesis is crucial in many practical applications, such as digital entertainment and law enforcement. Previous methods relying on many photo–sketch pairs have made great progress. State-of-the-art face sketch synthesis algorithms adopt Bayesian inference (BI) (e.g., Markov random fields) to select local sketch patches around corresponding position from a set of training data. However, these methods have two limitations: 1) they depend on many training photo–sketch pairs and 2) they cannot tackle nonfacial factors (e.g., hairpins, glasses, backgrounds, and image size) if these factors are excluded in training data. In this paper, we propose a novel face sketch synthesis method that is capable of handling nonfacial factors only using a single photo–sketch pair from coarse to fine. Our method proposes a cascaded image synthesis (CIS) strategy and integrates sparse representation-based greedy search (SRGS) and BI for face sketch synthesis. We first apply SRGS to select candidate sketch patches from the whole training photo–sketch pairs sampled from the only photo–sketch pair. We then employ BI to estimate an initial sketch. Afterward, the input photo and the estimated initial sketch are taken as an additional photo–sketch pair for training. Finally, we adopt CIS with the given two photo–sketch pairs to further improve the quality of the initial sketch. The experimental results on several databases demonstrate that our algorithm outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Image tag refinement is the task of refining initial tags of an image such that the refined tags can better reflect the content of the image and, therefore, can help users better access that image. The quality of tag refinement depends on the quality of concept representations that build a mapping from concepts to visual images. While good progress was made in the past decade on tag refinement, the previous approaches only achieved a limited success due to their limited concept representations. In this paper, we show that the visual appearances of a concept consist of both a generic view and a specific view, and therefore we can comprehensively represent a concept by two components. To ensure a clean concept representation, this representation is learned on clean click-through data, where noises are greatly reduced. In the framework, a coarse-to-fine image tag refinement is proposed, which: 1) first generates an efficient star graph to find candidate tags but missing in the initial tag list of an input image and 2) guided by this view-dependent concept representation, formulates a probabilistic objective function to eliminate irrelevant tags. Extensive experiments on two widely used standard data sets (MIRFlickr-25K and NUS-WIDE-270K) demonstrate the effectiveness of our approach. [ABSTRACT FROM PUBLISHER]

Generalized Hough voting (HV) has shown promising results in both object and action detection. However, most existing HV methods will suffer when insufficient training data are provided. We propose propagative HV to address this limitation and apply it to human activity analysis. Instead of training a discriminative classifier for local feature voting, we match individual local features to propagate the label and spatiotemporal configuration information of local features via HV. To enable a fast local feature matching, we index the local features using random projection trees (RPTs). RPTs can reveal the low-dimension manifold structure to provide adaptive local feature matching. Moreover, as the RPT index can be built in either labeled or unlabeled dataset, it can be applied to different tasks, such as activity search (limited training) and recognition (sufficient training). The superior performances on benchmarked datasets validate that our propagative HV can outperform state-of-the-art techniques in various activity analysis tasks, such as activity search, recognition, and prediction. [ABSTRACT FROM AUTHOR]

Boosting algorithms, especially AdaBoost, have attracted great attention in computer vision. In the early version of boosting algorithms, the weak classifier selection and the strong classifier learning are linked together. It has been demonstrated that decoupling of these two processes can provide more flexibility for training a better classifier. In these studies, linear discriminant analysis (LDA) has been adopted to select weak classifiers independently based on class separability rather than a training error that occurs normally in AdaBoost. It is observed that LDA is successful only if a large number of labeled training samples is available. However, a large-scale labeled training set is not always available in many computer vision applications such as object classification. To tackle this problem, this paper proposes semisupervised subspace learning combined with a boosting framework for object classification, through which unlabeled data can participate in the boosting training to compensate for the lack of enough labeled data. With the proposed framework, this paper develops three various approaches that utilize unlabeled data in different ways. According to the experiments on several public image data sets, the proposed methods achieve superior performance over AdaBoost and existing semisupervised algorithms. [ABSTRACT FROM PUBLISHER]

In this paper, a new network-transmission-based (NTB) algorithm is proposed for human activity recognition in videos. The proposed NTB algorithm models the entire scene as an error-free network. In this network, each node corresponds to a patch of the scene and each edge represents the activity correlation between the corresponding patches. Based on this network, we further model people in the scene as packages, while human activities can be modeled as the process of package transmission in the network. By analyzing these specific package transmission processes, various activities can be effectively detected. The implementation of our NTB algorithm into abnormal activity detection and group activity recognition are described in detail in this paper. Experimental results demonstrate the effectiveness of our proposed algorithm. [ABSTRACT FROM PUBLISHER]

Relevance feedback (RF) schemes have been widely designed to improve the performance of content-based image retrieval. Despite the success, it is not appropriate to require the user to label a large number of samples in RF. Collaborative image retrieval (CIR) aims to reduce the labeling efforts of the user by resorting to the auxiliary information. Support vector machine (SVM) active learning can select ambiguous samples as the most informative ones for the user to label with the help of the optimal hyperplane of SVM, and thus alleviate the labeling efforts of conventional RF. However, the optimal hyperplane of SVM is usually unstable and inaccurate with small-sized training data, and this is always the case in image retrieval since the user would not like to label a large number of feedback samples and cannot label each sample accurately all the time. In this paper, we propose a novel active learning method, i.e., geometric optimum experimental design (GOED), to select multiple representative samples in the database as the most informative ones for the user to label. Especially, GOED can alleviate the small-sized training data problem by leveraging the geometric structure of unlabeled samples in the reproducing kernel Hilbert space and thus further enhance the performance of image retrieval. Different from the conventional manifold regularization framework, the new method can effectively select the most informative samples for the user to label in image retrieval. By minimizing the expected average prediction variance on the test data, GOED has a clear geometric interpretation to select a set of the most representative samples in the database iteratively with the global optimum. Compared with the popular SVM active learning, our method is label-independent and can effectively avoid various potential problems caused by insufficient and inexactly labeled samples in RF, and is more appropriate and useful for image retrieval. Extensive experiments on both synthetic datasets and a real-world image database have been conducted to show the advantages of the proposed GOED for CIR. [ABSTRACT FROM PUBLISHER]