Search

Numerous tracking approaches attempt to improve target representation through target-aware or distractor-aware. However, the unbalanced considerations of target or distractor information make it diffcult for these methods to benefit from the two aspects at the same time. In this paper, we propose a target-distractor aware model with discriminative enhancement learning loss to learn target representation, which can better distinguish the target in complex scenes. Firstly, to enlarge the gap between the target and distractor, we design a discriminative enhancement learning loss. By highlighting the hard negatives that are similar to the target and shrinking the easy negatives that are pure background, the features sensitive to the target or distractor representation can be more conveniently mined. On this basis, we further propose a target-distractor aware model. Unlike existing methods of preference target or distractor, we construct the target-specific feature space by activating the target-sensitive and the distractor-silence feature. Therefore, the appearance model can not only represent the target well but also suppress the background distractor. Finally, the target-distractor aware target representation model is integrated with a Siamese matching network for visual tracking for achieving robust and realtime visual tracking. Extensive experiments are performed on eight tracking benchmarks show that the proposed algorithm achieves favorable performance. [ABSTRACT FROM AUTHOR]

Visible-infrared person re-identification (VI-ReID) is challenging due to the large modality discrepancy between visible and infrared images. Existing methods mainly focus on learning modality-shared representations by embedding images from different modalities into a common feature space, in which some discriminative modality information is discarded. Different from these methods, in this paper, we propose a novel Modality-Specific Memory Network (MSMNet) to complete the missing modality information and aggregate visible and infrared modality features into a unified feature space for the VI-ReID task. The proposed model enjoys several merits. First, it can exploit the missing modality information to alleviate the modality discrepancy when only the single-modality input is provided. To the best of our knowledge, this is the first work to exploit the missing modality information completion and alleviate the modality discrepancy with the memory network. Second, to guide the learning process of the memory network, we design three effective learning strategies, including feature consistency, memory representativeness and structural alignment. By incorporating these learning strategies in a unified model, the memory network can be well learned to propagate identity-related information between modalities and boost the VI-ReID performance. Extensive experimental results on two standard benchmarks (SYSU-MM01 and RegDB) demonstrate that the proposed MSMNet performs favorably against state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Weakly supervised temporal action detection has better scalability and practicability than fully supervised action detection in reality deployment. However, it is difficult to learn a robust model without temporal action boundary annotations. In this paper, we propose an en-to-end Multi-Scale Structure-Aware Network (MSA-Net) for weakly supervised temporal action detection by exploring both the global structure information of a video and the local structure information of actions. The proposed SA-Net enjoys several merits. First, to localize actions with different durations, each video is encoded into feature representations with different temporal scales. Second, based on the multi-scale feature representation, the proposed model has designed two effective structure modeling mechanisms including global structure modeling and local structure modeling, which can effectively learn discriminative structure aware representations for robust and complete action detection. To the best of our knowledge, this is the first work to fully explore the global and local structure information in a unified deep model for weakly supervised action detection. And extensive experimental results on two benchmark datasets demonstrate that the proposed MSA-Net performs favorably against state-of-the-art methods. [ABSTRACT FROM AUTHOR]

To establish robust semantic correspondence between images covering different objects belonging to the same category, there are three important types of information including inter-image relationship, intra-image relationship and cycle consistency. Most existing methods only exploit one or two types of the above information and cannot make them enhance and complement each other. Different from existing methods, we propose a novel end-to-end Consistency Graph Modeling Network (CGMNet) for semantic correspondence by modeling inter-image relationship, intra-image relationship and cycle consistency jointly in a unified deep model. The proposed CGMNet enjoys several merits. First, to the best of our knowledge, this is the first work to jointly model the three kinds of information in a deep model for semantic correspondence. Second, our model has designed three effective modules including cross-graph module, intra-graph module and cycle consistency module, which can jointly learn more discriminative feature representations robust to local ambiguities and background clutter for semantic correspondence. Extensive experimental results show that our algorithm performs favorably against state-of-the-art methods on four challenging datasets including PF-PASCAL, PF-WILLOW, Caltech-101 and TSS. [ABSTRACT FROM AUTHOR]

Weakly supervised temporal sentence grounding has better scalability and practicability than fully supervised methods in real-world application scenarios. However, most of existing methods cannot model the fine-grained video-text local correspondences well and do not have effective supervision information for correspondence learning, thus yielding unsatisfying performance. To address the above issues, we propose an end-to-end Local Correspondence Network (LCNet) for weakly supervised temporal sentence grounding. The proposed LCNet enjoys several merits. First, we represent video and text features in a hierarchical manner to model the fine-grained video-text correspondences. Second, we design a self-supervised cycle-consistent loss as a learning guidance for video and text matching. To the best of our knowledge, this is the first work to fully explore the fine-grained correspondences between video and text for temporal sentence grounding by using self-supervised learning. Extensive experimental results on two benchmark datasets demonstrate that the proposed LCNet significantly outperforms existing weakly supervised methods. [ABSTRACT FROM AUTHOR]

Facial expression recognition, face synthesis, and face alignment are three coherently related tasks and can be solved in a joint framework. To achieve this goal, in this paper, we propose a novel end-to-end deep learning model by exploiting the expression code, geometry code and generated data jointly for simultaneous pose-invariant facial expression recognition, face image synthesis, and face alignment. The proposed deep model enjoys several merits. First, to the best of our knowledge, this is the first work to address these three tasks jointly in a unified deep model to complement and enhance each other. Second, the proposed model can effectively disentangle the global and local identity representation from different expression and geometry codes. As a result, it can automatically generate facial images with different expressions under arbitrary geometry codes. Third, these three tasks can further boost their performance for each other via our model. Extensive experimental results on three standard benchmarks demonstrate that the proposed deep model performs favorably against state-of-the-art methods on the three tasks. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation aims to transfer domain knowledge from existing well-defined tasks to new ones where labels are unavailable. In the real-world applications, domain discrepancy is usually uncontrollable especially for multi-modality data. Therefore, it is significantly motivated to deal with a multi-modality domain adaptation task. As labels are unavailable in a target domain, how to learn semantic multi-modality representations and successfully adapt the classifier from a source to the target domain remain open challenges in a multi-modality domain adaptation task. To deal with these issues, we propose a multi-modality adversarial network (MMAN), which applies stacked attention to learn semantic multi-modality representations and reduces domain discrepancy via adversarial training. Unlike the previous domain adaptation methods, which cannot make full use of source domain categories information, multi-channel constraint is employed to capture fine-grained categories of knowledge that could enhance the discrimination of target samples and boost target performance on single-modality and multi-modality domain adaptation problems. We apply the proposed MMAN to two applications including cross-domain object recognition and cross-domain social event recognition. The extensive experimental evaluations demonstrate the effectiveness of the proposed model for unsupervised domain adaptation. [ABSTRACT FROM AUTHOR]

The video highlight detection task is to localize key elements (moments of user's major or special interest) in a video. Most of existing highlight detection approaches extract features from the video segment as a whole without considering the difference of local features both temporally and spatially. Due to the complexity of video content, this kind of mixed features will impact the final highlight prediction. In temporal extent, not all frames are worth watching because some of them only contain the background of the environment without human or other moving objects. In spatial extent, it is similar that not all regions in each frame are highlights especially when there are lots of clutters in the background. To solve the above problem, we propose a novel three-dimensional (3-D) (spatial+temporal) attention model that can automatically localize the key elements in a video without any extra supervised annotations. Specifically, the proposed attention model produces attention weights of local regions along both the spatial and temporal dimensions of the video segment. The regions of key elements in the video will be strengthened with large weights. Thus, the more effective feature of the video segment is obtained to predict the highlight score. The proposed 3-D attention scheme can be easily integrated into a conventional end-to-end deep ranking model that aims to learn a deep neural network to compute the highlight score of each video segment. Extensive experimental results on the YouTube and SumMe datasets demonstrate that the proposed approach achieves significant improvement over state-of-the-art methods. With the proposed 3-D attention model, video highlights can be accurately retrieved in spatial and temporal dimensions without human supervision in several domains, such as gymnastics, parkour, skating, skiing, surfing, and dog activities, on the public datasets. [ABSTRACT FROM AUTHOR]

Relative attribute (RA) learning aims to learn the ranking function describing the relative strength of the attribute. Most of current learning approaches learn a linear ranking function for each attribute by use of the hand-crafted visual features. Different from the existing study, in this paper, we propose a novel deep relative attributes (DRA) algorithm to learn visual features and the effective nonlinear ranking function to describe the RA of image pairs in a unified framework. Here, visual features and the ranking function are learned jointly, and they can benefit each other. The proposed DRA model is comprised of five convolutional neural layers, five fully connected layers, and a relative loss function which contains the contrastive constraint and the similar constraint corresponding to the ordered image pairs and the unordered image pairs, respectively. To train the DRA model effectively, we make use of the transferred knowledge from the large scale visual recognition on ImageNet  [1] to the RA learning task. We evaluate the proposed DRA model on three widely used datasets. Extensive experimental results demonstrate that the proposed DRA model consistently and significantly outperforms the state-of-the-art RA learning methods. On the public OSR, PubFig, and Shoes datasets, compared with the previous RA learning results [2], the average ranking accuracies have been significantly improved by about $8\%$, $9\%$, and $14\%$, respectively. [ABSTRACT FROM PUBLISHER]

Most of existing multimodal event classification methods fuse the traditional hand-crafted features with some manually defined weights, which may be not suitable to the event classification task with large amounts of photos. Besides, the feature extraction and event classification model are always performed separately, which cannot capture the most useful features to describe the semantic concepts of complex events. To deal with these issues, we propose a novel discriminative multimodal embedding (DME) model for event classification in user generated photos by jointly learning the representation together with the classifier in a unified framework. In the proposed DME model, we can effectively resolve the multimodal, intra-class variation and inter-class confusion challenges by using the contrastive constraints on the multimodal event data. Extensive experimental results on two collected datasets demonstrate the effectiveness of the proposed DME model for event classification. [ABSTRACT FROM AUTHOR]