Search

Video captioning is a challenging task that aims to generate linguistic description based on video content. Most methods only incorporate visual features (2D/3D) as input for generating visual and non-visual words in the caption. However, generating non-visual words usually depends more on sentence-context than visual features. The wrong non-visual words can reduce the sentence fluency and even change the meaning of sentence. In this paper, we propose a syntax-guided hierarchical attention network (SHAN), which leverages semantic and syntax cues to integrate visual and sentence-context features for captioning. First, a globally-dependent context encoder is designed to extract the global sentence-context feature that facilitates generating non-visual words. Then, we introduce hierarchical content attention and syntax attention to adaptively integrate features in terms of temporality and feature characteristics respectively. Content attention helps focus on time intervals related to the semantic of current word, while cross-modal syntax attention uses syntax information to model importance of different features for target word’s generation. Moreover, such hierarchical attention can enhance the model interpretability for captioning. Experiments on MSVD and MSR-VTT datasets show the comparable performance of our method compared with current methods. [ABSTRACT FROM AUTHOR]

Cross-modal retrieval has received considerable attention in recent years. The core of cross-modal retrieval is to find a representation space to align data from different modalities according to their semantics. In this paper, we propose a cross-modal retrieval method that aligns data from different modalities by transferring one source modality to another target modality with augmented adversarial training. To preserve the semantic meaning in the modality transfer process, we employ the idea of conditional GANs and augment it. The key idea is to incorporate semantic information from the label space into the adversarial training process by sampling more semantic relevant and irrelevant source-target sample pairs. The augmented sample pairs improve the alignment from two aspects. First, relevant source-target sample pairs provide more training samples, leading to a better guidance of the alignment of fake targets and true paired targets. Second, relevant and irrelevant source-target sample pairs teach the discriminator to better distinguish true relevant pairs from fake relevant pairs, which guides the generator to better transfer from the source modality to the target modality. Extensive experiments compared with state-of-the-art methods show the promising power of our approach. [ABSTRACT FROM AUTHOR]

Visual dialog is a challenging task in multimedia understanding, which requires the dialog agent to answer a series of questions that are based on an input image. The critical issue to produce an exact answer is how to model the mutual semantic interaction among feature representations of the image, question-answer history, and current question. In this study, we propose a textual-visual Reference-Aware Attention Network (RAA-Net), which aims to effectively fuse $Q$ (question), $H$ (history), $V_{l}$ (local vision), and $V_{g}$ (global vision) to infer the exact answer. In the multimodal feature flows, RAA-Net first learns the textual context through multi-head attention between $Q$ and $H$ and then guides the textual reference semantics to the image to capture visual reference semantics by self- and cross- reference-aware attention in and between $V_{l}$ and $V_{g}$. In the proposed RAA-Net, we exploit the two-stage (intra- and inter-) visual reasoning mechanism on $V_{l}$ and $V_{g}$. Extensive experiments on the VisDial v0.9 and v1.0 datasets show that RAA-Net achieves state-of-the-art performance. Visualization results on both visual and textual attention maps further validate the remarkable interpretability achieved by our solution. [ABSTRACT FROM AUTHOR]

The semantic similarity among cross-modal data objects, e.g., similarities between images and texts, are recognized as the bottleneck of cross-modal retrieval. However, existing batch-style correlation learning methods suffer from prohibitive time complexity and extra memory consumption in handling large-scale high dimensional cross-modal data. In this paper, we propose a Cross-Modal Online Low-Rank Similarity function learning (CMOLRS) method, which learns a low-rank bilinear similarity measurement for cross-modal retrieval. We model the cross-modal relations by relative similarities on the training data triplets and formulate the relative relations as convex hinge loss. By adapting the margin in hinge loss with pair-wise distances in feature space and label space, CMOLRS effectively captures the multi-level semantic correlation and adapts to the content divergence among cross-modal data. Imposed with a low-rank constraint, the similarity function is trained by online learning in the manifold of low-rank matrices. The low-rank constraint not only endows the model learning process with faster speed and better scalability, but also improves the model generality. We further propose fast-CMOLRS combining multiple triplets for each query instead of standard process using single triplet at each model update step, which further reduces the times of gradient updates and retractions. Extensive experiments are conducted on four public datasets, and comparisons with state-of-the-art methods show the effectiveness and efficiency of our approach. [ABSTRACT FROM AUTHOR]

Multi-view representation learning plays a fundamental role in multimedia data analysis. Some specific inter-view alignment principles are adopted in conventional models, where there is an assumption that different views share a common latent subspace. However, when dealing views on diverse semantic levels, the view-specific characteristics are neglected, and the divergent inconsistency of similarity measurements hinders sufficient information sharing. This paper proposes a hybrid deep network by introducing tensor factorization into the multi-view deep auto-encoder. The network adopts skeleton-embedding process for unsupervised multi-view subspace learning. It takes full consideration of view-specific characteristics, and leverages the strength of both shallow and deep architectures for modeling low- and high-level views, respectively. We first formulate the high-level-view semantic distribution as the underlying skeleton structure of the learned subspace, and then infer the local tangent structures according to the affinity propagation of low-level-view geometric correlations. As a consequence, more discriminative subspace representation can be learned from global semantic pivots to local geometric details. Experimental comparisons on three benchmark image datasets show the promising performance and flexibility of our model. [ABSTRACT FROM AUTHOR]

Cross-modal retrieval has attracted intensive attention in recent years, where a substantial yet challenging problem is how to measure the similarity between heterogeneous data modalities. Despite using modality-specific representation learning techniques, most existing shallow or deep models treat different modalities equally and neglect the intrinsic modality heterogeneity and information imbalance among images and texts. In this paper, we propose an online similarity function learning framework to learn the metric that can well reflect the cross-modal semantic relation. Considering that multiple CNN feature layers naturally represent visual information from low-level visual patterns to high-level semantic abstraction, we propose a new asymmetric image–text similarity formulation which aggregates the layer-wise visual–textual similarities parameterized by different bilinear parameter matrices. To effectively learn the aggregated similarity function, we develop three different similarity combination strategies, i.e., average kernel, multiple kernel learning, and layer gating. The former two kernel-based strategies assign uniform weights on different layers to all data pairs; the latter works on the original feature representation and assigns instance-aware weights on different layers to different data pairs, and they are all learned by preserving the bi-directional relative similarity expressed by a large number of cross-modal training triplets. The experiments conducted on three public datasets well demonstrate the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

Recently, thanks to the state-of-the-art techniques in Generative Adversarial Networks, a lot of work achieves remarkable performance on learning the mapping between an input image and an output image without any paired relation. However, traditional methods on image-to-image translation merely consider the visual appearance properties, they fail to maintain the true semantics of an image during the transfer learning procedure from source to target domain. We propose a new approach that utilizes GAN to translate unpaired images between domains and remain high level semantic abstraction aligned. Our model controls the hierarchical semantics of images by processing semantic information on label level and spatial level respectively by constructing label and attention consistent losses. The experimental results on several benchmark datasets show that generated samples are both visually similar with target images and semantically consistent with their source counterparts. Furthermore, the experiment also suggests that our method can effectively improve the classification performance in unsupervised domain adaptation problem. [ABSTRACT FROM AUTHOR]

Data from real applications involve multiple modalities representing content with the same semantics from complementary aspects. However, relations among heterogeneous modalities are simply treated as observation-to-fit by existing work, and the parameterized modality specific mapping functions lack flexibility in directly adapting to the content divergence and semantic complicacy in multimodal data. In this paper, we build our work based on the Gaussian process latent variable model (GPLVM) to learn the non-parametric mapping functions and transform heterogeneous modalities into a shared latent space. We propose multimodal Similarity Gaussian Process latent variable model (m-SimGP), which learns the mapping functions between the intra-modal similarities and latent representation. We further propose multimodal distance-preserved similarity GPLVM (m-DSimGP) to preserve the intra-modal global similarity structure, and multimodal regularized similarity GPLVM (m-RSimGP) by encouraging similar/dissimilar points to be similar/dissimilar in the latent space. We propose m-DRSimGP, which combines the distance preservation in m-DSimGP and semantic preservation in m-RSimGP to learn the latent representation. The overall objective functions of the four models are solved by simple and scalable gradient decent techniques. They can be applied to various tasks to discover the nonlinear correlations and to obtain the comparable low-dimensional representation for heterogeneous modalities. On five widely used real-world data sets, our approaches outperform existing models on cross-modal content retrieval and multimodal classification. [ABSTRACT FROM AUTHOR]

In this paper, we detect communities from trajectories. Existing algorithms for trajectory clustering usually rely on simplex representation and a single proximity-related metric. Unfortunately, additional information markers (e.g., social interactions or semantics in the spatial layout) are ignored, leading to the inability to fully discover the communities in trajectory database. This is especially true for human-generated trajectories, where additional fine-grained markers (e.g., movement velocity at certain locations, or the sequence of semantic spaces visited) are especially useful in capturing latent relationships among community members. To overcome this limitation, we propose TODMIS, a general framework for Trajectory-based cOmmunity Detection by diffusion modeling on Multiple Information Sources. TODMIS combines additional information with raw trajectory data and construct the diffusion process on multiple similarity metrics. It also learns the consistent graph Laplacians by constructing the multi-modal diffusion process and optimizing the heat kernel coupling on each pair of similarity matrices from multiple information sources. Then, dense sub-graph detection is used to discover the set of distinct communities (including community size) on the coupled multi-graph representation. At last, based on the community information, we propose a novel model for online recommendation. We evaluate TODMIS and our online recommendation methods using different real-life datasets. Experimental results demonstrate the effectiveness and efficiency of our methods. [ABSTRACT FROM AUTHOR]

With the explosive growth of web data, effective and efficient technologies are in urgent need for retrieving semantically relevant contents of heterogeneous modalities. Previous studies devote efforts to modeling simple cross-modal statistical dependencies, and globally projecting the heterogeneous modalities into a measurable subspace. However, global projections cannot appropriately adapt to diverse contents, and the naturally existing multilevel semantic relation in web data is ignored. We study the problem of semantic coherent retrieval, where documents from different modalities should be ranked by the semantic relevance to the query. Accordingly, we propose TINA, a correlation learning method by adaptive hierarchical semantic aggregation. First, by joint modeling of content and ontology similarities, we build a semantic hierarchy to measure multilevel semantic relevance. Second, with a set of local linear projections and probabilistic membership functions, we propose two paradigms for local expert aggregation, i.e., local projection aggregation and local distance aggregation. To learn the cross-modal projections, we optimize the structure risk objective function that involves semantic coherence measurement, local projection consistency, and the complexity penalty of local projections. Compared to existing approaches, a better bias-variance tradeoff is achieved by TINA in real-world cross-modal correlation learning tasks. Extensive experiments on widely used NUS-WIDE and ICML-Challenge for image–text retrieval demonstrate that TINA better adapts to the multilevel semantic relation and content divergence, and, thus, outperforms state of the art with better semantic coherence. [ABSTRACT FROM PUBLISHER]

Semantic Inpainting on Segmentation Map (SISM) aims to manipulate segmentation maps by semantics. Recent works show SISM provides semantic-aware auxiliary information for better style or structure manipulations. Providing structural assistance, segmentation maps have been broadly used as an intermediate interface to achieve better image manipulation. Mainstream solutions of image manipulation use Generative Adversarial Net (GAN) globally, locally or jointly. It is also applicable to SISM. However, the discriminator of global GAN is easier fooled, because the majority of its input is the same as the ground-truth, which is hard to fully mitigate the inconsistency between inpainted areas and the context. The inconsistency is more difficult for local GAN to address, due to the lack of context in its input. To mitigate the inconsistency, we propose a novel Multi-Expansion (MEx) loss. It is implemented by the adversarial loss on MEx areas. Each MEx area has the inpainted area as dominance and keeps knowledge of the scene context, so the consistency of the SISM results can be boosted. We propose an approximation of MEx loss, i.e., A-MEx loss, to further enhance the stability and usability. Besides performing well on SISM tasks, MEx loss also performs impressively on natural image inpainting. Extensive experiments on the two tasks demonstrate the advantages of our model over existing methods on four challenging datasets, such as a 2.59% increase in Hamm on SISM in Cityscape and a decrease of 5.00% FID on natural image inpainting in CMP Facade. The code of our work is available at: https://github.com/he159ok/AMEx-MEx-Loss. [ABSTRACT FROM AUTHOR]