Search

Video captioning is a challenging task that aims to generate linguistic description based on video content. Most methods only incorporate visual features (2D/3D) as input for generating visual and non-visual words in the caption. However, generating non-visual words usually depends more on sentence-context than visual features. The wrong non-visual words can reduce the sentence fluency and even change the meaning of sentence. In this paper, we propose a syntax-guided hierarchical attention network (SHAN), which leverages semantic and syntax cues to integrate visual and sentence-context features for captioning. First, a globally-dependent context encoder is designed to extract the global sentence-context feature that facilitates generating non-visual words. Then, we introduce hierarchical content attention and syntax attention to adaptively integrate features in terms of temporality and feature characteristics respectively. Content attention helps focus on time intervals related to the semantic of current word, while cross-modal syntax attention uses syntax information to model importance of different features for target word’s generation. Moreover, such hierarchical attention can enhance the model interpretability for captioning. Experiments on MSVD and MSR-VTT datasets show the comparable performance of our method compared with current methods. [ABSTRACT FROM AUTHOR]

As a deep learning model, a deep belief network (DBN) consists of multiple restricted Boltzmann machines (RBMs). Based on DBN, many intelligent fault diagnosis methods are proposed. However, these methods seldom considered the appearance of new working conditions during the operation of real machines. Varying working conditions lead to a change of feature distributions and finally result in low diagnosis accuracies. Therefore, we propose a distribution-invariant DBN (DIDBN) to learn distribution-invariant features directly from raw vibration data. DIDBN consists of a locally connected RBM (LCRBM) layer, a fully connected RBM layer, and an RBM layer with a mean discrepancy maximum (MDM-RBM). The LCRBM layer is designed with a locally connected structure. By proposing MDM, the MDM-RBM layer is able to obtain features that have close distributions under varying working conditions. Followed by a softmax classifier, DIDBN is able to recognize faults. The proposed method is applied to two diagnosis cases. Results verify that DIDBN is able to learn distribution-invariant features and achieve higher diagnosis accuracies than recently proposed methods. Moreover, the reason why DIDBN is able to learn distribution-invariant features is explained by visualizing the feature learning process. [ABSTRACT FROM AUTHOR]

Cross-modal retrieval has received considerable attention in recent years. The core of cross-modal retrieval is to find a representation space to align data from different modalities according to their semantics. In this paper, we propose a cross-modal retrieval method that aligns data from different modalities by transferring one source modality to another target modality with augmented adversarial training. To preserve the semantic meaning in the modality transfer process, we employ the idea of conditional GANs and augment it. The key idea is to incorporate semantic information from the label space into the adversarial training process by sampling more semantic relevant and irrelevant source-target sample pairs. The augmented sample pairs improve the alignment from two aspects. First, relevant source-target sample pairs provide more training samples, leading to a better guidance of the alignment of fake targets and true paired targets. Second, relevant and irrelevant source-target sample pairs teach the discriminator to better distinguish true relevant pairs from fake relevant pairs, which guides the generator to better transfer from the source modality to the target modality. Extensive experiments compared with state-of-the-art methods show the promising power of our approach. [ABSTRACT FROM AUTHOR]

Visual dialog is a challenging task in multimedia understanding, which requires the dialog agent to answer a series of questions that are based on an input image. The critical issue to produce an exact answer is how to model the mutual semantic interaction among feature representations of the image, question-answer history, and current question. In this study, we propose a textual-visual Reference-Aware Attention Network (RAA-Net), which aims to effectively fuse $Q$ (question), $H$ (history), $V_{l}$ (local vision), and $V_{g}$ (global vision) to infer the exact answer. In the multimodal feature flows, RAA-Net first learns the textual context through multi-head attention between $Q$ and $H$ and then guides the textual reference semantics to the image to capture visual reference semantics by self- and cross- reference-aware attention in and between $V_{l}$ and $V_{g}$. In the proposed RAA-Net, we exploit the two-stage (intra- and inter-) visual reasoning mechanism on $V_{l}$ and $V_{g}$. Extensive experiments on the VisDial v0.9 and v1.0 datasets show that RAA-Net achieves state-of-the-art performance. Visualization results on both visual and textual attention maps further validate the remarkable interpretability achieved by our solution. [ABSTRACT FROM AUTHOR]

The semantic similarity among cross-modal data objects, e.g., similarities between images and texts, are recognized as the bottleneck of cross-modal retrieval. However, existing batch-style correlation learning methods suffer from prohibitive time complexity and extra memory consumption in handling large-scale high dimensional cross-modal data. In this paper, we propose a Cross-Modal Online Low-Rank Similarity function learning (CMOLRS) method, which learns a low-rank bilinear similarity measurement for cross-modal retrieval. We model the cross-modal relations by relative similarities on the training data triplets and formulate the relative relations as convex hinge loss. By adapting the margin in hinge loss with pair-wise distances in feature space and label space, CMOLRS effectively captures the multi-level semantic correlation and adapts to the content divergence among cross-modal data. Imposed with a low-rank constraint, the similarity function is trained by online learning in the manifold of low-rank matrices. The low-rank constraint not only endows the model learning process with faster speed and better scalability, but also improves the model generality. We further propose fast-CMOLRS combining multiple triplets for each query instead of standard process using single triplet at each model update step, which further reduces the times of gradient updates and retractions. Extensive experiments are conducted on four public datasets, and comparisons with state-of-the-art methods show the effectiveness and efficiency of our approach. [ABSTRACT FROM AUTHOR]

Alpha channel is being applied in an increasing number of mobile web applications on mobile devices that require ultra-low power consumption in all cases including compute-intensive video encoding and decoding. Thus, we propose an ultra-low coding complexity and high efficiency alpha channel lossless coding approach. A novel coding framework and four new coding schemes are proposed for alpha channel coding. The framework fuses a string matching technique and a proposed prediction coding scheme named bit-depth preserving prediction (BDPP) together to reduce the correlations within and between repeated identical patterns and neighboring pixels. To achieve a good tradeoff between complexity and efficiency, either the unmatchable bytes are coded directly or the BDPP residuals of unmatchable bytes are coded by a proposed bytewise entropy coding scheme named 0.5-1-2byte-size-code. The other string matching parameters are coded by another proposed bytewise entropy coding scheme named byte-size multi-variable-length-code. To speed up the string-matching search, we apply a fast string search scheme that combines special position search and hash-based search. For the selected typical 236 alpha test images, compared with x265 in the fastest configuration and lossless mode, the proposed lossless approach achieves 14.33% less total compressed bytes with only 2.75% encoding and 1.83% decoding runtime. The proposed approach also outperforms the conventional lossless coding techniques such as LZ4HC, ZLIB, and PNG. [ABSTRACT FROM AUTHOR]

Multi-view representation learning plays a fundamental role in multimedia data analysis. Some specific inter-view alignment principles are adopted in conventional models, where there is an assumption that different views share a common latent subspace. However, when dealing views on diverse semantic levels, the view-specific characteristics are neglected, and the divergent inconsistency of similarity measurements hinders sufficient information sharing. This paper proposes a hybrid deep network by introducing tensor factorization into the multi-view deep auto-encoder. The network adopts skeleton-embedding process for unsupervised multi-view subspace learning. It takes full consideration of view-specific characteristics, and leverages the strength of both shallow and deep architectures for modeling low- and high-level views, respectively. We first formulate the high-level-view semantic distribution as the underlying skeleton structure of the learned subspace, and then infer the local tangent structures according to the affinity propagation of low-level-view geometric correlations. As a consequence, more discriminative subspace representation can be learned from global semantic pivots to local geometric details. Experimental comparisons on three benchmark image datasets show the promising performance and flexibility of our model. [ABSTRACT FROM AUTHOR]

Cross-modal retrieval has attracted intensive attention in recent years, where a substantial yet challenging problem is how to measure the similarity between heterogeneous data modalities. Despite using modality-specific representation learning techniques, most existing shallow or deep models treat different modalities equally and neglect the intrinsic modality heterogeneity and information imbalance among images and texts. In this paper, we propose an online similarity function learning framework to learn the metric that can well reflect the cross-modal semantic relation. Considering that multiple CNN feature layers naturally represent visual information from low-level visual patterns to high-level semantic abstraction, we propose a new asymmetric image–text similarity formulation which aggregates the layer-wise visual–textual similarities parameterized by different bilinear parameter matrices. To effectively learn the aggregated similarity function, we develop three different similarity combination strategies, i.e., average kernel, multiple kernel learning, and layer gating. The former two kernel-based strategies assign uniform weights on different layers to all data pairs; the latter works on the original feature representation and assigns instance-aware weights on different layers to different data pairs, and they are all learned by preserving the bi-directional relative similarity expressed by a large number of cross-modal training triplets. The experiments conducted on three public datasets well demonstrate the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

The Marx circuit structure based on fast high-power semiconductor devices is the classical way of generating picosecond pulses. However, most reported studies suggest that as the number of stages in the Marx circuit increases, the output pulse amplitude tends to saturate. This has of late made high-voltage picosecond-pulse generation a challenge. In this paper, as the number of stages increases, the internal resistance of the avalanche transistor tends to decrease with increasing current flowing through the avalanche transistor. A step-by-step wiring and debugging of the avalanche transistor-based Marx circuit demonstrates this experimentally. That is, for a certain range in the number of stages, saturation of the output pulse amplitude for the Marx circuit of avalanche transistors is not apparent. Introducing a parallel structure for the transistors not only increases the current level of the entire circuit but also further reduces the equivalent resistance of the avalanche transistors, thereby improving the output efficiency. Drawing on microstrip transmission theory, and combining the Marx circuit and avalanche transistor parallel structure, component parameters and circuit topology of the picosecond-pulse generator were redesigned, and a 30-level Marx circuit with a parallel structure of two avalanche transistors was developed. The picosecond-pulse generator outputs a pulse with adjustable voltage amplitude of 0.9–3.1 kV, a 350-ps full-width at half-maximum, a 150-ps rise time, and an adjustable high-stability repetition rate of up to 10 kHz. The number of pulses is precisely controlled. The generator is an all-solid-state compact device with high frequency suitable for research needs. [ABSTRACT FROM AUTHOR]

This paper proposes a universal string matching (USM) approach to screen content coding (SCC). USM uses a primary reference buffer and a secondary reference buffer for string matching and includes three modes: general string (GS) mode, constrained string 1 (CS1) mode, and constrained string 2 (CS2) mode. The CS1 mode and CS2 mode are constrained cases of the GS mode. Due to the diversity of the screen content, each of the three modes plays an indispensable role in coding some types of screen content. When using USM to code a coding unit (CU), one of the three modes is selected to code the CU. Compared with high-efficiency video coding (HEVC) SCC reference software HM-16.6 + SCM-5.2 of full frame search range for intrablock copy, USM achieves an average Y BD-rate of –28.4% for five text and graphics with motion (TGM) sequences from the audio video coding standard SCC common test condition (CTC) test suite and –5.8% for eight TGM test sequences from the HEVC SCC CTC test suite in all intraconfigurations, with a nearly 10% decrease in encoding runtime and almost the same decoding runtime. [ABSTRACT FROM PUBLISHER]

Different kinds of features describe different aspects of image data, and each feature can be treated as a view when we take it as a particular understanding of images. Leveraging multiple views provides a richer and comprehensive description than using only a single view. However, multiview data are often represented by high-dimensional heterogeneous features, so it is meaningful to find a low-dimensional consensus representation from multiple views. In this paper, we propose an unsupervised multiview dimensionality reduction method for images based on bilevel latent space learning. As different views have different physical meanings and statistical properties, they are not directly comparable. Therefore, we learn the comparable representation for each view in the first level. The shared and the private nature of multiview data are exploited to accurately preserve the information of each view. Then, we fuse different views into a low-dimensional representation by conducting joint matrix factorization in the second level. To guarantee the low-dimensional representation to be compact and discriminative, the intrinsic geometric structure of data is utilized. Besides, our method considers resisting the outliers and noise contained in multiview data, which may influence the learned representation and deteriorate its semantic consistency. We design appropriate optimization objectives to learn the latent spaces in different levels. Compared with the existing methods, our method could provide a more flexible multiview learning strategy that not only accurately captures the information of each view but also is robust to outliers and noise, which can obtain a more discriminative and compact low-dimensional representation. Experiments on two real-world image data sets demonstrate the advantages of our method over the existing multiview dimensionality reduction methods. [ABSTRACT FROM AUTHOR]