Search

Visual question answering (VQA) has gained increasing attention in both natural language processing and computer vision. The attention mechanism plays a crucial role in relating the question to meaningful image regions for answer inference. However, most existing VQA methods: 1) learn the attention distribution either from free-form regions or detection boxes in the image, which is intractable in answering questions about the foreground object and background form, respectively and 2) neglect the prior knowledge of human attention and learn the attention distribution with an unguided strategy. To fully exploit the advantages of attention, the learned attention distribution should focus more on the question-related image regions, such as human attention for both the questions, about the foreground object and background form. To achieve this, this article proposes a novel VQA model, called adversarial learning of supervised attentions (ALSAs). Specifically, two supervised attention modules: 1) free form-based and 2) detection-based, are designed to exploit the prior knowledge for attention distribution learning. To effectively learn the correlations between the question and image from different views, that is, free-form regions and detection boxes, an adversarial learning mechanism is implemented as an interplay between two supervised attention modules. The adversarial learning reinforces the two attention modules mutually to make the learned multiview features more effective for answer inference. The experiments performed on three commonly used VQA datasets confirm the favorable performance of ALSA.

Learning the representation for social images has recently made remarkable achievements for many tasks, such as cross-modal retrieval and multilabel classification. However, since social images contain both multimodal contents (e.g., visual images and textual descriptions) and social relations among images, simply modeling the content information may lead to suboptimal embedding. In this paper, we propose a novel multimodal representation learning model for social images, that is, correlational multimodal variational autoencoder (CMVAE) via triplet network. Specifically, in order to mine the highly nonlinear correlation between the visual content and the textual content, a CMVAE is proposed to learn a unified representation for the multiple modalities of social images. Both common information in all modalities and private information in each modality are encoded for the representation learning. To incorporate the social relations among images, we employ the triplet network to embed multiple types of social links in the representation learning. Then, a joint embedding model is proposed to combine the social relations for representation learning of the multimodal contents. Comprehensive experiment results on four datasets confirm the effectiveness of our method in two tasks, namely, multilabel classification and cross-modal retrieval. Our method outperforms the state-of-the-art multimodal representation learning methods with significant improvement of performance.

Abstract Multi-view representation learning for social images has recently made remarkable achievements in many tasks, such as cross-view classification and cross-modal retrieval. Since social images usually contain link information besides the multi-modal contents (e.g., text description, and visual content), simply employing the data content may result in sub-optimal multi-view representation of the social images. In this paper, we propose a Deep Multi-View Embedding Model (DMVEM) to learn joint embeddings for the three views including the visual content, the associated text descriptions, and their relations. To effectively encode the link information, a weighted relation network is built based on the linkages between social images, which is then embedded into a low dimensional vector space using the Skip-Gram model. The learned vector is regarded as the third view besides the visual content and text description. To learn a joint representation from the three views, a deep learning model with three-branch nonlinear neural network is proposed. A three-view bi-directional loss function is used to capture the correlation between the three views. The stacked autoencoder is adopted to preserve the self-structure and reconstructability of the learned representation for each view. Comprehensive experiments are conducted in the tasks of image-to-text, text-to-image, and image-to-image searches. Compared to the state-of-the-art multi-view embedding methods, our approach achieves significant improvement of performance. Highlights • We investigate the problem of multi-view learning for the linkage-embedded social images using deep model. • The learned representation can effectively capture both the content and the network structure. • We propose a deep multi-view model DMVEM to effectively learn a joint representation from the three views of social image. • The experimental results demonstrate the superiority of the proposed model against the state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

There is growing interest in social image classification because of its importance in web-based image application. Though there are many approaches on image classification, it is still a great problem to integrate multi-modal contents of social images simultaneously for classification, since the textual content and visual content are represented in two heterogeneous feature spaces. In this study, a multi-modal learning algorithm is proposed to fuse the multiple features through their correlation seamlessly. Specifically, two classification modules based on the kernel ridge regression (KRR) are learned for the two types of features, and they are integrated via a joint model. With the joint model, the classification based on visual features can be reinforced by the classification based on textual features, and vice verse. Then, an efficient optimization method is proposed to resolving the object function. The query image can be classified based on both of the textual features and visual features by combing the results of the two classifiers. Two methods are proposed to combine the classification results to obtain the final result. To evaluate the approach, extensive experiments are conducted on the real-world datasets, and the result demonstrates the superiority of our approach. [ABSTRACT FROM AUTHOR]

In a real e-commerce website, usually only a small number of users will give ratings to the items they purchased, and this can lead to the very sparse user-item rating data. The data sparsity issue will greatly limit the recommendation performance of most recommendation algorithms. However, a user may register accounts in many e-commerce websites. If such users’ historical purchasing data on these websites can be integrated, the recommendation performance could be improved. But it is difficult to align the users and items between these websites, and thus how to effectively borrow the users’ rating data of one website (source domain) to help improve the recommendation performance of another website (target domain) is very challenging. To this end, this paper extended the traditional one-dimensional psychometrics model to multidimension. The extended model can effectively capture users’ multiple interests. Based on this multidimensional psychometrics model, we further propose a novel transfer learning algorithm. It can effectively transfer users’ rating preferences from the source domain to the target domain. Experimental results show that the proposed method can significantly improve the recommendation performance. [ABSTRACT FROM AUTHOR]

• Define three types of features extracted from URLs, domains, etc. • Exploit a method to balance the majority and minority class samples. • Adopt an improved DAE-based method to reduce the dimension of the dataset. • Boost the detection performance by using the improved ELM-based classifier. • Do experiments to verify the feasibility and effectiveness of the proposed approach. In this paper, a novel approach based on non-inverse matrix online sequence extreme learning machine (NIOSELM) for phishing detection is presented, which takes into account three types of features to comprehensively characterize a website. For the NIOSELM algorithm, we use Sherman Morriso Woodbury equation to avoid the matrix inversion operation, and introduce the idea of online sequence extreme learning machine (OSELM) to update the training model. In order to reduce the dependence of the detection model on the majority class, we use Adaptive Synthetic Sampling (ADASYN) algorithm to generate the synthetic minority class samples to balance the distribution between the samples of the majority and minority classes. Furthermore, an improved denoising auto-encoder (SDAE) is designed to reduce the dimension of the experimental dataset. The experimental results show the efficiency and feasibility of the proposed detection mechanism. Moreover, the overall detection performance of NIOSELM is better than that of other existing methods, especially in training speed and the detection accuracy. [ABSTRACT FROM AUTHOR]