Search

The main task of cross-modal analysis is to learn discriminative representation shared across different modalities. In order to pursue aligned representation, conventional approaches tend to construct and optimize a linear projection or train a complex architecture of deep layers, yet it is difficult to compromise between accuracy and efficiency on modeling multimodal data. This paper proposes a novel graph-embedding learning framework implemented by neural networks. The learned embedding directly approximates the cross-modal aligned representation to perform cross-modal retrieval and image classification combining text information. Proposed framework extracts learned representation from a graph model and, simultaneously, trains a classifier under semi-supervised settings. For optimization, unlike previous methods based on the graph Laplacian regularization, a sampling strategy is adopted to generate training pairs to fully explore the inter-modal and intra-modal similarity relationship. Experimental results on various datasets show that the proposed framework outperforms other state-of-the-art methods on cross-modal retrieval. The framework also demonstrates convincing improvements on the new issue of image classification combining text information on Wiki dataset.

Scarcity of labeled samples is the main obstacle for hyperspectral image classification tasks when labeling data is considerably costly and time-consuming in real-world scenarios. To alleviate any underfitting problem that may occur due to lack of training data, semisupervised classification frameworks explore the intrinsic information of unlabeled samples and bridge labeled and unlabeled data. In this letter, we propose a novel framework that learns underlying manifold representation and semisupervised classifier simultaneously. It avoids explicit eigenvector decomposition and directly samples via iterating random walk on the similarity graph, which makes it feasible to implement on huge graphs. To verify the efficacy of embedding the learning process, we compare the proposed method with other dimensionality reduction and manifold-learning-based approaches. Experimental results show that compared to the methods using traditional semisupervised strategies, the graph embedding method gives a better result.

To combat the well-known Hughes phenomenon occurred in hyperspectral classification, most of the previous works adopt dimensionality reduction or manifold learning technique before supervised learning. While in this paper, we propose a different scheme: First, we design a pixel-wise classifier based on Convolutional Neural Network that could directly mapping observed spectrum to class distribution. Then, we conduct superpixel segmentation on the prediction map that learned by previous model and output the final classification results by spatial and spectral factors jointly. Varied from other deep learning method, our classification framework learns and infers spectrum efficiently via deep hierarchy with convolutional and pooling layers, thus forming a direct relationship between high-order data and class distribution. Moreover, superpixel segmentation helps further boost the accuracy of the classification by combining the spatial information. In experimental studies, multiple hyperspectral datasets with various context and spatial resolution are used to validate the proposed method. The experimental results show that the proposed method is efficient and competitive in practical uses.