Search

Visible-infrared person re-identification (VI-ReID) task aims to retrieve the same person between visible and infrared images. VI-ReID is challenging as the images captured by different spectra present large cross-modality discrepancy. Many methods adopt a two-stream network and design additional constraint conditions to extract shared features for different modalities. However, the interaction between the feature extraction processes of different modalities is rarely considered. In this paper, a partially interactive collaboration method is proposed to exploit the complementary information of different modalities to reduce the modality gap for VI-ReID. Specifically, the proposed method is achieved in a partially interactive-shared architecture: collaborative shallow layers and shared deep layers. The collaborative shallow layers consider the interaction between modality-specific features of different modalities, encouraging the feature extraction processes of different modalities constrain each other to enhance feature representations. The shared deep layers further embed the modality-specific features to a common space to endow them the same identity discriminability. To ensure the interactive collaborative learning implement effectively, the conventional loss and collaborative loss are utilized jointly to train the whole network. Extensive experiments on two publicly available VI-ReID datasets verify the superiority of the proposed PIC method. Specifically, the proposed method achieves a rank-1 accuracy of 83.6% and 57.5% on RegDB and SYSU-MM01 datasets, respectively. [ABSTRACT FROM AUTHOR]

Remote sensing images (RSIs) containing various ground objects have been applied in many fields. To make semantic understanding of RSIs objective and interactive, the task remote sensing visual question answering (VQA) has appeared. Given an RSI, the goal of remote sensing VQA is to make an intelligent agent answer a question about the remote sensing scene. Existing remote sensing VQA methods utilized a nonspatial fusion strategy to fuse the image features and question features, which ignores the spatial information of images and word-level information of questions. A novel method is proposed to complete the task considering these two aspects. First, convolutional features of the image are included to represent spatial information, and the word vectors of questions are adopted to present semantic word information. Second, attention mechanism and bilinear technique are introduced to enhance the feature considering the alignments between spatial positions and words. Finally, a fully connected layer with softmax is utilized to output an answer from the perspective of the multiclass classification task. To benchmark this task, a RSIVQA dataset is introduced in this article. For each of more than 37 000 RSIs, the proposed dataset contains at least one or more questions, plus corresponding answers. Experimental results demonstrate that the proposed method can capture the alignments between images and questions. The code and dataset are available at https://github.com/spectralpublic/RSIVQA. [ABSTRACT FROM AUTHOR]

The land cover classification has been an important task in remote sensing. With the development of various sensors technologies, carrying out classification work with multisource remote sensing (MSRS) data has shown an advantage over using a single type of data. Hyperspectral images (HSIs) are able to represent the spectral properties of land cover, which is quite common for land cover understanding. Light detection and ranging (LiDAR) images contain altitude information of the ground, which is greatly helpful with urban scene analysis. Current HSI and LiDAR fusion methods perform feature extraction and feature fusion separately, which cannot well exploit the correlation of data sources. In order to make full use of the correlation of multisource data, an unsupervised feature extraction-fusion network for HSI and LiDAR, which utilizes feature fusion to guide the feature extraction procedure, is proposed in this article. More specifically, the network takes multisource data as input and directly output the unified fused feature. A multimodal graph is constructed for feature fusion, and graph-based loss functions including Laplacian loss and t-distributed stochastic neighbor embedding (t-SNE) loss are utilized to constrain the feature extraction network. Experimental results on several data sets demonstrate the proposed network can achieve more excellent classification performance than some state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Person re-identification (ReID) task aims to retrieve the same person across multiple spatially disjoint camera views. Due to huge image changes caused by various factors such as posture variation and illumination transformation, images of different persons may share the more similar appearances than images of the same one. Learning discriminative representations to distinguish details of different persons is significant for person ReID. Many existing methods learn discriminative representations resorting to a human body part location branch which requires cumbersome expert human annotations or complex network designs. In this article, a novel bidirectional interaction network is proposed to explore discriminative representations for person ReID without any human body part detection. The proposed method regards multiple convolutional features as responses to various body part properties and exploits the inter-layer interaction to mine discriminative representations for person identities. Firstly, an inter-layer bilinear pooling strategy is proposed to feasibly exploit the pairwise feature relations between two convolution layers. Secondly, to explore interaction of multiple layers, an effective bidirectional integration strategy consisting of two different multi-layer interaction processes is designed to aggregate bilinear pooling interaction of multiple convolution layers. The interaction of multiple layers is implemented in a layer-by-layer nesting policy to ensure the two interaction processes are different and complementary. Extensive experiments validate the superiority of the proposed method on four popular person ReID datasets including Market-1501, DukeMTMC-ReID, CUHK03-NP and MSMT17. Specifically, the proposed method achieves a rank-1 accuracy of 95.1% and 88.2% on Market-1501 and DukeMTMC-ReID, respectively. [ABSTRACT FROM AUTHOR]

Remote sensing image (RSI) classification is one of the most important fields in RSI processing. It is well known that RSIs are very complicated due to its various kinds of contents. Therefore, it is very difficult to distinguish different scene categories with similar visual contents, like desert and bare land. To address hard negative categories, an attribute-cooperated convolutional neural network (ACCNN) is proposed to exploit attributes as additional guiding information. First, the classification branch extracts convolutional neural network feature, which is then utilized to recognize the RSI scene categories. Second, the attribute branch is proposed to make the network distinguish scene categories efficiently. The proposed attribute branch shares feature extraction layers with the classification branch and makes the classification branch aware of extra attribute information. Finally, the relationship branch constraints the relationship between the classification branch and the attribute branch. To exploit the attribute information, three attribute-classification data sets are generated (AC-AID, AC-UCM, and AC-Sydney). Experimental results show that the proposed method is competitive to state-of-the-art methods. The data sets are available at https://github.com/CrazyStoneonRoad/Attribute-Cooperated-Classification-Data sets. [ABSTRACT FROM AUTHOR]

Cross-domain scene classification refers to the scene classification task in which the training set (termed source domain) and the test set (termed target domain) come from different distributions. Various domain adaptation methods have been developed to reduce the distribution discrepancy between different domains. However, current domain adaptation methods assume that the source domain and target domain share the same categories. In reality, it is hard to find a source domain that can completely cover all the categories of target domain. In this article, we propose to use multiple complementary source domains to form the categories of target domain. A multisource compensation network (MSCN) is proposed to tackle these challenges: distribution discrepancy and category incompleteness. First, a pretrained convolutional neural network (CNN) is exploited to learn the feature representation for each domain. Second, a cross-domain alignment module is developed to reduce the domain shift between source and target domains. Domain shift is reduced by mapping the two domain features into a common feature space. Finally, a classifier complement module is proposed to align categories in multiple sources and learn a target classifier. Two cross-domain classification data sets are constructed using four heterogeneous remote sensing scene classification data sets. Extensive experiments are conducted on these datasets to validate the effectiveness of the proposed method. The proposed method can achieve 81.23% and 81.97% average accuracies on two-source-complementary data set and three-source-complementary data set, respectively. [ABSTRACT FROM AUTHOR]

Remote sensing (RS) scene classification is a challenging task due to various land covers contained in RS scenes. Recent RS classification methods demonstrate that aggregating the multilayer convolutional features, which are extracted from different hierarchical layers of a convolutional neural network, can effectively improve classification accuracy. However, these methods treat the multilayer convolutional features as equally important and ignore the hierarchical structure of multilayer convolutional features. Multilayer convolutional features not only provide complementary information for classification but also bring some interference information (e.g., redundancy and mutual exclusion). In this paper, a gated bidirectional network is proposed to integrate the hierarchical feature aggregation and the interference information elimination into an end-to-end network. First, the performance of each convolutional feature is quantitatively analyzed and a superior combination of convolutional features is selected. Then, a bidirectional connection is proposed to hierarchically aggregate multilayer convolutional features. Both the top–down direction and the bottom–up direction are considered to aggregate multilayer convolutional features into the semantic-assist feature and appearance-assist feature, respectively, and a gated function is utilized to eliminate interference information in the bidirectional connection. Finally, the semantic-assist feature and appearance-assist feature are merged for classification. The proposed method can compete with the state-of-the-art methods on four RS scene classification data sets (AID, UC-Merced, WHU-RS19, and OPTIMAL-31). [ABSTRACT FROM AUTHOR]

Image captioning has attracted more and more attention in remote sensing filed since it provides more specific information than traditional tasks, such as classification. Though image captioning has gained some developments in recent years, it is difficult to describe the image in one simple sentence. To relieve the limitation, a novel captioning task is proposed and a novel framework is proposed to solve the novel task. The proposed framework uses semantic embedding to measure the image representation and the sentence representation. The captioning performance is improved by a proposed sentence representation (collective representation). Experimental results and human evaluations on three captioning data sets in remote sensing field demonstrate that the proposed framework can lead to advancement in image captioning results. [ABSTRACT FROM AUTHOR]