Search

The multispectral (MS) and the panchromatic (PAN) images represent complementary and synergistic spatial spectral information, how to make optimal use of the advantages of them has become a hot research topic. This paper proposes a selectable Transformer and Gist CNN network (STGC-Net). It designs a subspace similar recombination module (SSR-Module) based on non-negative matrix factorization (NMF) and the self-attention mechanism for feature decomposition. This can alleviate the redundant information of multi-modal data and extract their own singular and common features. Considering that the MS and the PAN images exhibit different advantageous properties, a selectable self-attention spectral feature extraction module (S 3 FE-Module) and a multi-stream Gist spatial feature extraction module (MGSFE-Module) are proposed for the different singular features. The former can refine the Transformer's input and simultaneously characterize the sequence information between channels for the MS image. The latter introduces the positional relationship between local features while extracting spatial features for the PAN image, thereby improving the accuracy of scene classification. Experimental results indicate that the proposed method performs better than the other methods. The relevant code of this paper is provided at: https://github.com/ru-willow/ST-GC-Net. • A subspace similar recombination module is proposed. • The selectable self-attention spectral feature extraction module is designed. • The multi-stream Gist spatial feature extraction module is introduced. [ABSTRACT FROM AUTHOR]

Recently, a massive number of deep learning-based approaches have been successfully applied to various remote sensing image (RSI) recognition tasks. However, most existing advances of deep learning methods in the RSI field heavily rely on the features extracted by the manually designed backbone network, which severely hinders the potential of deep learning models due to the complexity of RSI and the limitation of prior knowledge. In this paper, we research a new design paradigm for the backbone architecture in RSI recognition tasks, including scene classification, land-cover classification and object detection. A novel one-shot architecture search framework based on a weight-sharing strategy and an evolutionary algorithm is proposed, called RSBNet, which consists of three stages: Firstly, a supernet constructed in a layer-wise search space is pretrained on a self-assembled large-scale RSI dataset based on an ensemble single-path training strategy. Next, the pre-trained supernet is equipped with different recognition heads through the switchable recognition module and respectively fine-tuned on the target dataset to obtain task-specific supernet. Finally, we search for the optimal backbone architecture for different recognition tasks based on the evolutionary algorithm without any network training. Extensive experiments have been conducted on five benchmark datasets for different recognition tasks, the results show the effectiveness of the proposed search paradigm and demonstrate that the searched backbone is able to flexibly adapt different RSI recognition tasks and achieve impressive performance. [ABSTRACT FROM AUTHOR]

Detecting changed regions between two given synthetic aperture radar images is very important to monitor change of landscapes, change of ecosystem and so on. This can be formulated as a classification problem and addressed by learning a classifier, traditional machine learning classification methods very easily stick to local optima which can be caused by noises of data. Hence, we propose an unsupervised algorithm aiming at constructing a classifier based on self-paced learning. Self-paced learning is a recently developed supervised learning approach and has been proven to be capable to overcome effectively this shortcoming. After applying a pre-classification to the difference image, we uniformly select samples using the initial result. Then, self-paced learning is utilized to train a classifier. Finally, a filter is used based on spatial contextual information to further smooth the classification result. In order to demonstrate the efficiency of the proposed algorithm, we apply our proposed algorithm on five real synthetic aperture radar images datasets. The results obtained by our algorithm are compared with five other state-of–the-art algorithms, which demonstrates that our algorithm outperforms those state-of-the-art algorithms in terms of accuracy and robustness. [ABSTRACT FROM AUTHOR]

Recently few-shot object detection (FSOD) in remote sensing images (RSIs) has drawn increasing attention. However, the current FSOD methods in RSIs merely focus on the detection performance of few-shot novel classes while ignoring the severe degradation of the base class performance. Generalized few-shot object detection (G-FSOD) aims to solve the FSOD problem without forgetting previous knowledge. In this paper, we focus on the G-FSOD in RSIs and propose a Generalized Few-Shot Detector (G-FSDet) that can learn novel knowledge without forgetting. Through the comprehensive analysis of each component in the detector, a novel efficient transfer-learning framework is presented as the foundation of our G-FSDet, which is more suitable for FSOD in remote sensing scenes. Considering the greater intra-class diversity and lower inter-class separability of geospatial objects, we design a metric-based discriminative loss to learn a more discriminative classifier in the few-shot fine-tuning stage. Furthermore, a representation compensation module is proposed to alleviate the catastrophic forgetting problem by decoupling the representation learning of previous and novel knowledge. Extensive experiments on DIOR and NWPU VHR-10.v2 datasets demonstrate that our proposed G-FSDet achieves competitive novel class performance with minor degradation in the base class, reaching state-of-the-art overall performance among all few-shot settings. The source code is available at (https://github.com/RSer-XDU/G-FSDet). [ABSTRACT FROM AUTHOR]

For remote sensing image (RSI) captioning tasks, two-stage RSI captioning methods have achieved high performance because they introduce the results of other RSI tasks, such as image classification as prior information. However, most previous works treat image classification in the two-stage RSI captioning method as a single-label classification task, which cannot adequately describe the entire contents of complex RSIs and may cause semantic ambiguity. To settle this problem and further refine image feature representation, we introduce multi-label classification into two-stage RSI captioning to provide sufficient and accurate prior semantic information and propose a multi-label semantic feature fusion (MLSFF) framework. Specifically, we design a robust multi-label semantic attribute extractor to extract multi-label semantic attributes of RSIs. To obtain discriminative feature representations for RSI captioning, we propose two cross-modal semantic feature fusion operators that fuse the extracted semantic attributes and the image feature extracted by the convolutional neural network. The results of extensive numerical experiments show that the proposed method can achieve state-of-the-art performance on the UCM-Captions, Sydney-Captions, and RSICD datasets. Specifically, on the UCM-Captions dataset, our method achieves a gain of 8.2% in S m score over the SAT (LAM) method (Zhang et al., 2019c). On the Sydney-Captions dataset, our method improves the S m score by 17.4% compared with the TCE loss-based method (Li et al., 2020a). On the RSICD, our method outperforms the multi-level attention method (Li et al., 2020b) by 3.2% in terms of the S m score. Code is available at https://github.com/xtye5025/MLSFF. [ABSTRACT FROM AUTHOR]

In recent years, the panchromatic (PAN) and the multispectral (MS) remote sensing images classification has become a research hotspot. In this paper, we propose a spatial-channel progressive fusion residual network (SCPF-ResNet) for multi-resolution remote sensing classification. Firstly, for the inputs of the proposed network, the interactive data fusion strategy (IDFS) combines generalized-intensity-hue-saturation (GIHS), and discrete wavelet transform (DWT) to interfuse patch pairs of the PAN and the MS images, so as to increase the similarity between them, thus reduce the difference in information between them. Secondly, for the branches of feature extraction, we design an adaptive spatial attention module (ASA-Module) and an adaptive channel attention module (ACA-Module) to strengthen spatial features from both larger-sized with smaller-sized targets and spectral features among channels. Finally, we insert the ASA-Module and ACA-Module into the residual modules to form a triple-branch network and use the common spatial-channel features extracted by the Fusion_Branch to gradually enhance the pure independent features extracted by the PAN_Branch and the MS_Branch, respectively. The experimental results indicate that SCPF-ResNet can achieve competitive performance. • An interactive data fusion strategy is proposed. • Two adaptive attention modules to highlight the advantages of two different data. • A triple-branch progressive fusion residual network for classification is proposed. [ABSTRACT FROM AUTHOR]

Tremendous progresses have been made in remote sensing image captioning (RSIC) task in recent years, yet there still some unresolved problems: (1) facing the gap between the visual features and semantic concepts, (2) reasoning the higher-level relationships between semantic concepts. In this work, we focus on injecting high-level visual-semantic interaction into RSIC model. Firstly, the semantic concept extractor (SCE), end-to-end trainable, precisely captures the semantic concepts contained in the RSIs. In particular, the visual-semantic co-attention (VSCA) is designed to grain coarse concept-related regions and region-related concepts for multi-modal interaction. Furthermore, we incorporate the two types of attentive vectors with semantic-level relational features into a consensus exploitation (CE) block for learning cross-modal consensus-aware knowledge. The experiments on three benchmark data sets show the superiority of our approach compared with the reference methods. • We propose an end-to-end trainable network architecture for remote sensing image captioning task, aiming to extract explicit high-level semantic from remote sensing image, learn discriminative cross-modal linguistic-aware features, and exploit cross-modal alignment. • To capture the explicit semantic concepts, semantic concept extractor is designed to generate a series of semantic words. The designed network based on visual features and semantic concepts is effective for visual-semantic features extracting. • For modeling the visual features and semantic concepts, a visual-semantic co-attention module with two parallel attentive branches is designed, which learns the linguistic awareness between visual features and enhances the region-related semantic contents. • In order to facilitate the alignment between cross-modal features, a consensus exploitation is devised for exploring the multi-modal consensus-aware representations based on the constructed concept correlation graph. [ABSTRACT FROM AUTHOR]

• A dual branch attention fusion deep network for classification is proposed. • An adaptive center offset sampling strategy for each patch is proposed. • Two attention based modules to highlight the advantages of two different data. • The feature level fusion is achieved by the proposed network. In recent years, with the diversification of acquisition methods of very high resolution panchromatic (PAN) and multispectral (MS) remote sensing images, multiresolution remote sensing classification has become a research hotspot. In this paper, from the perspective of data–driven deep learning, we design a dual–branch attention fusion deep network (DBAF–Net) for the multiresolution classification. It aims to integrate the feature–level fusion and classification into an end–to–end network model. In the process of establishing a training sample library, unlike the traditional pixel–centric sampling strategy with fixed patch size, we propose an adaptive center–offset sampling strategy (ACO–SS), which allows each patch to adaptively determine the neighborhood range by finding the texture structure of the pixel to be classified. And the neighborhood range is not symmetrical with this pixel, we expect to capture the neighborhood information that is more conducive to its classification. In network structure, based on the captured patches by ACO–SS, we design a spatial attention module (SA–module) for PAN data and a channel attention module (CA–module) for MS data, thus highlighting the spatial resolution advantages of PAN data and the multi–channel advantages of MS data, respectively. Then these two features are interfused to improve and strengthen the fusion features in both spatial and channel. The quantitative and qualitative experimental results verify the robustness and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Hyperspectral pansharpening (HP) can fuse different kinds of remote sensing images for the downstream classification or detection task. Deep learning methods have made rapid development in HP. However, most deep learning-based HP methods design the network architecture without considering the specific downstream task. When the pansharpening image is transferred to the downstream tasks, it may lead to poor performance on the downstream tasks. To solve this problem, a novel classification-oriented multi-task network (CMNet) is proposed for HP. CMNet combines the classification and HP networks in a multi-task learning way. In the classification network, class-specific and spatial-difference contrastive constraints are defined to enforce the extraction module to extract discriminative features for classification. In the HP network, a fusion module with an identical structure to the extraction module is constructed to extract fused features for reconstruction. Then, multi-gradient learning is designed to optimize the HP network under the guidance of the classification network. It adopts gradient mapping to consider the conflicting or collaborative situations between these different tasks. In this way, the fusion module in the HP network learns discriminative feature extraction ability from the classification network, which makes the HP network realize classification-oriented HP. Experimental results demonstrate that the proposed method markedly improves the pansharpening performance and the pansharpening is also able to elevate the classification performance. [ABSTRACT FROM AUTHOR]

Considering that saliency areas in images play important roles in human visual perception, in this paper we propose a Learning Compressive Sampling (LCS) scheme and its physical implementation for more efficient image acquisition via a new Multiscale and Steerable Support Value Transform (MS2VT). The key idea is to learn geometric saliency maps of images by MS2VT, which is deduced from a mapped least square-support vector machine. Because MS2VT can produce a multiscale, multidirectional, undecimated, dyadic and aliasing transform with shift-invariant and anisotropy properties, the obtained support values can reveal the geometric and saliency information of images. The learned saliency map is then used to formulate a variable density compressive function, to realize a simple, fast and efficient sampling, which aims to allocate more sensing resources to saliency attention areas but fewer to non-salient regions. Several experiments are taken on some natural images and remote sensing images to compare our proposed LCS scheme with traditional samplings when saliency information is not used. Moreover, the performance of MS2VT based saliency detection scheme is also compared with other related saliency detection approaches. The experimental results indicate that it can obtain high quality images, especially in preserving more detailed edges, contours and complex structures, even at low sampling ratios. [ABSTRACT FROM AUTHOR]