Search

Oriented object detection has gained increasing attention due to its ability to detect objects with arbitrary orientations in the field of remote sensing (RS) images. However, the laborious task of annotating oriented bounding boxes (OBBs) presents significant challenges for training a fully supervised arbitrary-oriented object detector. Some existing approaches apply annotated horizontal bounding boxes (HBBs) as weakly supervised signals, in which even HBB annotations require significant human efforts. In this article, we propose a point-to-mask-to-HBB-to-OBB (PMHO) method that achieves weakly supervised oriented object detection by requiring only single-point object annotations. Specifically, we first take the input images and the given annotated points as prompts to obtain the initial segmentation masks, and then present a neighboring mask combination scheme to address the over-segmentation issue and a point-centric mask selection strategy to filtrate the related masks. Based on positive and negative proposal bags transferred from the mask regions, the pseudo-HBB generation network which consists of a classification branch for classification and an instance branch for the localization of individual proposals, aims to generate pseudo HBBs for each object. For further refinement of pseudo HBBs, we present a pseudo-HBB filtering strategy with K-means clustering on features extracted by a CNN model pretrained on a large-scale offline RS dataset. To train the oriented object detector, an inscribed ellipse constraint is proposed to measure the regression loss between predicted OBBs and the given pseudo HBBs. Extensive experiments including OBB detection, pseudo-HBB generation, and ablation studies are conducted on public DIOR and DIOR-R datasets, which demonstrates that our method achieves state-of-the-art performance.

Dozens of spectral bands of hyperspectral images (HSIs) have been successfully reconstructed from only three color band images using deep neural networks according to their powerful nonlinear mapping capability. However, the existing deep-learning-based approaches tend to directly reconstruct HSIs from RGB inputs without emphasizing the discriminative intrinsic properties of different materials, resulting in certain distortion in reconstructed spectra. In this article, an intrinsic image decomposition (IID)-based spectral super-resolution (SSR) framework is proposed to reconstruct spectra of pixels from their reflectance feature and shading feature separately, by which the intrinsic properties can be emphasized during spectral reconstruction. Specifically, a dual hierarchical regression network (DHRNet) is designed for the proposed IID-based SSR task, in which a shading feature extraction module (SFEM) based on dense structure and a reflectance feature extraction module (RFEM) with attention mechanism are first, respectively, designed to reconstruct spectral information from reflectance feature and shading feature, and a feature enhancement module (FEM) is consequently devised to further improve the coarse combined estimation. Ultimately, a novel hybrid loss combining smooth $\boldsymbol {l}_{1}$ loss, spectral angel mapper (SAM), and gradient prior is also presented to restrain the spectral distortion while enhancing the sharpness of the reconstructed HSI. Experimental results over three datasets demonstrate the superiority of our proposed framework.

Metalearning has been widely applied to solve the few-shot object detection (FSOD) problem in natural scenes, which performs similarity measurement and information aggregation of the support set and the query set. However, regarding remote sensing images (RSIs), many difficulties caused by their disparities need to be further addressed, such as inconsistencies in imaging scale, direction, and background between support and query images. These result in feature misalignment and attention bias, interfering with model performance. In this article, a multilevel information interaction (MLII) strategy is proposed for FSOD to alleviate feature misalignment and attention bias. Information interactions are conducted within multiple scales of features and highlight similar regions of query and support features. A semantic enhancement module (SEM) is proposed to assist MLII in extracting key information and achieving more discriminative feature representation. Moreover, a feature cross-aggregation module (FCM) with separate classification losses is designed to train the detector to identify objects that coexist in query and support images. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art few-shot object detectors over commonly used benchmark datasets, i.e., DIOR and NWPU-10.

Image-to-image (I2I) translation methods based on generative adversarial networks (GANs) have shown general solutions for aerial visible-to-infrared image translation (AVIT) task. Though the existing approaches have made impressive results, they still struggle to produce diverse or high-realism translated aerial infrared images (AIIs). In this article, a novel model is proposed to achieve both diverse and realistic AVIT, named DR-AVIT. Specifically, we introduce disentangled representation learning to disentangle the image representation of aerial visible images (AVIs) and AIIs into a domain-invariant semantic structure space and two domain-specific imaging style spaces. By leveraging this disentanglement, our model can perform the translation process conditioned on semantic structure information derived from the input AVI and randomly sampled imaging style features from the AII domain to obtain diverse outputs. Furthermore, a new constraint is present to encourage GANs to learn efficient mappings between AVI and AII domains by integrating geometry-consistency constraint and a dual learning framework, named dual geometry-consistency constraint. Coping with these two designs, our method exhibits superiority in both realism and diversity of the translation results over several state-of-the-art I2I translation methods on AVIID dataset and two new benchmark datasets for AVIT, which are obtained by extracting data from publicly available datasets. Codes of DR-AVIT and proposed benchmark datasets are available at https://github.com/silver-hzh/DR-AVIT.

Recent developments in adversarial attack have witnessed the success of background attack against object detectors. However, most existing methods attack detectors by luring targets into background. Therefore, an innovative background attack framework via dual-adversarial-induced error identification (BADEI) is proposed to attack detectors by deceiving background as targets, as well as deceiving targets as background, where the attack performance can be greatly enhanced by these two kinds of induced error identification. Specifically, a mechanism that generates the adversarial background is proposed to result in dual error detection, where the background can conceal the specified targets and cause the misclassification of the adversarial pattern in the background as a specific category. Moreover, an unoccluded training strategy (UTS) that leverages the target mask of an image is introduced to strategically place adaptive adversarial background beneath the targets while optimizing and updating the pixel values of the background outside the target region, which can enhance attack effectiveness for adversarial background, significantly degrade the targets’ average accuracy, and enhance the robustness of background. Finally, a dual deceptive loss function (D2LF) is carefully formulated to generate false negatives (FNs) and false positives (FPs) to achieve untargeted attacks for hiding objects as well as targeted attacks for erroneously recognizing objects. Extensive experiments and comparative analysis of various victim network models on two datasets (including the dataset for object detection in aerial images (DOTA) and RSOD dataset) confirm that the proposed framework exhibits superior performance over the state-of-the-art methods in both digital and physical scenarios.

Object detection in remote sensing images identifies and extracts the acquired Earth surface information, providing data support and research basis for multiple fields. Remote sensing image object detection based on knowledge distillation (KD) can transfer the knowledge of a large teacher model to a smaller student model, achieving the effect of low parameter volume and high accuracy. Mainstream methods directly imitate teacher features to improve student performance, ignoring the generation of high-ranking features through teacher features instructing student feature maps in this knowledge transfer process. In this article, an adaptive composite feature generation (ACFG) strategy is proposed to achieve end-to-end trainable KD for object detection in remote sensing images, in which the robustness of feature points under composite masks is improved through adaptive feature mapping. In particular, a composite mask generator (CMG) module is proposed to select student instance-related features and point background features. Furthermore, a global and local projection layer (GLPL) module is proposed to connect the local information and global information of the feature map under the mask generator to adaptively realize the global recovery mapping of the feature map with partial feature points. Finally, balanced decoupling loss (BDL) is improved to handle foreground and background loss separately, so that the two decoupled features can better enable the student model to learn instance-related information. Note that the proposed ACFG is capable of conducting KD for both single-stage and two-stage object detectors. Experimental results using both anchor-based and anchor-free detectors on the DIOR dataset and DOTA dataset demonstrate that the proposed ACFG clearly achieved better performance than several state-of-the-art (SOTA) algorithms for KD.

Band selection (BS), which effectively reduces spectral dimensionality, stands out as a leading focus within hyperspectral image (HSI) analysis. Self-representation (SR) has surfaced as a favored technique in this domain due to its applicability to BS and unsupervised nature. However, the existing SR-based BS approaches only leverage either spatial or spectral relationships, with few integrating both while concentrating on the representation level rather than the selection level. In addition, employing all spatial pixels for spatial relationship utilization leads to considerable computational complexity. Therefore, this article proposes joint spatial and spectral graph-based consistent SR (JSSGCSR) to more effectively exploit spatial and spectral relationships for BS, which separately conducts SR to handle each view of spatial and spectral graphs to better consider two different structure characteristics, and ultimately integrates two SR results to achieve a unified and robust representative band set by imposing consistent sparsity pattern on their joint representation coefficients. In addition, the spatial and spectral relationships are integrated into different data spaces, that is, spectral graph SR and spatial graph SR are, respectively, conducted in the original HSI and the segmented and pooled HSI, which not only reduces the influence of superpixel segmentation on spectral relationships, but also improves the efficiency of spatial relationship utilization. Experimental results on three benchmark datasets have demonstrated the effectiveness of the proposed JSSGCSR in HSI classification tasks.

Graph convolutional networks (GCNs) have garnered extensive attention in the realm of hyperspectral image (HSI) classification. However, due to the problem of oversmoothing caused by deep GCN, most of the existing GCN-based methods are limited to constructing shallow networks, thus only able to extract superficial features. Moreover, when existing shallow GCNs extend to a more deeper structure, the number of learnable parameters increases linearly, thus leading to poor generalization performance under limited training samples. To address the aforementioned issues, a separable deep GCN integrated with convolutional neural network and prototype learning (SDGCP) is proposed for HSI classification, which can extract effective global structural information of HSI without increasing the number of trainable parameters. Specifically, the spectral and spatial features, adaptively selected by the attention module, are encoded into the structure of a graph by the graph encoder with the assistance of the pixel-to-region mapping obtained from the simple linear iterative clustering (SLIC). Then, a separable deep graph convolution module, composed of feature extraction and deep feature propagation, is adopted to capture the long-range contextual relationships from HSI encoded as graph data, which is combined with locally complementary information extracted by CNN after decoding. Finally, to further boost the performance of classification under limited labeled samples, prototype learning with regularization terms is utilized to enhance the intraclass compactness and interclass separability of feature representations. Extensive experiments on three standard HSI datasets demonstrate the superiority of the proposed SDGCP over the state-of-the-art (SOTA) methods.

Feature representation is crucial for hyperspectral image (HSI) classification. However, existing convolutional neural network (CNN)-based methods are limited by the convolution kernel and only focus on local features, which causes it to ignore the global properties of HSIs. Transformer-based networks can make up for the limitations of CNNs because they emphasize the global features of HSIs. How to combine the advantages of these two networks in feature extraction is of great importance in improving classification accuracy. Therefore, a cross-attention fusion network bridging CNN and Transformer (CAF-Former) is proposed, which can fully utilize the advantages of CNN in local features and Transformer’s long time-dependent feature learning for hyperspectral classification. In order to fully explore the local and global information within an HSI, a Dynamic-CNN branch is proposed to effectively encode local features of pixels, while a Gaussian Transformer branch is constructed to accurately model the global features and long-range dependencies. Moreover, in order to fully interact with local and global features, a cross-attention fusion (CAF) module is proposed as a bridge to fuse the features extracted by the two branches. Experiments over several benchmark datasets demonstrate that the proposed CAF-Former significantly outperforms both CNN-based and Transformer-based state-of-the-art networks for HSI classification.

Aerial person detection (APD) is vital for enhancing search and rescue (SaR) operations, particularly when locating victims in remote, poorly-lit areas. Despite advancements in detection technologies, achieving a balance between detection speed and accuracy on mobile devices in “edge AI” continues to pose challenges. In this article, a lightweight distillation network (APDNet) is proposed for edge deployment of APD, which enables real-time inference as well as minimizes accuracy loss during model transfer. The proposed APDNet employs a distillation network between varying-depth backbones and integrates an 8-bit quantized optimizer to reduce the floating-point operations of network parameters. Specifically, in the teach-assistant distillation (TAD) stage, small student models using random weight initialization are trained with pseudo-labels generated by deeper teacher models, facilitating consistent learning for a more accurate, lighter model. Moreover, a low-precision quantization (LPQ) stage incorporates an offline, quantization-aware training strategy that dynamically adjusts the ranges of weight and activation function float-point values, reducing computational complexity. In order to compensate for the potential accuracy decline, a pluggable tracker updates the position and feature information of persons frame-by-frame, with tracking results integrated with detection outputs to enhance accuracy. Extensive experiments on the Heridal, Manipal-UAV, and VTSaR datasets confirm the effectiveness of APDNet, demonstrating its superior performance in edge-based APD.

Learning discriminative features is crucial for hyperspectral image (HSI) classification. Though metric learning has been applied to learn effective features in HSI classification tasks, existing metric loss functions only consider distance among features of sample pairs but ignore the feature centers and boundaries in the embedding feature space, which limits the discrimination of learned features. In this article, a novel metric loss function named center-boundary metric loss (CBML) is proposed to learn more discriminative features so as to improve HSI classification performance. Unlike the existing metric loss functions, CBML not only considers the distance between sample pairs to enhance intraclass similarity and interclass separability but also pays more attention to the feature centers and boundaries in the embedding feature space that could greatly determine and affect the category of features. Specifically, CBML forces the distance of a sample to its corresponding feature center to be explicitly smaller than that to samples from other classes by a predefined threshold. As a result, the boundaries of different classes will separate an actual distance, which improves the discrimination of learned features. Moreover, in order to improve the training efficiency, a cross mini-batch sampling strategy is further proposed to break through the limitation within the mini-batch by using features between several contiguous mini-batches to sample pairs without increasing the size of the mini-batch. Accordingly, the sampling range of sample pairs is greatly expanded, and the training data is more fully exploited. Experimental results over four benchmark datasets with a typical network for HSI classification demonstrate our proposed method outperforms several state-of-the-arts.

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. ASTN introduces a novel adversarial training scheme with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. In our experiments on 393 trials collected from 21 subjects, the proposed ASTN achieved an AUC 0.85, clearly outperforming conventional learning methods.

Convolutional neural networks (CNNs) have been demonstrated to be powerful tools to automatically learn effective features from large datasets. Though features learned in CNNs are approximately scale-, translation-, and position-invariant, and their capacity in dealing with image rotations remains limited. In this article, a novel cyclic polar coordinate convolutional layer (CPCCL) is proposed for CNNs to handle the problem of rotation invariance for feature learning. First, the proposed CPCCL converts rotation variation into translation variation using polar coordinates transformation, which can easily be handled by CNNs. Moreover, cyclic convolution is designed to completely handle the translation variation converted from rotation variation by conducting convolution in a cyclic shift mode. Note that the proposed CPCCL is capable of generalization and can be used as a preprocessing layer for classification CNNs to learn the rotation-invariant feature. Extensive experiments over three benchmark datasets demonstrate that the proposed CPCCL can clearly handle the rotation-sensitive problem in traditional CNNs and outperforms several state-of-the-art rotation-invariant feature learning algorithms.

Spectral super-resolution (SR), which reconstructs high spatial-resolution hyperspectral images (HSIs) from RGB inputs, has been demonstrated to be one of the effective computational imaging techniques to acquire HSIs. Though deep neural networks have shown their superiority in such a complex mapping problem, existing networks generally involve a very complex structure with huge amounts of parameters, resulting in giant memory occupation. In this article, a lightweight multiresolution feature fusion network (MRFN) is proposed, which adopts a multiresolution feature extraction and fusion framework to fully explore RGB inputs in different scales of resolution. Specifically, a lightweight feature extraction module (LFEM), which adopts cheap convolution and attention mechanisms, is constructed to explore different scales of features under a lightweight structure. Moreover, a hybrid loss function is proposed by encountering not only pixel-value level reconstruction error but also spectral continuity and fidelity. Experiments over three benchmark datasets, i.e., CAVE, Interdisciplinary Computational Vision Laboratory (ICVL), and NTIRE2022 datasets, have demonstrated that the proposed MRFN can reconstruct HSIs from RGB inputs in higher quality with fewer parameters and computational floating-point operations (FLOPs) compared with several state-of-the-art networks.

With the wide application of remote sensing images (RSIs) in military and civil fields, remote sensing object detection (RSOD) has gradually become a hot research direction. However, we observe two main challenges for RSOD, namely, the complicated background and the small objects issues. Given the different appearances of generic objects and remote sensing objects, the detection algorithms designed for the former usually cannot perform well for the latter. We propose a novel global-to-local scale-aware detection network (GLSANet) for RSOD, aiming to solve the abovementioned two challenges. First, we design a global semantic information interaction module (GSIIM) to excavate and reinforce the high-level semantic information in the deep feature map, which alleviates the obstacles of complex background on foreground objects. Second, we optimize the feature pyramid network to improve the performance of multiscale object detection in RSIs. Finally, a local attention pyramid (LAP) is introduced to highlight the feature representation of small objects gradually while suppressing the background and noise in the shallower feature maps. Extensive experiments on three public datasets demonstrate that the proposed method achieves superior performance compared with the state-of-the-art detectors, especially on small object detection datasets. Specifically, our algorithm reaches 94.57% mean average precision (mAP) on the NWPU VHR-10 dataset, 95.93% mAP on the RSOD dataset, and 77.9% mAP on the DIOR dataset.

Band selection (BS), which can reduce the spectral dimensionality effectively, has become one of the most popular topics in hyperspectral image (HSI) analysis. Recently, sparse representation-based BS has emerged as a popular tool. The existing sparse models mainly focus on minimizing reconstruction error and sparsity, while do not fully exploit the unique correlations among hundreds of continuous bands, which may cause representative bands missed and highly correlated bands selected. Therefore, this article proposes the spectral correlation-based diverse BS (SCDBS) for HSIs to improve representativeness and diversity of the selected bands. Specifically, a correlation derived weight is used to perform weighted sparse reconstruction to select the bands that are more correlated with the whole HSI, and a correlation minimization term is designed to remove the highly correlated bands simultaneously. In addition, the proposed method imposes an adjustable sparse constraint by using an $\ell _{2,0< p\leq 1}$ norm, which extends and unifies the commonly used $\ell _{2,1}$ norm to provide more flexible sparsity level. To optimize the proposed BS model, an iteration algorithm with relatively low computational cost is designed, of which the convergence is theoretically presented. Experimental results on three benchmark datasets have demonstrated that the proposed SCDBS outperforms state-of-the-art methods in HSI classification.

Patch-based physical attacks have increasingly aroused concerns. However, most existing methods focus on obscuring targets captured on the ground, and some of these methods are simply extended to deceive aerial detectors. They smear the targeted objects in the physical world with the elaborated adversarial patches, which can only slightly sway the aerial detectors’ prediction and with weak attack transferability. To address the above issues, a novel contextual background attack (CBA) framework is proposed to fool aerial detectors in the physical world, which can achieve strong attack efficacy and transferability in real-world scenarios even without smudging the interested objects at all. Specifically, the targets of interest, i.e., the aircraft in aerial images, are adopted to mask adversarial patches. The pixels outside the mask area are optimized to make the generated adversarial patches closely cover the critical contextual background area for detection, which contributes to gifting adversarial patches with more robust and transferable attack potency in the real world. To further strengthen the attack performance, the adversarial patches are forced to be outside targets during training, by which the detected objects of interest, both on and outside patches, benefit the accumulation of attack efficacy. Consequently, the sophisticatedly designed patches are gifted with solid fooling efficacy against objects both on and outside the adversarial patches simultaneously. Extensive proportionally scaled experiments are performed in physical scenarios, demonstrating the superiority and potential of the proposed framework for physical attacks. We expect that the proposed physical attack method will serve as a benchmark for assessing the adversarial robustness of diverse aerial detectors and defense methods. The code has been released at https://github.com/JiaweiLian/CBA.

Despite deep neural networks (DNNs) have been widely applied in remote sensing (RS) scene classification and achieved satisfying performance, the vulnerability of DNNs toward adversarial examples significantly degrades their performance. Moreover, the relatively limited labeled samples of RS scene classification make DNNs more likely to overfit, leading to weak generalizability and noise sensitivity. This may result in DNNs being more vulnerable to adversarial examples. Consequently, the defense of adversarial examples is of crucial importance to improve both the generalizability and robustness of DNNs in the RS scene classification task. However, few studies have been conducted on defense for RS scene classification, especially ignoring the intrinsic characteristics of RS images. In this article, an effective defense framework for RS scene classification, named reconstruction-assisted and distance-optimized adversarial training (RDAT), is proposed to defend adversarial examples. To solve the problems caused by high interclass similarity, a distance-optimized (DO) strategy is designed for adversarial training (AT) to strengthen the learning of underfitting content, increase the interclass distance, and improve the robustness of the networks. Furthermore, to generate high-quality samples for AT, a reconstruction-assisted (RA) block is proposed to eliminate adversarial perturbations in adversarial examples. Specifically, in this block, by Swin Transformer (SwinT) block and multiscale convolution (MSC) block, SwinT-MSC-UNet (SMUNet) is constructed to fully extract global and multiscale local features to adapt to the characteristics of RS images with a large variance of ground object scales. Extensive experiments on the benchmark datasets, that is, UC Merced (UCM) and aerial image dataset (AID), have demonstrated that the proposed RDAT can effectively resist multiple adversarial attacks and yield superior results than other defense methods for RS scene classification.

Recently, the Transformer-based technique has emerged as a promising solution for modeling contextual information in remote sensing (RS) scenes and has found widespread applications in RS scene classification. However, how to make full use of intermediate features learned in Transformers is of crucial importance in the RS scene classification tasks. Therefore, this article proposes a hierarchical feature fusion of transformer with patch dilating (HFFT-PD), which aims to capture rich contextual information from hierarchical features to enhance the performance of RS scene classification. Specifically, the HFFT-PD model consists of a hierarchical transformer merging (HTM) block and a lightweight adaptive channel compression (LACC) module, in which the HTM is specially designed for the Transformer architecture to bridge the semantic gaps between features from different hierarchical blocks, and the LACC accounts for the significance of distinct channels in the ultimate classification features. In addition, a brand-new Patch Dilating strategy is uniquely designed for the Transformer paradigm, functioning as a reassembly operator predicated on patch features. Contrasting with conventional upsampling techniques, Patch Dilating facilitates upsampling without requiring supplementary information, while concurrently preserving the semantic content of local spatial structure. Extensive and rigorous experiments conducted on the UC Merced land-use dataset (UCM), aerial image dataset (AID), and NWPU-45 datasets, with training ratios of 80%, 50%, and 20%, respectively, demonstrate that our proposed HFFT-PD outperforms the baseline at least by 0.59%, 0.44%, and 0.99%, respectively, showcasing the significant superiority of our HFFT-PD over contemporary state-of-the-art methodologies.

Transformer-based networks, which can well model the global characteristics of inputted data using the attention mechanism, have been widely applied to hyperspectral image (HSI) classification and achieved promising results. However, the existing networks fail to explore complex local land cover structures in different scales of shapes in hyperspectral remote sensing images. Therefore, a novel network named multiscale and cross-level attention learning (MCAL) network is proposed to fully explore both the global and local multiscale features of pixels for classification. To encounter local spatial context of pixels in the transformer, a multiscale feature extraction (MSFE) module is constructed and implemented into the transformer-based networks. Moreover, a cross-level feature fusion (CLFF) module is proposed to adaptively fuse features from the hierarchical structure of MSFEs using the attention mechanism. Finally, the spectral attention module (SAM) is implemented prior to the hierarchical structure of MSFEs, by which both the spatial context and spectral information are jointly emphasized for hyperspectral classification. Experiments over several benchmark datasets demonstrate that the proposed MCAL obviously outperforms both the convolutional neural network (CNN)-based and transformer-based state-of-the-art networks for hyperspectral classification.

Remote sensing cross-modal text-image retrieval (RSCTIR) has recently received unprecedented attention due to its advantages of flexible input and efficient query on enormous remote sensing (RS) images. However, most RSCTIR methods focus obsessively on the cross-modal semantic alignment between text and image modalities and are easily stuck in the dilemma of information redundancy, leading to the degradation of retrieval accuracy. To address the issues, we construct a novel RSCTIR framework based on mask-guided relation modeling with entity loss (MGRM-EL), to fully explore uni-modal feature learning on entities and relations in the cross-modal model learning process. Specifically, we take advantage of the Transformer encoder architecture for its ability to capture long-distance dependencies from the global view and build two uni-modal (visual and textual) Transformer encoders combined with convolutional neural network (CNN) to extract the spatial interregion relations of images as well as long-term interword relations of texts for prominent feature embedding of visual and semantic representations. A mask-guided attention strategy is further introduced to learn the salient regions and words, with the aim of enhancing the RSCTIR model’s uni-modal learning ability and eliminating unnecessary and redundant information about each modality. Unlike existing methods that simply compute the semantic similarity between images and texts on the loss functions, we present a novel uni-modal entity loss, which treats each image as an image entity and merges similar texts into a text entity, to learn the independent distribution of entities in each modality. We conduct extensive experiments on public RSCTIR benchmarks including RSICD and RSITMD datasets, which demonstrate the state-of-the-art performance of the proposed method on the RSCTIR task.

Transformer has been widely applied in image processing tasks as a substitute for convolutional neural networks (CNNs) for feature extraction due to its superiority in global context modeling and flexibility in model generalization. However, the existing transformer-based methods for semantic segmentation of remote sensing (RS) images are still with several limitations, which can be summarized into two main aspects: 1) the transformer encoder is generally combined with CNN-based decoder, leading to inconsistency in feature representations; and 2) the strategies for global and local context information utilization are not sufficiently effective. Therefore, in this article, a global-local transformer segmentor (GLOTS) framework is proposed for the semantic segmentation of RS images to acquire consistent feature representations by adopting transformers for both encoding and decoding, in which a masked image modeling (MIM) pretrained transformer encoder is adopted to learn semantic-rich representations of input images and a multiscale global-local transformer decoder is designed to fully exploit the global and local features. Specifically, the transformer decoder uses a feature separation-aggregation module (FSAM) to utilize the feature adequately at different scales and adopts a global-local attention module (GLAM) containing global attention block (GAB) and local attention block (LAB) to capture the global and local context information, respectively. Furthermore, a learnable progressive upsampling strategy (LPUS) is proposed to restore the resolution progressively, which can flexibly recover the fine-grained details in the upsampling process. The experiment results on the three benchmark RS datasets demonstrate that the proposed GLOTS is capable of achieving better performance with some state-of-the-art methods, and the superiority of the proposed framework is also verified by ablation studies. The code will be available at https://github.com/lyhnsn/GLOTS.

Hyperspectral remote sensing is well-known for its extraordinary spectral distinguishability to discriminate different materials. However, the cost of hyperspectral image (HSI) acquisition is much higher compared to traditional RGB imaging. In addition, spatial and temporal resolutions are sacrificed to obtain very high spectral resolution owing to the limitations of sensor technologies. Therefore, in this paper, HSIs are reconstructed using easily acquired RGB images and a convolutional neural network (CNN). As a result, high spatial and temporal resolution RGB images can be inherited to HSIs. Specifically, a two-stage CNN, referred to as the spectral super-resolution network (SSR-Net), is designed to learn the transformation model between RGB images and HSIs from training data, including a band prediction network (BP-Net) to estimate hyperspectral bands from RGB images and a refinement network (RF-Net) to further reduce spectral distortion in the band prediction step. As a result, the learned joint features in the proposed SSR-Net can directly predict HSIs from their corresponding scenes in RGB images without prior knowledge. Experimental results obtained on several benchmark datasets demonstrate that the proposed SSR-Net outperforms several state-of-the-art methods by ensuring higher quality in HSI reconstruction, and significantly improves the performance of traditional RGB images in classification.