Search

The intelligent segmentation of high-resolution remote sensing (HRS) image, also called as dense prediction task for HRS image, has been and will continue to be important research in the remote sensing community. In recent years, the growing wave of artificial intelligence (AI) technology has introduced innovative paradigms to this domain, yielding outstanding results and overcoming many challenges with conventional segmentation techniques. This paper provides a comprehensive review of these intelligent segmentation methodologies, including traditional pattern recognition, convolution neural network (CNN)-based, and Transformer-based techniques. However, the explosive but incomplete development of intelligent segmentation techniques also poses more challenges for earth observation experts, the most of which is the technical interpretability. Consequently, we consider these segmentation techniques in the aspect of explainable artificial intelligence (XAI). Data-centric XAI thinks the practical applications of the segmentation model while model-centric XAI will facilitate the understanding of decision-making processes and the adjustment of structural features. Moreover, this review identifies novel research questions and provides constructive insights and recommendations to HRS image segmentation tasks, which may shed new light on the intelligent segmentation methods within the remote sensing image understanding community. [ABSTRACT FROM AUTHOR]

Automatic building detection from remote sensing images holds significant applications in various domains, including cartography, land-use change detection, urban planning, and 3D intelligent city construction. Driven by the innovation of deep learning technology, convolutional neural networks (CNNs) have emerged as a powerful framework for automatic interpretation, exhibiting hierarchical feature generation and robust representation for building extraction. However, the inherent computational limitations of convolutional operators hinder the acquisition of an effective global receptive field. Recent advancements in deep learning have witnessed the adoption of the multi-head self-attention (MHSA) mechanism by Transformer, presenting the superiority of global context construction. Nonetheless, this network structure encounters difficulties in effectively modelling two-dimensional spatial structure features with scale variation in semantic segmentation and often has huge parameters with high algorithm complexity. To address the above problems, this paper proposed a hybrid deep neural network synergistically harnessing the superiority of CNN and Transformer to extract buildings from remote sensing images. Meanwhile, an improved MHSA mechanism is developed to facilitate the construction of multiscale global representations, concurrently reducing computational complexity. Furthermore, the cross-level MHSA modules the semantic correlation between the deep and shallow-level features to refine spatial fine-grained information. The proposed method was comprehensively evaluated on two building datasets via quantitative analysis and visualization. The experimental results confirmed that the developed network and modules outperform other state-of-the-art methods and can effectively improve the accuracy (93.4% F1-score on the WHU dataset, 89.38% F1-score on the CAMB dataset) with high efficiency for building extraction. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural network (CNN) and Transformer have been successfully applied in hyperspectral image (HSI) classification. Despite their success, some challenges persist. Popular HSI classification methods typically employ only a single patch as the network input, which may not provide robust information for various datasets or within different scenarios of a specific dataset. Large-scale patches are usually conducive to classifying large-scale scenes, while small-scale patches excel at classifying boundary locations. In addition, the majority of methods do not distinguish between the central pixel and its adjacent pixels, although the information from the central position is crucial. In this paper, a novel double-branch hierarchical partial convolution network (DBHP-Net) is proposed to address the aforementioned challenges. Concretely, DBHP-Net employs a new convolution, named TConv, to emphasize mining the spatial-channel features of the central position. Then, the TConvFormer Block is built by adopting the advanced engineering techniques of Transformer. Furthermore, we construct our model in a hierarchical style. The double-branch structure with large-scale and small-scale patches provides complementary information from the dataset. Experimental results based on randomly selected samples and spatially disjoint samples of two benchmark datasets demonstrate that our method outperforms other state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods. [ABSTRACT FROM AUTHOR]

Vision Transformers (ViTs) have become one of the main architectures in deep learning with the self-attention mechanism, and are becoming an alternative to Convolutional Neural Networks (CNNs) for remote sensing scene classification tasks. However, the earlier self-attention layer of ViTs focuses on local features rather than global features, and the deeper self-attention layer focuses on global features but ignores the different impact of different frequency information. This will greatly increase the training and computational cost due to the quadratic complexity of the self-attention mechanism on the long sequence representation. In this paper, we propose a hybrid CNN – vision transformer structure (HCVNet), which uses convolutional layers to replace the earlier self-attention layers, and a novel Frequency Multi-head Self Attention (F-MSA) mechanism to replace the deeper self-attention layers. Specifically, F-MSA is a dual-stream structure that reduces the computational cost and improves the classification performance by encoding the high/low frequency information separately. In addition, a Semantic-aware Localization (SaL) module is introduced, which can guide the selection of crop by learning prior knowledge, avoiding the issue of pure background sampling. Our method performed an accuracy value of 97.20 ± 0.02% on the Aerial Image Dataset and 93.89 ± 0.03% on the NWPU-RESISC45 Dataset, with low complexity costs. [ABSTRACT FROM AUTHOR]

Wildfires pose a serious threat to the environment, ecosystems, property, biodiversity, and human life. Early detection of wildland fires is crucial for effective firefighting and mitigation. In this paper, we propose an ensemble learning method, called CT-Fire, which combines the deep CNN RegNetY and the vision transformer EfficientFormer v2 to recognize and detect forest fires on ground and aerial images. Testing results showed that CT-Fire achieved excellent performance with fast speed and accuracy of 99.62% and 87.77% using ground and aerial images, respectively. CT-Fire also outperformed benchmark CNNs and vision transformer methods, showing its accurate reliability in detecting wildfires. It also surpassed various challenges, including the detection of very small wildfires, background complexity, image quality, and wildland fire variability in terms of intensity, size, and shape. [ABSTRACT FROM AUTHOR]

Accurately segmenting roads is challenging due to substantial intra-class variations, indistinct inter-class distinctions, and occlusions caused by shadows, trees, and buildings. To address these challenges, it is crucial to simultaneously pay attention to important texture details and perceive global geometric context information. Recent research has shown that CNN-Transformer hybrid structures outperform using CNN or Transformer alone. CNN excels at extracting local detail features and Transformer naturally has the ability to perceive global context information. In this paper, we propose a dual-branch network module called ConSwin, which combines the advantages of ResNet and Swin Transformer for better road information extraction. Taking ConSwin as the basic encoding module, we build an hourglass-shaped network with an encoder-decoder structure. In order to better transfer texture and structural detail information between the encoder and decoder, we also design two novel connection blocks. We conduct comparative experiments on the Massachusetts and CHN6-CUG datasets, and the proposed method outperforms state-of-the-art methods in terms of overall accuracy, IOU, and F1 metrics. Other experiments verify the effectiveness of our method, while visualization results demonstrate its ability to obtain better road representations. [ABSTRACT FROM AUTHOR]

Most existing remote sensing disaster assessment methods rely on convolutional neural networks (CNNs). Although CNNs can extract effective semantic features, determining global spatial relationships remains limited due to the locality of convolutional operations. The recently developed transformer-based method can extract global information from images effectively by encoding image tokens. However, its consumption of computational resources varies markedly with increasing image resolution. In this paper, we propose a novel transformer-based neural network for disaster assessment problems. The network uses a multihead neighbourhood attention (MNA) transformer as the base layer of the encoder to achieve more efficient self-attention computation. In addition, the bitemporal feature fusion module (BFFM) performs differential enhancement and injects the change information to the decoder via skip connections. The multiscale tokenizer generates multiscale image tokens to mitigate the loss of detail during encoding. Experimental results on three datasets show that the proposed method outperforms existing methods. [ABSTRACT FROM AUTHOR]

Building detection and change detection using remote sensing images can help urban and rescue planning. Moreover, they can be used for building damage assessment after natural disasters. Currently, most of the existing models for building detection use only one image (pre-disaster image) to detect buildings. This is based on the idea that post-disaster images reduce the model's performance because of the presence of destroyed buildings. In this paper, we propose a siamese model, called SiamixFormer, which uses pre- and post-disaster images as input. Our model has two encoders and has a hierarchical transformer architecture. The output of each stage in both encoders is given to a temporal transformer for feature fusion in a way that query is generated from pre-disaster images and (key, value) is generated from post-disaster images. To this end, temporal features are also considered in feature fusion. Another advantage of using temporal transformers in feature fusion is that they can better maintain large receptive fields generated by transformer encoders compared with CNNs. Finally, the output of the temporal transformer is given to a simple MLP decoder at each stage. The SiamixFormer model is evaluated on xBD, and WHU datasets, for building detection and on LEVIR-CD and CDD datasets for change detection and could outperform the state-of-the-art. [ABSTRACT FROM AUTHOR]

Hyperspectral images carry numerous spectral bands, and their wealth of band data is a valuable source of information for the accurate classification of ground objects. Three-dimensional (3D) convolution, although an excellent spectral information extraction method, is limited by its huge number of parameters and long model training time. To allow better integration of 3D convolution with the most popular transformer models currently available, a new architecture called mobile 3D convolutional vision transformer (MDvT) is proposed. The MDvT introduces inverted residual structure to reduce the number of model parameters and balance the data mining efficiency of low-dimensional data input. Simultaneously, a square patch is used to cut the sequence of tokens to accelerate the model operation. Through extensive experiments, we evaluated the classification overall performance of the proposed MDvT on the WHU-Hi and Pavia University datasets, and demonstrated significant improvements in classification accuracy and model runtime compared with classical deep learning models. It is worth noting that compared with directly integrating 3D convolution into the transformer model, the MDvT architecture improves the accuracy while reducing the time to train an epoch by approximately 58.54%. To facilitate the reproduction of the work in this paper, the model code is available at . [ABSTRACT FROM AUTHOR]