Search

The neural network-based remote sensing image change detection method faces a large amount of imaging interference and severe class imbalance problems under high-resolution conditions, which bring new challenges to the accuracy of the detection network. In this work, to address the imaging interference caused by different imaging angles and times, the siamese strategy and multi-head self-attention mechanism are used to reduce the imaging differences between the dual-temporal images and fully exploit the inter-temporal information. Secondly, a learnable multi-part feature learning module is used to adaptively exploit features from different scales to obtain more comprehensive features. Finally, a mixed loss function strategy is used to ensure that the network converges effectively and excludes the adverse interference of a large number of negative samples to the network. Extensive experiments show that our method outperforms numerous methods on LEVIR-CD, WHU, and DSIFN datasets. [ABSTRACT FROM AUTHOR]

Remote sensing images semantic segmentation is a difficult instance of image understanding. Due to the regional variability and uncertainty of real-world ground cover features, the semantic segmentation of remote sensing images becomes a challenging task. In this paper, we propose an end-to-end multi-source remote sensing image semantic segmentation network (MCENet) aiming at the problems of intra-class inconsistency and inter-class indistinguishability in remote sensing images. Firstly, we design a collaborative enhanced fusion module to mine complementary characteristics of multi-source remote sensing images. Among them, the collaborative fusion module is used to solve the problem of intra-class difference, and the enhanced aggregation module is used to solve the problem of inter-class similarity. Secondly, a multi-scale decoder is proposed to improve the robustness of the model for small targets and large-scale changes by learning scale invariance features. Experimental results show that our method achieved 2.2% and 1.11% mean intersection over union (mIoU) score improvements compared with other methods on the US3D and ISPRS Potsdam data sets, respectively. In addition, the method proposed in this paper also has strong competitiveness in terms of parameter quantity and inference speed. [ABSTRACT FROM AUTHOR]

Due to all-weather and all-time work characteristics, synthetic aperture radar (SAR) images have been widely used in remote sensing. There is great difficulty in understanding SAR images because they are quite different from optical images in imaging mechanism, geometric characteristics, and radiation characteristics. It can greatly improve the readability of SAR images if we can translate them into optical image styles. In this paper, we propose a variational generative adversarial network for SAR images to optical images translation (S2O-VGAN). To demonstrate the validity of the proposed model, a new large-scale dataset called SARGB is proposed. Experimental results based on the proposed dataset with several evaluations show the superiority of the proposed model over the existing methods on SAR-optical image translation. [ABSTRACT FROM AUTHOR]

Global context information is essential for the semantic segmentation of remote sensing (RS) images. However, most existing methods rely on a convolutional neural network (CNN), which is challenging to directly obtain the global context due to the locality of the convolution operation. Inspired by the Swin transformer with powerful global modeling capabilities, we propose a novel semantic segmentation framework for RS images called ST-U-shaped network (UNet), which embeds the Swin transformer into the classical CNN-based UNet. ST-UNet constitutes a novel dual encoder structure of the Swin transformer and CNN in parallel. First, we propose a spatial interaction module (SIM), which encodes spatial information in the Swin transformer block by establishing pixel-level correlation to enhance the feature representation ability of occluded objects. Second, we construct a feature compression module (FCM) to reduce the loss of detailed information and condense more small-scale features in patch token downsampling of the Swin transformer, which improves the segmentation accuracy of small-scale ground objects. Finally, as a bridge between dual encoders, a relational aggregation module (RAM) is designed to integrate global dependencies from the Swin transformer into the features from CNN hierarchically. Our ST-UNet brings significant improvement on the ISPRS-Vaihingen and Potsdam datasets, respectively. The code will be available at https://github.com/XinnHe/ST-UNet. [ABSTRACT FROM AUTHOR]

The appropriate setting of hyperparameter is a key factor to determine the performance of the deep learning model. Efficient hyperparametric optimization algorithm can not only improve the efficiency and speed of model hyperparametric optimization, but also reduce the application threshold of deep learning model. Therefore, we propose a parameter learning algorithm-based co-evolutionary for remote sensing scene classification. First, a co-evolution framework is proposed to optimize the optimizer's hyperparameters and weight parameters of the convolutional neural networks (CNNs) simultaneously. Second, with the strategy of co-evolution with two populations, the hyperparameters can learn within the population and the weights of CNN can be updated by utilizing information between the populations. Finally, the parallel computing mechanism is adapted to speed up the learning process, as the two populations can evolve simultaneously. Extensive experiments on three public datasets demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Currently, the majority of publicly raindrop scattering database are calculated with spherical particles. However, as the raindrop grows, the bottom of the raindrop gradually flattens out. The differences of raindrop shape will lead to a difference of the calculated scattering parameters from reality, and further influencing the accuracy of calculations in the fields of radar detection, microwave transmission and satellite remote sensing. In this regard, we developed a non-spherical raindrop scattering parameter database (RP database) with frequencies from 3GHz to 1000GHz. It has been published in an open-access repository. We compared the RP database with the database of Ekelund et al., and found that their difference is basically under 5%; Further the comparisons were also made between the scattering parameters of ellipsoids, spherical raindrops and the non-spherical raindrops, both spherical and ellipsoidal particles have clear difference from non-spherical particles. Finally, we analyzed the group particle attenuation at different frequencies. • A database of the Raindrop Scattering Properties covering the Millimeter and Sub-Millimeter band is established. • The actual shape of the rain droplet is considered in the scattering simulation. • The appicablity of different shape model of rain droplet is systematically analyzed. [ABSTRACT FROM AUTHOR]

Compared with a single image, in a complex environment, image fusion can utilize the complementary information provided by multiple sensors to significantly improve the image clarity and the information, more accurate, reliable, comprehensive access to target and scene information. It is widely used in military and civil fields, such as remote sensing, medicine, security and other fields. In this paper, we propose an end-to-end fusion framework based on structural similarity preserving GAN (SSP-GAN) to learn a mapping of the fusion tasks for visible and infrared images. Specifically, on the one hand, for making the fusion image natural and conforming to visual habits, structure similarity is introduced to guide the generator network produce abundant texture structure information. On the other hand, to fully take advantage of shallow detail information and deep semantic information for achieving feature reuse, we redesign the network architecture of multi-modal image fusion meticulously. Finally, a wide range of experiments on real infrared and visible TNO dataset and RoadScene dataset prove the superior performance of the proposed approach in terms of accuracy and visual. In particular, compared with the best results of other seven algorithms, our model has improved entropy, edge information transfer factor, multi-scale structural similarity and other evaluation metrics, respectively, by 3.05%, 2.4% and 0.7% on TNO dataset. And our model has also improved by 0.7%, 2.82% and 1.1% on RoadScene dataset. [ABSTRACT FROM AUTHOR]

Image captioning is a task generating the natural semantic description of the given image, which plays an essential role for machines to understand the content of the image. Remote sensing image captioning is a part of the field. Most of the current remote sensing image captioning models failed to fully utilize the semantic information in images and suffered the overfitting problem induced by the small size of the dataset. To this end, we propose a new model using the Transformer to decode the image features to target sentences. For making the Transformer more adaptive to the remote sensing image captioning task, we additionally employ dropout layers, residual connections, and adaptive feature fusion in the Transformer. Reinforcement Learning is then applied to enhance the quality of the generated sentences. We demonstrate the validity of our proposed model on three remote sensing image captioning datasets. Our model obtains all seven higher scores on the Sydney Dataset and Remote Sensing Image Caption Dataset (RSICD), four higher scores on UCM dataset, which indicates that the proposed methods perform better than the previous state of the art models in remote sensing image caption generation. [ABSTRACT FROM AUTHOR]

The convolutional neural networks (CNNs) have shown powerful feature representation capability, which provides novel avenues to improve scene classification of remote sensing imagery. Although we can acquire large collections of satellite images, the lack of rich label information is still a major concern in the remote sensing field. In addition, remote sensing data sets have their own limitations, such as the small scale of scene classes and lack of image diversity. To mitigate the impact of the existing problems, a Siamese CNN, which combines the identification and verification models of CNNs, is proposed in this letter. A metric learning regularization term is explicitly imposed on the features learned through CNNs, which enforce the Siamese networks to be more robust. We carried out experiments on three widely used remote sensing data sets for performance evaluation. Experimental results show that our proposed method outperforms the existing methods. [ABSTRACT FROM AUTHOR]

Feature pyramid networks are widely applied in remote sensing images for object detection to deal with the challenge of large scale variation in objects. However, the feature pyramid-based object detector for remote sensing images ignores the channel information loss, feature misalignment, and additional computational overhead to eliminate the aliasing effect, leading to inadequate feature extraction for multi-scale objects in remote sensing images. To address these challenges, an Informative Feature Pyramid Network (Info-FPN) is proposed. Specifically, we propose a Pixel Shuffle-based lateral connection Module (PSM) for the complete preservation of channel information in the feature pyramid. Then, to alleviate the problem of confusion caused by feature misalignment, a Feature Alignment Module (FAM) is proposed to achieve aligned feature fusion by template matching and learning feature offsets in the feature fusion stage. To eliminate the aliasing effect, we design a Semantic Encoder Module (SEM), which reduces the parameters and computation of model with the desirable detection accuracy. Extensive experiments on two challenging remote sensing datasets, namely DOTA and HRSC2016, prove the effectiveness of the proposed method which achieves comparable detection performance with the state-of-the-art FPN-based method. • A new feature pyramid network is proposed for remote sensing images detection. • Reducing the channel information loss for remote sensing images using Pixel Shuffle. • Offset learning and template matching align multi-scale features. • Simple encoder can replace redundant calculations that eliminate aliasing effects. • Our model achieves state-of-the-art performance for remote sensing images detection. [ABSTRACT FROM AUTHOR]

Recently, very high resolution (VHR) panchromatic and multispectral (MS) remote-sensing images can be acquired easily. However, it is still a challenging task to fuse and classify these VHR images. Generally, there are two ways for the fusion and classification of panchromatic and MS images. One way is to use a panchromatic image to sharpen an MS image, and then classify a pan-sharpened MS image. Another way is to extract features from panchromatic and MS images, respectively, and then combine these features for classification. In this paper, we propose a superpixel-based multiple local convolution neural network (SML-CNN) model for panchromatic and MS images classification. In order to reduce the amount of input data for the CNN, we extend simple linear iterative clustering algorithm for segmenting MS images and generating superpixels. Superpixels are taken as the basic analysis unit instead of pixels. To make full advantage of the spatial-spectral and environment information of superpixels, a superpixel-based multiple local regions joint representation method is proposed. Then, an SML-CNN model is established to extract an efficient joint feature representation. A softmax layer is used to classify these features learned by multiple local CNN into different categories. Finally, in order to eliminate the adverse effects on the classification results within and between superpixels, we propose a multi-information modification strategy that combines the detailed information and semantic information to improve the classification performance. Experiments on the classification of Vancouver and Xi’an panchromatic and MS image data sets have demonstrated the effectiveness of the proposed approach. [ABSTRACT FROM PUBLISHER]

Semantic segmentation is a fundamental task in the field of remote sensing image intelligent interpretation and computer vision. Multispectral remote sensing images have attracted more and more researchers' attention because they can accurately describe different types of reflection spectra. However, inaccurate multispectral feature description leads to edge semantic ambiguity and misclassification of small objects. In this article, we propose a novel network named edge-aware and spectral–spatial information aggregation net (ESSANet) to capture both high-level semantic features and low-level edge details for semantic segmentation of remote sensing images. Specifically, on the one hand, in order to improve the representation ability of discriminant features, we design a two-stream spectral–spatial feature extraction network via 3D hybrid convolution and multi-level aggregation network. On the other hand, in order to eliminate the effect of edge semantic ambiguity, we develop a siamese edge-aware structure and multi-stage edge loss function. Experimental results show that our method achieved 3.5% and 4.09% mean intersection over union (mIoU) score improvements and 2.59% and 3.32% Kappa score improvements compared with the competitive baseline algorithm on the SEN12MS and US3D datasets, respectively. In addition, the method proposed in this paper also achieves a better trade-off between speed and accuracy. [ABSTRACT FROM AUTHOR]

Convolutional neural networks have achieved remarkable success in the field of computer vision. However, due to their high storage and expensive computations, recently, there has been a lot of work focusing on reducing the complexity of convolutional neural networks. In this work, we propose a random filter pruning method by means of evolutionary multiobjective optimization algorithms to accelerate the Siamese ResNet-50 for remote sensing scene classification. We have conduct experiments on NWPU-RESISC45, UC Merced Land-Use and SIRI-WHU datasets for performance evaluation of the proposed method. The experimental results demonstrate that the classification performance of our pruned model has been improved while keeping a certain degree of sparsity of the model. [ABSTRACT FROM AUTHOR]

The remote sensing data is difficult to collect and lack of diversity, which extremely limits the performance of object detection on remote sensing images. In this paper, a multi-branch conditional generative adversarial network (MCGAN) is proposed to augment data for object detection in optical remote sensing images, which is the first GANs-based data augmentation framework proposed for this topic. We use MCGAN to generate the diverse objects based on the existing remote sensing datasets. The multi-branch dilated convolution and the classification branch are adopted into MCGAN to help the generator to generate the diverse and high-quality images. Meanwhile, an adaptive samples selection strategy based on the Faster R-CNN is proposed to select the samples for data augmentation from the objects generated by MCGAN, which can ensure the quality of new augmented training sets and improve the diversity of samples. Experiments based on NWPU VHR-10 and DOTA show that the objects generated by MCGAN have the higher quality compared with the objects generated by WGAN and LSGAN. And the mean average precision detected by the state-of-the-art object detection models used in the experiments has the satisfactory improvement after the MCGAN based data augmentation, which indicates that data augmentation by MCGAN can effectively improve the accuracy of remote sensing images object detection. [ABSTRACT FROM AUTHOR]

Existing remote sensing semantic segmentation methods generally ignore the structural information of objects that is vital in the human visual recognition system. The absence of overall structural information often results in weak perceptions of subtle textures and fragmented predictions, especially for complex and variable ground object scenarios. Besides, they still suffer from the semantic ambiguity caused by the unclear object boundary features in remote sensing images. In this paper, we propose a novel remote sensing semantic segmentation framework, called CSBNet, which aims to enhance the capacity of class-guided structural interaction and boundary perception simultaneously. It consists of a class-guided structure interaction module (CSIM), a Transformer-based context aggregation module (TCAM) and a class-guided boundary supervision module (CBSM). The CSIM has the ability to progressively extract the class-specific structural features, i.e. , refining the structural information of each class by iteratively exchanging information between initial coarse class tokens and contexts. Meanwhile, the TCAM is constructed to provide CSIM with more discriminative multi-scale contexts without losing spatial features. In particular, the CBSM plays an auxiliary role, which applies the boundary information obtained from the class tokens to supervise the segmentation of boundary regions. When tested on the ISPRS dataset, LoveDA dataset, UAVid dataset, our method significantly outperforms the state-of-the-art remote sensing semantic segmentation approaches. [ABSTRACT FROM AUTHOR]

• We proposed a new drought monitoring index; • We corroborated the applicability of this newly-proposed drought index; • We characterized spatial and temporal patterns of droughts across China. In this study, we proposed a new integrated remote sensing drought monitoring indices, i.e. Multiple Remote Sensing Drought Index integrated by Principal Component Analysis (PSDI), Multiple Remote Sensing Drought Index integrated by multiple linear regression (MRSDI) and Multiple Remote Sensing drought index integrated by gradient boosting method (GBMDI), based on the Precipitation Condition Index (PCI), Temperature Condition Index (TCI), Vegetation Condition Index (VCI), and Soil Moisture Condition Index (SMCI). The monitoring performance of PSDI, MRSDI and GBMDI was compared and verified based on the real-world observed droughts during 2002 to 2016. We also evidenced drought monitoring performance of the PSDI MRSDI and GBMDI by comparison between PSDI, MRSDI, GBMDI and SPEI, SPI and PDSI based on the in situ observed meteorological data. We found that the spatiotemporal characteristics of droughts monitored by the PSDI, MRSDI and GBMDI were generally in good agreement with those by the SPI and SPEI. The GBMDI performs better than PSDI and MRSDI in describing drought processes and spatial patterns of droughts of different drought intensities. Comparison between the real-world observed drought-affected croplands and those monitored by PSDI, MRSDI and GBMDI indicated better drought monitoring performance of GBMDI than PSDI and MRSDI in monitoring droughts across widespread drought-affected regions. Besides, the trend of GBMDI is in good agreement with standardized crop yield. Therefore GBMDI should be the first choice in drought monitoring practice. The GBMDI developed in this study can help to provide an alternative drought monitoring index for large-scale drought monitoring across China and also in other regions of the globe. [ABSTRACT FROM AUTHOR]

Image captioning, i.e., generating the natural semantic descriptions of given image, is an essential task for machines to understand the content of the image. Remote sensing image captioning is a part of the field. Most of the current remote sensing image captioning models suffered the overfitting problem and failed to utilize the semantic information in images. To this end, we propose a V ariational Autoencoder and R einforcement Learning based T wo-stage M ulti-task Learning M odel (VRTMM) for the remote sensing image captioning task. In the first stage, we finetune the CNN jointly with the Variational Autoencoder. In the second stage, the Transformer generates the text description using both spatial and semantic features. Reinforcement Learning is then applied to enhance the quality of the generated sentences. Our model surpasses the previous state of the art records by a large margin on all seven scores on Remote Sensing Image Caption Dataset. The experiment result indicates our model is effective on remote sensing image captioning and achieves the new state-of-the-art result. • Introducing VAE to regularize the shared encoder and extract image features more effectively by reconstructing input images. • Improving the performance of image caption significantly by virtue of low-level and high-level image features simultaneously. • Enhancing the final text description quality by adding self-attention to spatial features. • Our proposed model outperforms the state-of-the-art models in the remote sensing image captioning. [ABSTRACT FROM AUTHOR]