Search

The scene classification of high spatial resolution (HSR) imagery involves labeling an HSR image with a specific high-level semantic class according to the composition of the semantic objects and their spatial relationships. As such, scene classification has attracted increased attention in recent years, and many different algorithms have now been proposed for the cross-scene classification task. However, the recently proposed scene classification methods based on deep convolutional neural networks (CNNs) still suffer from domain shift problems, because of the training data and validation data not following the assumption of independent and identical distributions. The employment of generative adversarial networks has been found to be an effective way to bridge the domain shift/gap. However, the existing cross-scene classification methods do not use the classification information in the target domain, and the domain classifier is task-independent for different scene classification tasks. In this article, to solve this problem, domain adaptation via a task-specific classifier (DATSNET) framework is proposed for HSR image scene classification. Task-specific classifiers and minimizing and maximizing, “ i.e., minimaxing,” of the classifier discrepancy are integrated in the DATSNET framework. The task-specific classifiers are proposed to align the distributions of the source domain features and target domain features by utilizing task-specific decision boundaries in the target domain. In order to align the two task-specific classifiers’ feature distributions, minimaxing the defined discrepancy between the different classifiers in an adversarial manner is proposed to obtain better task-specific classifier boundaries in the target domain and a better-aligned feature distribution in both domains. The experimental results obtained with different remote sensing cross-scene classification tasks demonstrate that the proposed method achieves a significantly improved performance compared with the other state-of-the-art remote sensing cross-scene classification algorithms. [ABSTRACT FROM AUTHOR]

Deep learning (DL) is currently the dominant approach to image classification and segmentation, but the performances of DL methods are remarkably influenced by the quantity and quality of the ground truth (GT) used for training. In this article, a DL method is presented to deal with the semantic segmentation of very-high-resolution (VHR) remote-sensing data in the case of scarce GT. The main idea is to combine a specific type of deep convolutional neural networks (CNNs), namely fully convolutional networks (FCNs), with probabilistic graphical models (PGMs). Our method takes advantage of the intrinsic multiscale behavior of FCNs to deal with multiscale data representations and to connect them to a hierarchical Markov model (e.g., making use of a quadtree). As a consequence, the spatial information present in the data is better exploited, allowing a reduced sensitivity to GT incompleteness to be obtained. The marginal posterior mode (MPM) criterion is used for inference in the proposed framework. To assess the capabilities of the proposed method, the experimental validation is conducted with the ISPRS 2D Semantic Labeling Challenge datasets on the cities of Vaihingen and Potsdam, with some modifications to simulate the spatially sparse GTs that are common in real remote-sensing applications. The results are quite significant, as the proposed approach exhibits a higher producer accuracy than the standard FCNs considered and especially mitigates the impact of scarce GTs on minority classes and small spatial details. [ABSTRACT FROM AUTHOR]

The use of deep learning for water extraction requires precise pixel-level labels. However, it is very difficult to label high-resolution remote-sensing images at the pixel level. Therefore, we study how to utilize point labels to extract water bodies and propose a novel method called the neighbor feature aggregation network (NFANet). Compared with pixel-level labels, point labels are much easier to obtain, but they will lose much information. In this article, we take advantage of the similarity between the adjacent pixels of a local water body, and propose a neighbor sampler to resample remote-sensing images. Then, the sampled images are sent to the network for feature aggregation. In addition, we use an improved recursive training algorithm to further improve the extraction accuracy, making the water boundary more natural. Furthermore, our method utilizes neighboring features instead of global or local features to learn more representative features. The experimental results show that the proposed NFANet method not only outperforms other studied weakly supervised approaches, but also obtains similar results as the state-of-the-art ones. [ABSTRACT FROM AUTHOR]

The profound impact of deep learning and particularly of convolutional neural networks (CNNs) in automatic image processing has been decisive for the progress and evolution of remote sensing (RS) hyperspectral imaging (HSI) processing. Indeed, CNNs have stated themselves as the current state of the art, reaching unparalleled results in HSI classification. However, most CNNs were designed for RGB images, and their direct application to HSI data analysis could lead to nonoptimal solutions. Moreover, CNNs perform classification based on the identification of specific features, neglecting the spatial relationships between different features (i.e., their arrangement) due to pooling techniques. The capsule network (CapsNet) architecture is an attempt to overcome this drawback by nesting several neural layers within a capsule, connected by dynamic routing, both to identify not only the presence of a feature but also its instantiation parameters and to learn the relationships between different features. Although this mechanism improves the data representations, enhancing the classification of HSI data, it still acts as a black box, without control of the most relevant features for classification purposes. Indeed, important features could be discriminated against. In this article, a new multiple attention-guided CapsNet is proposed to improve feature processing for RS-HSIs’ classification, both to improve computational efficiency (in terms of parameters) and increase accuracy. Hence, the most representative visual parts of the images are identified using a detailed feature extractor coupled with attention mechanisms. Extensive experimental results have been obtained on five real datasets, demonstrating the great potential of the proposed method compared to other state-of-the-art classifiers. [ABSTRACT FROM AUTHOR]

Accurate digitization of synoptic ocean features is crucial for climate studies and the operational forecasting of ocean and coupled ocean–atmosphere systems. Today, for some North Atlantic operational regional models, skilled human experts visualize and extract the gulf stream and rings (warm and cold eddies) through an extensive and knowledge-based manual process. To automate this task, we develop a dynamics-inspired deep learning system that extracts the Gulf Stream and rings from concurrent satellite images of sea surface temperature (SST) and sea surface height (SSH). We pose the above problem as a multilabel semantic image segmentation problem. A novel deep convolutional neural network architecture named W-Net, with two parallel encoder–decoder branches, is developed to perform the segmentation. The W-Net’s one branch is the SST branch (accepts SST image as input) and another is the SSH branch (accepts SSH as input), and the final output is a segmentation of gulf stream, warm rings, and cold rings. A dataset consisting of SST, SSH, and manual feature annotation (ground truth) from 2014 to 2018 is used for training. For gulf stream, we obtain 82.7% raw test accuracy and a low error of 4.39% in the detected path length. For the Rings, we obtain more than 71% raw eddy detection accuracy. A detailed ablation study and an examination of both SST and SSH parts of the network are presented to understand how the deep neural network learns to segment the gulf stream’s meandering path and Rings accurately. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have achieved tremendous success in computer vision tasks, such as building extraction. However, due to domain shift, the performance of the CNNs drops sharply on unseen data from another domain, leading to poor generalization. As it is costly and time-consuming to acquire dense annotations for remote-sensing (RS) images, developing algorithms that can transfer knowledge from a labeled source domain to an unlabeled target domain is of great significance. To this end, we propose a novel full-level domain adaptation network (FDANet) for building extraction by combining image-, feature-, and output-level information effectively. At the input level, a simple Wallis filter method is employed to transfer source images into target-like ones whereby alleviating radiometric discrepancy and achieving image-level alignment. To further reduce domain shift, adversarial learning is used to enforce feature distribution consistency constraints between the source and target images. In this way, feature-level alignment can be embedded effectively. At the output level, a mean-teacher model is introduced to enforce transformation-consistent constraint for the target output so that the regularization effect is enhanced and the uncertain predictions can be suppressed as much as possible. To further improve the performance, a novel self-training strategy is also employed by using pseudo labels. The effectiveness of the proposed FDANet is verified on three diverse high-resolution aerial datasets with different resolutions and scenarios. Extensive experimental results and ablation studies demonstrated the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Recently, due to the excellent representation ability of convolutional neural networks (CNNs), object detection in remote sensing images has undergone remarkable development. However, when trained with a small number of samples, the performance of the object detectors drops sharply. In this article, we focus on the following three main challenges of few-shot object detection in remote sensing images: 1) since the sample number of novel classes is far less than base classes, object detectors would fail to quickly adapt to the features of novel classes, which would result in overfitting; 2) the scarcity of samples in novel classes leads to a sparse orientation space, while the objects in remote sensing images usually have arbitrary orientations; and 3) the distribution of object instances in remote sensing images is scattered and, therefore, it is hard to identify foreground objects from the complex background. To tackle these problems, we propose a simple yet effective method named prototype-CNN (P-CNN), which mainly consists of three parts: a prototype learning network (PLN) converting support images to class-aware prototypes, a prototype-guided region proposal network (P-G RPN) for better generation of region proposals, and a detector head extending the head of Faster region-based CNN (R-CNN) to further boost the performance. Comprehensive evaluations on the large-scale DIOR dataset demonstrate the effectiveness of our P-CNN. The source code is available at https://github.com/Ybowei/P-CNN. [ABSTRACT FROM AUTHOR]

The complementarity of synthetic aperture radar (SAR) and optical images allows remote sensing observations to “see” unprecedented discoveries. Image matching plays a fundamental role in the fusion and application of SAR and optical images. However, both the geometric imaging pattern and the physical radiation mechanism of these two sensors are significantly different, so that the images show complex geometric distortion and nonlinear radiation differences. This phenomenon brings great challenges to image matching, which neither the handcrafted descriptors nor the deep learning-based methods have adequately addressed. In this article, a novel image-based matching method for SAR to optical images via an image-based convolutional network with spatial pyramid aggregated pooling (SPAP) and an attention mechanism is proposed, namely MAP-Net. The original image is embedded through the convolutional neural network to generate the feature map. Through the information extraction and abstraction of the original imagery, the embedded features containing the high-level semantic information are more robust to the geometric distortion and radiation variation among the different modal images, which is beneficial to the matching of cross-modal images. The adoption of the SPAP module makes the network more capable of integrating global and local contextual information. The attention block weights the dense features generated from the network to extract the key features that are invariant, distinguishable, repeatable, and suitable for the image matching task. In the experiments, five sets of multisource and multiresolution SAR and optical images with wide and varied ground coverage were used to evaluate the accuracy of MAP-Net, compared to both handcrafted and deep learning-based methods. The experimental results show that the MAP-Net method is superior to the current state-of-the-art image matching methods for SAR to optical images. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) denoising plays an important role in image quality improvement and related applications. Convolutional neural network (CNN)-based image denoising methods have been predominant due to advances made in the field of deep learning in recent years. Spatial and spectral information are crucial to HIS denoising, along with their correlations. However, existing methods fail to consider the global dependence and correlation between spatial and spectral information. Accordingly, in this article, we propose a novel dual-attention denoising network to overcome these limitations. We design two parallel branches to process the spatial and spectral information separately. The position attention module is applied to the spatial branch to formulate the interdependencies on the feature map, while the channel attention module is applied to the spectral branch to simulate the spectral correlation before the two branches are combined. A multiscale structure is also employed to extract and fuse the multiscale features following the fusion of spatial and spectral information. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively when compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The binary segmentation of roads in very high resolution (VHR) remote sensing images (RSIs) has always been a challenging task due to factors such as occlusions (caused by shadows, trees, buildings, etc.) and the intraclass variances of road surfaces. The wide use of convolutional neural networks (CNNs) has greatly improved the segmentation accuracy and made the task end-to-end trainable. However, there are still margins to improve in terms of the completeness and connectivity of the results. In this article, we consider the specific context of road extraction and present a direction-aware residual network (DiResNet) that includes three main contributions: 1) an asymmetric residual segmentation network with deconvolutional layers and a structural supervision to enhance the learning of road topology (DiResSeg); 2) a pixel-level supervision of local directions to enhance the embedding of linear features; and 3) a refinement network to optimize the segmentation results (DiResRef). Ablation studies on two benchmark data sets (the Massachusetts data set and the DeepGlobe data set) have confirmed the effectiveness of the presented designs. Comparative experiments with other approaches show that the proposed method has advantages in both overall accuracy and F1-score. The code is available at: https://github.com/ggsDing/DiResNet. [ABSTRACT FROM AUTHOR]

This article investigates the application of deep learning (DL) to the fusion of omnidirectional (O-D) infrared (IR) sensors and O-D visual sensors to improve the intelligent perception of autonomous robotic systems. Recent techniques primarily focus on O-D and conventional visual sensors for applications in localization, mapping, and tracking. The robotic vision systems have not sufficiently utilized the combination of O-D IR and O-D visual sensors, coupled with DL, for the extraction of vegetation material. We will be showing the contradiction between current approaches and our deep vegetation learning sensor fusion. This article introduces two architectures: 1) the application of two autoencoders feeding into a four-layer convolutional neural network (CNN) and 2) two deep CNN feature extractors feeding a deep CNN fusion network (DeepFuseNet) for the fusion of O-D IR and O-D visual sensors to better address the number of false detects inherent in indices-based spectral decomposition. We compare our DL results to our previous work with normalized difference vegetation index (NDVI) and IR region-based spectral fusion, and to traditional machine learning approaches. This work proves that the fusion of the O-D IR and O-D visual streams utilizing our DeepFuseNet DL approach outperforms both the previous NVDI fused with far-IR region segmentation and traditional machine learning approaches. Experimental results of our method validate a 92% reduction in false detects compared to traditional indices-based detection. This article contributes a novel method for the fusion of O-D visual and O-D IR sensors using two CNN feature extractors feeding into a deep CNN (DeepFuseNet). [ABSTRACT FROM AUTHOR]

Subpixel mapping (SPM) is an effective way to solve the mixed pixel problem, which is a ubiquitous phenomenon in remotely sensed imagery, by characterizing subpixel distribution within the mixed pixels. In fact, the majority of the classical and state-of-the-art SPM algorithms can be viewed as a convolution process, but these methods rely heavily on fixed and handcrafted kernels that are insufficient in characterizing a geographically realistic distribution image. In addition, the traditional SPM approach is based on the prerequisite of abundance images derived from spectral unmixing (SU), during which process uncertainty inherently exists and is propagated to the SPM. In this article, a kernel-learnable convolutional neural network (CNN) framework for subpixel mapping (SPMCNN-F) is proposed. In SPMCNN-F, the kernel is learnable during the training stage based on the given training sample pairs of low- and high-resolution patches for learning a geographically realistic prior, instead of fixed priors. The end-to-end mapping structure enables direct subpixel information extraction from the original coarse image, avoiding the uncertainty propagation from the SU. In the experiments undertaken in this study, two state-of-the-art super-resolution networks were selected as application demonstrations of the proposed SPMCNN-F method. In experiment part, three hyperspectral image data sets were adopted, two in a synthetic coarse image approach and one in a real coarse image approach, for the validation. Additionally, a new data set with pairs of Moderate-resolution Imaging Spectroradiometer (MODIS) and Landsat images were adopted in a real coarse image approach, for further validation of SPMCNN-F in large-scale area. The restored fine distribution images obtained in all the experiments showed a perceptually better reconstruction quality, both qualitatively and quantitatively, confirming the superiority of the proposed SPM framework. [ABSTRACT FROM AUTHOR]

Recently, the graph convolutional network (GCN) has drawn increasing attention in the hyperspectral image (HSI) classification. Compared with the convolutional neural network (CNN) with fixed square kernels, GCN can explicitly utilize the correlation between adjacent land covers and conduct flexible convolution on arbitrarily irregular image regions; hence, the HSI spatial contextual structure can be better modeled. However, to reduce the computational complexity and promote the semantic structure learning of land covers, GCN usually works on superpixel-based nodes rather than pixel-based nodes; thus, the pixel-level spectral–spatial features cannot be captured. To fully leverage the advantages of the CNN and GCN, we propose a heterogeneous deep network called CNN-enhanced GCN (CEGCN), in which CNN and GCN branches perform feature learning on small-scale regular regions and large-scale irregular regions, and generate complementary spectral–spatial features at pixel and superpixel levels, respectively. To alleviate the structural incompatibility of the data representation between the Euclidean data-oriented CNN and non-Euclidean data-oriented GCN, we propose the graph encoder and decoder to propagate features between image pixels and graph nodes, thus enabling the CNN and GCN to collaborate in a single network. In contrast to other GCN-based methods that encode HSI into a graph during preprocessing, we integrate the graph encoding process into the network and learn edge weights from training data, which can promote the node feature learning and make the graph more adaptive to HSI content. Extensive experiments on three data sets demonstrate that the proposed CEGCN is both qualitatively and quantitatively competitive compared with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Physical and/or economic constraints cause acquired seismic data to be incomplete; however, complete data are required for many subsequent seismic processing procedures. Data reconstruction is a crucial and long-standing topic in the exploration seismology field. We extended our previous works on deep learning (DL)-based irregularly and regularly missing 2-D data reconstruction to 3-D data. A key motivation is that the 3-D convolutional neural network (CNN) can take full advantage of the 3-D nature of the data, and the additional dimension allows more information to contribute to the data reconstruction. DL also avoids many assumptions (e.g., linearity, sparsity, and low-rank) limiting conventional nonintelligent reconstruction methods. We built an artificial neural network (ANN) based on an end-to-end U-Net encoder–decoder-style 3-D CNN. The ANN was trained on large quantities of various synthetic and field 3-D seismic data using a mean-squared-error (MSE) loss function and an Adam optimizer. We demonstrated that the developed 3-D CNN reconstruction method appears to outperform the 2-D CNN for 3-D restoration. We benchmarked the ANN’s generalization capacity for recovery of irregularly and regularly sampled 3-D data on several typical seismic data sets, particularly those with high missing percentages or large gaps. An ANN trained with irregularly sampled data can be partly applied to regularly sampled cases. We investigated how a key parameter, i.e., the learning rate, can be experimentally determined. In the context of the presented examples, our methodology provided a substantial improvement over an open-source state-of-the-art rank-reduction-based approach in terms of data fidelity and efficiency. [ABSTRACT FROM AUTHOR]

This article addresses the problem of semantic segmentation with limited cross-modality data in large-scale urban scenes. Most prior works have attempted to address this issue by using multimodal deep neural networks (DNNs). However, their ability to effectively blending different properties across multimodalities and robustly learning representations from complex scenes remains limited, particularly in the absence of sufficient and well-annotated training images. This leads to a challenge related to cross-modality learning with multimodal DNNs. To this end, we introduce two novel plug-and-play units in the network: self-generative adversarial networks (GANs) module and mutual-GANs module, to learn perturbation-insensitive feature representations and to eliminate the gap between multimodalities, respectively, yielding more effective and robust information transfer. Furthermore, a patchwise progressive training strategy is devised to enable effective network learning with limited samples. We evaluate the proposed network on two multimodal (hyperspectral and multispectral) overhead image data sets and achieve a significant improvement in comparison with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Current hyperspectral image classification assumes that a predefined classification system is closed and complete, and there are no unknown or novel classes in the unseen data. However, this assumption may be too strict for the real world. Often, novel classes are overlooked when the classification system is constructed. The closed nature forces a model to assign a label given a new sample and may lead to overestimation of known land covers (e.g., crop area). To tackle this issue, we propose a multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (named MDL4OW) where unknown classes may exist. The reconstructed data are compared with the original data; those failing to be reconstructed are considered unknown based on the assumption that they are not well represented in the latent features due to the lack of labels. A threshold needs to be defined to separate the unknown and known classes; we propose two strategies based on the extreme value theory for few- and many-shot scenarios. The proposed method was tested on real-world hyperspectral images; state-of-the-art results were achieved, e.g., improving the overall accuracy by 4.94% for the Salinas data. By considering the existence of unknown classes in the open world, our method achieved more accurate hyperspectral image classification, especially under the few-shot context. [ABSTRACT FROM AUTHOR]

First arrival time picking is one of the critical processing steps of acoustic emission (AE)/microseismic (MS) monitoring for studying rock fracture processes. Because of massive monitoring data, the automatic arrival time picking technique is particularly desired. Inspired by recent successful applications of machine learning (ML) in earthquake phase identification, we propose a deep learning (DL)-based P-wave first arrival time picking method named AE Network (AEnet) for laboratory AE monitoring data. Our approach consists of two steps: classification and picking. The convolutional neural network (CNN) is used to classify each sample point of acoustic waveforms into either noise or signal. Different from prior DL-based phase picking studies using raw waveforms, we combine the waveform and high-order statistics as the input to enrich the input data features and accelerate the CNN model learning process. Our approach is examined using the laboratory AE monitoring data and the performance of each component of AEnet is also analyzed. The results show that the CNN model can classify the sample points accurately for the picking procedure. With this classification result, we pick the first arrival time of each trace using the curve fitting method and an unsupervised clustering algorithm. To evaluate the performance of AEnet, we apply Akaike Information Criterion-Short Term Averaging/Long Term Averaging Method (AIC-STA/LTA), one of the most popular and traditional picking methods, on the same waveforms and use the manual picks as the reference. Error analysis results show that AEnet outperforms AIC-STA/LTA. [ABSTRACT FROM AUTHOR]

Nowadays, deep learning (DL) finds application in a large number of scientific fields, among which the estimation and the enhancement of signals disrupted by the noise of different natures. In this article, we address the problem of the estimation of the interferometric parameters from synthetic aperture radar (SAR) data. In particular, we combine convolutional neural networks together with the concept of residual learning to define a novel architecture, named Φ-Net, for the joint estimation of the interferometric phase and coherence. Φ-Net is trained using synthetic data obtained by an innovative strategy based on the theoretical modeling of the physics behind the SAR acquisition principle. This strategy allows the network to generalize the estimation problem with respect to: 1) different noise levels; 2) the nature of the imaged target on the ground; and 3) the acquisition geometry. We then analyze the Φ-Net performance on an independent data set of synthesized interferometric data, as well as on real InSAR data from the TanDEM-X and Sentinel-1 missions. The proposed architecture provides better results with respect to state-of-the-art InSAR algorithms on both synthetic and real test data. Finally, we perform an application-oriented study on the retrieval of the topographic information, which shows that Φ-Net is a strong candidate for the generation of high-quality digital elevation models at a resolution close to the one of the original single-look complex data. [ABSTRACT FROM AUTHOR]

The large volumes of Sentinel-1 data produced over Europe are being used to develop pan-national ground motion services. However, simple analysis techniques like thresholding cannot detect and classify complex deformation signals reliably making providing usable information to a broad range of nonexpert stakeholders a challenge. Here, we explore the applicability of deep learning approaches by adapting a pretrained convolutional neural network (CNN) to detect deformation in a national-scale velocity field. For our proof-of-concept, we focus on the U.K. where previously identified deformation is associated with coal-mining, ground water withdrawal, landslides, and tunneling. The sparsity of measurement points and the presence of spike noise make this a challenging application for deep learning networks, which involve calculations of the spatial convolution between images. Moreover, insufficient ground truth data exist to construct a balanced training data set, and the deformation signals are slower and more localized than in previous applications. We propose three enhancement methods to tackle these problems: 1) spatial interpolation with modified matrix completion; 2) a synthetic training data set based on the characteristics of the real U.K. velocity map; and 3) enhanced overwrapping techniques. Using velocity maps spanning 2015–2019, our framework detects several areas of coal mining subsidence, uplift due to dewatering, slate quarries, landslides, and tunnel engineering works. The results demonstrate the potential applicability of the proposed framework to the development of automated ground motion analysis systems. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are becoming increasingly popular in modern remote sensing image processing tasks and exhibit outstanding capability for hyperspectral image (HSI) classification. However, for the existing CNN-based HSI-classification methods, most of them only consider single-scale feature extraction, which may neglect some important fine information and cannot guarantee to capture optimal spatial features. Moreover, many state-of-the-art methods have a huge number of network parameters needed to be tuned, which will cause high computational cost. To address the aforementioned two issues, a novel multiscale residual network (MSRN) is proposed for HSI classification. Specifically, the proposed MSRN introduces depthwise separable convolution (DSC) and replaces the ordinary depthwise convolution in DSC with mixed depthwise convolution (MDConv), which mixes up multiple kernel sizes in a single depthwise convolution operation. The DSC with mixed depthwise convolution (MDSConv) can not only explore features at different scales from each feature map but also greatly reduce learnable parameters in the network. In addition, a multiscale residual block (MRB) is designed by replacing the convolutional layer in an ordinary residual block with the MDSConv layer. The MRB is used as the major unit of the proposed MSRN. Furthermore, to enhance further the feature representation ability, the proposed network adds a high-level shortcut connection (HSC) on the cascaded two MRBs to aggregate lower level features and higher level features. Experimental results on three benchmark HSIs demonstrate the superiority of the proposed MSRN method over several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Multispectral remote sensing (RS) images are often contaminated by the haze that degrades the quality of RS data and reduces the accuracy of interpretation and classification. Recently, the emerging deep convolutional neural networks (CNNs) provide us new approaches for RS image dehazing. Unfortunately, the power of CNNs is limited by the lack of sufficient hazy-clean pairs of RS imagery, which makes supervised learning impractical. To meet the data hunger of supervised CNNs, we propose a novel haze synthesis method to generate realistic hazy multispectral images by modeling the wavelength-dependent and spatial-varying characteristics of haze in RS images. The proposed haze synthesis method not only alleviates the lack of realistic training pairs in multispectral RS image dehazing but also provides a benchmark data set for quantitative evaluation. Furthermore, we propose an end-to-end RSDehazeNet for haze removal. We utilize both local and global residual learning strategies in RSDehazeNet for fast convergence with superior performance. Channel attention modules are incorporated to exploit strong channel correlation in multispectral RS images. Experimental results show that the proposed network outperforms the state-of-the-art methods for synthetic data and real Landsat-8 OLI multispectral RS images. [ABSTRACT FROM AUTHOR]

The trade-off between feature representation power and spatial localization accuracy is crucial for the dense classification/semantic segmentation of remote sensing images (RSIs). High-level features extracted from the late layers of a neural network are rich in semantic information, yet have blurred spatial details; low-level features extracted from the early layers of a network contain more pixel-level information but are isolated and noisy. It is therefore difficult to bridge the gap between high- and low-level features due to their difference in terms of physical information content and spatial distribution. In this article, we contribute to solve this problem by enhancing the feature representation in two ways. On the one hand, a patch attention module (PAM) is proposed to enhance the embedding of context information based on a patchwise calculation of local attention. On the other hand, an attention embedding module (AEM) is proposed to enrich the semantic information of low-level features by embedding local focus from high-level features. Both proposed modules are lightweight and can be applied to process the extracted features of convolutional neural networks (CNNs). Experiments show that, by integrating the proposed modules into a baseline fully convolutional network (FCN), the resulting local attention network (LANet) greatly improves the performance over the baseline and outperforms other attention-based methods on two RSI data sets. [ABSTRACT FROM AUTHOR]

Recently, multimodal cloud samples are utilized to learn completed feature representations for cloud classification. However, the existing methods neglect the related information from other multimodal cloud samples in the learning process, which leads to inadequate learning. In this article, we propose a novel deep model to learn heterogeneous deep features (HDFs) for multimodal ground-based remote sensing cloud classification. Specifically, we first design the convolutional neural network (CNN) extractor to combine the visual information and the multimodal information (MI) to obtain the CNN-based features of multimodal cloud samples. Afterward, we treat the CNN-based features of multimodal cloud samples as the nodes of graph, and utilize the similarity between nodes as the adjacency matrix. We feed the graph and the adjacency matrix into the graph convolutional network (GCN) extractor to obtain the GCN-based features that could capture correlations among multimodal cloud samples using graph convolutional layers. After obtaining CNN-based features and GCN-based features, we concatenate the two kinds of heterogeneous features to represent the multimodal cloud samples. As a result, the concatenated feature contains the visual information, the MI and the related information among multimodal cloud samples. We conduct a series of experiments on the multimodal ground-based cloud database (MGCD), and the experimental results verify that the proposed HDF outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this article, we propose a convolutional neural network with mapping layers (MCNN) for hyperspectral image (HSI) classification. The proposed mapping layers map the input patch into a low-dimensional subspace by multilinear algebra. We use our mapping layers to reduce the spectral and spatial redundancies and maintain most energy of the input. The feature extracted by our mapping layers can also reduce the number of following convolutional layers for feature extraction. Our MCNN architecture avoids the declining accuracy with increasing layers phenomenon of deep learning models for HSI classification and also saves the training time for its effective mapping layers. Furthermore, we impose the 3-D convolutional kernel on the convolutional layer to extract the spectral–spatial features for HSI. We tested our MCNN on three data sets of Indian Pines, University of Pavia, and Salinas, and we achieved the classification accuracy of 98.3%, 99.5%, and 99.3%, respectively. Experimental results demonstrate that the proposed MCNN can significantly improve classification accuracy and save much time consumption. [ABSTRACT FROM AUTHOR]

Generative adversarial networks (GANs) are famous for generating samples by training a generator and a discriminator via an adversarial procedure. For hyperspectral image classification, the collection of samples is always difficult. However, directly applying GAN to hyperspectral image classification exists two problems. One is that the generated samples lack discriminative information. Meanwhile, the discriminator has no discriminative ability for multiclassification. Another is that spatial and spectral information requires to be considered in hyperspectral image classification simultaneously. To address these problems, a novel multiclass spatial–spectral GAN (MSGAN) method is proposed. In MSGAN, two generators are devised to generate the samples containing spatial and spectral information, respectively, and the discriminator is devised to extract joint spatial–spectral features and output multiclass probabilities. Moreover, novel adversarial objectives for multiclass are defined. The discriminator is devised to predict training samples belonging to true classes and generated samples belonging to all the classes with the same probability. The generators are devised to make the discriminator mistake. By adversarial learning between the discriminator and generators, the classification performance of the discriminator is promoted with the assistance of discriminative generated samples. Experimental results on hyperspectral images demonstrate that the proposed method achieves encouraging classification performance compared with several state-of-the-art methods, especially with the limited training samples. [ABSTRACT FROM AUTHOR]

Cloud detection in remote sensing images is a challenging but significant task. Due to the variety and complexity of underlying surfaces, most of the current cloud detection methods have difficulty in detecting thin cloud regions. In fact, it is quite meaningful to distinguish thin clouds from thick clouds, especially in cloud removal and target detection tasks. Therefore, we propose a method based on multiscale features-convolutional neural network (MF-CNN) to detect thin cloud, thick cloud, and noncloud pixels of remote sensing images simultaneously. Landsat 8 satellite imagery with various levels of cloud coverage is used to demonstrate the effectiveness of our proposed MF-CNN model. We first stack visible, near-infrared, short-wave, cirrus, and thermal infrared bands of Landsat 8 imagery to obtain the combined spectral information. The MF-CNN model is then used to learn the multiscale global features of input images. The high-level semantic information obtained in the process of feature learning is integrated with low-level spatial information to classify the imagery into thick, thin and noncloud regions. The performance of our proposed model is compared to that of various commonly used cloud detection methods in both qualitative and quantitative aspects. Compared to other cloud detection methods, the experimental results show that our proposed method has a better performance not only in thick and thin clouds but also in the entire cloud regions. [ABSTRACT FROM AUTHOR]

The challenges in hyperspectral image (HSI) classification lie in the existence of noisy spectral information and lack of contextual information among pixels. Considering the three different levels in HSIs, i.e., subpixel, pixel, and superpixel, offer complementary information, we develop a novel HSI feature learning network (HSINet) to learn consistent features by self-supervision for HSI classification. HSINet contains a three-layer deep neural network and a multifeature convolutional neural network. It automatically extracts the features such as spatial, spectral, color, and boundary as well as context information. To boost the performance of self-supervised feature learning with the likelihood maximization, the conditional random field (CRF) framework is embedded into HSINet. The potential terms of unary, pairwise, and higher order in CRF are constructed by the corresponding subpixel, pixel, and superpixel. Furthermore, the feedback information derived from these terms are also fused into the different-level feature learning process, which makes the HSINet-CRF be a trainable end-to-end deep learning model with the back-propagation algorithm. Comprehensive evaluations are performed on three widely used HSI data sets and our method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In recent studies, synthetic aperture radar (SAR) automatic target recognition (ATR) algorithms based on convolutional neural network (CNN) have achieved high recognition rates in the moving and stationary target acquisition and recognition (MSTAR) data set. However, the correlation between clutter in the training and test data sets is ignored in these algorithms, although most of them used only the center part of the images by removing a lot of the clutter but not everything, which may result in better performance than what would be achieved in the operational scenarios. To tackle this problem, we propose a target segmentation method based on morphological operations to generate data sets without clutter. Then, we design the large-margin softmax (LM-softmax) batch-normalization CNN (LM-BN-CNN) structure, which utilizes the LM-softmax classifier in the last layer to increase the separability of samples after clutter removal. In addition, this structure performs BN with constant mean and variance to increase the convergence speed and reduce overfitting. Experiments on the MSTAR data set have shown that LM-BN-CNN obtains better performance than the available CNNs designed for SAR ATR of ground vehicles, and it has robustness to large depression angle variation, configuration variants, and version variants. [ABSTRACT FROM AUTHOR]