Search

Recent progress in remote sensing image (RSI) super-resolution (SR) has exhibited remarkable performance using deep neural networks, e.g., Convolutional Neural Networks and Transformers. However, existing SR methods often suffer from either a limited receptive field or quadratic computational overhead, resulting in sub-optimal global representation and unacceptable computational costs in large-scale RSI. To alleviate these issues, we develop the first attempt to integrate the Vision State Space Model (Mamba) for RSI-SR, which specializes in processing large-scale RSI by capturing long-range dependency with linear complexity. To achieve better SR reconstruction, building upon Mamba, we devise a Frequency-assisted Mamba framework, dubbed FMSR, to explore the spatial and frequent correlations. In particular, our FMSR features a multi-level fusion architecture equipped with the Frequency Selection Module (FSM), Vision State Space Module (VSSM), and Hybrid Gate Module (HGM) to grasp their merits for effective spatial-frequency fusion. Recognizing that global and local dependencies are complementary and both beneficial for SR, we further recalibrate these multi-level features for accurate feature fusion via learnable scaling adaptors. Extensive experiments on AID, DOTA, and DIOR benchmarks demonstrate that our FMSR outperforms state-of-the-art Transformer-based methods HAT-L in terms of PSNR by 0.11 dB on average, while consuming only 28.05% and 19.08% of its memory consumption and complexity, respectively., Comment: Frequency-Assisted Mamba for Remote Sensing Image Super-Resolution

Hyperspectral images play a crucial role in precision agriculture, environmental monitoring or ecological analysis. However, due to sensor equipment and the imaging environment, the observed hyperspectral images are often inevitably corrupted by various noise. In this study, we proposed a truncated diffusion model, called TDiffDe, to recover the useful information in hyperspectral images gradually. Rather than starting from a pure noise, the input data contains image information in hyperspectral image denoising. Thus, we cut the trained diffusion model from small steps to avoid the destroy of valid information.

Recently, convolutional networks have achieved remarkable development in remote sensing image Super-Resoltuion (SR) by minimizing the regression objectives, e.g., MSE loss. However, despite achieving impressive performance, these methods often suffer from poor visual quality with over-smooth issues. Generative adversarial networks have the potential to infer intricate details, but they are easy to collapse, resulting in undesirable artifacts. To mitigate these issues, in this paper, we first introduce Diffusion Probabilistic Model (DPM) for efficient remote sensing image SR, dubbed EDiffSR. EDiffSR is easy to train and maintains the merits of DPM in generating perceptual-pleasant images. Specifically, different from previous works using heavy UNet for noise prediction, we develop an Efficient Activation Network (EANet) to achieve favorable noise prediction performance by simplified channel attention and simple gate operation, which dramatically reduces the computational budget. Moreover, to introduce more valuable prior knowledge into the proposed EDiffSR, a practical Conditional Prior Enhancement Module (CPEM) is developed to help extract an enriched condition. Unlike most DPM-based SR models that directly generate conditions by amplifying LR images, the proposed CPEM helps to retain more informative cues for accurate SR. Extensive experiments on four remote sensing datasets demonstrate that EDiffSR can restore visual-pleasant images on simulated and real-world remote sensing images, both quantitatively and qualitatively. The code of EDiffSR will be available at https://github.com/XY-boy/EDiffSR, Comment: Submitted to IEEE TGRS

Optical-flow-based and kernel-based approaches have been extensively explored for temporal compensation in satellite Video Super-Resolution (VSR). However, these techniques are less generalized in large-scale or complex scenarios, especially in satellite videos. In this paper, we propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies since we observed that these discrepancies offer distinct and mutually complementary properties. Specifically, we devise a Short-term Temporal Difference Module (S-TDM) to extract local motion representations from RGB difference maps between adjacent frames, which yields more clues for accurate texture representation. To explore the global dependency in the entire frame sequence, a Long-term Temporal Difference Module (L-TDM) is proposed, where the differences between forward and backward segments are incorporated and activated to guide the modulation of the temporal feature, leading to a holistic global compensation. Moreover, we further propose a Difference Compensation Unit (DCU) to enrich the interaction between the spatial distribution of the target frame and temporal compensated results, which helps maintain spatial consistency while refining the features to avoid misalignment. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches. Code will be available at https://github.com/XY-boy/LGTD, Comment: Accepted by IEEE TCSVT

In the fields of image restoration and image fusion, model-driven methods and data-driven methods are the two representative frameworks. However, both approaches have their respective advantages and disadvantages. The model-driven methods consider the imaging mechanism, which is deterministic and theoretically reasonable; however, they cannot easily model complicated nonlinear problems. The data-driven methods have a stronger prior knowledge learning capability for huge data, especially for nonlinear statistical features; however, the interpretability of the networks is poor, and they are over-dependent on training data. In this paper, we systematically investigate the coupling of model-driven and data-driven methods, which has rarely been considered in the remote sensing image restoration and fusion communities. We are the first to summarize the coupling approaches into the following three categories: 1) data-driven and model-driven cascading methods; 2) variational models with embedded learning; and 3) model-constrained network learning methods. The typical existing and potential coupling methods for remote sensing image restoration and fusion are introduced with application examples. This paper also gives some new insights into the potential future directions, in terms of both methods and applications.

The data fusion technology aims to aggregate the characteristics of different data and obtain products with multiple data advantages. To solves the problem of reduced resolution of PolSAR images due to system limitations, we propose a fully polarimetric synthetic aperture radar (PolSAR) images and single-polarization synthetic aperture radar SAR (SinSAR) images fusion network to generate high-resolution PolSAR (HR-PolSAR) images. To take advantage of the polarimetric information of the low-resolution PolSAR (LR-PolSAR) image and the spatial information of the high-resolution single-polarization SAR (HR-SinSAR) image, we propose a fusion framework for joint LR-PolSAR image and HR-SinSAR image and design a cross-attention mechanism to extract features from the joint input data. Besides, based on the physical imaging mechanism, we designed the PolSAR polarimetric loss function for constrained network training. The experimental results confirm the superiority of fusion network over traditional algorithms. The average PSNR is increased by more than 3.6db, and the average MAE is reduced to less than 0.07. Experiments on polarimetric decomposition and polarimetric signature show that it maintains polarimetric information well.

Hyperspectral images are crucial for many research works. Spectral super-resolution (SSR) is a method used to obtain high spatial resolution (HR) hyperspectral images from HR multispectral images. Traditional SSR methods include model-driven algorithms and deep learning. By unfolding a variational method, this paper proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior, resulting in physically interpretable networks. Unlike the fully data-driven CNN, auxiliary spectral response function (SRF) is utilized to guide CNNs to group the bands with spectral relevance. In addition, the channel attention module (CAM) and reformulated spectral angle mapper loss function are applied to achieve an effective reconstruction model. Finally, experiments on two types of datasets, including natural and remote sensing images, demonstrate the spectral enhancement effect of the proposed method. And the classification results on the remote sensing dataset also verified the validity of the information enhanced by the proposed method.

Satellite-based retrieval has become a popular PM2.5 monitoring method currently. To improve the retrieval performance, multiple variables are usually introduced as auxiliary variable in addition to aerosol optical depth (AOD). Different kinds of variables are usually at different resolutions varying from sub-kilometers to dozens of kilometers. Generally, when doing the retrieval, variables at different resolutions are resampled to the same resolution as the AOD product to keep the scale consistency. A deficiency of doing this is that the information contained in the scale difference is discarded. To fully utilize the information contained at different scales, a dual-scale retrieval method is proposed in this study. At the first stage, variables which influence PM2.5 concentration at large scale were used for PM2.5 retrieval at coarse resolution. Then at the second stage, variables which affect PM2.5 distribution in finer scale, were used for the further PM2.5 retrieval at high resolution (sub-km level resolution) with the retrieved PM2.5 at the first stage at coarser resolution also as input. In this study, four different retrieval models including multiple linear regression (MLR), geographically weighted regression (GWR), random forest (RF) and generalized regression neural network (GRNN) are adopted to test the performance of the dual-scale retrieval method. Compared with the traditional retrieval method, the proposed dual-scale retrieval method can achieve PM2.5 mapping at finer resolution and with higher accuracy. Dual-scale retrieval can fully utilize the information contained at different scales, thus achieving a higher resolution and accuracy. It can be used for the generation of quantitative remote sensing products in various fields, and promote the improvement of the quality of quantitative remote sensing products.

Satellite-derived aerosol optical depth (AOD) has been increasingly employed for the estimation of ground-level PM2.5, which is often achieved by modeling the relationship between AOD and PM2.5. To evaluate the accuracy of PM2.5 estimation, the cross-validation (CV) technique has been widely used. There have been several CV-based validation approaches applied for the AOD-PM2.5 models. However, the applicable conditions of these validation approaches still remain unclear. Additionally, is there space to develop better validation approaches for the AOD-PM2.5 models? The contributions of this study can be summarized as two aspects. Firstly, we comprehensively analyze and assess the existing validation approaches, and give the suggestions for applicable conditions of them. Then, the existing validation approaches do not take the distance to monitoring station into consideration. A new validation approach considering the distance to monitoring station is proposed in this study. Among the existing validation approaches, the sample-based CV is used to reflect the overall prediction ability; the site-based CV and the region-based CV have the potentials to evaluate spatial prediction performance; the time-based CV and the historical validation are capable of evaluating temporal prediction accuracy. In addition, the validation results indicate that the proposed validation approach has shown great potentials to better evaluate the accuracy of PM2.5 estimation. This study provides application implications and new perspectives for the validation of AOD-PM2.5 models., Comment: 3 figures, 2 tables

The existence of hybrid noise in hyperspectral images (HSIs) severely degrades the data quality, reduces the interpretation accuracy of HSIs, and restricts the subsequent HSIs applications. In this paper, the spatial-spectral gradient network (SSGN) is presented for mixed noise removal in HSIs. The proposed method employs a spatial-spectral gradient learning strategy, in consideration of the unique spatial structure directionality of sparse noise and spectral differences with additional complementary information for better extracting intrinsic and deep features of HSIs. Based on a fully cascaded multi-scale convolutional network, SSGN can simultaneously deal with the different types of noise in different HSIs or spectra by the use of the same model. The simulated and real-data experiments undertaken in this study confirmed that the proposed SSGN performs better at mixed noise removal than the other state-of-the-art HSI denoising algorithms, in evaluation indices, visual assessments, and time consumption., Comment: Accept by IEEE TGRS

The integration of satellite-derived aerosol optical depth (AOD) and station-measured PM2.5 provides a promising approach for obtaining spatial PM2.5 data. Several spatiotemporal models, which considered spatial and temporal heterogeneities of AOD-PM2.5 relationship, have been widely adopted for PM2.5 estimation. However, they generally described the complex AOD-PM2.5 relationship based on a linear hypothesis. Previous machine learning models yielded great superiorities for fitting the nonlinear AOD-PM2.5 relationship, but seldom allowed for its spatiotemporal variations. To simultaneously consider the nonlinearity and spatiotemporal heterogeneities of AOD-PM2.5 relationship, geographically and temporally weighted neural networks (GTWNNs) were developed for satellite-based estimation of ground-level PM2.5 in this study. Using satellite AOD products, NDVI data, and meteorological factors over China as input, GTWNNs were set up with station PM2.5 measurements. Then the spatial PM2.5 data of those locations with no ground stations could be obtained. The proposed GTWNNs have achieved a better performance compared with previous spatiotemporal models, i.e., daily geographically weighted regression and geographically and temporally weighted regression. The sample-based and site-based cross-validation R2 values of GTWNNs are 0.80 and 0.79, respectively. On this basis, the spatial PM2.5 data with a resolution of 0.1 degree were generated in China. This study implemented the combination of geographical law and neural networks, and improved the accuracy of satellite-based PM2.5 estimation., Comment: 5 figures, 3 tables

Destriping is a classical problem in remote sensing image processing. Although considerable effort has been made to remove stripes, few of the existing methods can eliminate stripe noise with arbitrary orientations. This situation makes the removal of oblique stripes in the higher-level remote sensing products become an unfinished and urgent issue. To overcome the challenging problem, we propose a novel destriping model which is self-adjusted to different orientations of stripe noise. First of all, the oriented variation model is designed to accomplish the stripe orientation approximation. In this model, the stripe direction is automatically estimated and then imbedded into the constraint term to depict the along-stripe smoothness of the stripe component. Mainly based on the oriented variation model, a whole destriping framework is proposed by jointly employing an L1-norm constraint and a TV regularization to separately capture the global distribution property of stripe component and the piecewise smoothness of the clean image. The qualitative and quantitative experimental results of both orientation and destriping aspects confirm the effectiveness and stability of the proposed method., Comment: 14 pages, 14 figures

In this study, we evaluated the performance of the MODIS C6.1 AOD products and compared them with the C6 products over regions in China with multifarious underlying surfaces during 2001 to 2016. The AOD retrievals were validated against 20 AERONET sites, and the results show that the R for DT retrievals in C6.1 is 0.946, while the fraction within the EE can be considered relatively low at only 54.03%. DB retrievals in C6.1 have a slightly lower R value, but the other criteria are superior to DT. Comparing the results over urban and vegetation areas in C6.1, the overall quality of the DB retrievals is better than the DT retrievals in urban areas. The performance of DT is significantly superior to DB in the LEV areas. For the HEV areas, DB performs synthetically better than DT. In the spatial distribution aspect in C6.1, most of the DB AOD values are less than those of DT, and the relationship between DT and DB varies with the different land cover types. For the AOD coverage in C6.1, DT retrievals with high coverage mainly distribute in east-central China. However, the effects of high surface reflectance lead to low AOD coverage in the southwest. In contrast, the AOD coverage of DB tends to be high in areas where the main land cover type is bare soil and tends to be low in areas affected by snow. In terms of the comparison between C6.1 and C6, the overestimation of DT over urban areas in C6 is effectively mitigated in C6.1. However, a nearly systematic decline in DT is discovered in C6.1 as well. With respect to DB, consistent AOD coverage distribution is observed, with only subtle distinction. The AOD coverage of DB in C6.1 appears higher than that in C6 in the middle, south, and northeast of China. The quality of the DB retrievals in C6.1 increases slightly compared to C6, and the most remarkable improvement is observed for the coarse aerosol particles., Comment: This paper is under review

In the field of spatial-spectral fusion, the model-based method and the deep learning (DL)-based method are state-of-the-art. This paper presents a fusion method that incorporates the deep neural network into the model-based method for the most common case in the spatial-spectral fusion: PAN/multispectral (MS) fusion. Specifically, we first map the gradient of the high spatial resolution panchromatic image (HR-PAN) and the low spatial resolution multispectral image (LR-MS) to the gradient of the high spatial resolution multispectral image (HR-MS) via a deep residual convolutional neural network (CNN). Then we construct a fusion framework by the LR-MS image, the gradient prior learned from the gradient network, and the ideal fused image. Finally, an iterative optimization algorithm is used to solve the fusion model. Both quantitative and visual assessments on high-quality images from various sources demonstrate that the proposed fusion method is superior to all the mainstream algorithms included in the comparison in terms of overall fusion accuracy.

Satellite aerosol products have been widely used to retrieve ground PM2.5 concentration because of its wide coverage and continuously spatial distribution. While more and more studies focus on the retrieval algorithm, we find that the relationship between PM2.5 concentration and satellite AOD has not been fully discussed in China. Is satellite AOD always a good indicator for PM2.5 in different regions and can AOD still be employed to retrieve PM2.5 with pollution conditions changing in these years remain unclear. In this study, the relationships between PM2.5 and AOD were investigated in 368 cities in China for a continuous period from February 2013 to December 2017, at different time and regional scales. Pearson correlation coefficients and PM2.5/AOD ratio were used as the indicator. Firstly, we concluded the relationship of PM2.5 and AOD in terms of spatiotemporal variations. Then the impact of meteorological factors, aerosol size and topography were discussed. Finally, a GWR retrieval experiment was conducted to find out how was the retrieval accuracy changing with the varying of PM2.5-AOD relationship. We found that spatially the correlation is higher in Beijing-Tianjin-Hebei and Chengyu region and weaker in coastal areas such as Yangtze River Delta and Pearl River Delta. The PM2.5/AOD ratio has obvious North-South difference with a high ratio in north China and a lower ratio in south China. Temporally, PM2.5/AOD ratio is higher in winter and lower in summer, the correlation coefficient tends to be higher in May and September. As for interannual variations from 2013 to 2017, we detected a declining tendency on PM2.5/AOD ratio. The accuracy of GWR retrievals were decreasing too, which may imply that AOD may not be a good indicator for PM2.5 in the future.

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based method for this task is proposed, by learning a non-linear end-to-end mapping between the noisy and clean HSIs with a combined spatial-spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multi-scale feature extraction and multi-level feature representation are respectively employed to capture both the multi-scale spatial-spectral feature and fuse the feature representations with different levels for the final restoration. The simulated and real-data experiments demonstrate that the proposed HSID-CNN outperforms many of the mainstream methods in both the quantitative evaluation indexes, visual effects, and HSI classification accuracy., Comment: Accepted by IEEE TGRS, available codes: https://github.com/WHUQZhang/HSID-CNN

Satellite remote sensing has been reported to be a promising approach for the monitoring of atmospheric PM2.5. However, the satellite-based monitoring of ground-level PM2.5 is still challenging. First, the previously used polar-orbiting satellite observations, which can be usually acquired only once per day, are hard to monitor PM2.5 in real time. Second, many data gaps exist in satellite-derived PM2.5 due to the cloud contamination. In this paper, the hourly geostationary satellite (i.e., Himawari-8) observations were adopted for the real-time monitoring of PM2.5 in a deep learning architecture. On this basis, the satellite-derived PM2.5 in conjunction with ground PM2.5 measurements are incorporated into a spatio-temporal fusion model to fill the data gaps. Using Wuhan Urban Agglomeration as an example, we have successfully derived the real-time and seamless PM2.5 distributions. The results demonstrate that Himawari-8 satellite-based deep learning model achieves a satisfactory performance (out-of-sample cross-validation R2=0.80, RMSE=17.49 ug/m3) for the estimation of PM2.5. The missing data in satellite-derive PM2.5 are accurately recovered, with R2 between recoveries and ground measurements of 0.75. Overall, this study has inherently provided an effective strategy for the real-time and seamless monitoring of ground-level PM2.5., Comment: Paper presented in ISPRS TC III Symposium (May 7-10, 2018 Beijing,China)

Because of the internal malfunction of satellite sensors and poor atmospheric conditions such as thick cloud, the acquired remote sensing data often suffer from missing information, i.e., the data usability is greatly reduced. In this paper, a novel method of missing information reconstruction in remote sensing images is proposed. The unified spatial-temporal-spectral framework based on a deep convolutional neural network (STS-CNN) employs a unified deep convolutional neural network combined with spatial-temporal-spectral supplementary information. In addition, to address the fact that most methods can only deal with a single missing information reconstruction task, the proposed approach can solve three typical missing information reconstruction tasks: 1) dead lines in Aqua MODIS band 6; 2) the Landsat ETM+ Scan Line Corrector (SLC)-off problem; and 3) thick cloud removal. It should be noted that the proposed model can use multi-source data (spatial, spectral, and temporal) as the input of the unified framework. The results of both simulated and real-data experiments demonstrate that the proposed model exhibits high effectiveness in the three missing information reconstruction tasks listed above., Comment: To be published in IEEE Transactions on Geoscience and Remote Sensing

Pan-sharpening is a fundamental and significant task in the field of remote sensing imagery processing, in which high-resolution spatial details from panchromatic images are employed to enhance the spatial resolution of multi-spectral (MS) images. As the transformation from low spatial resolution MS image to high-resolution MS image is complex and highly non-linear, inspired by the powerful representation for non-linear relationships of deep neural networks, we introduce multi-scale feature extraction and residual learning into the basic convolutional neural network (CNN) architecture and propose the multi-scale and multi-depth convolutional neural network (MSDCNN) for the pan-sharpening of remote sensing imagery. Both the quantitative assessment results and the visual assessment confirm that the proposed network yields high-resolution MS images that are superior to the images produced by the compared state-of-the-art methods.

Almost all remote sensing atmospheric PM2.5 estimation methods need satellite aerosol optical depth (AOD) products, which are often retrieved from top-of-atmosphere (TOA) reflectance via an atmospheric radiative transfer model. Then, is it possible to estimate ground-level PM2.5 directly from satellite TOA reflectance without a physical model? In this study, this challenging work are achieved based on a machine learning model. Specifically, we establish the relationship between PM2.5, satellite TOA reflectance, observation angles, and meteorological factors in a deep learning architecture (denoted as Ref-PM modeling). Taking the Wuhan Urban Agglomeration (WUA) as a case study, the results demonstrate that compared with the AOD-PM modeling, the Ref-PM modeling obtains a competitive performance, with out-of-sample cross-validated R2 and RMSE values of 0.87 and 9.89 ug/m3 respectively. Also, the TOA-reflectance-derived PM2.5 have a finer resolution and larger spatial coverage than the AOD-derived PM2.5. This work updates the traditional cognition of remote sensing PM2.5 estimation and has the potential to promote the application in atmospheric environmental monitoring., Comment: This paper is under review

In this paper, to break the limit of the traditional linear models for synthetic aperture radar (SAR) image despeckling, we propose a novel deep learning approach by learning a non-linear end-to-end mapping between the noisy and clean SAR images with a dilated residual network (SAR-DRN). SAR-DRN is based on dilated convolutions, which can both enlarge the receptive field and maintain the filter size and layer depth with a lightweight structure. In addition, skip connections and residual learning strategy are added to the despeckling model to maintain the image details and reduce the vanishing gradient problem. Compared with the traditional despeckling methods, the proposed method shows superior performance over the state-of-the-art methods on both quantitative and visual assessments, especially for strong speckle noise., Comment: 18 pages, 13 figures, 7 tables

The interactions between PM2.5 and meteorological factors play a crucial role in air pollution analysis. However, previous studies that have researched the relationships between PM2.5 concentration and meteorological conditions have been mainly confined to a certain city or district, and the correlation over the whole of China remains unclear. Whether or not spatial and seasonal variations exit deserves further research. In this study, the relationships between PM2.5 concentration and meteorological factors were investigated in 74 major cities in China for a continuous period of 22 months from February 2013 to November 2014, at season, year, city, and regional scales, and the spatial and seasonal variations were analyzed. The meteorological factors were relative humidity (RH), temperature (TEM), wind speed (WS), and surface pressure (PS). We found that spatial and seasonal variations of their relationships with PM2.5 do exist. Spatially, RH is positively correlated with PM2.5 concentration in North China and Urumqi, but the relationship turns to negative in other areas of China. WS is negatively correlated with PM2.5 everywhere expect for Hainan Island. PS has a strong positive relationship with PM2.5 concentration in Northeast China and Mid-south China, and in other areas the correlation is weak. Seasonally, the positive correlation between PM2.5 concentration and RH is stronger in winter and spring. TEM has a negative relationship with PM2.5 in autumn and the opposite in winter. PS is more positively correlated with PM2.5 in autumn than in other seasons. Our study investigated the relationships between PM2.5 and meteorological factors in terms of spatial and seasonal variations, and the conclusions about the relationships between PM2.5 and meteorological factors are more comprehensive and precise than before., Comment: 3 tables, 13 figures