Search

Pan-sharpening aims to obtain high resolution multispectral (HRMS) images by integrating the information in the panchromatic and multispectral images. Existing pan-sharpening methods have demonstrated impressive sharpening performance. However, these methods inherently overlook the complementary characteristics and interaction between diverse source images, resulting in sharpened outcomes accompanied by distortion. To solve the above problems, we construct a novel cross fusion model with fine-grained detail reconstruction from the perspective of frequency-domain. The motivation of the model is twofold: (1) to reconstruct spatial detail representations from diverse source images, laying the foundation for the generation of fine details in the subsequent fused images; and (2) to enhance the interaction between diverse source features during the fusion process in order to attain high-fidelity fusion outcomes. Based on the theoretical model, we develop a frequency-spectral dual domain cross fusion network (CF2N) utilizing the deep learning technique. Consequently, the CF2N consist of two main stages, namely frequency-domain dominated detail reconstruction (FD2R) and frequency-spectral cross fusion (FSCF). Specifically, a more reasonable reconstruction of fine frequency details in HRMS can be achieved by performing adaptive weighted fusion of frequency details in the FD2R stage. Furthermore, the FSCF module, which seamlessly integrates frequency- and spectral-domain details in a highly interactive cross fusion manner. As a result, the CF2N possesses the capability to attain high frequency-spectral fidelity results with excellent interpretability. Extensive experiments show the superior performance of ours over state of the art, while maintaining high efficiency. All implementations of this work will be published at our website.

Change detection is an important method of analyzing information about changes in geographical features. However, existing deep learning feature difference methods often lead to the loss of detailed information. Differences in features can arise from factors like illumination or geometric variations rather than actual change regions, resulting in inaccurate change detection. This leads to poor detection of fine-grained boundaries and internal hole problems. To alleviate this, we propose a novel change detection network guided by change boundary awareness and incorporating the concept of boundary-to-center. Our network introduces a change boundary-aware module to capture boundary information of change regions. This module enhances boundaries, reducing the influence of noise in feature differences and providing rich contextual information to improve the accuracy of change boundaries. Additionally, we propose a bitemporal feature aggregation module (BFAM) based on spatial-temporal features. The BFAM aggregates multiple receptive fields features and complements texture information. Both modules utilize the SimAM attention mechanism to enhance the finegrained nature of the features. In addition, we introduce a deep feature extraction module to extract deep features and minimize information loss during the decoupling process. The proposed change detection network in this article is guided by change boundary perception, progressively integrating semantic and spatial texture information to refine edges and enhance internal integrity. The performance and efficiency of B2CNet have been validated on four publicly available remote sensing image change detection datasets. Through extensive experiments, the effectiveness of the proposed method has been demonstrated. For example, in terms of IOU for LEVIR, WHU, SYSU, and HRCUS datasets, the improvements compared to the baseline are 1.89%, 2.86%, 4.70%, and 3.79%, respectively.

In the field of remote sensing, it is not feasible to obtain high spatial resolution multispectral (HRMS) images from a single satellite sensor. The existing methods use pansharpening techniques to obtain HRMS images by fusing panchromatic (PAN) and multispectral (MS) images. However, due to the scale difference between PAN and MS images, most pansharpening methods often use explicit sampling methods to integrate features at different scales. These explicit-based sampling techniques represent pixels as discrete points through predefined functions, rendering it challenging to fit the distribution among diverse modal data, this results in the loss of image texture details during the fusion process. Implicit neural networks can enhance the generative capability of images by incorporating pixel coordinate information, which is crucial for the fusion of remote sensing images with different spatial resolutions. Inspired by implicit neural representation, we propose a progressive implicit neural feature fusion network (PINFNet) for remote sensing images. A progressive implicit neural feature fusion is proposed; it establishes a coordinate modal relationship between the spatial and spectral information through the guidance of the high spatial features in PAN images. This enables the proposed PINFNet to progressively learn and integrate spatial and spectral information at different scales. Our method, as opposed to discrete sampling techniques, is capable of establishing a continuous representation between diverse modal data, which in turn preserves more texture detail information. Extensive experiments have shown that this approach outperforms state-of-the-art methods while maintaining high efficiency.

The fast data acquisition rate due to the shorter revisit periods and wider observation coverage of satellites results in large amounts of remote sensing images every day. This brings the challenge of how to accurately search the images with similar visual content as the query image. Content-based image retrieval (CBIR) is a solution to this challenge, its performance heavily depends on the effectiveness of the image representation features and similarity evaluation metrics. Ideal image feature representations have dispersed interclass, compact intraclass distribution. However, the neural networks employed by many CBIR methods are trained with cross entropy loss, which does not directly optimize the metrics that evaluates interclass variance over intraclass variance, hence, their feature representations are suboptimal. Meanwhile, the traditional distance metrics used by many CBIR methods cannot index the similarity of feature representations well in high-dimensional space. For better CBIR performance, we propose a discriminative feature learning approach with distinguishable distance metrics for remote sensing image classification and retrieval. By balancing the diagonal elements and nondiagonal elements of the within-class scatter matrix of deep linear discriminant analysis, our proposed loss function, balanced deep linear discriminant analysis, can better optimize the Rayleigh–Ritz quotient, which measures interclass variance over intraclass variance. In addition, the proposed distance metrics, reciprocal exponential distance (RED), is more capable of maintaining distance contrast in high dimensionality, therefore, it can better index similarity for feature representations in high dimensionality. Both visual interpretations and quantitative metrics of extensive experiments demonstrated the effectiveness of our approach.

Onboard real-time processing of remote sensing satellites is an important means of rapidly obtaining information, and the fusion processing of panchromatic and multispectral data is of great significance for optical satellites. In order to ensure the effect, most traditional algorithms perform statistical analysis or transformation on the entire image first and then perform subsequent processing. There are problems such as high algorithm complexity and resource occupation, and it is difficult to apply to onboard scenarios where the volume and power consumption are strictly limited. Aiming at the requirements of onboard fusion, a real-time processing approach for high-resolution optical satellites is proposed. First, through the implementation of real-time geometric positioning, ROI extraction is completed while the camera is imaging, avoiding the disadvantages of traditional methods for processing large amounts of data; then, based on the principle of object-space consistency, by fine-tuning virtual sensor parameters, the registration of a panchromatic multispectral image is completed in the sensor correction step, so that the relative accuracy of the two can meet the fusion requirements, and time-consuming pixel-level registration processing is avoided; finally, according to the characteristics of the algorithms and embedded hardware, an efficient algorithm mapping strategy is formulated, and deep optimization is implemented to achieve a significant improvement in performance. Experiments show that the performance of this method is improved by 156.23 times compared with the traditional method. Moreover, after building a parallel pipeline, it can meet the real-time fusion processing requirement of completing a 5000 × 5000 pixels ROI area every 2.4 s.

This article proposes an efficient and high-fidelity panchromatic (PAN) spectral decomposition method based on smoothing filtering for multispectral (MS) and hyperspectral (HS) image pansharpening. The proposed method assumes that the high-frequency spatial details at the same scale are the same for different spectral images captured by the satellite at the same time. When the PAN image is prefiltered and down-sampled to the MS scale, it will have the same high-frequency spatial detail as the MS image, with only low-frequency spectral differences. Prefiltering for antialiasing when downsampling. Then, the spectral decomposition coefficients from down-sampled PAN image to MS image can be calculated on MS scale. The low-frequency spectral information of an image at different scales is the same, the spectral decomposition coefficients on the MS scale can be up-sampled to the PAN scale, and the original PAN image can be spectrally decomposed to obtain the sharpened image. The proposed method only decomposes the low-frequency spectrum, and the high-frequency spatial details are the same as the original PAN image, the spatial detail is well preserved. This article verifies the effect of the proposed method on MS and HS image sharpening through experiments, and the results show that the proposed method is better than the comparison method. This article also controls other variables to compare with the HPM method. The results show that the hybrid quality with no reference and spatial distortion index ($D_{s}$) of the proposed method are better than the HPM method.

According to statistics, more than 50% of optical satellite images are covered by clouds or fog. Moreover, the cloud cover rate is much higher in large bodies of water and nearby areas than inland areas due to the high amount of water evaporation and condensation. Therefore, ship detection from optical remote sensing images based on water surface analysis is more susceptible to cloud and fog interference, which affects the detection accuracy. In the time-sensitive application field of remote sensing images, due to the massive parameters in the large-scale network model, the detection speed is slow, and a lightweight detection model is commonly used. However, it is difficult for the lightweight detection model to achieve both high efficiency and accuracy for ship detection in a cloud-covered environment. To solve these problems, this manuscript proposes a lightweight algorithm called the fog remote sensing ship detection network (FRS-Net) suitable for ship detection from remote sensing images in thin-cloud and fog-covered environments. FRS-Net is developed based on the deep learning algorithm and can effectively improve ship detection accuracy under thin-cloud and fog-covered environment. First, for the allocation strategy of anchor boxes, by using the K-means clustering algorithm, FRS-Net simplifies the number of anchor boxes by utilizing the shape characteristics of the ship, which improves the time efficiency and the detection accuracy. Second, the FRS-Net network can meet the detection accuracy requirement and has a fast inference speed. The FRS-Net network is mainly composed of a backbone extraction network, a feature fusion network, and a prediction network. Experimental results on the ship detection in optical remote sensing image dataset demonstrate the detection accuracy and computational efficiency of FRS-Net. The recognition mean average precision achieved 43.20% for ship detection under thin-cloud and fog-covered environment, with an efficiency of up to 424 frames per second. FRS-Net has the potential to be applied in future scenarios such as embedded processing and onboard processing, where computing capabilities are strictly limited and the timeliness requirement is high.

This paper proposes an efficient and high-fidelity image fusion method based on adaptive smoothing filtering for panchromatic (PAN) and multispectral (MS) image fusion. The scale ratio reflects the ratio of spatial resolution between the panchromatic image and the multispectral image. When facing a multiscale fusion task, traditional methods are unable to simultaneously handle the problems of spectral resolution loss resulting from high scale ratios and the issue of reduced spatial resolution due to low scale ratios. To adapt to the fusion of panchromatic and multispectral satellite images of different scales, this paper improves the problem of the insufficient filtering of high-frequency information of remote sensing images of different scales by the classic filter-based intensity modulation (SFIM) model. It uses Gaussian convolution kernels instead of traditional mean convolution kernels and builds a Gaussian pyramid to adaptively construct convolution kernels of different scales to filter out high-frequency information of high-resolution images. It can adaptively process panchromatic multispectral images of different scales, iteratively filter the spatial information in panchromatic images, and ensure that the scale transformation is consistent with the definition of multispectral images. Using 15 common fusion methods, this paper compares the experimental results of ZY-3 with scale ratio 2.7 and SV-1 with scale ratio 4 data. The results show that the method proposed in this paper retains good spatial information for image fusion at different scales and has good spectral preservation.

Opium poppy is a medicinal plant, and its cultivation is illegal without legal approval in China. Unmanned aerial vehicle (UAV) is an effective tool for monitoring illegal poppy cultivation. However, targets often appear occluded and confused, and it is difficult for existing detectors to accurately detect poppies. To address this problem, we propose an opium poppy detection network, YOLOHLA, for UAV remote sensing images. Specifically, we propose a new attention module that uses two branches to extract features at different scales. To enhance generalization capabilities, we introduce a learning strategy that involves iterative learning, where challenging samples are identified and the model’s representation capacity is enhanced using prior knowledge. Furthermore, we propose a lightweight model (YOLOHLA-tiny) using YOLOHLA based on structured model pruning, which can be better deployed on low-power embedded platforms. To evaluate the detection performance of the proposed method, we collect a UAV remote sensing image poppy dataset. The experimental results show that the proposed YOLOHLA model achieves better detection performance and faster execution speed than existing models. Our method achieves a mean average precision (mAP) of 88.2% and an F1 score of 85.5% for opium poppy detection. The proposed lightweight model achieves an inference speed of 172 frames per second (FPS) on embedded platforms. The experimental results showcase the practical applicability of the proposed poppy object detection method for real-time detection of poppy targets on UAV platforms.

Spatial and spectral information are essential sources of information in remote sensing applications, and the fusion of panchromatic and multispectral images effectively combines the advantages of both. Due to the existence of two main classes of fusion methods—component substitution (CS) and multi-resolution analysis (MRA), which have different advantages—mixed approaches are possible. This paper proposes a fusion algorithm that combines the advantages of generalized intensity–hue–saturation (GIHS) and non-subsampled shearlet transform (NSST) with principal component analysis (PCA) technology to extract more spatial information. Therefore, compared with the traditional algorithms, the algorithm in this paper uses PCA transformation to obtain spatial structure components from PAN and MS, which can effectively inject spatial information while maintaining spectral information with high fidelity. First, PCA is applied to each band of low-resolution multispectral (MS) images and panchromatic (PAN) images to obtain the first principal component and to calculate the intensity of MS. Then, the PAN image is fused with the first principal component using NSST, and the fused image is used to replace the original intensity component. Finally, a fused image is obtained using the GIHS algorithm. Using the urban, plants and water, farmland, and desert images from GeoEye-1, WorldView-4, GaoFen-7 (GF-7), and Gaofen Multi-Mode (GFDM) as experimental data, this fusion method was tested using the evaluation mode with references and the evaluation mode without references and was compared with five other classic fusion algorithms. The results showed that the algorithms in this paper had better fusion performances in both spectral preservation and spatial information incorporation.

With the rapid development of wind generation, the box substation is widely applied in the wind farm. However the box substation is more prone to break down than the conventional power transformer, due to the uncertainty of wind power. It is difficult for the existing transformer condition assessment to fulfill the requirement of “few people on duty” operation model, which is promoted by both the domestic and foreign wind farms. To improve the safety and reliability of the wind generation, an on-line condition assessment of the box substation in wind farm based on hypothesis testing is proposed. Firstly, an ensemble learning based on least squares support vector regression (EL-LSSVR) is proposed. Secondly, EL-LSSVR is applied to the historical data of the normal box substation, to establish a normal hot spot temperature model. Then the normal hot spot temperature model learns from the real-time data of the evaluated box substation on-line, to establish a real-time hot spot temperature model. Thirdly, the hypothesis testing is employed to analyze the significance between the real-time hot spot temperature and the normal hot spot temperature, which are generated by the real-time hot spot temperature model and the normal hot spot temperature model respectively. The condition assessment of the box substation is implemented according to the significance. Finally, the proposed method is validated by the condition assessment experiment of the box substation, which is installed at a wind farm in South China.

Land cover classification is a multiclass segmentation task to classify each pixel into a certain natural or human-made category of the earth’s surface, such as water, soil, natural vegetation, crops, and human infrastructure. Limited by hardware computational resources and memory capacity, most existing studies preprocessed original remote sensing images by downsampling or cropping them into small patches less than 512 × 512 pixels before sending them to a deep neural network. However, downsampling incurs a spatial detail loss, renders small segments hard to discriminate, and reverses the spatial resolution progress obtained by decades of efforts. Cropping images into small patches causes a loss of long-range context information, and restoring the predicted results to their original size brings extra latency. In response to the above weaknesses, we present an efficient lightweight semantic segmentation network termed MKANet. Aimed at the characteristics of top view high-resolution remote sensing imagery, MKANet utilizes sharing kernels to simultaneously and equally handle ground segments of inconsistent scales, and also employs a parallel and shallow architecture to boost inference speed and friendly support image patches more than 10× larger. To enhance boundary and small segment discrimination, we also propose a method that captures category impurity areas, exploits boundary information, and exerts an extra penalty on boundaries and small segment misjudgments. Both visual interpretations and quantitative metrics of extensive experiments demonstrate that MKANet obtains a state-of-the-art accuracy on two land-cover classification datasets and infers 2× faster than other competitive lightweight networks. All these merits highlight the potential of MKANet in practical applications.

Since the data generation rate of high-resolution satellites is increasing rapidly, to relieve the stress of data downloading and processing systems while enhancing the time efficiency of information acquisition, it is important to deploy on-board edge computing on satellites. However, the volume, weight, and computability of on-board systems are strictly limited by the harsh space environment. Therefore, it is very difficult to match the computability and the requirements of diversified intelligent applications. Currently, this problem has become the first challenge of the practical deployment of on-board edge computing. To match the actual requirements of the Luojia3 satellite of Wuhan University, this manuscript proposes a three-level edge computing architecture based on a System-on-Chip (SoC) for low power consumption and expandable on-board processing. First, a transfer level is designed to focus on hardware communications and Input/Output (I/O) works while maintaining a buffer to store image data for upper levels temporarily. Second, a processing framework that contains a series of libraries and Application Programming Interfaces (APIs) is designed for the algorithms to easily build parallel processing applications. Finally, an expandable level contains multiple intelligent remote sensing applications that perform data processing efficiently using base functions, such as instant geographic locating and data picking, stream computing balance model, and heterogeneous parallel processing strategy that are provided by the architecture. It is validated by the performance improvement experiment that following this architecture, using these base functions can help the Region of Interest (ROI) system geometric correction fusion algorithm to be 257.6 times faster than the traditional method that processes scene by scene. In the stream computing balance experiment, relying on this architecture, the time-consuming algorithm ROI stabilization production can maintain stream computing balance under the condition of insufficient computability. We predict that based on this architecture, with the continuous development of device computability, the future requirements of on-board computing could be better matched.

This paper presents a near real-time automatic sub-pixel registration method of high-resolution panchromatic (PAN) and multispectral (MS) images using a graphics processing unit (GPU). In the first step, the method uses differential geo-registration to enable accurate geographic registration of PAN and MS images. Differential geo-registration normalizes PAN and MS images to the same direction and scale. There are also some residual misalignments due to the geometrical configuration of the acquisition instruments. These residual misalignments mean the PAN and MS images still have deviations after differential geo-registration. The second step is to use differential rectification with tiny facet primitive to eliminate possible residual misalignments. Differential rectification corrects the relative internal geometric distortion between PAN and MS images. The computational burden of these two steps is large, and traditional central processing unit (CPU) processing takes a long time. Due to the natural parallelism of the differential methods, these two steps are very suitable for mapping to a GPU for processing, to achieve near real-time processing while ensuring processing accuracy. This paper used GaoFen-6, GaoFen-7, ZiYuan3-02 and SuperView-1 satellite data to conduct an experiment. The experiment showed that our method’s processing accuracy is within 0.5 pixels. The automatic processing time of this method is about 2.5 s for 1 GB output data in the NVIDIA GeForce RTX 2080Ti, which can meet the near real-time processing requirements for most satellites. The method in this paper can quickly achieve high-precision registration of PAN and MS images. It is suitable for different scenes and different sensors. It is extremely robust to registration errors between PAN and MS.

With the advent of very-high-resolution remote sensing images, semantic change detection (SCD) based on deep learning has become a research hotspot in recent years. SCD aims to observe the change in the Earth’s land surface and plays a vital role in monitoring the ecological environment, land use and land cover. Existing research mainly focus on single-task semantic change detection; the problem they face is that existing methods are incapable of identifying which change type has occurred in each multi-temporal image. In addition, few methods use the binary change region to help train a deep SCD-based network. Hence, we propose a dual-task semantic change detection network (GCF-SCD-Net) by using the generative change field (GCF) module to locate and segment the change region; what is more, the proposed network is end-to-end trainable. In the meantime, because of the influence of the imbalance label, we propose a separable loss function to alleviate the over-fitting problem. Extensive experiments are conducted in this work to validate the performance of our method. Finally, our work achieves a 69.9% mIoU and 17.9 Sek on the SECOND dataset. Compared with traditional networks, GCF-SCD-Net achieves the best results and promising performances.

Synthetic aperture radar (SAR) is an important means to detect ocean oil spills which cause serious damage to the marine ecosystem. However, the look-alikes, which have a similar behavior to oil slicks in SAR images, will reduce the oil spill detection accuracy. Therefore, a novel oil spill detection method based on multiple features of polarimetric SAR data is proposed to improve the detection accuracy in this paper. In this method, the self-similarity parameter, which is sensitive to the randomness of the scattering target, is introduced to enhance the discrimination ability between oil slicks and look-alikes. The proposed method uses the Random Forest classification combing self-similarity parameter with seven well-known features to improve oil spill detection accuracy. Evaluations and comparisons were conducted with Radarsat-2 and UAVSAR polarimetric SAR datasets, which shows that: (1) the oil spill detection accuracy of the proposed method reaches 92.99% and 82.25% in two datasets, respectively, which is higher than three well-known methods. (2) Compared with other seven polarimetric features, self-similarity parameter has the better oil spill detection capability in the scene with lower wind speed close to 2⁻3 m/s, while, when the wind speed is close to 9⁻12 m/s, it is more suitable for oil spill detection in the downwind scene where the microwave incident direction is similar to the sea surface wind direction and performs well in the scene with incidence angle range from 29.7° to 43.5°.