Pansharpening via Frequency-Aware Fusion Network With Explicit Similarity Constraints

The process of fusing a high spatial resolution (HR) panchromatic (PAN) image and a low spatial resolution (LR) multispectral (MS) image to obtain an HRMS image is known as pansharpening. With the development of convolutional neural networks (CNNs), the performance of pansharpening methods has been improved; however, blurry effects and spectral distortion still exist in their fusion results due to the insufficiency in detail learning and the frequency mismatch between MS and PAN. Therefore, the improvement of spatial details at the premise of reducing spectral distortion is still a challenge. In this article, we propose a frequency-aware fusion network (FAFNet) together with a novel high-frequency feature similarity (HFS) loss to address the above-mentioned problems. FAFNet is mainly composed of two kinds of blocks, where the frequency-aware blocks (FABs) aim to extract features in the frequency domain with the help of discrete wavelet transform (DWT) layers, and the frequency fusion blocks (FFBs) reconstruct and transform the features from the frequency domain to the spatial domain with the assistance of inverse DWT (IDWT) layers. Finally, the fusion results are obtained through a convolutional block (CB). To learn the correspondence, we also propose an HFS loss to constrain the high-frequency (HF) features derived from PAN and MS branches, so that HF features of PAN can reasonably be used to supplement that of MS. Experimental results on three datasets at both reduced and full resolutions demonstrate the superiority of the proposed method compared with several state-of-the-art pansharpening models. The codes are available at <uri>https://github.com/YinghuiXing/FAFNet</uri>.