Search

Recently, transform-based tensor nuclear norm (TNN) methods have received increasing attention as a powerful tool for multi-dimensional visual data (color images, videos, and multispectral images, etc.) recovery. Especially, the redundant transform-based TNN achieves satisfactory recovery results, where the redundant transform along spectral mode can remarkably enhance the low-rankness of tensors. However, it suffers from expensive computational cost induced by the redundant transform. In this paper, we propose a learnable spatial-spectral transform-based TNN model for multi-dimensional visual data recovery, which not only enjoys better low-rankness capability but also allows us to design fast algorithms accompanying it. More specifically, we first project the large-scale original tensor to the small-scale intrinsic tensor via the learnable semi-orthogonal transforms along the spatial modes. Here, the semi-orthogonal transforms, serving as the key building block, can boost the spatial low-rankness and lead to a small-scale problem, which paves the way for designing fast algorithms. Secondly, to further boost the low-rankness, we apply the learnable redundant transform along the spectral mode to the small-scale intrinsic tensor. To tackle the proposed model, we apply an efficient proximal alternating minimization-based algorithm, which enjoys a theoretical convergence guarantee. Extensive experimental results on real-world data (color images, videos, and multispectral images) demonstrate that the proposed method outperforms state-of-the-art competitors in terms of evaluation metrics and running time.

Transformer holds significance in deep learning (DL) research. Node embedding (NE) and positional encoding (PE) are usually two indispensable components in a Transformer. The former can excavate hidden correlations from the data, while the latter can store locational relationships between nodes. Recently, the Transformer has been applied for hyperspectral image (HSI) classification because the model can capture long-range dependencies to aggregate global features for representation learning. In an HSI, adjacent pixels tend to be homogeneous, while the NE does not identify the positional information of pixels. Therefore, PE is crucial for Transformers to understand locational relationships between pixels. However, in this area, most Transformer-based methods randomly generate PEs without considering their physical meaning, which leads to weak representations. This article proposes a new graph generative structure-aware Transformer (GraphGST) to solve the above-mentioned PE problem when implementing HSI classification. In our GraphGST, a new absolute PE (APE) is established to acquire pixels’ absolute positional sequences (APSs) and is integrated into the Transformer architecture. Moreover, a generative mechanism with self-supervised learning is developed to achieve cross-view contrastive learning (CL), aiming to enhance the representation learning of the Transformer. The proposed GraphGST model can capture local-to-global correlations, and the extracted APSs can complement the spectral features of pixels to assist in NE. Several experiments with real HSIs are conducted to evaluate the effectiveness of our GraphGST. The proposed method demonstrates very competitive performance compared with other state-of-the-art (SOTA) approaches. Our source codes will be provided in the following link https://github.com/yuanchaosu/TGRS-graphGST.

Recently, the transform-based tensor nuclear norm (TNN) methods have shown promising performance and drawn increasing attention in tensor completion (TC) problems. The main idea of these methods is to exploit the low-rank structure of frontal slices of the tensor under the transform. However, the transforms in TNN methods usually treat all modes equally and do not consider the different traits of different modes (i.e., spatial and spectral/temporal modes). To address this problem, we suggest a new low-rank tensor representation based on the coupled nonlinear transform (called CoNoT) for a better low-rank approximation. Concretely, spatial and spectral/temporal transforms in the CoNoT, respectively, exploit the different traits of different modes and are coupled together to boost the implicit low-rank structure. Here, we use the convolutional neural network (CNN) as the CoNoT, which can be learned solely from an observed multidimensional image in an unsupervised manner. Based on this low-rank tensor representation, we build a new multidimensional image completion model. Moreover, we also propose an enhanced version (called Ms-CoNoT) to further exploit the spatial multiscale nature of real-world data. Extensive experiments on real-world data substantiate the superiority of the proposed models against many state-of-the-art methods both qualitatively and quantitatively.

In this article, we propose a novel tensor learning and coding model for third-order data completion. The aim of our model is to learn a data-adaptive dictionary from given observations and determine the coding coefficients of third-order tensor tubes. In the completion process, we minimize the low-rankness of each tensor slice containing the coding coefficients. By comparison with the traditional predefined transform basis, the advantages of the proposed model are that: 1) the dictionary can be learned based on the given data observations so that the basis can be more adaptively and accurately constructed and 2) the low-rankness of the coding coefficients can allow the linear combination of dictionary features more effectively. Also we develop a multiblock proximal alternating minimization algorithm for solving such tensor learning and coding model and show that the sequence generated by the algorithm can globally converge to a critical point. Extensive experimental results for real datasets such as videos, hyperspectral images, and traffic data are reported to demonstrate these advantages and show that the performance of the proposed tensor learning and coding method is significantly better than the other tensor completion methods in terms of several evaluation metrics.

Thick clouds in remote sensing (RS) images deteriorate the visual quality and hinder subsequent applications. The emerging multitemporal RS images with rich temporal information bring the opportunity for cloud removal. How to effectively exploit the rich temporal information of the multitemporal RS images remains challenging. As multitemporal RS images with the same geographic scene, the spatial gradient of RS images at different time nodes has a resemblance, which can guide the reconstruction of the cloudy region. Motivated by this, we suggest a gradient domain fidelity with respect to the guided gradient for thick cloud removal in multitemporal RS images, which faithfully preserves the fine edges and textures compared with the original pixel domain fidelity. Armed with the gradient domain fidelity, we propose a low-rank tensor ring decomposition model (termed as TRGFid) for the thick cloud removal problem. In the proposed model, the guided gradient of the cloudy region is availably estimated by using the Regression method from the cloud-free region of different time nodes. Moreover, we develop an efficient proximal alternating minimization (PAM)-based algorithm for solving the proposed nonconvex model. Extensive simulated and real experiments show that the proposed method outperforms its competitors and preserves fine edges and textures.

Cloud removal is an important task in remotely sensed images (RSIs) processing, which is beneficial for downstream applications, such as unmixing, fusion, and target detection. Multitemporal RSIs (MRSIs), which contains abundant spatial–spectral–temporal (SST) information, potentially bring new opportunities for cloud removal. However, how to effectively and efficiently explore the rich information of MRSIs remains a challenge. Inspired by the low-rankness of MRSIs, we propose an Unsupervised Domain Factorization Network (UnDFN) for thick cloud removal, which allows us to effectively and efficiently exploit the rich SST information of MRSIs. In the UnDFN framework, we first factorize RSI for each time node of MRSIs into its corresponding spatial factor and spectral factor. Due to powerful expressive ability, untrained neural networks (NNs) are leveraged to faithfully capture the spatial and spectral factors. Especially, motivated by the low-rankness of the concatenated spatial factors of all time nodes, a low-rank spatial factor module is elaborately designed to effectively and efficiently capture the spatial factors of all time nodes as compared with separately using networks to capture spatial factors for each time node. Extensive experiments on simulated and real MRSIs of different satellites (including Sentinel-2 and Landsat-8) substantiate that the proposed UnDFN achieves state-of-the-art performance in thick cloud removal compared to other methods.

Deep learning models, such as convolutional neural networks (CNNs), have made significant progress in hyperspectral image (HSI) classification. However, these models require a large number of parameters, which occupy a lot of storage space and suffer from overfitting, thus resulting in performance loss. To solve the above problems, in this article, we propose a new compression network [namely, a Hybrid Fully Connected Tensorized Compression Network (HybridFCTCN)] by considering the high dimensionality of HSI data. First, using the low-rank fully connected tensor network decomposition (FCTND), three novel units, i.e., FCTN-FC, FCTNConv2D, and FCTNConv3D, are designed to compress the weight tensor of standard fully connected (FC) layer and kernel tensor of convolutional layer, reducing their parameters. In the novel units, the intrinsic correlation of the decomposed factors is adequately exploited by the FC structures, which enhances their feature extraction and classification abilities. Then, benefiting from the hybrid network backbone composed of the FCTNConv3D and FCTNConv2D units, HybridFCTCN can extract more discriminative features with fewer parameters, while it has great generalization capability and robustness, enabling better HSI classification. Finally, the rank of above-designed units is defined, and its determination is discussed to facilitate the application of the proposed model. Extensive experiments on three widely used HSI datasets reveal that the proposed model achieves state-of-the-art classification performance for different training sample sizes with a very small number of parameters.

Recently, untrained neural network priors (UNNPs) have received increasing attention for multi-dimensional image recovery. However, previous studies are based on over-parameterized untrained neural networks, which results in unstable behavior. In this letter, we propose an untrained low-rank neural network prior (ULRNNP) for multi-dimensional image recovery, which enjoys the powerful representation ability and stable behavior. More specifically, the elaborately designed nonlinear Tucker decomposition module implicitly imposes low-rank constraints on the feature tensor and can more compactly represent the feature tensor. Attributed to the suggested nonlinear Tucker decomposition module, ULRNNP can simultaneously enjoy strong representation ability and stable behavior. The friendly stable behavior allows us to design a friendly stopping criteria without the reference ground truth image as compared with classic UNNP-based methods. Extensive experiments on different multi-dimensional image datasets validate the superior performance of the proposed ULRNNP over state-of-the-art methods.

Recently, the transform-based tensor nuclear norm (TNN) framework has yielded promising results for hyperspectral image (HSI) denoising as compared with previous original-domain tensor-based models. However, the TNN framework only exploits the low-rankness of each band of HSIs (tensors) under a single spectral transform. The correlation between all bands under the transform (i.e., the global low-rankness of the transformed tensor) and the sparsity of the transformed HSI, which are beneficial for HSI denoising, is usually neglected in the TNN framework. In this article, we propose to reconcile sparse and low-tensor-ring (TR)-rank priors in the learned transformed domain (called T-RSTR model) for HSI denoising. In T-RSTR, the transform-based low-TR-rank and sparse regularizers are designed to characterize the global low-rankness and sparsity of the transformed tensors, respectively, and then the transform-based low-TR-rank and sparse regularizers are organically integrated and benefit from each other for substantially boosting denoising performance. To tackle the T-RSTR model, we elaborately design a proximal alternating minimization-based algorithm with the theoretical convergence. Extensive numerical results demonstrate that T-RSTR is superior to the competing methods.

Recently emerged deep learning methods have achieved great success in single image rain streaks removal. However, existing methods ignore an essential factor in the rain streaks generation mechanism, i.e., the motion blur leading to the line pattern appearances. Thus, they generally produce overderaining or underderaining results. In this article, inspired by the generation mechanism, we propose a novel rain streaks removal framework using a kernel-guided convolutional neural network (KGCNN), achieving state-of-the-art performance with a simple network architecture. More precisely, our framework consists of three steps. First, we learn the motion blur kernel by a plain neural network, termed parameter network, from the detail layer of a rainy patch. Then, we stretch the learned motion blur kernel into a degradation map with the same spatial size as the rainy patch. Finally, we use the stretched degradation map together with the detail patches to train a deraining network with a typical ResNet architecture, which produces the rain streaks with the guidance of the learned motion blur kernel. Experiments conducted on extensive synthetic and real data demonstrate the effectiveness of the proposed KGCNN, in terms of rain streaks removal and image detail preservation.

We propose a novel image denoising model based on the total generalized variation (TGV) regularization. In the model, a spatially dependent regularization parameter is utilized to adaptively fit the local image features, resulting in further exploitation of the denoising potential of the TGV regularization. The proposed model is formulated under a joint optimization framework, by which the estimations of the restored image and the regularization parameter are achieved simultaneously. Furthermore, the model is general purpose that can handle various types of noise occurring in image processing. An alternating minimization-based numerical scheme is especially developed, which leads to an efficient algorithmic solution to the nonconvex optimization problem. Numerical experiments are reported to illustrate the effectiveness of our model in terms of both peak signal-to-noise ratio and visual perception.

Singular value decomposition (SVD)-based approaches, e.g., truncated SVD and Tikhonov regularization methods, are effective ways to solve problems of small or moderate size. However, SVD, in the sense of computation, is expensive when it is applied in large-sized cases. A multilevel method (MLM) combining SVD-based methods with the thresholding technique for signal restoration is proposed in this paper. Our MLM will transfer large-sized problems to small- or moderate-sized problems in order to make the SVD-based methods available. The linear systems on the coarsest level in the multilevel process will be solved by the Tikhonov regularization method. No presmoothers are implemented in the multilevel process to avoid damaging the parameter choice on the coarsest level. Furthermore, the soft-thresholding denoising technique is employed for the postsmoothers aiming to eliminate the leaving high-frequency information due to the lack of presmoothers. Finally, computational experiments show that our method outperforms other SVD-based methods in signal restoration ability at a shorter CPU-time consumption.