Search

We propose an online learning algorithm tailored for a class of machine learning models within a separable stochastic approximation framework. The central idea of our approach is to exploit the inherent separability in many models, recognizing that certain parameters are easier to optimize than others. This paper focuses on models where some parameters exhibit linear characteristics, which are common in machine learning applications. In our proposed algorithm, the linear parameters are updated using the recursive least squares (RLS) algorithm, akin to a stochastic Newton method. Subsequently, based on these updated linear parameters, the nonlinear parameters are adjusted using the stochastic gradient method (SGD). This dual-update mechanism can be viewed as a stochastic approximation variant of block coordinate gradient descent, where one subset of parameters is optimized using a second-order method while the other is handled with a first-order approach. We establish the global convergence of our online algorithm for non-convex cases in terms of the expected violation of first-order optimality conditions. Numerical experiments demonstrate that our method achieves significantly faster initial convergence and produces more robust performance compared to other popular learning algorithms. Additionally, our algorithm exhibits reduced sensitivity to learning rates and outperforms the recently proposed slimTrain algorithm (Newman et al. 2022). For validation, the code has been made available on GitHub.

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution image from its corresponding low-resolution input. A common technique to enhance the reconstruction quality is Non-Local Attention (NLA), which leverages self-similar texture patterns in images. However, we have made a novel finding that challenges the prevailing wisdom. Our research reveals that NLA can be detrimental to SISR and even produce severely distorted textures. For example, when dealing with severely degrade textures, NLA may generate unrealistic results due to the inconsistency of non-local texture patterns. This problem is overlooked by existing works, which only measure the average reconstruction quality of the whole image, without considering the potential risks of using NLA. To address this issue, we propose a new perspective for evaluating the reconstruction quality of NLA, by focusing on the sub-pixel level that matches the pixel-wise fusion manner of NLA. From this perspective, we provide the approximate reconstruction performance upper bound of NLA, which guides us to design a concise yet effective Texture-Fidelity Strategy (TFS) to mitigate the degradation caused by NLA. Moreover, the proposed TFS can be conveniently integrated into existing NLA-based SISR models as a general building block. Based on the TFS, we develop a Deep Texture-Fidelity Network (DTFN), which achieves state-of-the-art performance for SISR. Our code and a pre-trained DTFN are available on GitHub† for verification.

Blind Super-Resolution (BlindSR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) images without prior knowledge of the image degradation process. This is a challenging problem in real-world applications, where the degradation can be complex and unknown. Recent unsupervised learning-based BlindSR methods can estimate the image degradation in an unsupervised manner, but they suffer from limited adaptability to different types and intensities of degradation. They tend to capture the average level of degradation across all training samples, resulting in over-smoothing or over-sharpening effects for some images. As a result, the final reconstruction may exhibit the mean effect. Moreover, existing synthetic datasets do not reflect the real-world degradation scenarios, making it difficult to evaluate the performance of BlindSR methods. To address these issues, we propose a novel Degradation Intensity Estimation Module (DIEM) method, which can estimate the pixel-level degradation information of the input image more specifically and use it to guide image reconstruction. Furthermore, we construct a benchmark dataset under real scenarios, which is closer to the real-world BlindSR problem than existing synthetic datasets, and can provide a more reasonable evaluation of BlindSR methods. Extensive experimental results demonstrate that our DIEM-guided BlindSR method can achieve state-of-the-art image reconstruction results. Our code and pre-trained models have been uploaded to GitHub for validation.

Blind super-resolution (BlindSR) has recently attracted attention in the field of remote sensing. Due to the lack of paired data, most works assume that the acquired remote sensing images are high-resolution (HR) and use predefined degradation models to synthesize low-resolution (LR) images for training and evaluation. However, these acquired remote sensing images are often degraded by various factors, which still require super-resolution (SR) reconstruction to meet practical needs. Using them as ground-truth (GT) images will limit the model’s ability to restore fine details, resulting in blurry and noisy reconstructions. To overcome these limitations, we propose an unsupervised degradation-aware network which transforms natural images into the degraded domain as real-world remote sensing images. It uses natural images containing rich texture information as a reference for fine-grained restoration of the network, enabling the network to produce clearer reconstructions. Furthermore, we discovered the remarkable capability of the patchwise discriminator to perceive the degradation type of different regions within the acquired remote sensing image. Inspired by this finding, we design a novel degradation representation module (DRM) that can estimate the degradation information from LR images and guide the network to perform adaptive restoration. Comprehensive experimental results demonstrate that our proposed unsupervised blind super-resolution framework achieves state-of-the-art (SOTA) restoration performance. Our code and pretrained models have been uploaded to GitHub (https://github.com/55Dupup/UDASR) for validation.

This article presents a novel online identification algorithm for nonlinear regression models. The online identification problem is challenging due to the presence of nonlinear structure in the models. Previous works usually ignore the special structure of nonlinear regression models, in which the parameters can be partitioned into a linear part and a nonlinear part. In this article, we develop an efficient recursive algorithm for nonlinear regression models based on analyzing the equivalent form of variable projection (VP) algorithm. By introducing the embedded point iteration step, the proposed recursive algorithm can properly exploit the coupling relationship of linear parameters and nonlinear parameters. In addition, we theoretically prove that the proposed algorithm is mean-square bounded. Numerical experiments on synthetic data and real-world time series verify the high efficiency and robustness of the proposed algorithm.

Effective fault detection and diagnosis (FDD) is crucial in modern industry applications, but developing accurate and interpretable FDD models for complex processes remains a challenge. In this article, we propose a novel approach that partitions measurements into a principal component and a residual subspace, formulating the FDD model-building problem as a separable nonlinear optimization problem. To improve model interpretability, we incorporate sparse regularizers to derive sparse loadings. Taking advantage of the special separable structure presented in such problems, we present an efficient variable projection-based algorithm that significantly reduces the dimension of the parameters by solving a least-squares problem with an orthonormal constraint. This results in a reduced problem that only contains loading parameters. We further establish a significant theorem, which is essential for computing the gradient of the reduced objective function and addressing the coupling between different parts of the parameters, leading to improved identification performance of the proposed algorithm. Numerical results on synthetic and real-world datasets demonstrate that our algorithm achieves faster convergence speed and better evaluation metrics.

Separable nonlinear least-squares (SNLLS) problems arise frequently in many research fields, such as system identification and machine learning. The variable projection (VP) method is a very powerful tool for solving such problems. In this paper, we consider the regularization of ill-conditioned SNLLS problems based on the VP method. Selecting an appropriate regularization parameter is difficult because of the nonlinear optimization procedure. We propose to determine the regularization parameter using the weighted generalized cross-validation method at every iteration. This makes the original objective function changing during the optimization procedure. To circumvent this problem, we use an inequation to produce a consistent demand of decreasing at successive iterations. The approximation of the Jacobian of the regularized problem is also discussed. The proposed regularized VP algorithm is tested by the parameter estimation problem of several statistical models. Numerical results demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]