Search

Separable nonlinear models (SNLMs) are of great importance in system modeling, signal processing, and machine learning because of their flexible structure and excellent description of nonlinear behaviors. The online identification of such models is quite challenging, and previous related work usually ignores the special structure where the estimated parameters can be partitioned into a linear and a nonlinear part. In this brief, we propose an efficient first-order recursive algorithm for SNLMs by introducing the variable projection (VP) step. The proposed algorithm utilizes the recursive least-squares method to eliminate the linear parameters, resulting in a reduced function. Then, the stochastic gradient descent (SGD) algorithm is employed to update the parameters of the reduced function. By considering the tight coupling relationship between linear parameters and nonlinear parameters, the proposed first-order VP algorithm is more efficient and robust than the traditional SGD algorithm and alternating optimization algorithm. More importantly, since the proposed algorithm just uses the first-order information, it is easier to apply it to large-scale models. Numerical results on examples of different sizes confirm the effectiveness and efficiency of the proposed algorithm.

Deep neural networks have achieved breakthrough improvement in various application fields. Nevertheless, they usually suffer from a time-consuming training process because of the complicated structures of neural networks with a huge number of parameters. As an alternative, a fast and efficient discriminative broad learning system (BLS) is proposed, which takes the advantages of flat structure and incremental learning. The BLS has achieved outstanding performance in classification and regression problems. However, the previous studies ignored the reason why the BLS can generalize well. In this article, we focus on the interpretation from the viewpoint of the frequency domain. We discover the existence of the frequency principle in BLS, i.e., the BLS preferentially captures low-frequency components quickly and then fits the high frequencies during the incremental process of adding feature nodes and enhancement nodes. The frequency principle may be of great inspiration for expanding the application of BLS.

In this article, we study the recursive algorithms for a class of separable nonlinear models (SNLMs) in which the parameters can be partitioned into a linear part and a nonlinear part. Such models are very common in machine learning, system identification, and signal processing. Utilizing the special structure of the SNLMs, we propose a recursive variable projection (RVP) algorithm, in which at each recursion, the linear parameters of the model are eliminated, and the nonlinear parameters are updated by the recursive Levenberg-Marquart algorithm. Then, based on the updated nonlinear parameters, the linear parameters are updated by the recursive least-squares algorithm. According to a convergence analysis of the RVP algorithm, the parameter estimation error is mean-square bounded. Numerical examples confirm the satisfactory performance of the proposed algorithm.

In this paper, we consider the term selection problem for a class of separable nonlinear models. The strategy is a two-step process in which the nonlinear parameters of the model are first optimized by a variable projection method, and then the least absolute shrinkage and selection operator are adopted to obtain a sparse solution by picking out the critical terms automatically. This process may be repeated several times. The proposed algorithm is tested on parameter estimation problems for an exponential model and a neural network-based model. The numerical results show that the proposed algorithm can pick out the appropriate terms from the overparameterized model and the obtained parsimonious model performs better than other methods.

Separable nonlinear models are very common in various research fields, such as machine learning and system identification. The variable projection (VP) approach is efficient for the optimization of such models. In this paper, we study various VP algorithms based on different matrix decompositions. Compared with the previous method, we use the analytical expression of the Jacobian matrix instead of finite differences. This improves the efficiency of the VP algorithms. In particular, based on the modified Gram-Schmidt (MGS) method, a more robust implementation of the VP algorithm is introduced for separable nonlinear least-squares problems. In numerical experiments, we compare the performance of five different implementations of the VP algorithm. Numerical results show the efficiency and robustness of the proposed MGS method-based VP algorithm.