Search

In this paper, we study the influence of ill-conditioned regression matrix on two hyper-parameter estimation methods for the kernel-based regularization method: the empirical Bayes (EB) and the Stein’s unbiased risk estimator (SURE). First, we consider the convergence rate of the cost functions of EB and SURE, and we find that they have the same convergence rate but the influence of the ill-conditioned regression matrix on the scale factor are different: for upper bounds, the scale factor for SURE contains one more factor cond(ΦTΦ) than that of EB, where Φ is the regression matrix and cond(•) denotes the condition number of a matrix. This finding indicates that when Φ is ill-conditioned, i.e., cond(ΦTΦ) is large, the cost function of SURE converges slower than that of EB. Then we consider the convergence rate of the optimal hyper-parameters of EB and SURE, and we find that they are both asymptotically normally distributed and have the same convergence rate, but the influence of the ill-conditioned regression matrix on the scale factor are different. In particular, for the ridge regression case, we show that the optimal hyperparameter of SURE converges slower than that of EB with a factor of 1/n2, as cond(ΦTΦ) goes to ∞, where n is the FIR model order.

An estimated state-space model can possibly be improved by further iterations with estimation data. This contribution specifically studies if models obtained by subspace estimation can be improved by subsequent re-estimation of the B, C, and D matrices (which involves linear estimation problems). Several tests are performed, which shows that it is generally advisable to do such further re-estimation steps using the maximum likelihood criterion. Stated more succinctly in terms of MATLAB functions, ssest generally outperforms n4sid., 18th IFAC Symposium on System Identification, Stockholm, Sweden, July 9-11, 2018

Regularization methods with regularization matrix in quadratic form have received increasing attention. For those methods, the design and tuning of the regularization matrix are two key issues that are closely related. For systems with complicated dynamics, it would be preferable that the designed regularization matrix can bring the hyper-parameter estimation problem certain structure such that a locally optimal solution can be found efficiently. An example of this idea is to use the so-called multiple kernel Chen et al. (2014) for kernel-based regularization methods. In this paper, we propose to use the multiple regularization matrix for the filter-based regularization. Interestingly, the marginal likelihood maximization with the multiple regularization matrix is also a difference of convex programming problem, and a locally optimal solution could be found with sequential convex optimization techniques.

In this paper, we study the asymptotic properties of the generalized cross validation (GCV) hyperparameter estimator and establish its connection with the Stein’s unbiased risk estimators (SURE) as well as the mean squared error (MSE). It is shown that as the number of data goes to infinity, the GCV has the same asymptotic property as the SURE does and both of them converge to the best hyperparameter in the MSE sense. We illustrate the efficacy of the result by Monte Carlo simulations.

We explore the problem of identification of LPV models when the scheduling variables are not known in advance and the model parameters exhibit a dynamic dependence on them. We consider an affine ARX model structure whose parameters vary with time. We solve for the model’s parameters and scheduling variables in two steps. In the first step, we use the measured input-output data to realize a parameter trajectory by solving a regularized Hankel matrix rank minimization problem. The regularization penalty is guided by the prior knowledge regarding the nature of system’s time variation. In the second step, the scheduling variables are estimated as parameters of a sparse ARX structure relating the model’s parameters to the measured input-output variables. The effectiveness of the proposed approach is illustrated with two practical examples.

Regularization of simple linear regression models for system identification is a recent much-studied problem. Several parameterizations (“kernels”) of the regularization matrix have been suggested together with different ways of estimating (“tuning”) its parameters. This contribution defines an asymptotic view on the problem of tuning and selection of kernels. It is shown that the SURE approach to parameter tuning provides an asymptotically consistent estimate of the optimal (in a MSE sense) hyperparameters. At the same time it is shown that the common marginal likelihood (empirical Bayes) approach does not enjoy that property.

A new nonparametric approach for system identification has been recently proposed where the impulse response is modeled as the realization of a zero-mean Gaussian process whose covariance (kernel) has to be estimated from data. In this scheme, quality of the estimates crucially depends on the parametrization of the covariance of the Gaussian process. A family of kernels that have been shown to be particularly effective in the system identification framework is the family of Diagonal/Correlated (DC) kernels. Maximum entropy properties of a related family of kernels, the Tuned/Correlated (TC) kernels, have been recently pointed out in the literature. In this paper we show that maximum entropy properties indeed extend to the whole family of DC kernels. The maximum entropy interpretation can be exploited in conjunction with results on matrix completion problems in the graphical models literature to shed light on the structure of the DC kernel. In particular, we prove that the DC kernel admits a closed-form factorization, inverse and determinant. These results can be exploited both to improve the numerical stability and to reduce the computational complexity associated with the computation of the DC estimator., Comment: Extends results of 2014 IEEE MSC Conference Proceedings (arXiv:1406.5706)

This paper studies the asymptotic properties of the hyperparameter estimators including the leave- $k$ -out cross validation (LKOCV) and r-fold cross validation (RFCV), and discloses their relation with the Stein's unbiased risk estimators (SURE) as well as the mean squared error (MSE). It is shown that as the number of data goes to infinity, the LKOCV shares the same asymptotic best hyperparameter minimizing the MSE estimator as the SURE does if the input is bounded and the ratio between the training data and the whole data tends to zero. We illustrate the efficacy of the theoretical result by Monte Carlo simulations.

The first order stable spline (SS-1) kernel is used extensively in regularized system identification. In particular, the stable spline estimator models the impulse response as a zero-mean Gaussian process whose covariance is given by the SS-1 kernel. In this paper, we discuss the maximum entropy properties of this prior. In particular, we formulate the exact maximum entropy problem solved by the SS-1 kernel without Gaussian and uniform sampling assumptions. Under general sampling schemes, we also explicitly derive the special structure underlying the SS-1 kernel (e.g. characterizing the tridiagonal nature of its inverse), also giving to it a maximum entropy covariance completion interpretation. Along the way similar maximum entropy properties of the Wiener kernel are also given.

The kernel-based regularization method has two core issues: kernel design and hyperparameter estimation. In this paper, we focus on the second issue and study the properties of several hyperparameter estimators including the empirical Bayes (EB) estimator, two Steins unbiased risk estimators (SURE) (one related to impulse response reconstruction and the other related to output prediction) and their corresponding Oracle counterparts, with an emphasis on the asymptotic properties of these hyperparameter estimators. To this goal, we first derive and then rewrite the first order optimality conditions of these hyperparameter estimators, leading to several insights on these hyperparameter estimators. Then we show that as the number of data goes to infinity, the two SUREs converge to the best hyperparameter minimizing the corresponding mean square error, respectively, while the more widely used EB estimator converges to another best hyperparameter minimizing the expectation of the EB estimation criterion. This indicates that the two SUREs are asymptotically optimal in the corresponding MSE senses but the EB estimator is not. Surprisingly, the convergence rate of two SUREs is slower than that of the EB estimator, and moreover, unlike the two SUREs, the EB estimator is independent of the convergence rate of Phi(T)Phi/N to its limit, where Phi is the regression matrix and N is the number of data. A Monte Carlo simulation is provided to demonstrate the theoretical results. (C) 2018 Elsevier Ltd. All rights reserved. Funding Agencies|National Natural Science Foundation of China [61773329, 61603379]; central government of China; Shenzhen Science and Technology Innovation Council [Ji-20170189, Ji-20160207]; Chinese University of Hong Kong, Shenzhen [PF. 01.000249, 2014.0003.23]; Swedish Research Council [2014-5894]; National Key Basic Research Program of China (973 Program) [2014CB845301]; Presidential Fund of the Academy of Mathematics and Systems Science, CAS [2015-hwyxqnrc-mbq]

In the last years, nonparametric linear dynamical systems modeling has regained attention in the system identification world. In particular, the application of regularization techniques that were already widely used in statistics and machine learning, has proven beneficial for the estimation of the impulse response of linear systems. The low-rank approximation of the impulse response obtained by the truncated singular value decomposition (SVD) also leads to reduced complexity estimates. In this paper, the link between regularization and SVD truncation for finite impulse response (FIR) model estimation is made explicit. The SVD truncation is reformulated as a regularization problem with a specific choice of the regularization matrix. Both approaches (regularization and SVD truncation) are applied on a FIR modeling example and compared with the classic prediction error method/maximum likelihood approach. The results show the advantage of these techniques for impulse response estimation.

The stable spline kernel and the diagonal correlated kernel are two kernels that have been tested extensively in kernel-based regularization methods for LTI system identification. As shown in our recent works, although these two kernels are introduced in different ways, they share some common features, e.g., they all belong to the class of exponentially convex locally stationary kernels, and state-space model induced kernels. In this work, we further show that similar to the derivation of the stable spline kernel, the continuous-time diagonal correlated kernel can be derived by applying the same “stable” coordinate change to a “generalized” first order spline kernel, and thus can be interpreted as a stable generalized first order spline kernel. This interpretation provides new facets to understand the properties of the diagonal correlated kernel. Due to this interpretation, new eigendecompositions, explicit expression of the norm, and new maximum entropy interpretation of the diagonal correlated kernel are derived accordingly.

Inspired by ideas taken from the machine learning literature, new regularization techniques have been recently introduced in linear system identification. In particular, all the adopted estimators solve a regularized least squares problem, differing in the nature of the penalty term assigned to the impulse response. Popular choices include atomic and nuclear norms (applied to Hankel matrices) as well as norms induced by the so called stable spline kernels. In this paper, a comparative study of estimators based on these different types of regularizers is reported. Our findings reveal that stable spline kernels outperform approaches based on atomic and nuclear norms since they suitably embed information on impulse response stability and smoothness. This point is illustrated using the Bayesian interpretation of regularization. We also design a new class of regularizers defined by "integral" versions of stable spline/TC kernels. Under quite realistic experimental conditions, the new estimators outperform classical prediction error methods also when the latter are equipped with an oracle for model order selection. (C) 2016 Elsevier Ltd. All rights reserved. Funding Agencies|MIUR FIRB project [RBFR12M3AC]; Progetto di Ateneo [CPDA147754/14]; Linnaeus Center CADICS; Swedish Research Council; ERC advanced grant LEARN [267381]; European Research Council; Swedish Research Council (VR) [2014-5894]

System identification with regularization methods has attracted increasing attention recently and is a complement to the current standard maximum likelihood/prediction error method. In this paper, we focus on the kernel-based regularization method and give a spectral analysis of the so-called diagonal correlated (DC) kernel, one family of kernel structures that has been proven useful for linear time-invariant system identification. In particular, using the theory of Bessel functions, we derive the eigenvalues and corresponding eigenfunctions of the DC kernel. Accordingly, we derive the Karhunen-Loeve expansion of the stochastic process whose covariance function is the DC kernel.

Intrigued by some recent results on impulse response estimation by kernel and nonparametric techniques, we revisit the old problem of transfer function estimation from input-output measurements. We formulate a classical regularization approach, focused on finite impulse response (FIR) models, and find that regularization is necessary to cope with the high variance problem. This basic, regularized least squares approach is then a focal point for interpreting other techniques, like Bayesian inference and Gaussian process regression. The main issue is how to determine a suitable regularization matrix (Bayesian prior or kernel). Several regularization matrices are provided and numerically evaluated on a data bank of test systems and data sets. Our findings based on the data bank are as follows. The classical regularization approach with carefully chosen regularization matrices shows slightly better accuracy and clearly better robustness in estimating the impulse response than the standard approach - the prediction error method/maximum likelihood (PEM/ML) approach. If the goal is to estimate a model of given order as well as possible, a low order model is often better estimated by the PEM/ML approach, and a higher order model is often better estimated by model reduction on a high order regularized FIR model estimated with careful regularization. Moreover, an optimal regularization matrix that minimizes the mean square error matrix is derived and studied. The importance of this result lies in that it gives the theoretical upper bound on the accuracy that can be achieved for this classical regularization approach. CADICS

Segmentation of time-varying systems and signals into models whose parameters are piecewise constant in time is an important and well studied problem. Here it is formulated as a least-squares problem with sum-of-norms regularization over the state parameter jumps. a generalization of L1-regularization. A nice property of the suggested formulation is that it only has one tuning parameter, the regularization constant which is used to trade-off fit and the number of segments. CADICS

This paper treats several aspects relevant to the identification of continuous-time output error (OE) models based on non-uniformly sampled output data. The exact method for doing this is well known in the time domain, where the continuous-time system is discretized, simulated and the result is fitted in a mean square sense to measured data. The material presented here is based on a method proposed in a companion paper (Gillberg & Ljung, 2010) which deals with the same topic but for the case of uniformly sampled data. In this text it will be shown how that method suggests that the output should be reconstructed using a B-spline with uniformly distributed knots. This representation can then be used to directly identify the continuous-time system without proceeding via discretization. Only the relative degree of the model is used to choose the order of the spline.

International audience; The Two-Stage Algorithm (TSA) has been extensively used and adapted for the identification of Hammerstein systems. It is essentially based on a particular formulation of Hammerstein systems in the form of bilinearly parameterized linear regressions. This paper has been motivated by a somewhat contradictory fact: though the optimality of the TSA has been established by Bai in 1998 only in the case of some special weighting matrices, the unweighted TSA is usually used in practice. It is shown in this paper that the unweighted TSA indeed gives the optimal solution of the weighted nonlinear least squares problem formulated with a particular weighting matrix. This provides a theoretical justification of the unweighted TSA, and also leads to a generalization of the obtained result to the case of colored noise with noise whitening. Numerical examples of identification of Hammerstein systems are presented to validate the theoretical analysis.

International audience; The Two-Stage Algorithm (TSA) has been extensively used and adapted for the identification of Hammerstein systems. It is essentially based on a particular formulation of Hammerstein systems in the form of bilinearly parameterized linear regressions. This paper has been motivated by a somewhat contradictory fact: though the optimality of the TSA has been established by Bai in 1998 only in the case of some special weighting matrices, the unweighted TSA is usually used in practice. It is shown in this paper that the unweighted TSA indeed gives the optimal solution of the weighted nonlinear least-squares problem formulated with a particular weighting matrix. This provides a theoretical justification of the unweighted TSA, and leads to a generalization of the obtained result to the case of colored noise with noise whitening. Numerical examples of identification of Hammerstein systems are presented to validate the theoretical analysis.

The basic least squares method for identifying linear systems has been extensively studied. Conditions for convergence involve issues about noise assumptions and behavior of the sample covariance matrix of the regressors. Lai and Wei proved in 1982 convergence for essentially minimal conditions on the regression matrix: All eigenvalues must tend to infinity, and the logarithm of the largest eigenvalue must not tend to infinity faster than the smallest eigenvalue. In this contribution we revisit this classical result with respect to assumptions on the noise: How much unstructured disturbances can be allowed without affecting the convergence? The answer is that the norm of these disturbances must tend to infinity slower than the smallest eigenvalue of the regression matrix.

Most of existing results on regularized system identification focus on regularized impulse response estimation. Since the impulse response model is a special case of orthonormal basis functions, it is interesting to consider if it is possible to tackle the regularized system identification using more compact orthonormal basis functions. In this paper, we explore two possibilities. First, we construct reproducing kernel Hilbert space of impulse responses by orthonormal basis functions and then use the induced reproducing kernel for the regularized impulse response estimation. Second, we extend the regularization method from impulse response estimation to the more general orthonormal basis functions estimation. For both cases, the poles of the basis functions are treated as hyperparameters and estimated by empirical Bayes method. Then we further show that the former is a special case of the latter, and more specifically, the former is equivalent to ridge regression of the coefficients of the orthonormal basis functions., 6 pages, final submission of an contribution for European Control Conference 2015, uploaded on March 20, 2015

The Direct Weight Optimization (DWO) approach to estimating a regression function is studied here for the class of approximately linear functions, i.e., functions whose deviation from an affine function is bounded by a known constant. Upper and lower bounds for the asymptotic maximum MSE are given, some of which also hold in the non-asymptotic case and for an arbitrary fixed design. Their coincidence is then studied. Particularly, under mild conditions, it can be shown that there is always an interval in which the DWO-optimal estimator is optimal among all estimators. Experiment design issues are also studied.

Closed loop subspace identification has become a focus of interest with several recent developments. Notably are the innovation estimation approach (Qin and Ljung, 2003), the state space approach with ARX pre-estimates (SSARX, (Jansson, 2003)), and the whitening filter approach (Chiuso and Picci, 2004). All these approaches use an extended future horizon to form the projection or regression from which an observable subspace is extracted. Yet there are other methods such as OKID of (Phan and Longman, 1992) and that of (Ljung and McKelvey, 1996) that do not use an extended horizon in the projection or regression step. Instead, a single high order ARX model is used. A natural question is whether the future horizon is necessary and if so what role does it play in these steps. In this paper we investigate the role of the future horizon using the whitening filter approach of (Chiuso and Picci, 2004), which works for both open-loop and closed-loop data. We conclude that the role of future horizon in this algorithm is merely extending the order of a bank of already high order ARX models. The difference from a single ARX model is insignificant if the ARX order or past horizon is sufficiently high. The role of future horizon is mainly in the model reduction step where it serves to elevate the order of the Hankel matrix. We complement the analysis with simulations.

This paper considers the identification of stochastic Wiener dynamic systems, that is linear dynamic systems with process noise, where the measurable output signal is a nonlinear function of the output from the linear system corrupted with additive measurement noise. It is shown how stochastic Wiener system identification can be viewed as a particular non-linear model errors-in-variables problem, for which there exists a large literature. We compare the maximum likelihood method with prediction error minimization methods based on the conditional mean predictor for Wiener systems. Related methods have previously been studied in the framework of identification of non-linear error-in-variables models. We extend these results by taking the input signal to the Wiener system into consideration. For example, the input will affect the variance of the prediction errors. Hence, a prediction error method with a variance weighting is derived to obtain more reliable parameter estimates. An advantage with the prediction error method is that for certain special cases we can avoid numerical integration. We also discuss how the unscented transform can be used to obtain an approximate predictor for the prediction error method. The numerical evaluation of these methods is performed on a simple first order FIR system with a cubic nonlinearity, for which some illustrative analytic properties are derived.

Nonlinear systems can be approximated by linear time-invariant (LTI) models in many ways. Here, LTI models that are optimal approximations in the mean-square error sense are analyzed. A necessary and sufficient condition on the input signal for the optimal LTI approximation of an arbitrary nonlinear finite impulse response (NFIR) system to be a linear finite impulse response (FIR) model is presented. This condition says that the input should be separable of a certain order, i.e., that certain conditional expectations should be linear. For the special case of Gaussian input signals, this condition is closely related to a generalized version of Bussgang's classic theorem about static nonlinearities. It is shown that this generalized theorem can be used for structure identification and for the identification of generalized Wiener-Hammerstein systems.

A general framework for estimating nonlinear functions and systems is described and analyzed in this paper. Identification of a system is seen as estimation of a predictor function. The considered predictor function estimate at a particular point is defined to be affine in the observed outputs and the estimate is defined by the weights in this expression. For each given point, the maximal mean-square error (or an upper bound) of the function estimate over a class of possible true functions is minimized with respect to the weights, which is a convex optimization problem. This gives different types of algorithms depending on the chosen function class. It is shown how the classical linear least squares is obtained as a special case and how unknown-but-bounded disturbances can be handled. Most of the paper deals with the method applied to locally smooth predictor functions. It is shown how this leads to local estimators with a finite bandwidth, meaning that only observations in a neighborhood of the target point will be used in the estimate. The size of this neighborhood (the bandwidth) is automatically computed and reflects the noise level in the data and the smoothness priors. The approach is applied to a number of dynamical systems to illustrate its potential.

We address a problem of non-parametric estimation of an unknown regression function f : (i1=2;1=2) ! R at a fixed point x0 2 (i1=2;1=2) on the basis of observations (xi;yi), i = 1;::;n such that yi = f(xi) + ei ; where ei » N(0;ae 2 ) is unobservable, Gaussian i.i.d. random noise and xi 2 (i1=2;1=2) are given design points. Recently, the Direct Weight Optimization (DWO) method has been proposed to solve a problem of such kind. The properties of the method have been studied for the case when the unknown function f is continuously dierentiable with Lipschitz continuous derivative having a priori known Lipschitz constant L. The minimax optimality and adaptivity with respect to the design have been established for the resulting estimator. However, in order to implement the approach, both L and ae are to be known. The subject of the submission is the study of an adaptive version of the DWO estimator which uses a data-driven choice of the method parameter L. Copyright c ∞ 2004 IFAC

Approximations of slightly nonlinear systems with linear time-invariant (LTI) models are often used in applications. Here, LTI models that are optimal approximations in the mean-square error sense are studied. It is shown that these models can be very sensitive to small nonlinearities. Furthermore, the significance of the distribution of the input process is discussed. From the examples studied here, it seems that LTI approximations for inputs with distributions that are Gaussian or almost Gaussian are less sensitive to small nonlinearities.

An expression for the variance of the estimated spectrum based on auto-regressions is developed. This expression is asymptotic in the number of data, but exact in the model order. As the order tends to infinity it converges to the well known result that the variance is proportional to the model order times the square of the spectrum itself. The exact expression gives insight into the character of this convergence, its speed and its dependence on the poles of the underlying AR-process.

We present a result that can be viewed as a generalization of Bussgang's classical theorem about static nonlinearities with Gaussian inputs. This result is used to characterize the best linear approximation of a nonlinear finite impulse response (NFIR) system with a Gaussian input. The best linear approximation is here defined as the causal and stable LTI system that minimizes the mean-square error. Furthermore, we discuss how this characterization can be used for structure identification and for identification of generalized Hammerstein and Wiener systems.

This report describes how parameter estimation can be performed in linear DAE systems. Both time domain and frequency domain identification are examined. The results are exemplified on a small syst ...

In this contribution we discuss some variance properties of a two-step ARX estimation scheme. An expression for the co-variance of the final low order model is calculated and it is discussed how one should minimize this covariance. The implication of the results is that identification of the dynamics of a system could very easily be performed with standard linear least squares (two times), even if the measurement noise is heavily colored. We also show a numerical example, where this two-step estimation scheme gives a variance which is close (but not equal) to the the Cramer-Rao lower bound. Moreover, we show that the point estimate of the covariance is close to the one obtained through Monte Carlo simulations.

The sequence of estimates formed by the LMS algorithm for a standard linear regression estimation problem is considered. It is known since earlier that smoothing these estimates by simple averaging will lead to, asymptotically, the recursive least-squares algorithm. In this paper, it is first shown that smoothing the LMS estimates using a matrix updating will lead to smoothed estimates with optimal tracking properties, also in case the true parameters are slowly changing as a random walk. The choice of smoothing matrix should be tailored to the properties of the random walk. Second, it is shown that the same accuracy can be obtained also for a modified algorithm, SLAMS, which is based on averages and requires much less computations.

In this paper the effect of some weighting matrices on the asymptotic variance of the estimates of linear discrete time state space systems estimated using subspace methods is investigated. The analysis deals with systems with white or without observed inputs and refers to the Larimore type of subspace procedures. The main result expresses the asymptotic variance of the system matrix estimates in canonical form as a function of some of the user choices, clarifying the question on how to choose them optimally. It is shown, that the CCA weighting scheme leads to optimal accuracy. The expressions for the asymptotic variance can be implemented more efficiently as compared to the ones previously published.

Mechanisms for adapting models, filters, decisions, regulators, and so on to changing properties of a system or a signal are of fundamental importance in many modern signal processing and control a ...

The sequence of estimates formed by the LMS algorithm for a standard linear regression estimation problem is considered. In this paper it is first shown that smoothing the LMS estimates using a matrix updating will lead to smoothed estimates with optimal tracking properties, also in the case the true parameters are slowly changing as a random walk. The choice of smoothing matrix should be tailored to the properties of the random walk. Second, it is shown that the same accuracy can be obtained also for a modified algorithm, SLAMS, which is based on averages and requires much less computations.

This paper deals with the problem of estimating the variance of an undermodeled model. Undermodeling means that the model class used is not flexible enough to describe the underlying system. The proposed solution to the problem is an algorithm that is based on the bootstrap. A simulation example shows that the variance estimates based on the proposed algorithm are in very good agreement with Monte Carlo simulations.

This contribution describes a common family of estimation methods for system identification, viz, prediction-error methods. The basic ideas behind these methods are described. An overview of typica ...

Asymptotic variance expressions are analyzed for models that are identified on the basis of closed-loop data. The considered methods comprise the classical 'direct' method, as well as the more recently developed indirect methods, employing coprime factorized models, dual Youla/Kucera parametrizations and the two-stage approach. The variance expressions are compared with the open-loop situation, and evaluated in terms of their relevance for subsequent model-based control design.

Expressions for the variance of an estimated frequency function are necessary for many issues in model validation and experiment design. A general result is that a simple expression for this variance can be obtained asymptotically as the model order tends to infinity. This expression shows that the variance is inversely proportional to the signal-to-noise ratio frequency by frequency. Still, for low order models the actual variance may be quite different. We derive an exact expression for the variance, which is not asymptotic in the model order. This expression applies to a restricted class of models: AR-models, as well as fixed pole models with a polynomial noise model. It brings out the character of the simple approximation and the convergence rate to the limit as the model order increases. It also provides nonasymptotic lower bounds for the general case. The calculations are illustrated by numerical examples.

The so-called accelerated convergence is an ingenuous idea to improve the asymptotic accuracy in stochastic approximation (gradient based) algorithms. The estimates obtained from the basic algorith ...

Identification of systems operating in closed loop has long been of prime interest in industrial applications. The problem offers many possibilities, and also some fallacies, and a wide variety of approaches have been suggested, many quite recently. The purpose of the current contribution is to place most of these approaches in a coherent framework, thereby showing their connections and display similarities and differences in the asymptotic properties of the resulting estimates. The common framework is created by the basic prediction error method, and it is shown that most of the common methods correspond to different parameterizations of the dynamics and noise models. The so-called indirect methods, e.g., are indeed ''direct'' methods employing noise models that contain the regulator. The asymptotic properties of the estimates then follow from the general theory and take different forms as they are translated to the particular parameterizations. We also study a new projection approach to closed-loop identification with the advantage of allowing approximation of the open-loop dynamics in a given, and user-chosen frequency domain norm, even in the case of an unknown, nonlinear regulator.

The determination of the right resolution parameter when estimating frequency functions of linear systems is a trade-off between bias and variance. Traditional non-parametric approaches, like “window-closing” employ a global resolution parameter — the window width — that is tuned by ad hoc methods, usually visual inspection of the results. This paper suggests a method that tunes such parameters by an automatic procedure. A further benefit is that the tuning can be performed locally, i.e., that different resolutions can be used in different frequency bands. The ideas are based on local polynomial regression and a data-driven bandwidth selector. The advantages of the proposed method are illustrated in numerical examples.

The present contribution deals with the accuracy of the transfer function of state-space parametric models estimated under the prediction error identification framework. More precisely, we intend to propagate the Cramer-Rao bound usually available on the covariance matrix of the state-space parameter estimates to that of the coefficients of the corresponding input-to-output transfer function. A natural way to solve this problem is to take advantage of the Jacobian matrix of the state-space to transfer function transformation while applying Gauss' formula for evaluating the covariance of the transfer function coefficients. Here, we focus on the computational aspects of the evaluation of this Jacobian matrix.

A Bayesian formulation of system identification problems has become popular recently; this is mainly due to the introduction of a family of prior descriptions (kernels) which encode structural properties of dynamical systems such as stability. The simplest instance of this kernel prescribes that the impulse response coefficients are independent random variables with exponentially decaying variance. Selecting the most suitable kernel within this class, which involves tuning the rate at which variance decay, is an important step. This paper studies the properties of the so-called “marginal likelihood” approach providing an interpretation in terms of Mean Squared Error properties of the resulting estimators.

The presence of abrupt changes, such as impulsive and load disturbances, commonly occur in applications, but make the state estimation problem considerably more difficult than in the standard setting with Gaussian process disturbance. Abrupt changes often introduce a jump in the state, and the problem is therefore readily and often treated by change detection techniques. In this paper, we take a different approach. The state smoothing problem for linear state space models is here formulated as a constrained least-squares problem with sum-of-norms regularization, a generalization of l1-regularization. This novel formulation can be seen as a convex relaxation of the well known generalized likelihood ratio method by Willsky and Jones. Another nice property of the suggested formulation is that it only has one tuning parameter, the regularization constant which is used to trade off fit and the number of jumps. Good practical choices of this parameter along with an extension to nonlinear state space models are given. CADICS

In this paper we discuss the role of criterion minimization as a means for parameter estimation. Most traditional methods, such as maximum likelihood and prediction error identification are based on these principles. However, somewhat surprisingly, it turns out that it is not always “optimal” to try to find the absolute minimum point of the criterion. The reason is that “stopped minimization” (where the iterations have been terminated before the absolute minimum has been reached) has more or less identical properties as using regularization (adding a parametric penalty term). Regularization is known to have beneficial effects on the variance of the parameter estimates. How does one know when to terminate the iterations then? A useful criterion would be to stop iterations when the criterion function applied to a validation data set no longer decreases. However, we show in this paper, that applying this technique extensively may lead to the fact that the resulting estimate is an unregularized estimate for the total data set: Estimation + validation data.

In this contribution aspects of inter-sample input signal behavior are examined. The starting point is that parametric identification always is performed on basis of discrete-time data. This is valid for identification of discrete-time models as well as continuous-time models. The usual assumptions on the input signal are; i) it is band-limited, ii) it is piecewise constant or iii) it is piecewise linear. One point made in this paper is that if a discrete-time model is used, the best possible (in the model structure) adjustment to data is made. This is independent of the assumption on the input signal. However, a transformation of the obtained discrete model to a continuous one is not possible without additional assumptions on the input signal. The other point made is that the frequency functions of the discrete models very well coincides with the frequency functions of the discretized continuous time models and the continuous time transfer function fitted in the frequency domain.

It is a fundamental problem of identification to be able—even before the data have been analyzed—to decide if all the free parameters of a model structure can be uniquely recovered from data. This is the issue of global identifiability. In this contribution we show how global identifiability for an arbitrary model structure (basically with analytic non-linearities) can be analyzed using concepts and algorithms from differential algebra. It is shown how the question of global structural identifiability is reduced to the question of whether the given model structure can be rearranged as a linear regression. An explicit algorithm to test this is also given. Furthermore, the question of ‘persistent excitation’ for the input can also be tested explicitly is a similar fashion. The algorithms involved are very well suited for implementation in computer algebra. One such implementation is also described.

In this companion paper, the choice of kernels for estimating the impulse response of linear stable systems is considered from a classical, “frequentist”, point of view. The kernel determines the regularization matrix in a regularized least squares estimate of an FIR model. The quality is assessed from a mean square error (MSE) perspective, and measures and algorithms for optimizing the MSE are discussed. The ideas are tested on the same data bank as used in Part I of the companion papers. The resulting findings and conclusions in the two papers are very similar despite the different perspectives.