Search

The universal approximation properties with respect to $L ^p $-type criteria of three important families of reservoir computers with stochastic discrete-time semi-infinite inputs is shown. First, it is proved that linear reservoir systems with either polynomial or neural network readout maps are universal. More importantly, it is proved that the same property holds for two families with linear readouts, namely, trigonometric state-affine systems and echo state networks, which are the most widely used reservoir systems in applications. The linearity in the readouts is a key feature in supervised machine learning applications. It guarantees that these systems can be used in high-dimensional situations and in the presence of large datasets. The $L ^p $ criteria used in this paper allow the formulation of universality results that do not necessarily impose almost sure uniform boundedness in the inputs or the fading memory property in the filter that needs to be approximated., 11 pages

Many recurrent neural network machine learning paradigms can be formulated using state-space representations. The classical notion of canonical state-space realization is adapted in this paper to accommodate semi-infinite inputs so that it can be used as a dimension reduction tool in the recurrent networks setup. The so-called input forgetting property is identified as the key hypothesis that guarantees the existence and uniqueness (up to system isomorphisms) of canonical realizations for causal and time-invariant input/output systems with semi-infinite inputs. Additionally, the notion of optimal reduction coming from the theory of symmetric Hamiltonian systems is implemented in our setup to construct canonical realizations out of input forgetting but not necessarily canonical ones. These two procedures are studied in detail in the framework of linear fading memory input/output systems. Finally, the notion of implicit reduction using reproducing kernel Hilbert spaces (RKHS) is introduced which allows, for systems with linear readouts, to achieve dimension reduction without the need to actually compute the reduced spaces introduced in the first part of the paper., Comment: 31 pages

A new explanation of the geometric nature of the reservoir computing (RC) phenomenon is presented. RC is understood in the literature as the possibility of approximating input-output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logarithmically reduced with respect to the original system. This reservoir system is able to approximate any element in the fading memory filters class just by training a different linear readout for each different filter. Explicit expressions for the probability distributions needed in the generation of the projected reservoir system are stated, and bounds for the committed approximation error are provided., IEEE Transactions on Neural Networks and Learning Systems, 33 (11), ISSN:2162-237X, ISSN:2162-2388

Echo state networks (ESNs) have been recently proved to be universal approximants for input/output systems with respect to various $L ^p$-type criteria. When $1\leq p< \infty$, only $p$-integrability hypotheses need to be imposed, while in the case $p=\infty$ a uniform boundedness hypotheses on the inputs is required. This note shows that, in the last case, a universal family of ESNs can be constructed that contains exclusively elements that have the echo state and the fading memory properties. This conclusion could not be drawn with the results and methods available so far in the literature., 6 pages letter

We propose a new approach based on a generalization of the logit model to improve prediction accuracy in U.S. bank failures. Mixed-data sampling (MIDAS) is introduced in the context of a logistic regression. We also mitigate the class-imbalance problem in data and adjust the classification accuracy evaluation. In applying the suggested model to the period from 2004 to 2016, we show that it correctly classifies significantly more bank failure cases than the classic logit model, in particular for long-term forecasting horizons. Some of the largest recent bank failures in the United States that had been previously misclassified are now correctly predicted.

The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs makes natural the introduction of the network forecasting capacity, that measures the possibility of forecasting time series values using network states. Generic bounds for memory and forecasting capacities are formulated in terms of the number of neurons of the nonlinear recurrent network and the autocovariance function or the spectral density of the input. These bounds generalize well-known estimates in the literature to a dependent inputs setup. Finally, for the particular case of linear recurrent networks with independent inputs it is proved that the memory capacity is given by the rank of the associated controllability matrix, a fact that has been for a long time assumed to be true without proof by the community., Comment: 27 pages, 1 figure. To appear in Physica D

This paper addresses the reservoir design problem in the context of delay-based reservoir computers for multidimensional input signals, parallel architectures, and real-time multitasking. First, an approximating reservoir model is presented in those frameworks that provides an explicit functional link between the reservoir parameters and architecture and its performance in the execution of a specific task. Second, the inference properties of the ridge regression estimator in the multivariate context is used to assess the impact of finite sample training on the decrease of the reservoir capacity. Finally, an empirical study is conducted that shows the adequacy of the theoretical results with the empirical performances exhibited by various reservoir architectures in the execution of several nonlinear tasks with multidimensional inputs. Our results confirm the robustness properties of the parallel reservoir architecture with respect to task misspecification and parameter choice that had already been documented in the literature., Comment: 24 pages, 6 figures

This paper shows that echo state networks are universal uniform approximants in the context of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times. This result guarantees that any fading memory input/output system in discrete time can be realized as a simple finite-dimensional neural network-type state-space model with a static linear readout map. This approximation is valid for infinite time intervals. The proof of this statement is based on fundamental results, also presented in this work, about the topological nature of the fading memory property and about reservoir computing systems generated by continuous reservoir maps., 28 pages

We propose a new approach based on a generalization of the classic logit model to improve prediction accuracy in US bank failures. We introduce mixed-data sampling (Midas) aggregation to construct financial predictors in a logistic regression. This allows relaxing the limitation of conventional annual aggregation in financial studies. Moreover, we suggest an algorithm to reweight observations in the log-likelihood function to mitigate the class-imbalance problem, that is, when one class of observations is severely undersampled. We also address the issue of the classification accuracy evaluation when imbalance of the classes is present. When applying the suggested model to the period from 2004 to 2016, we show that it correctly classifies more bank failure cases than the reference logit model introduced in the literature, in particular for long-term forecasting horizons. This improvement has a strong significant impact both in statistical and economic terms. Some of the largest recent bank failures in the US that were previously misclassified are now correctly predicted.

We introduce a new strategy for the prediction of linear temporal aggregates; we call it ‘hybrid’ and study its performance using asymptotic theory. This scheme consists of carrying out model parameter estimation with data sampled at the highest available frequency and the subsequent prediction with data and models aggregated according to the forecasting horizon of interest. We develop explicit expressions that approximately quantify the mean square forecasting errors associated with the different prediction schemes and that take into account the estimation error component. These approximate estimates indicate that the hybrid forecasting scheme tends to outperform the so-called ‘all-aggregated’ approach and, in some instances, the ‘all-disaggregated’ strategy that is known to be optimal when model selection and estimation errors are neglected. Unlike other related approximate formulas existing in the literature, those proposed in this paper are totally explicit and require neither assumptions on the second-order stationarity of the sample nor Monte Carlo simulations for their evaluation. Copyright © 2014 John Wiley & Sons, Ltd.

This paper extends the notion of information processing capacity for non-independent input signals in the context of reservoir computing (RC). The presence of input autocorrelation makes worthwhile the treatment of forecasting and filtering problems for which we explicitly compute this generalized capacity as a function of the reservoir parameter values using a streamlined model. The reservoir model leading to these developments is used to show that, whenever that approximation is valid, this computational paradigm satisfies the so called separation and fading memory properties that are usually associated with good information processing performances. We show that several standard memory, forecasting, and filtering problems that appear in the parametric stochastic time series context can be readily formulated and tackled via RC which, as we show, significantly outperforms standard techniques in some instances.

The aim of this presentation is to show how various ideas coming from the nonlinear stability theory of functional differential systems, stochastic modeling, and machine learning, can be put together in order to create an approximating model that explains the working mechanisms behind a certain type of reservoir computers. Reservoir computing is a recently introduced brain-inspired machine learning paradigm capable of excellent performances in the processing of empirical data. We focus on time-delay based reservoir computers that have been physically implemented using optical and electronic systems and have shown unprecedented data processing rates. Reservoir computing is well-known for the ease of the associated training scheme but also for the problematic sensitivity of its performance to architecture parameters. The reservoir design problem is addressed, which remains the biggest challenge in the applicability of this information processing scheme. Our results use the information available regarding the optimal reservoir working regimes in order to construct a functional link between the reservoir parameters and its performance. This function is used to explore various properties of the device and to choose the optimal reservoir architecture, thus replacing the tedious and time consuming parameter scannings used so far in the literature.

We apply a quadratic hedging scheme developed by Foellmer, Schweizer, and Sondermann to European contingent products whose underlying asset is modeled using a GARCH process and show that local risk-minimizing strategies with respect to the physical measure do exist, even though an associated minimal martingale measure is only available in the presence of bounded innovations. More importantly, since those local risk-minimizing strategies are in general convoluted and difficult to evaluate, we introduce Girsanov-like risk-neutral measures for the log-prices that yield more tractable and useful results. Regarding this subject, we focus on GARCH time series models with Gaussian innovations and we provide specific sufficient conditions that have to do with the finiteness of the kurtosis, under which those martingale measures are appropriate in the context of quadratic hedging. When this equivalent martingale measure is adapted to the price representation we are able to recover out of it the classical pricing formulas of Duan and Heston-Nandi, as well as hedging schemes that improve the performance of those proposed in the literature., 25 pages

This paper introduces a method based on the use of various linear and nonlinear state space models that uses non-synchronous data to extract global stochastic financial trends (GST). These models are specifically constructed to take advantage of the intraday arrival of closing information coming from different international markets in order to improve the quality of volatility description and forecasting performances. A set of three major asynchronous international stock market indices is used in order to empirically show that this forecasting scheme is capable of significant performance improvements when compared with those obtained with standard models like the dynamic conditional correlation (DCC) family.

We study different implementations of the sparse portfolio construction and rebalancing method introduced by Brodie et al. (Brodie, J., I. Daubechies, C. De Mol, D. Giannone, and I. Loris. 2009. “Sparse and Stable Markowitz Portfolios.” PNAS 106 (30): 12267–12272). This technique is based on the use of a l1-norm (sum of the absolute values) type penalization on the portfolio weights vector that regularizes the Markowitz portfolio selection problem by automatically eliminating the dynamical redundancies present in the time evolution of asset prices. We make specific recommendations as to the different estimation techniques for the parameters needed in the use of the method and we prove its good performance in realistic situations involving different rebalancing frequencies and transaction costs. Our empirical findings show that the beneficial effects of the use of sparsity constraints are robust with respect to the choice of trend and covariance estimation methods used in its implementation.

International audience; Reservoir computing is a recently introduced machine learning paradigm that has already shown excellent performances in the processing of empirical data. We study a particular kind of reservoir computers called time-delay reservoirs that are constructed out of the sampling of the solution of a time-delay diFFerential equation and show their good performance in the forecasting of the conditional covariances associated to multivariate discrete-time nonlinear stochastic processes of VEC-GARCH type as well as in the prediction of factual daily market realized volatilities computed with intraday quotes, using as training input daily log-return series of moderate size. We tackle some problems associated to the lack of task-universality for individually operating reservoirs and propose a solution based on the use of parallel arrays of time-delay reservoirs.

Numerous empirical proofs indicate the adequacy of the time discrete auto-regressive stochastic volatility models introduced by Taylor (Taylor S. J. [1982]; Taylor S. J. [1986]; Taylor S. J. [2005]) in the dynamical description of the log-returns of financial assets. The pricing and hedging of contingent products that use these models for their underlying assets is a complicated task due to the incomplete nature of the corresponding market and the non-observability of the associated volatility process. In this paper we introduce new pricing kernels for this setup and apply two existing volatility filtering techniques available in the literature for these models, namely Kalman filtering and the hierarchical-likelihood approach, in order to implement various pricing and dynamical hedging strategies. An extensive empirical analysis using both historical returns and options data illustrates the advantages of this model when compared with more standard approaches, namely Black-Scholes and GARCH.

We introduce a new strategy for the prediction of linear temporal aggregates, we call it "hybrid", and study its performance using asymptotic theory. This scheme consists of carrying out model parameter estimation with data sampled at the highest available frequency and the subsequent prediction with data and models aggregated according to the forecasting horizon of interest. We develop explicit expressions that approximately quantify the mean square forecasting errors associated to the different prediction schemes and that take into account the estimation error component. These approximate estimates indicate that the hybrid forecasting scheme tends to outperform the so called "all-aggregated" approach and, in some instances, the "all-disaggregated'' strategy that is known to be optimal when model selection and estimation errors are neglected. Unlike other related approximate formulas existing in the literature, the ones proposed in this paper are totally explicit and require neither assumptions on the second order stationarity of the sample nor Monte Carlo simulations for their evaluation.

The estimation of multivariate GARCH time series models is a difficult task mainly due to the significant overparameterization exhibited by the problem and usually referred to as the "curse of dimensionality". For example, in the case of the VEC family, the number of parameters involved in the model grows as a polynomial of order four on the dimensionality of the problem. Moreover, these parameters are subjected to convoluted nonlinear constraints necessary to ensure, for instance, the existence of stationary solutions and the positive semidefinite character of the conditional covariance matrices used in the model design. So far, this problem has been addressed in the literature only in low dimensional cases with strong parsimony constraints. In this paper we propose a general formulation of the estimation problem in any dimension and develop a Bregman-proximal trust-region method for its solution. The Bregman-proximal approach allows us to handle the constraints in a very efficient and natural way by staying in the primal space and the Trust-Region mechanism stabilizes and speeds up the scheme. Preliminary computational experiments are presented and confirm the very good performances of the proposed approach., Comment: 35 pages, 5 figures

One of the main themes of this book is the notion of symmetry. The main goal in the chapters at the core of this book is explaining to the reader how the symmetries of a Hamiltonian dynamical system can be used to simplify or reduce the study of that system. From the mathematical point of view the description of symmetries is implemented via the use of Lie group actions and, more generally, pseudogroups and groupoids. In the following two chapters we review all the material concerning these topics that will be needed in the rest of the book.