Search

This article develops a novel passive stochastic gradient algorithm. In passive stochastic approximation, the stochastic gradient algorithm does not have control over the location where noisy gradients of the cost function are evaluated. Classical passive stochastic gradient algorithms use a kernel that approximates a Dirac delta to weigh the gradients based on how far they are evaluated from the desired point. In this article, we construct a multikernel passive stochastic gradient algorithm. The algorithm performs substantially better in high dimensional problems and incorporates variance reduction. We analyze the weak convergence of the multikernel algorithm and its rate of convergence. In numerical examples, we study the multikernel version of the passive least mean squares algorithm for transfer learning to compare the performance with the classical passive version. [ABSTRACT FROM AUTHOR]

This article compiles several aspects of the dynamics of stochastic approximation algorithms with Markov iterate-dependent noise when the iterates are not known to be stable beforehand. We achieve the same by extending the lock-in probability (i.e., the probability of convergence of the iterates to a specific attractor of the limiting ordinary differential equation (o.d.e.) given that the iterates are in its domain of attraction after a sufficiently large number of iterations (say) $n_0$) framework to such recursions. Specifically, with the more restrictive assumption of Markov iterate-dependent noise supported on a bounded subset of the Euclidean space, we give a lower bound for the lock-in probability. We use these results to prove almost sure convergence of the iterates to the specified attractor when the iterates satisfy an asymptotic tightness condition. The novelty of our approach is that if the state space of the Markov process is compact, we prove almost sure convergence under much weaker assumptions compared to the work by Andrieu et al., which solves the general state-space case under much restrictive assumptions by providing sufficient conditions for stability of the iterates. We also extend our single-timescale results to the case where there are two separate recursions over two different timescales. This, in turn, is shown to be useful in analyzing the tracking ability of general adaptive algorithms. Additionally, we show that our results can be used to derive a sample complexity estimate of such recursions, which then can be used for step-size selection. [ABSTRACT FROM AUTHOR]

We consider the stochastic generalized Nash equilibrium problem (SGNEP) with expected-value cost functions. Inspired by Yi and Pavel (2019), we propose a distributed generalized Nash equilibrium seeking algorithm based on the preconditioned forward–backward operator splitting for SGNEPs, where, at each iteration, the expected value of the pseudogradient is approximated via a number of random samples. Our main contribution is to show almost sure convergence of the proposed algorithm if the pseudogradient mapping is restricted (monotone and) cocoercive. [ABSTRACT FROM AUTHOR]

We study gradient methods for solving distributed convex optimization problems over a network when the communication bandwidth between the nodes is limited, and so information that is exchanged between the nodes must be quantized. This imperfect communication poses a fundamental challenge, and if not properly accounted for may prevent the convergence of these algorithms. Our first contribution in this article is to propose a modified consensus-based gradient method for solving such problems using random (dithered) quantization. This algorithm can be interpreted as a distributed variant of a two-time-scale stochastic approximation algorithm. We then study the convergence and derive upper bounds on the rates of convergence of the proposed method as a function of the bandwidth available between the nodes and the underlying network topology for both convex and strongly convex objective functions. Our results complement existing literature, where such convergence and explicit formulas of the convergence rates are missing. Finally, we provide numerical simulations to compare the convergence properties of the distributed gradient methods with and without quantization for solving the well-known regression problems over networks, for both quadratic and absolute loss functions. [ABSTRACT FROM AUTHOR]

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools, and techniques that are popular in multiagent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/control problem becomes more tractable, function approximations affect stability and convergence. In this article, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze policy gradient methods and noisy value iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided that they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between the limiting set and the approximation errors is established. Finally, experimental results that support the theory are presented. [ABSTRACT FROM AUTHOR]

We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs Q-learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g., conditional value-at-risk, optimized certainty equivalent, and absolute semideviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance ε > 0 for optimal Q-value estimation gap and learning rate k ∈ (1/2,1], the overall convergence rate of our algorithm is Ω((ln(1/δε)/ε2)1/k + (ln(1/ε))1/(1−k)) with probability at least 1−ε. [ABSTRACT FROM AUTHOR]

An online reinforcement learning algorithm called renewal Monte Carlo (RMC) is presented. RMC works for infinite horizon Markov decision processes with a designated start state. RMC is a Monte Carlo algorithm that retains the key advantages of Monte Carlo—viz., simplicity, ease of implementation, and low bias—while circumventing the main drawbacks of Monte Carlo—viz., high variance and delayed updates. Given a parameterized policy $\pi _\theta$ , the algorithm consists of three parts: estimating the expected discounted reward $\mathsf {R}_{\theta }$ and the expected discounted time $\mathsf {T}_\theta$ over a regenerative cycle; estimating the derivatives $\nabla _\theta \mathsf {R}_\theta$ and $\nabla _\theta \mathsf {T}_\theta$ ; and updating the policy parameters using stochastic approximation to find the roots of $\mathsf {R}_\theta \nabla _\theta \mathsf {T}_\theta - \mathsf {T}_\theta \nabla _\theta \mathsf {R}_\theta$. It is shown that under mild technical conditions, RMC converges to a locally optimal policy. It is also shown that RMC works for postdecision state models as well. An approximate version of RMC is proposed where a regenerative cycle is defined as successive visits to a prespecified “renewal set”. It is shown that if the value function of the system is locally Lipschitz on the renewal set, then RMC converges to an approximate locally optimal policy. Three numerical experiments are presented to illustrate RMC and compare it with other state-of-the-art reinforcement learning algorithms. [ABSTRACT FROM AUTHOR]

We introduce deterministic perturbation (DP) schemes for the recently proposed random directions stochastic approximation, and propose new first-order and second-order algorithms. In the latter case, these are the first second-order algorithms to incorporate DPs. We show that the gradient and/or Hessian estimates in the resulting algorithms with DPs are asymptotically unbiased, so that the algorithms are provably convergent. Furthermore, we derive convergence rates to establish the superiority of the first-order and second-order algorithms, for the special case of a convex and quadratic optimization problem, respectively. Numerical experiments are used to validate the theoretical results. [ABSTRACT FROM AUTHOR]

This paper proposes an iterative learning control (ILC) algorithm with gain adaptation for discrete-time stochastic systems. The algorithm is based on Kesten's accelerated stochastic approximation (SA) algorithm. The gain adaptation uses only tracking error information, and, hence, is a data-driven adaptation approach. If stochastic noises account for a small proportion of the tracking error, the learning gain matrix remains constant with a high probability. If stochastic noises dominate the tracking error, the learning gain matrix is decreasing. Therefore, the new ILC algorithm converges more quickly than existing SA-based algorithms. In addition, the classic P-type ILC law for noise-free systems is a special case of the new ILC algorithm. The behaviors of the proposed ILC algorithm are demonstrated through examples. [ABSTRACT FROM AUTHOR]

In this paper, a novel distributed stochastic approximation algorithm (DSAA) is proposed to seek roots of the sum of local functions, each of which is associated with an agent from multiple agents connected over a network. At each iteration, each agent updates its estimate for the root utilizing the noisy observations of its local function and the information derived from the neighboring agents. The key difference of the proposed algorithm from the existing ones consists in the expanding truncations (so it is called the DSAAWET), by which the boundedness of the estimates can be guaranteed without imposing the growth-rate constraints on the local functions. The estimates generated by the DSAAWET are shown to converge almost surely to a consensus set, which belongs to a connected subset of the root set of the sum function. In comparison with the existing results, we impose weaker conditions on the local functions and on the observation noise. We then apply the proposed algorithm to two applications, one from signal processing and the other one from distributed optimization, and establish the almost sure convergence. Numerical simulation results are also included. [ABSTRACT FROM AUTHOR]

This paper investigates the interaction between control and communications in networked systems by studying a class of stochastic approximation algorithms that accommodate random network topology switching processes, time-varying functions, nonlinear dynamics, additive and nonadditive noises, and other uncertainties. Interaction among control strategy and the multiple stochastic processes introduces critical challenges in such problems. By modeling the random switching as a discrete-time Markov chain and studying multiple stochastic uncertainties in a unified framework, it is shown that under broad conditions, the algorithms are convergent. The performance of the algorithms is further analyzed by establishing their rate of convergence and asymptotic characterizations. Simulation case studies are conducted to evaluate the performance of the procedures in various aspects. [ABSTRACT FROM AUTHOR]

We are interested in understanding stability (almost sure boundedness) of stochastic approximation algorithms (SAs) driven by a “controlled Markov” process. Analyzing this class of algorithms is important, since many reinforcement learning (RL) algorithms can be cast as SAs driven by a “controlled Markov” process. In this paper, we present easily verifiable sufficient conditions for stability and convergence of SAs driven by a “controlled Markov” process. Many RL applications involve continuous state spaces. While our analysis readily ensures stability for such continuous state applications, traditional analyses do not. As compared to literature, our analysis presents a two-fold generalization: 1) the Markov process may evolve in a continuous state space and 2) the process need not be ergodic under any given stationary policy. Temporal difference (TD) learning is an important policy evaluation method in RL. The theory developed herein, is used to analyze generalized $\text{TD}(0)$ , an important variant of TD. Our theory is also used to analyze a TD formulation of supervised learning for forecasting problems. [ABSTRACT FROM AUTHOR]

Consensus conditions and convergence speeds are crucial for distributed consensus algorithms of networked systems. Based on a basic first-order average-consensus protocol with time-varying topologies and additive noises, this paper first investigates its critical consensus condition on network topology by stochastic approximation frameworks. A new joint-connectivity condition called extensible joint-connectivity that contains a parameter $\delta$ (termed the extensible exponent) is proposed. With this and a balanced topology condition, we show that a critical value of \delta$ for consensus is 1/2$ . Optimization on convergence rate of this protocol is further investigated. It is proved that the fastest convergence rate, which is the theoretic optimal rate among all controls, is of the order 1/t for the worst topologies, which are balanced and satisfy the extensible joint-connectivity condition. For practical implementation, certain open-loop control strategies are introduced to achieve consensus with a convergence rate of the same order as the fastest convergence rate. Furthermore, a consensus condition is derived for nonstationary and strongly correlated random topologies. The algorithms and consensus conditions are applied to distributed consensus computation of mobile ad-hoc networks; and their related critical exponents are derived from relative velocities of mobile agents for guaranteeing consensus. [ABSTRACT FROM PUBLISHER]

In this paper, the decentralized partially observable Markov decision process (Dec-POMDP) systems with discrete state and action spaces are studied from a gradient point of view. Dec-POMDPs have recently emerged as a promising approach to optimizing multiagent decision making in the partially observable stochastic environment. However, the decentralized nature of the Dec-POMDP framework results in a lack of shared belief state, which makes the decision maker impossible to estimate the system state based on local information. In contrast to the belief-based policy, this paper focuses on optimizing the decentralized observation-based policy, which is easily to be applied and does not have the sharing problem. By analyzing the gradient of the objective function, we have developed a centralized stochastic gradient policy iteration algorithm to find the optimal policy on the basis of gradient estimates from a single sample path. This algorithm does not need any specific assumption and can be applied to most practical Dec-POMDP problems. One numerical example is provided to demonstrate the effectiveness of the algorithm. [ABSTRACT FROM AUTHOR]

In this paper, the problem of identifying nonlinear systems under adaptive binary-valued output measurements is considered. We follow a nonparametric approach, which directly estimates the value of the nonlinear function representing the system at any fixed point with the help of a recursive kernel-based stochastic approximation algorithm with expanding truncations (SAAWET). The thresholds of the binary sensor are adaptively designed to achieve a sufficient richness of information in the output observations. The constructed estimates are proved to converge to the true values with probability one. Two numerical examples are given showing that the simulation results are consistent with the theoretical analysis. [ABSTRACT FROM AUTHOR]

We present new algorithms for simulation optimization using random directions stochastic approximation (RDSA). These include first-order (gradient) as well as second-order (Newton) schemes. We incorporate both continuous-valued as well as discrete-valued perturbations into both types of algorithms. The former are chosen to be independent and identically distributed (i.i.d.) symmetric uniformly distributed random variables (r.v.), while the latter are i.i.d. asymmetric Bernoulli r.v.s. Our Newton algorithm, with a novel Hessian estimation scheme, requires $N$-dimensional perturbations and three loss measurements per iteration, whereas the simultaneous perturbation Newton search algorithm of [1] requires $2N$ -dimensional perturbations and four loss measurements per iteration. We prove the asymptotic unbiasedness of both gradient and Hessian estimates and asymptotic (strong) convergence for both first-order and second-order schemes. We also provide asymptotic normality results, which in particular establish that the asymmetric Bernoulli variant of Newton RDSA method is better than 2SPSA of [1]. Numerical experiments are used to validate the theoretical results. [ABSTRACT FROM PUBLISHER]

We consider the classical TD(0) algorithm implemented on a network of agents wherein the agents also incorporate updates received from neighboring agents using a gossip-like mechanism. The combined scheme is shown to converge for both discounted and average cost problems. [ABSTRACT FROM AUTHOR]

This paper considers simulation-based optimization of the performance of a regime-switching stochastic system over a finite set of feasible configurations. Inspired by the stochastic fictitious play learning rules in game theory, we propose an adaptive random search algorithm that uses a smooth best-response sampling strategy and tracks the set of global optima, yet distributes the search so that most of the effort is spent on simulating the system performance at the global optima. The algorithm responds properly to the random unpredictable jumps of the global optimum even when the observations data are temporally correlated as long as a weak law of large numbers holds. Numerical examples show that the proposed scheme yields faster convergence and superior efficiency for finite sample lengths compared with several existing random search and pure exploration methods in the literature. [ABSTRACT FROM PUBLISHER]

This paper studies distributed stochastic approximation algorithms based on broadcast gossip on communication networks represented by digraphs. Weak convergence of these algorithms is proved, and an associated ordinary differential equation (ODE) is formulated connecting convergence points with local objective functions and network properties. Using these results, a methodology is proposed for network design, aimed at achieving the desired asymptotic behavior at consensus. Convergence rate of the algorithm is also analyzed and further improved using an attached stochastic differential equation. Simulation results illustrate the theoretical concepts. [ABSTRACT FROM AUTHOR]

We present a novel distributed algorithm for convex constrained optimization problems that are subject to noise corruption and uncertainties. The proposed scheme can be classified as a distributed stochastic approximation method, where a unique feature here is that we allow for multiple noise terms to appear in both the computation and communication stages of the distributed iterative process. Specifically, we consider problems that involve multiple agents optimizing a separable convex objective function subject to convex local constraints and linear coupling constraints. This is a richer class of problems compared to those that can be handled by existing distributed stochastic approximation methods which consider only consensus constraints and fewer sources of noise. The proposed algorithm utilizes the augmented Lagrangian (AL) framework, which has been widely used recently to solve deterministic optimization problems in a distributed way. We show that the proposed method generates sequences of primal and dual variables that converge to their respective optimal sets almost surely. [ABSTRACT FROM AUTHOR]

We develop a gradient algorithm for optimizing the performance of product-form networks through online adjustment of control parameters. The use of standard algorithms for finding optimal parameter settings is hampered by the prohibitive computational burden of calculating the gradient in terms of the stationary probabilities. The proposed approach instead relies on measuring empirical frequencies of the various states through simulation or online operation so as to obtain estimates for the gradient. Besides the reduction in computational effort, a further benefit of the online operation lies in the natural adaptation to slow variations in ambient parameters as commonly occurring in dynamic environments. On the downside, the measurements result in inherently noisy and biased estimates. We exploit mixing time results in order to overcome the impact of the bias and establish sufficient conditions for convergence to a globally optimal solution. We discuss our algorithm in the context of different systems, including queueing networks, loss networks, and wireless networks. We also illustrate how the algorithm can be used in such systems to optimize a service/cost trade-off, to map parameter regions that lead to systems meeting specified constraints, and to achieve target performance measures. For the latter application, we first identify which performance measures can be controlled depending on the set of configurable parameters. We then characterize the achievable region of performance measures in product-form networks, and finally we describe how our algorithm can be used to achieve the target performance in an online, distributed fashion, depending on the application context. [ABSTRACT FROM PUBLISHER]

Recently, Chen (2010) presented a new approach to recursive identification for ARMAX systems, which is a three-stage recursive scheme and assumes independent and identically distributed input signals. Here, we observe that, unless the time scale of the algorithm at one stage is reasonably faster than those at the previous stages, convergence to the true value may not take place. To remedy this issue, this note proposes a multi-time-scale modification of the algorithm in [5] such that convergence is achieved. In addition, the new scheme handles a wider class of input signals so that the input can be designed for some purpose.The advantage of the multi-time scale algorithm is verified with numerical examples. [ABSTRACT FROM PUBLISHER]

Recently, much progress has been made on particle swarm optimization (PSO). A number of works have been devoted to analyzing the convergence of the underlying algorithms. Nevertheless, in most cases, rather simplified hypotheses are used. For example, it often assumes that the swarm has only one particle. In addition, more often than not, the variables and the points of attraction are assumed to remain constant throughout the optimization process. In reality, such assumptions are often violated. Moreover, there are no rigorous rates of convergence results available to date for the particle swarm, to the best of our knowledge. In this paper, we consider a general form of PSO algorithms, and analyze asymptotic properties of the algorithms using stochastic approximation methods. We introduce four coefficients and rewrite the PSO procedure as a stochastic approximation type iterative algorithm. Then we analyze its convergence using weak convergence method. It is proved that a suitably scaled sequence of swarms converge to the solution of an ordinary differential equation. We also establish certain stability results. Moreover, convergence rates are ascertained by using weak convergence method. A centered and scaled sequence of the estimation errors is shown to have a diffusion limit. [ABSTRACT FROM AUTHOR]

Multi-dimensional stochastic optimization plays an important role in analysis and control of many technical systems. To solve the challenging multidimensional problems of nonstationary optimization, it is suggested to use a stochastic approximation algorithm (like SPSA) with perturbed input and constant step-size which has simple form. We get a finite bound of residual between estimates and time-varying unknown parameters when observations are made under an unknown but bounded noise. Applications of the algorithm are considered for a random walk, an optimization of UAV's flight, and a load balancing problem. [ABSTRACT FROM AUTHOR]

Stochastic approximation (SA) algorithms are proposed to identify a multi-input and multi-output (MIMO) Wiener system, in which the system input is taken to be a sequence of independent and identically distributed (i.i.d.) Gaussian random vectors uk\in{\cal N}(0,I). The algorithm for identifying the nonlinear part is designed with multi-variable kernel functions. Under suitable conditions, we show that the estimates of the coefficients of the linear subsystem and of the values of the nonlinear function converge to the respective true values with probability one. [ABSTRACT FROM AUTHOR]

In this note, we study the consensus problem for multiagent systems with measurement noises. Different from the existing approach, the consensus problem is converted to a root finding problem for which the stochastic approximation theory can be applied. By choosing an appropriate regression function, we propose a consensus algorithm which is applicable to systems with more general measurement noise processes, including stationary autoregressive and moving average (ARMA) processes and infinite moving average (MA) processes. Further, we establish a relationship between the convergence rate and the exponent of the step size of the algorithm. Particularly, strong convergence rate for systems with a leader-follower topology is studied. [ABSTRACT FROM AUTHOR]

This paper considers both synchronous and asynchronous consensus algorithms with noisy measurements. For stochastic approximation based consensus algorithms, the existing convergence analysis with dynamic topologies heavily relies on quadratic Lyapunov functions, whose existence, however, may be difficult to ensure for switching directed graphs. Our main contribution is to introduce a new analytical approach. We first show a fundamental role of ergodic backward products for mean square consensus in algorithms with additive noise. Subsequently, we develop ergodicity results for backward products of degenerating stochastic matrices converging to a 0–1 matrix via a discrete time dynamical system approach. These results complement the existing ergodic theory of stochastic matrices and provide an effective tool for analyzing consensus problems. Under a joint connectivity assumption, our approach may deal with switching topologies, delayed measurements and correlated noises, and it does not require the double stochasticity condition typically assumed for the existence of quadratic Lyapunov functions. [ABSTRACT FROM AUTHOR]

Due to the existence of various uncertainties, the design of distributed estimation algorithms with robustness and high accuracy is an urgent demand for sensor network applications. This paper is aimed at investigating the design of distributed parameter estimation algorithms and the analysis of their convergence properties in uncertain sensing and communication environments. Consensus-based distributed parameter estimation algorithms for both discrete-time and continuous-time cases are established, which are suitable for unreliable communication networks with stochastic communication noises, random link gains and Markovian signal losses. Under mild conditions on stochastic noises, gain function and topology-switching Markov chain, we establish both the mean square and almost sure convergence of the designed algorithms by use of probability limit theory, algebraic graph theory, stochastic differential equation theory and Markov chain theory. The effect of sensor-dependent gain functions on the convergence of the algorithm is also analyzed. [ABSTRACT FROM AUTHOR]

We study a class of random sampling-based algorithms for solving general non-differentiable optimization problems. These are iterative approaches that are based on sampling from and updating an underlying distribution function over the set of feasible solutions. In particular, we propose a novel and systematic framework to investigate the convergence and asymptotic convergence rates of these algorithms by exploiting their connections to the well-known stochastic approximation (SA) method. Such an SA framework unifies our understanding of these randomized algorithms and provides new insight into their design and implementation issues. Our preliminary numerical experiments indicate that new implementations of these algorithms based on the proposed framework may lead to improved performance over existing procedures. [ABSTRACT FROM PUBLISHER]

In this paper, an algorithm for decentralized multi-agent estimation of parameters in linear discrete-time regression models is proposed in the form of a combination of local stochastic approximation algorithms and a global consensus strategy. An analysis of the asymptotic properties of the proposed algorithm is presented, taking into account both the multi-agent network structure and the probabilities of getting local measurements and implementing exchange of inter-agent messages. In the case of non-vanishing gains in the stochastic approximation algorithms, an asymptotic estimation error covariance matrix bound is defined as the solution of a Lyapunov-like matrix equation. In the case of asymptotically vanishing gains, the mean-square convergence is proved and the rate of convergence estimated. In the discussion, the problem of additive communication noise is treated in a methodologically consistent way. It is also demonstrated how the consensus scheme in the algorithm can contribute to the overall reduction of measurement noise influence. Some simulation results illustrate the obtained theoretical results. [ABSTRACT FROM AUTHOR]

We introduce bounds on the finite-time performance of Markov chain Monte Carlo (MCMC) algorithms in solving global stochastic optimization problems defined over continuous domains. It is shown that MCMC algorithms with finite-time guarantees can be developed with a proper choice of the target distribution and by studying their convergence in total variation norm. This work is inspired by the concept of finite-time learning with known accuracy and confidence developed in statistical learning theory. [ABSTRACT FROM AUTHOR]