Search

Many real-world domains require safe decision making in the presence of uncertainty. In this work, we propose a deep reinforcement learning framework for approaching this important problem. We consider a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures, and we show that our formulation is equivalent to a distributionally robust safe reinforcement learning problem with robustness guarantees on performance and safety. We propose an efficient implementation that only requires access to a single training environment, and we demonstrate that our framework produces robust, safe performance on a variety of continuous control tasks with safety constraints in the Real-World Reinforcement Learning Suite.

In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original high-dimensional dynamics onto a computationally tractable low-dimensional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlinearity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier-Stokes equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman filter designed using the same ROM. Moreover, it yields accurate high-dimensional state estimates for reference trajectories corresponding to various physical parameter values, without direct knowledge of the latter., Comment: 21 pages, 16 figures

The design automation of analog circuits is a longstanding challenge in the integrated circuit field. This paper presents a deep reinforcement learning method to expedite the design of analog circuits at the pre-layout stage, where the goal is to find device parameters to fulfill desired circuit specifications. Our approach is inspired by experienced human designers who rely on domain knowledge of analog circuit design (e.g., circuit topology and couplings between circuit specifications) to tackle the problem. Unlike all prior methods, our method originally incorporates such key domain knowledge into policy learning with a graph-based policy network, thereby best modeling the relations between circuit parameters and design targets. Experimental results on exemplary circuits show it achieves human-level design accuracy (~99%) with 1.5x efficiency of existing best-performing methods. Our method also shows better generalization ability to unseen specifications and optimality in circuit performance optimization. Moreover, it applies to designing diverse analog circuits across different semiconductor technologies, breaking the limitations of prior ad-hoc methods in designing one particular type of analog circuits with conventional semiconductor technology., 8 pages, 5 figures, 2 tables, Thirty-Sixth AAAI Conference on Artificial Intelligence, The 1st Annual AAAI Workshop on AI to Accelerate Science and Engineering (AI2ASE)

Safety and robustness are two desired properties for any reinforcement learning algorithm. CMDPs can handle additional safety constraints and RMDPs can perform well under model uncertainties. In this paper, we propose to unite these two frameworks resulting in robust constrained MDPs (RCMDPs). The motivation is to develop a framework that can satisfy safety constraints while also simultaneously offer robustness to model uncertainties. We develop the RCMDP objective, derive gradient update formula to optimize this objective and then propose policy gradient based algorithms. We also independently propose Lyapunov based reward shaping for RCMDPs, yielding better stability and convergence properties., Comment: arXiv admin note: text overlap with arXiv:2010.04870

In this paper, we investigate the performance of two first-order optimization algorithms, obtained from forward Euler discretization of finite-time optimization flows. These flows are the rescaled-gradient flow (RGF) and the signed-gradient flow (SGF), and consist of non-Lipscthiz or discontinuous dynamical systems that converge locally in finite time to the minima of gradient-dominated functions. We propose an Euler discretization for these first-order finite-time flows, and provide convergence guarantees, in the deterministic and the stochastic setting. We then apply the proposed algorithms to academic examples, as well as deep neural networks training, where we empirically test their performances on the SVHN dataset. Our results show that our schemes demonstrate faster convergences against standard optimization alternatives.

This paper proposes a finite-time Newton seeking control design for systems described by unknown multivariable static maps. The Newton seeking system has an averaged system that implements a Newton continuous-time algorithm. The averaged Newton seeking system is shown to achieve finite-time stability of the unknown optimum of the static map. An averaging analysis demonstrates that the proposed Newton-seeking achieves finite-time practical stability of the optimum. A simulation study is used to demonstrate the effectiveness of the design method., Comment: 16 pages, 4 figures

In this paper, we focus on the problem of robustifying reinforcement learning (RL) algorithms with respect to model uncertainties. Indeed, in the framework of model-based RL, we propose to merge the theory of constrained Markov decision process (CMDP), with the theory of robust Markov decision process (RMDP), leading to a formulation of robust constrained-MDPs (RCMDP). This formulation, simple in essence, allows us to design RL algorithms that are robust in performance, and provides constraint satisfaction guarantees, with respect to uncertainties in the system's states transition probabilities. The need for RCMPDs is important for real-life applications of RL. For instance, such formulation can play an important role for policy transfer from simulation to real world (Sim2Real) in safety critical applications, which would benefit from performance and safety guarantees which are robust w.r.t model uncertainty. We first propose the general problem formulation under the concept of RCMDP, and then propose a Lagrangian formulation of the optimal problem, leading to a robust-constrained policy gradient RL algorithm. We finally validate this concept on the inventory management problem.

In this paper, we propose two discontinuous dynamical systems in continuous time with guaranteed prescribed finite-time local convergence to strict local minima of a given cost function. Our approach consists of exploiting a Lyapunov-based differential inequality for differential inclusions, which leads to finite-time stability and thus finite-time convergence with a provable bound on the settling time. In particular, for exact solutions to the aforementioned differential inequality, the settling-time bound is also exact, thus achieving prescribed finite-time convergence. We thus construct a class of discontinuous dynamical systems, of second order with respect to the cost function, that serve as continuous-time optimization algorithms with finite-time convergence and prescribed convergence time. Finally, we illustrate our results on the Rosenbrock function., Comment: Presented at workshop "Beyond First Order Methods in Machine Learning" of NeurIPS 2019

We present some results on stabilization for reduced-order models (ROMs) of partial differential equations. The stabilization is achieved using Lyapunov theory to design a new closure model that is robust to parametric uncertainties. The free parameters in the proposed ROM stabilization method are optimized using a model-free multi-parametric extremum seeking (MES) algorithm. The 3D Boussinesq equations provide a challenging numerical test-problem that is used to demonstrate the advantages of the proposed method., arXiv admin note: text overlap with arXiv:1510.01728

We study in this paper the problem of adaptive trajectory tracking control for nonlinear systems affine in the control with time-varying parametric uncertainties. We propose to use a modular approach, in the sense that we first design a robust nonlinear state feedback which renders the closed loop input to state stable (ISS) between an estimation error of the uncertain parameters and an output tracking error. Next, we complement this robust ISS controller with a model-free extremum seeking (ES) algorithm to estimate the time-varying model uncertainties. The combination of the ISS feedback and the ES algorithm gives an indirect adaptive controller. We show the efficiency of this approach on a two-link robot manipulator example.

We study in this paper the problem of adaptive trajectory tracking for nonlinear systems affine in the control with bounded state-dependent and time-dependent uncertainties. We propose to use a modular approach, in the sense that we first design a robust nonlinear state feedback which renders the closed loop input to state stable(ISS) between an estimation error of the uncertain parameters and an output tracking error. Next, we complement this robust ISS controller with a model-free multiparametric extremum seeking (MES) algorithm to estimate the model uncertainties. The combination of the ISS feedback and the MES algorithm gives an indirect adaptive controller. We show the efficiency of this approach on a two-link robot manipulator example.

In this work we study the problem of rope sway dynamics control for elevator systems. We choose to actuate the system with a semi-active damper mounted on the top of the elevator car. We propose nonlinear controllers based on Lyapunov theory, to actuate the semi-active damper and stabilize the rope sway dynamics. We study the stability of the proposed controllers, and test their performances on a numerical example.

Fault tolerant control (FTC) is the branch of control theory, dealing with the control of systems that become faulty during their operating life. Following the systems classification, as linear and nonlinear models, FTC can be classified in two different groups, linear FTC (LFTC) dealing with linear models, and the one of interest to us in this paper, nonlinear FTC (NFTC), which deals with nonlinear models. We present in this paper a survey of some of the results obtained in these last years on NFTC.