Search

In this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved. [ABSTRACT FROM AUTHOR]

Summary: In this paper, we consider the optimal control of unknown stochastic dynamical system for both the finite‐horizon and infinite‐horizon cases. The objective of this paper is to find an optimal controller to minimize the expected value of a function which depends on the random disturbance. Throughout this paper, it is assumed that the mean vector and covariance matrix of the disturbance distribution is unknown. An uncertainty set in the space of mean vector and the covariance matrix is introduced. For the finite‐horizon case, we derive a closed‐form expression of the unique optimal policy and the opponents policy that generates the worst‐case distribution. For the infinite‐horizon case, we simplify the Riccati equation obtained in the finite‐hozion setting to an algebraic Riccati equation, which can guarantee the existence of the solution of the Riccati equation. It is shown that the resulting optimal policies obtained in these two cases can stabilize the expected value of the system state under the worst‐case distribution. Furthermore, the unknown system matrices can also be explicitly computed using the adaptive dynamic programming technique, which can help compute the optimal control policy by solving the algebraic Riccati equation. Finally, a simulation example is presented to demonstrate the effectiveness of our theoretical results. [ABSTRACT FROM AUTHOR]

This paper deals with finite‐horizon minimum‐variance and covariance steering problems subject to constraints. The goal of the minimum variance problem is to steer the state mean of an uncertain system to a prescribed vector while minimizing the trace of its terminal state covariance whereas the goal in the covariance steering problem is to steer the covariance matrix of the terminal state to a prescribed positive definite matrix. The paper proposes a solution approach that relies on a stochastic version of the affine disturbance feedback control parametrization. In this control policy parametrization, the control input at each stage is expressed as an affine function of the history of disturbances that have acted upon the system. It is shown that this particular parametrization reduces the stochastic optimal control problems considered in this paper into tractable convex programs or difference of convex functions programs with essentially the same decision variables. In addition, the paper proposes a variation of this control parametrization that relies on truncated histories of past disturbances, which allows for sub‐optimal controllers to be designed that strike a balance between performance and computational cost. The suboptimality of the truncated policies is formally analyzed and closed form expressions are provided for the performance loss due to the use of the truncation scheme. Finally, the paper concludes with a comparative analysis of the truncated versions of the proposed policy parametrization and other standard policy parametrizations through numerical simulations. [ABSTRACT FROM AUTHOR]

This paper is concerned with quantum stochastic differential equations driven by the fermion field in noncommutative space Lp(풞) for 2≤p<∞$$ 2\le p<\infty $$. First, we investigate the existence and uniqueness of Lp$$ {L}&#x0005E;p $$‐solutions of quantum stochastic differential equations in an infinite time horizon by using the noncommutative Burkholder–Gundy inequality given by Pisier and Xu and the noncommutative generalized Minkowski inequality. Then, we investigate the stability, self‐adjointness, and Markov properties of Lp$$ {L}&#x0005E;p $$‐solutions and analyze the error of numerical schemes of quantum stochastic differential equations. The results of this paper can be utilized for investigating the optimal control of quantum stochastic systems with infinite dimensions. [ABSTRACT FROM AUTHOR]

We develop a new continuous‐time stochastic gradient descent method for optimizing over the stationary distribution of stochastic differential equation (SDE) models. The algorithm continuously updates the SDE model's parameters using an estimate for the gradient of the stationary distribution. The gradient estimate is simultaneously updated using forward propagation of the SDE state derivatives, asymptotically converging to the direction of steepest descent. We rigorously prove convergence of the online forward propagation algorithm for linear SDE models (i.e., the multidimensional Ornstein–Uhlenbeck process) and present its numerical results for nonlinear examples. The proof requires analysis of the fluctuations of the parameter evolution around the direction of steepest descent. Bounds on the fluctuations are challenging to obtain due to the online nature of the algorithm (e.g., the stationary distribution will continuously change as the parameters change). We prove bounds for the solutions of a new class of Poisson partial differential equations (PDEs), which are then used to analyze the parameter fluctuations in the algorithm. Our algorithm is applicable to a range of mathematical finance applications involving statistical calibration of SDE models and stochastic optimal control for long time horizons where ergodicity of the data and stochastic process is a suitable modeling framework. Numerical examples explore these potential applications, including learning a neural network control for high‐dimensional optimal control of SDEs and training stochastic point process models of limit order book events. [ABSTRACT FROM AUTHOR]

This paper is concerned with the investment and reinsurance problem between two insurance companies and a reinsurance company by constructing a two‐layer stochastic differential game. Insurance companies invest in a risk‐free asset, a defaultable bond, and a risky asset under the bi‐fractional Brownian motion environment; reinsurance companies invest in a risk‐free asset and a risky asset under the bi‐fractional Brownian motion environment. In order to maximize the expected utility of the insurance companies' relative wealth and the expected utility of the reinsurance company's wealth at the terminal time, we solve the Hamilton–Jacobi–Bellman (HJB) equations by using the differential game theory and stochastic optimal control theory and obtain the equilibrium investment–reinsurance and reinsurance premium strategies. Finally, we investigate the influence of the parameters on the equilibrium strategy through numerical examples and analyze its economic implications. [ABSTRACT FROM AUTHOR]

Summary: This paper deals with the second‐order necessary optimality conditions for discrete‐time stochastic optimal control problems under weakened convexity assumptions. Using a special variation of the control, and by virtue of a new discrete‐time backward stochastic equation, we establish a more general and constructive first‐order necessary optimality condition in the form of a global stochastic maximum principle. Moreover, by introducing a new discrete‐time backward stochastic matrix equation, the second‐order multipoint necessary optimality conditions of singular controls are derived, which covers and improves the classical second‐order necessary optimality conditions of discrete‐time stochastic systems. [ABSTRACT FROM AUTHOR]

This paper deals with the discrete-time mean-field stochastic optimal control problem under weakened convexity assumption. By introducing the concept of the γ-convex set, a special variation of the control is proposed. Using the new discrete-time backward stochastic equation of mean-field type, the global stochastic maximum principle is established. Moreover, as illustrations, one kind of discrete-time linear-quadratic (LQ) nonzero-sum stochastic game and a discrete-time mean-variance portfolio selection optimization problem are solved. [ABSTRACT FROM AUTHOR]

In the present note, we state and solve the linear quadratic (LQ) control problem for a class of McKean--Vlasov stochastic differential equations for which the control actions are of impulsive nature. After reformulating the original optimization problem using an orthogonal decomposition of the state variables, we introduce an adequately defined system of two coupled matrix Lyapunov-type differential equations with jumps. Such equations are central for the definition of the optimal feedback gains corresponding to the closed-loop representation of the optimal LQ control. [ABSTRACT FROM AUTHOR]

In this paper, we establish a second-order stochastic maximum principle for optimal stochastic control of stochastic differential equations of general mean-field type. The coefficients of the system are nonlinear and depend on the state process as well as of its probability law. The control variable is allowed to enter into both drift and diffusion terms. We establish a set of second-order necessary conditions for the optimal control in integral form. The control domain is assumed to be convex. The proof of our main result is based on the first- and second-order derivatives with respect to the probability law and by using a convex perturbation with some appropriate estimates. [ABSTRACT FROM AUTHOR]

In this paper, we study an optimal control problem of a mean-field forward--backward stochastic system with random jumps in progressive structure, where both regular and singular controls are considered in our formula. In virtue of the variational technology, the related stochastic maximum principle (SMP) has been obtained, and it is essentially different from that in the classical predictable structure. Specifically, there are three parts in our SMP, that is, continuous part, jump part, and impulse part, and they are, respectively, used to characterize the characteristics of the optimal controls at continuous time, jump time, and impulse time. This shows that the progressive structure can more accurately describe the characteristics of the optimal control at the jump time.We also give two linear--quadratic examples to show the significance of our results. [ABSTRACT FROM AUTHOR]

We study the "real world" calibration of a partially specified stochastic volatility model, where the analytic expressions of the asset price drift rate and of the stochastic variance drift are not specified. The model is calibrated matching the observed asset log returns and the priors assigned by the investor. No option price data are used in the calibration. The priors chosen for the asset price drift rate and for the stochastic variance drift are those suggested by the Heston model. For this reason, the model presented can be considered as an "enhanced" Heston model. The calibration problem is formulated as a stochastic optimal control problem and solved using the dynamic programming principle. The model presented and the Heston model are calibrated using synthetic and Standard & Poor 500 (S&P500) data. The calibrated models are used to produce 6, 12, and 24 months in the future synthetic and S&P500 forecasts. [ABSTRACT FROM AUTHOR]

In this paper, we consider the stochastic optimal control problem for moving average control system. The corresponding moving average stochastic differential equation is a kind of integral differential equations. We prove the existence and uniqueness of the solution of the moving average stochastic differential equations. We obtain the stochastic maximum principle of the moving average optimal control system by introducing a kind of generalized anticipated backward stochastic differential equations. We prove the existence and uniqueness of the solution of this adjoint equation, which is singular at 0. As an application, the linear quadratic moving average control problem is investigated to illustrate the main results. [ABSTRACT FROM AUTHOR]

Scenario‐based stochastic optimal control problems suffer from the curse of dimensionality as they can easily grow to six and seven figure sizes. First‐order methods are suitable as they can deal with such large‐scale problems, but may perform poorly and fail to converge within a reasonable number of iterations. To achieve a fast rate of convergence and high solution speeds, in this article, we propose the use of two proximal quasi‐Newtonian limited‐memory algorithms—minfbe applied to the dual problem and the Newton‐type alternating minimization algorithm (nama)—which can be massively parallelized on lockstep hardware such as graphics processing units. In particular, we use minfbe and nama to solve scenario‐based stochastic optimal control problems with affine dynamics, convex quadratic cost functions (with the stage cost functions being strongly convex in the control variable) and joint state‐input convex constraints. We demonstrate the performance of these methods, in terms of convergence speed and parallelizability, on large‐scale problems involving millions of variables. [ABSTRACT FROM AUTHOR]

Model‐based methods in autonomous driving and advanced driving assistance gain importance in research and development due to their potential to contribute to higher road safety. Parameters of vehicle models, however, are hard to identify precisely or they can change quickly depending on the driving conditions. In this paper, we address the problem of safe trajectory planning under parametric model uncertainties motivated by automotive applications. We use the generalized polynomial chaos expansions for efficient nonlinear uncertainty propagation and distributionally robust inequalities for chance constraints approximation. Inspired by the tube‐based model predictive control, an ancillary feedback controller is used to control the deviations of stochastic modes from the nominal solution, and therefore, decrease the variance. Our approach allows reducing conservatism related to nonlinear uncertainty propagation while guaranteeing constraints satisfaction with a high probability. The performance is demonstrated on the example of a trajectory optimization problem for a simplified vehicle model with uncertain parameters. [ABSTRACT FROM AUTHOR]

This paper studies a continuous‐time stochastic linear‐quadratic (SLQ) optimal control problem on infinite‐horizon. Combining the Kronecker product theory with an existing policy iteration algorithm, a data‐driven policy iteration algorithm is proposed to solve the problem. In contrast to most existing methods that need all information of system coefficients, the proposed algorithm eliminates the requirement of three system matrices by utilizing data of a stochastic system. More specifically, this algorithm uses the collected data to iteratively approximate the optimal control and a solution of the stochastic algebraic Riccati equation (SARE) corresponding to the SLQ optimal control problem. The convergence analysis of the obtained algorithm is given rigorously, and a simulation example is provided to illustrate the effectiveness and applicability of the algorithm. [ABSTRACT FROM AUTHOR]

The present paper considers the finite‐horizon indefinite linear quadratic (LQ) control problem for stochastic Takagi–Sugeno (T‐S) fuzzy systems with input delay. In this paper, we consider the presence of sensor data scheduling, which imposes a communication energy constraint and necessitates optimal state estimation for measurements. Then, by utilizing dynamic programming principles, the stochastic LQ problem under consideration can be solved, while the optimal control policy is developed in terms of the unique solutions to a set of coupled difference Riccati equations (CDREs). Specifically, for simple delay‐free case, the linear matrix inequalities based conditions are also proposed, whose feasibility is shown to be equivalent to the well‐posedness of the indefinite LQ control under consideration. As an application, our theoretic analysis is extended to study the intermittent observation model caused by random denial‐of‐service attack. [ABSTRACT FROM AUTHOR]

In this paper, we study a new class of noninstantaneous impulsive fractional stochastic differential systems driven by the Rosenblatt process with state‐dependent delay. We utilized the β$$ \beta $$‐resolvent family, fixed point technique, and solution operator to present the solvability of the proposed system. Further, we derived the existence of optimal multicontrol pairs for the considered system. Finally, the main results are validated with the aid of an example. [ABSTRACT FROM AUTHOR]

In this article, we study a risk‐sensitive stochastic optimal control problem driven by forward‐backward systems in which the control variable consists of two components: the continuous control and the impulse control. The control domain is assumed to be convex. We establish the maximum principle (i.e., necessary condition) for this kind of control problem. Under some additional assumptions, the necessary optimality conditions turn out to be sufficient. To explain the theoretical results, a linear‐quadratic risk‐sensitive control problem is solved by using the maximum principle derived and the optimal control is obtained. [ABSTRACT FROM AUTHOR]

This article studies optimal control of a discrete‐time stochastic differential equation of mean‐field type with coefficients dependent on function of the law and state of the process. A new version of the maximum principle for discrete‐time mean‐field type stochastic optimal control problems is established, using new discrete‐time mean‐field backward stochastic equations. The cost functional is also of mean‐field type. The study derives necessary first‐order and sufficient optimality conditions using adjoint equations that take the form of discrete‐time backward stochastic differential equations with a mean‐field component. An optimization problem for production and consumption choice is used as an example. [ABSTRACT FROM AUTHOR]

We study a stochastic control problem of mean‐field controlled stochastic differential systems driven by a fractional Brownian motion with Hurst parameter H∈(1/2,1)$$ H\in \left(1/2,1\right) $$. As a necessary condition of the optimal control we obtain a stochastic maximum principle. The associated adjoint mean‐field backward stochastic differential equation driven by a fractional Brownian motion and a classical Brownian motion. Applying the stochastic maximum principle to a mean‐field stochastic linear quadratic problem, we obtain the optimal control and prove that the necessary condition for the optimality of an admissible control is also sufficient under certain assumptions. [ABSTRACT FROM AUTHOR]

A class of fully coupled forward–backward stochastic difference equations with delay (FBSDDEs) over infinite horizon are considered in this article. By establishing a non‐homogeneous explicit relation between the forward and backward equations in terms of Riccati‐like difference equations, we derive the unique solution to the FBSDDEs under certain conditions. Then, we deduce that the FBSDDEs are solvable if and only if the corresponding stochastic delayed system is β$$ \beta $$‐degree open‐loop mean‐square exponentially stabilizable. Finally, as an application, the FBSDDEs are employed to demonstrate the maximum principle of the stochastic LQ optimal control problem. [ABSTRACT FROM AUTHOR]

Nonlinear model predictive control (NMPC) is one of the few control methods that can handle complex nonlinear systems with multi‐objectives and various constraints. However, the performance of NMPC highly depends on the model accuracy and the deterministic solutions may suffer from conservatism, for example, robust MPC only considers the worst‐case scenario, which yields the NMPC not working efficiently in uncertain stochastic cases. To address these issues, a model‐and data‐driven predictive control approach using Gaussian processes (GP‐MDPC) is synthesized in this paper, which copes with the tracking control problems of stochastic nonlinear systems subject to model uncertainties and chance constraints. Because GP has high flexibility to capture complex unknown functions and it inherently handles measurement noise, GP models are employed to approximate the unknowns, the predictions and uncertainty quantification provided by the GPs are then exploited to propagate the uncertainties through the nonlinear model and to formulate a finite‐horizon stochastic optimal control problem (FH‐SOCP). Specifically, given the GP models, closed‐loop simulations are executed offline to generate Monte Carlo samples, from which the back‐offs for constraint tightening are calculated iteratively. The tightened constraints then guarantee the satisfaction of chance constraints online. A tractable GP‐MDPC framework using back‐offs for handling nonlinear chance constrained tracking control problems is yielded, whose advantages include fast online evaluation, consideration of closed‐loop behaviour, and achievable trade‐off between conservatism and constraint violation. Comparisons are carried out to verify the effectiveness and superiority of the proposed GP‐MDPC scheme with back‐offs. [ABSTRACT FROM AUTHOR]

In this paper, we consider the indefinite stochastic optimal control problems of discrete‐time Markov jump linear systems. Firstly, we establish the new stochastic maximum principle, and by solving the forward‐backward stochastic difference equations with Markov jump (FBSDEs‐MJ), we derive the necessary and sufficient solvability condition of the indefinite control problem with non‐discounted cost, which is in an explicit analytical expression. Then, the optimal control is designed by a series of coupled generalized Riccati difference equations with Markov jump (GRDEs‐MJ) and linear recursive equations with Markov jump (LREs‐MJ). Moreover, based on the non‐discounted cost case, we deduce the optimal control problem with discounted cost. Finally, a numerical example for defined‐benefit (DB) pension fund with regime switching is exploited to illustrate the validity of the obtained results. [ABSTRACT FROM AUTHOR]

In this article, we study the relationship between maximum principle (MP) and dynamic programming principle (DPP) for stochastic recursive optimal control problem driven by G$$ G $$‐Brownian motion. Under the smooth assumption for the value function, we obtain the connection between MP and DPP under a reference probability Pt,x∗$$ {P}_{t,x}^{\ast } $$. Within the framework of viscosity solution, we establish the relation between the first‐order super‐jet, sub‐jet of the value function and the solution to the adjoint equation respectively. [ABSTRACT FROM AUTHOR]

The fault tolerant control (FTC) problem based on rational entropy performance criteria is researched for fuzzy descriptor stochastic distribution networked control (SDNC) systems with packet dropout. The independent Bernoulli distribution is employed to describe the packet dropout in the feedback channel. The static model and the dynamic model of descriptor SDNC systems are built up by the rational square‐root fuzzy logic model (FLM) and the T‐S fuzzy model, respectively. When the given output probability density function (PDF) is not known, the minimization of output randomness becomes an important control target. First, the unknown fault is estimated by developing the fuzzy fault estimation observer. Then, the fault tolerant controller with the fault compensation is designed so that the output of the descriptor SDNC system remains with the minimum uncertainty after the fault occurs. A simulation example is supplied to prove the proposed scheme. [ABSTRACT FROM AUTHOR]

This study covers the output‐feedback model predictive control (MPC) of nonlinear systems subjected to stochastic disturbances and state chance constraints. The stochastic optimal control problem is solved in a stochastic dynamic programming fashion, and the output‐feedback control is performed with the extended Kalman filter. The information state is summarized as a dynamic Gaussian belief model. Thus, the stochastic Bellman equation is transformed into a deterministic equation using this model. The resulting constrained Bellman equation is solved using the proposed constrained, approximate dynamic programming algorithm. The algorithm is proved to have a Q‐superlinear local convergence rate. Numerical experiments show that the proposed algorithm can attain good control performance and reasonable chance‐constraint satisfaction and is computationally efficient owing to its dynamic programming structure. [ABSTRACT FROM AUTHOR]

We study the stochastic optimal control problem for fully coupled forward‐backward stochastic differential equations (FBSDEs) with jump diffusions. A major technical challenge of such problems arises from the dependence of the (forward) diffusion term on the backward SDE and the presence of jump diffusions. Previously, this class of problems has been solved via only the stochastic maximum principle, which guarantees only the necessary condition of optimality and requires identifying unknown parameters in the corresponding variational inequality. Our paper provides an alternative approach, which constitutes the sufficient condition for optimality. Specifically, the original fully coupled FBSDE control problem (referred to as (P)) is converted into the terminal state‐constrained forward stochastic control problem (referred to as (P′)$$ \left({\mathbf{P}}^{\prime}\right) $$) that includes additional (possibly unbounded) control variables. Then (P′)$$ \left({\mathbf{P}}^{\prime}\right) $$ is solved via the backward reachability analysis, by which the value function of (P′)$$ \left({\mathbf{P}}^{\prime}\right) $$ is expressed as the zero‐level set of the value function for the auxiliary unconstrained (forward) control problem (referred to as (P′′)$$ \left({\mathbf{P}}^{\prime \prime}\right) $$). Unlike (P′$$ \Big({\mathbf{P}}^{\prime } $$), (P′′)$$ \left({\mathbf{P}}^{\prime \prime}\right) $$ is an unconstrained problem, which includes additional control variables as a consequence of the martingale representation theorem. We show that the value function for (P′′)$$ \left({\mathbf{P}}^{\prime \prime}\right) $$ is the unique viscosity solution to the associated integro‐type Hamilton‐Jacobi‐Bellman (HJB) equation. The viscosity solution analysis presented in our paper requires a new technique due to additional control variables in the Hamiltonian maximization and the presence of the nonlocal integral operator in terms of the (singular) Lévy measure. To solve the original problem (P), we reverse our approach. Specifically, we first solve (P′′)$$ \left({\mathbf{P}}^{\prime \prime}\right) $$ to obtain the value function using the verification theorem and the viscosity solution of the HJB equation. Then (P′)$$ \left({\mathbf{P}}^{\prime}\right) $$ is solved by characterizing the zero‐level set of the value function of (P′′)$$ \left({\mathbf{P}}^{\prime \prime}\right) $$, from which the optimal solution of (P) can be constructed. To illustrate the theoretical results of this paper, applications to the linear‐quadratic problem for fully coupled FBSDEs with jumps are also presented. [ABSTRACT FROM AUTHOR]

Owing to the characteristics of time‐delay, fault randomness, uncertainty, nonlinearity and unknown interference in industrial production processes, a stochastic fuzzy predictive fault‐tolerant control algorithm is put forward based on the traditional fault‐tolerant control algorithm. The main idea of this algorithm is to integrate actuator fault into the established Takagi‐Sugeno (T‐S) model to solve actuator fault under a certain probability for the nonlinear industrial processes with time‐varying delays by combining stochastic control theory and relevant theorems. First, the actuator fault under a certain probability is considered as a T‐S model, which can be used as a description of the fault situation in the nonlinear industrial processes. Afterwards, the augmented state space model is established through integrating state deviation and output tracking error. Second, the stochastic fuzzy predictive fault‐tolerant control law can be designed on the basis of the augmented model. Meanwhile, the actuator control mode for different faults is given. If the actuator fault is under a small probability, the control mode is switched to normal control; if the actuator fault is under a large probability, the control mode is switched to fault‐tolerant control. To this end, this control algorithm can reduce energy consumption and raw material consumption. On this basis, the designed control law can be solved by using the given stochastic stability conditions in terms of linear matrix inequality. Finally, the temperature control process of a strongly nonlinear continuous stirred tank reactor is selected as a simulation object to prove the feasibility and effectivity of this algorithm. [ABSTRACT FROM AUTHOR]

We introduce a general framework for Markov decision problems under model uncertainty in a discrete‐time infinite horizon setting. By providing a dynamic programming principle, we obtain a local‐to‐global paradigm, namely solving a local, that is, a one time‐step robust optimization problem leads to an optimizer of the global (i.e., infinite time‐steps) robust stochastic optimal control problem, as well as to a corresponding worst‐case measure. Moreover, we apply this framework to portfolio optimization involving data of the S&P500$S\&P\nobreakspace 500$. We present two different types of ambiguity sets; one is fully data‐driven given by a Wasserstein‐ball around the empirical measure, the second one is described by a parametric set of multivariate normal distributions, where the corresponding uncertainty sets of the parameters are estimated from the data. It turns out that in scenarios where the market is volatile or bearish, the optimal portfolio strategies from the corresponding robust optimization problem outperforms the ones without model uncertainty, showcasing the importance of taking model uncertainty into account. [ABSTRACT FROM AUTHOR]

We formulate and solve a multi‐player stochastic differential game between financial agents who seek to cost‐efficiently liquidate their position in a risky asset in the presence of jointly aggregated transient price impact, along with taking into account a common general price predicting signal. The unique Nash‐equilibrium strategies reveal how each agent's liquidation policy adjusts the predictive trading signal to the aggregated transient price impact induced by all other agents. This unfolds a quantitative relation between trading signals and the order flow in crowded markets. We also formulate and solve the corresponding mean field game in the limit of infinitely many agents. We prove that the equilibrium trading speed and the value function of an agent in the finite N‐player game converges to the corresponding trading speed and value function in the mean field game at rate O(N−2)$O(N^{-2})$. In addition, we prove that the mean field optimal strategy provides an approximate Nash‐equilibrium for the finite‐player game. [ABSTRACT FROM AUTHOR]

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision‐making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value‐ and policy‐based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. We then discuss in detail the application of these RL algorithms in a variety of decision‐making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo‐advising. Our survey concludes by pointing out a few possible future directions for research. [ABSTRACT FROM AUTHOR]

This study applies the theory of stochastic control to an environmental project planning to counteract against the sediment starvation problem in river environments. This can be considered as a time‐average inventory problem to time‐discretely control a continuous‐time system driven by a non‐smooth jump process under habit formation disturbing project implementation. The system is modeled such that the sediment storage dynamics are physically consistent with certain experimental results. Further, the habit formation is modeled as simple linear dynamics and serves as a constraint related to the replenishment amount of the sediment. We show that the time‐average control problem is not necessarily ergodic. Consequently, the effective Hamiltonian may become a non‐constant. Thereafter, ratcheting cases as extreme cases of the irreversible habit formation are considered, owing to them being unique exactly solvable non‐ergodic control problems. The optimality equation associated with a regularized and hence well‐defined control problem is verified. Furthermore, a finite difference scheme is examined against the exactly solvable case and then applied to more complicated cases. [ABSTRACT FROM AUTHOR]

The paper addresses the problem of accelerating predictive control of non‐linear system models augmented with Gaussian processes (GP‐MPC). Due to the non‐linear and stochastic prediction model, predictive control of GP‐based models requires to solve a stochastic optimization problem. Different model simplification methods have to be applied to reformulate this problem to a deterministic, non‐linear optimization task that can be handled by a numerical solver. As these problems are still complex, especially with exact moment calculations, real‐time implementation of GP‐MPC is extremely challenging. The existing solutions accelerate the computations at the solver level by linearizing the non‐linear optimization problem and applying sequential convexification. In contrast, this paper proposes a novel GP‐MPC solution approach that without linearization formulates a series of surrogate quadratic programs (QP‐s) to iteratively obtain the solution of the original non‐linear optimization problem. The first step is embedding the non‐linear mean‐variance dynamics of the GP‐MPC prediction model in a linear parameter‐varying (LPV) structure and rewriting the constraints in parameter‐varying form. By fixing the scheduling trajectory at a known variation (based on previously computed or initial state‐input trajectories), optimization of the input sequence for the remaining varying linear model reduces to a linearly constrained quadratic program. After solving the QP, the non‐linear prediction model is simulated for the new control input sequence and new scheduling trajectories are updated. The procedure is iterated until the convergence of the scheduling, that is, the solution of the QP converges to the solution of the original non‐linear optimization problem. By designing a reference tracking controller for a 4DOF robot arm, we illustrate that the convergence is remarkably fast and the approach is computationally advantageous compared to current solutions. The proposed method enables the application of GP‐MPC algorithms even with exact moment matching on fast dynamical systems and requires only a QP solver. This paper aims to discuss the approach of constrained modified feedback linearization model predictive control (CMFLMPC) for the spacecraft simulator. The simulation and experimental results demonstrate that the proposed hybrid controller has an insignificant calculative cost and facilitates the spacecraft to perform the regulation manoeuvre with sufficient precision in the presence of external torques and actuator saturations. [ABSTRACT FROM AUTHOR]

Summary: This paper studies the performance comparison of periodic and event‐based sampling for a class of linear stochastic systems with multiplicative noise, where the impulse control is adopted. By solving boundary value problems, we obtain the analytic expressions of the mean sampling time and the average state variance under the event‐based sampling. It is shown that the event‐based impulse control has substantially smaller average state variance than the periodic control under the same sampling frequency. Particularly, for the integrator case, the performance ratio of the two sampling methods is given explicitly. By simulation, it is demonstrated that the advantage of event‐based sampling over periodic sampling is most obvious for unstable systems, followed by critical stable systems, and least obvious for stable systems. [ABSTRACT FROM AUTHOR]

This paper deals with the optimal control of grid‐connected Battery Energy Storage Systems (BESSs) operating for energy arbitrage. An important issue is that BESSs degrade over time, according to their use, and thus they are usable only for a limited number of cycles. Therefore, the time horizon of the optimization depends on the actual operation of the BESS. We focus on Li‐ion batteries and use an empirical model to describe battery degradation. The BESS model includes an equivalent circuit for the battery and a simplified model for the power converter. In order to model the energy price variations, we use a linear stochastic model that includes the effect of the time‐of‐the‐day. The problem of maximizing the revenues obtained over the BESS lifetime is formulated as a stochastic optimal control problem with a long, operation‐dependent time horizon. First, we divide this problem into a finite set of subproblems, such that for each one of them, the State of Health (SoH) of the battery is approximately constant. Next, we reformulate approximately every subproblem into the minimization of the ratio of two long‐time average‐cost criteria and use a value‐iteration‐type algorithm to derive the optimal policy. Finally, we present some numerical results and investigate the effects of the energy loss parameters, degradation parameters, and price dynamics on the optimal policy. [ABSTRACT FROM AUTHOR]

The anomalous Zeeman effect made it clear that charged particles like the electron possess a magnetic dipole moment. Classically, this could be understood if the charged particle possesses an eigenrotation, that is, spin. Within Nelson's stochastic mechanics, it was shown that the model of a rotating charged ball is able to reproduce the well‐known spin expectation values. This classically motivated model of intrinsic rotation described in terms of a stochastic process is revisited here within the formalism of stochastic optimal control theory. Quantum Hamilton equations (QHE) for spinning particles are derived, which reduce to their classical counterpart in the zero quantum noise limit. These equations enable the calculation of the common spin expectation values without the use of the wave function. They also offer information on the orientation dynamics of the magnetic moment of charged particles beyond the behavior of the spin averages. [ABSTRACT FROM AUTHOR]

This article investigates the fault estimation and fault tolerant control (FTC) problems for linear stochastic uncertain systems. By introducing the fictitious noise, the fault is augmented as part of the systems state, and then a robust estimator is proposed to simultaneously obtain the state and fault estimation. Based on the estimated information, the active FTC is presented to eliminate the impact of the fault. Finally, a simulation example is conducted to demonstrate the effectiveness of our main method. [ABSTRACT FROM AUTHOR]

In dealer markets, dealers provide prices at which they agree to buy and sell the assets and securities they have in their scope. With ever increasing trading volume, this quoting task has to be done algorithmically in most markets such as foreign exchange (FX) markets or corporate bond markets. Over the last 10 years, many mathematical models have been designed that can be the basis of quoting algorithms in dealer markets. Nevertheless, in most (if not all) models, the dealer is a pure internalizer, setting quotes and waiting for clients. However, on many dealer markets, dealers also have access to an interdealer market or even public trading venues where they can hedge part of their inventory. In this paper, we propose a model taking this possibility into account therefore allowing dealers to externalize part of their risk. The model displays an important feature well known to practitioners that within a certain inventory range, the dealer internalizes the flow by appropriately adjusting the quotes and starts externalizing outside of that range. The larger the franchise, the wider is the inventory range suitable for pure internalization. The model is illustrated numerically with realistic parameters for USDCNH spot market. [ABSTRACT FROM AUTHOR]

Practical industrial process is usually a dynamic process including uncertainty. Stochastic constraints can be used for industrial process modeling, when system sate and/or control input constraints cannot be strictly satisfied. Thus, optimal control of nonlinear systems with stochastic constraints can be available to address practical industrial process problems with integer‐valued control inputs. In general, obtaining an analytical solution of this optimal control problem is very difficult due to the discrete nature of the control inputs and the complexity of stochastic constraints. To obtain a numerical solution, this problem is formulated as a finite dimensional static constrained optimization problem. First, the integer‐valued control input is relaxed into a continuous‐valued control input by imposing a penalty term on the objective function. Convergence results show that the relaxed solution obtained is an integer solution as long as the penalty parameter is sufficiently large. In addition, it should be noted that no any constraint is introduced in the novel relaxation technique, which can effectively avoid introducing any extra extreme point for the original optimal control problem. Next, a novel smooth approximation function is used to construct a subset of feasible region for this optimal control problem. It is proved that the smooth approximation can converge uniformly to the stochastic constraints as the adjusting parameter reduces. Following that, a numerical computation method is proposed for solving the original optimal control problem. Finally, in order to illustrate the effectiveness of the proposed method, an electric vehicle energy management problem is extended by considering some stochastic constraints. The numerical results show that the proposed method is less conservative compared with the existing typical approaches and can obtain a stable and robust performance when considering the initial condition small perturbations. [ABSTRACT FROM AUTHOR]

In this paper, we study partially observed optimal stochastic singular control problems of general Mckean–Vlasov type with correlated noises between the system and the observation. The control variable has two components, the first being absolutely continuous and the second is a bounded variation, nondecreasing continuous on the right with left limits. The dynamic system is governed by Itô‐type controlled stochastic differential equation. The coefficients of the dynamic depend on the state process and of its probability law and the continuous control variable. In terms of a classical convex variational techniques, we establish a set of necessary conditions of optimal singular control in the form of maximum principle. Our main result is proved by applying Girsanov's theorem and the derivatives with respect to probability law in Lions' sense. To illustrate our theoretical result, we study partially observed linear‐quadratic singular control problem of McKean–Vlasov type. [ABSTRACT FROM AUTHOR]

In this article, we establish a cluster‐based gradient method (CGM) by combining K‐means clustering algorithm and stochastic gradient descent (SGD). By clustering the sampling solutions, we use the cluster centroids to represent sampling data and give an estimate to the full gradient. It is well known that the full gradient descent (FGD) can provide the steepest descent direction for finding a local minimum of the desired stochastic control problems. However, the huge computational requirements, which is proportional to the product of sample size and the numerical cost for each sample, often makes FGD cost prohibitive for large scale optimization problems. To reduce the formidable cost and the risks of getting stuck in a local minimum, SGD is proposed and can be regarded as a stochastic approximation of FGD. This, however, would result in a slow convergence due to the incorrect approximation of the iteratively update parameters. Our study shows that CGM could provide a good stochastic approximation to the full gradient with small sample size while has a more stable and faster convergence than SGD. To verify our algorithm, a stochastic elliptic control problem is selected and tested. The numerical results validate our method as a reliable gradient descent method with great potential applications in optimization problems. [ABSTRACT FROM AUTHOR]

This article focuses on a stochastic recursive optimal control problem involving impulse control under partially observed information and obtains the related maximum principle. Unlike the classical literature, both the regular control (considering the nonconvexity of the domain) and the impulse control (where the domain is convex) are considered in the framework. In virtue of two types of variational methods, spike variation and convex variation, the Pontryagin maximum principle is established. In addition, to demonstrate the validity of the results, this article investigates a differential game problem as an application. [ABSTRACT FROM AUTHOR]

In this paper, the adaptive multi‐dimensional Taylor network (MTN) control problem is investigated for nonlinear stochastic systems with full state time‐varying constraints and the finite‐time output constraint. By combining the MTN‐based approximation method and the adaptive backstepping control method, a novel adaptive MTN control scheme is provided by constructing the time‐varying barrier Lyapunov function (TVBLF). To implement the finite‐time output constraint, the finite‐time performance function (FTPF) is introduced in the control scheme. The proposed scheme can ensure that the tracking error finally converges to a small neighborhood of the origin in the finite‐time and all signals in the closed‐loop system are semi‐globally uniformly ultimately bounded (SGUUB) in probability. Finally, two simulation examples are presented to show the effectiveness of the provided control scheme. [ABSTRACT FROM AUTHOR]

In this paper, we explore the dynamics of working hours and wages in a model economy where a firm and its workforce are linked to each other by an implicit contract. Specifically, we develop a deterministic and a stochastic framework in which the firm sets its level of labor utilization by considering that workers' earnings tend to adjust in the direction of a fixed level. Without any uncertainty about firm's profitability, we show that the existence and the properties of stationary solutions rely on the factors that usually determine the enforceability of contracts and we demonstrate that wages move countercyclically towards the allocation preferred by the firm. Moreover, we show that adding uncertainty does not overturn the countercyclical pattern of wages but is helpful in explaining their dynamic behavior in response to demand shocks as well as their typical stickiness observed at the macrolevel. [ABSTRACT FROM AUTHOR]

Environmental management optimizing a long‐run objective is an ergodic control problem whose resolution can be achieved by solving an associated non‐local Hamilton–Jacobi–Bellman (HJB) equation having an effective Hamiltonian. Focusing on sediment storage management as a modern engineering problem, we formulate, analyze, and compute a new ergodic control problem under discrete observations: a simple but non‐trivial mathematical problem. We give optimality and comparison results of the corresponding HJB equation having unique non‐smoothness and discontinuity. To numerically compute HJB equations, we propose a new fast‐sweep method resorting to neither pseudo‐time integration nor vanishing discount. The optimal policy and the effective Hamiltonian are then computed simultaneously. Convergence rate of numerical solutions is computationally analyzed. An advanced robust control counterpart where the dynamics involve uncertainties is also numerically considered. [ABSTRACT FROM AUTHOR]

This paper is concerned with a special case of stochastic distributed optimal control, where the objective is to design a structurally constrained controller for a system subject to state and input power constraints. The structural constraints are induced by the directed communication between local controllers over a strongly connected graph. Based on the information structure present, that is, who knows what and when, we provide a control synthesis with the optimal control law consisting of two parts: one that is based on the common information between the subsystems and one that uses more localized information. The developed method is applicable to an arbitrary number of physically interconnected subsystems. [ABSTRACT FROM AUTHOR]

The original pharmacokinetics (PK) two‐compartment model illustrates the change law of the drug in the central and peripheral chambers after administration, respectively. Although this deterministic model has derived many practical conclusions, it is based on simplifications and neglects noise, which is inherent to pharmacological processes. The actual pharmacological processes are always influenced by factors that cannot be entirely understood or modeled explicitly. Modeling without considering these phenomena may impact the accuracy and the related conclusions. A stochastic type of model can capture such noise. We proposed a novel two‐compartment PK model concerning drug administration through intravenous route by combining optimal control theory and stochastic analysis. The selection of the objective function was based on the goal of obtaining the best possible therapeutic effect with the least possible drug dosage. Firstly, we extended the original PK two‐compartment model based on optimal control and proved the existence and uniqueness of the switch in the control. Moreover, considering the possible uncertain factors, we added disturbances to the distance between the drug concentration and equilibrium point of the dynamic system and extended the model to a stochastic differential equation model. Qualitative and quantitative analyses showed that optimal controls were bang‐bang, that is, alternating the drug dosages at a full dose with rest‐periods in‐between. Our analysis provided a schedule for optimal dosage and timing. The solutions of the model provided estimates of the drug concentration at any given time. Finally, we simulated the model using R and showed that the numerical method is stable. [ABSTRACT FROM AUTHOR]

This article studies the maximum principle for stochastic optimal control problems of forward‐backward stochastic difference systems (FBSΔSs) where the uncertainty is modeled by a discrete time, finite state process, rather than white noises. Two distinct forms of FBSΔSs are investigated. The first one is described by a partially coupled forward‐backward stochastic difference equation (FBSΔE) and the second one is described by a fully coupled FBSΔE. We deduce the adjoint difference equation by adopting an appropriate representation of the product rule and a proper formulation of the backward stochastic difference equation (BSΔE). Finally, the maximum principle for this optimal control problem with the convex control domain is established. [ABSTRACT FROM AUTHOR]