Search

This paper studies continuous-time reinforcement learning for controlled jump-diffusion models by featuring the q-function (the continuous-time counterpart of Q-function) and the q-learning algorithms under the Tsallis entropy regularization. Contrary to the conventional Shannon entropy, the general form of Tsallis entropy renders the optimal policy not necessary a Gibbs measure, where some Lagrange multiplier and KKT multiplier naturally arise from certain constraints to ensure the learnt policy to be a probability distribution. As a consequence,the relationship between the optimal policy and the q-function also involves the Lagrange multiplier. In response, we establish the martingale characterization of the q-function under Tsallis entropy and devise two q-learning algorithms depending on whether the Lagrange multiplier can be derived explicitly or not. In the latter case, we need to consider different parameterizations of the q-function and the policy and update them alternatively. Finally, we examine two financial applications, namely an optimal portfolio liquidation problem and a non-LQ control problem. It is interesting to see therein that the optimal policies under the Tsallis entropy regularization can be characterized explicitly, which are distributions concentrate on some compact support. The satisfactory performance of our q-learning algorithm is illustrated in both examples.

This paper studies an infinite horizon optimal tracking portfolio problem using capital injection in incomplete market models. We consider the benchmark process modelled by a geometric Brownian motion with zero drift driven by some unhedgeable risk. The relaxed tracking formulation is adopted where the portfolio value compensated by the injected capital needs to outperform the benchmark process at any time, and the goal is to minimize the cost of the discounted total capital injection. In the first part, we solve the stochastic control problem when the market model is known, for which the equivalent auxiliary control problem with reflections and the associated HJB equation with a Neumann boundary condition are studied. In the second part, the market model is assumed to be unknown, for which we consider the exploratory formulation of the control problem with entropy regularizer and develop the continuous-time q-learning algorithm for the stochastic control problem with state reflections. In an illustrative example, we show the satisfactory performance of the q-learning algorithm., Comment: Optimal tracking portfolio, capital injection, incomplete market, stochastic control with reflection, continuous-time reinforcement learning, q-learning

This paper studies a Merton's optimal consumption problem in an extended formulation incorporating the tracking of a benchmark process described by a geometric Brownian motion. We consider a relaxed tracking formulation such that the wealth process compensated by a fictitious capital injection outperforms the benchmark at all times. The fund manager aims to maximize the expected utility of consumption deducted by the cost of the capital injection, where the latter term can also be regarded as the expected largest shortfall of the wealth with reference to the benchmark. By introducing an auxiliary state process with reflection, we formulate and tackle an equivalent stochastic control problem by means of the dual transform and probabilistic representation, where the dual PDE can be solved explicitly. On the strength of the closed-form results, we can derive and verify the optimal feedback control for the primal control problem, allowing us to discuss some new and interesting financial implications induced by the additional risk-taking from the capital injection and the goal of tracking., Comment: Keywords: Benchmark tracking, capital injection, expected largest shortfall, consumption and portfolio choice, Neumann boundary condition

This paper studies stochastic control problems motivated by optimal consumption with wealth benchmark tracking. The benchmark process is modeled by a combination of a geometric Brownian motion and a running maximum process, indicating its increasing trend in the long run. We consider a relaxed tracking formulation such that the wealth compensated by the injected capital always dominates the benchmark process. The stochastic control problem is to maximize the expected utility of consumption deducted by the cost of the capital injection under the dynamic floor constraint. By introducing two auxiliary state processes with reflections, an equivalent auxiliary control problem is formulated and studied, which leads to the HJB equation with two Neumann boundary conditions. We establish the existence of a unique classical solution to the dual PDE using some novel probabilistic representations involving the local time of some dual processes together with a tailor-made decomposition-homogenization technique. The proof of the verification theorem on the optimal feedback control can be carried out by some stochastic flow analysis and technical estimations of the optimal control., Comment: Keywords: Relaxed benchmark tracking, optimal consumption, Neumann boundary conditions, probabilistic representation, reflected diffusion process