Search

Abstract: Due to its simplicity, the completion-of-squares technique is quite popular in linear optimal control. However, this simple technique is limited to linear quadratic Gaussian systems. In this note, by interpreting the completion-of-squares from a new angle, we extend this technique to performance optimization of general Markov systems with the long run average criterion, leading to a new approach to performance optimization based on direct comparisons of the performance of two policies. [Copyright &y& Elsevier]

Abstract: In this paper, we study the th-bias optimality problem for finite continuous-time Markov decision processes (MDPs) with a multichain structure. We first provide th-bias difference formulas for two policies and present some interesting characterizations of an th-bias optimal policy by using these difference formulas. Then, we prove the existence of an th-bias optimal policy by using th-bias optimal policy iteration algorithms, and show that such an th-bias optimal policy can be obtained in a finite number of policy iterations. [Copyright &y& Elsevier]

Abstract: The solution of Markov Decision Processes (MDPs) often relies on special properties of the processes. For two-level MDPs, the difference in the rates of state changes of the upper and lower levels has led to limiting or approximate solutions of such problems. In this paper, we solve a two-level MDP without making any assumption on the rates of state changes of the two levels. We first show that such a two-level MDP is a non-standard one where the optimal actions of different states can be related to each other. Then we give assumptions (conditions) under which such a specially constrained MDP can be solved by policy iteration. We further show that the computational effort can be reduced by decomposing the MDP. A two-level MDP with M upper-level states can be decomposed into one MDP for the upper level and M to MDPs for the lower level, depending on the structure of the two-level MDP. The upper-level MDP is solved by time aggregation, a technique introduced in a recent paper [Cao, X.-R., Ren, Z. Y., Bhatnagar, S., Fu, M., & Marcus, S. (2002). A time aggregation approach to Markov decision processes. Automatica, 38(6), 929–943.], and the lower-level MDPs are solved by embedded Markov chains. [Copyright &y& Elsevier]

We propose a unified framework to Markov decision problems and performance sensitivity analysis for multichain Markov processes with both discounted and average-cost performance criteria. With the fundamental concept of performance potentials, we derive both performance-gradient and performance-difference formulas, which play the central role in performance optimization. The standard policy iteration algorithms for both discounted- and average-reward MDPs can be established using the performance-difference formulas in a simple and intuitive way; and the performance-gradient formulas together with stochastic approximation may lead to new optimization schemes. This sensitivity-based point of view of performance optimization provides some insights that link perturbation analysis, Markov decision processes, and reinforcement learning together. The research is an extension of the previous work on ergodic Markov chains (Cao, Automatica 36 (2000) 771). [Copyright &y& Elsevier]

We propose a time aggregation approach for the solution of infinite horizon average cost Markov decision processes via policy iteration. In this approach, policy update is only carried out when the process visits a subset of the state space. As in state aggregation, this approach leads to a reduced state space, which may lead to a substantial reduction in computational and storage requirements, especially for problems with certain structural properties. However, in contrast to state aggregation, which generally results in an approximate model due to the loss of Markov property, time aggregation suffers no loss of accuracy, because the Markov property is preserved. Single sample path-based estimation algorithms are developed that allow the time aggregation approach to be implemented on-line for practical systems. Some numerical and simulation examples are presented to illustrate the ideas and potential computational savings. [Copyright &y& Elsevier]

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s , S) policy in inventory control, the well-known c μ -rule and the recently discovered c / μ -rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results. [ABSTRACT FROM AUTHOR]

Nowadays stocks and futures are traded in electronic order-driven markets, in which orders to buy and sell are centralized in a Limit Order Book (LOB), and LOB provides more information about the stocks than their prices, such as the price dynamics and the prediction probability of the next move. In this paper, we investigate how and why Limit Order Book can improve the trading decision making. We propose a conditional binomial model for a stock price with conditional upward probability based on observations. We formulate the stock portfolio management as an optimization problem with observations, and optimal schemes are given for single-stock and multi-stock portfolios with observations. Compared with the strategies that do not consider the observations, we show that the final expected wealth achieves significant improvements. Finally, two real-data-based examples are conducted to demonstrate the efficiency of our schemes. Our work points to a new direction in improving the final wealth in portfolio management; and it illustrates the advantages of event-based optimization approach, which was successfully applied in discrete event dynamics systems. [ABSTRACT FROM AUTHOR]

Abstract: It is well known that stochastic control systems can be viewed as Markov decision processes (MDPs) with continuous state spaces. In this paper, we propose to apply the policy iteration approach in MDPs to the optimal control problem of stochastic systems. We first provide an optimality equation based on performance potentials and develop a policy iteration procedure. Then we apply policy iteration to the jump linear quadratic problem and obtain the coupled Riccati equations for their optimal solutions. The approach is applicable to linear as well as nonlinear systems and can be implemented on-line on real world systems without identifying all the system structure and parameters. [Copyright &y& Elsevier]