Search

Cao's work shows that, by defining an α-dependent equivalent infinitesimal generator Aα, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and Aα-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via Aα and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neurodynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

A modular approach of the estimation-based design in adaptive linear control systems has been extended to the adaptive robust control of strict-feedback stochastic nonlinear systems with additive standard Wiener noises and constant unknown parameters. By using Ito’s differentiation rule, nonlinear damping and adaptive Backstepping procedure, the input-to-state stable controller of global stabilization in probability is developed, which guarantees that system states are bounded and the system has a robust stabilization. According to Swapping technique, we develop two filters and convert dynamic parametric models into static ones to which the gradient update law is designed. Transient performance of the system is estimated by the norm of error. Results of simulation show the effectiveness of the control algorithms. The modular design, which has a concise hierarchy, is more flexible and versatile than a Lyapunov-based algorithm.

The optimization problems of Markov control processes (MCPs) with exact knowledge of system parameters, in the form of transition probabilities or infinitesimal transition rates, can be solved by using the concept of Markov performance potential which plays an important role in the sensitivity analysis of MCPs. In this paper, by using an equivalent infinitesimal generator, we first introduce a definition of discounted Poisson equations for semi-Markov control processes (SMCPs), which is similar to that for MCPs, and the performance potentials of SMCPs are defined as solution of the equation. Some related optimization techniques based on performance potentials for MCPs may be extended to the optimization of SMCPs if the system parameters are known with certainty. Unfortunately, exact values of the distributions of the sojourn times at some states or the transition probabilities of the embedded Markov chain for a large-scale SMCP are generally difficult or impossible to obtain, which leads to the uncertainty of the semi-Markov kernel, and thereby to the uncertainty of equivalent infinitesimal transition rates. Similar to the optimization of uncertain MCPs, a potential-based policy iteration method is proposed in this work to search for the optimal robust control policy for SMCPs with uncertain infinitesimal transition rates that are represented as compact sets. In addition, convergence of the algorithm is discussed.

This paper is concerned with exponential stability of a class of single-tool cutting systems, which is described as a Markovian jump system with state time-varying delays. A sufficient condition for the exponential stability in mean-square is proposed based on the Lyapunov functional method. We also obtain the influence of delay parameters on exponential decreasing rate of the states. A numerical example is provided to demonstrate the effectiveness of the proposed approach.

Semi-Markov control problems with unknown kernel are considered, a reinforcement learning based online adaptive optimization algorithm is proposed. First an event-driven stochastic switching model is introduced to formulate the semi-Markov control problems. Then by utilizing the features of event-driven policy an optimization algorithm that combines policy gradient estimation and stochastic approximation is derived. This algorithm can converge to global optimization without the explicit knowledge of the semi-Markov kernel. Moreover, this algorithm does not require the computation of performance potentials or other related quantities (e.g. Q-factors) and therefore saves computational cost significantly. Simulation results demonstrate the effectiveness of the proposed algorithm.

Motivated by optimization of network communication systems, this paper presents an event-driven semi-Markov switching state-space control process with hierarchical dynamic architectures. First, the semi-Markov kernel of the switching control process is constructed, and the sensitivity formula for performance derivatives under average criterion is derived. Then, an online optimization algorithm that combines policy gradient estimation and stochastic approximation is proposed. This analytic model is with constructional flexibility and scalability, and the proposed optimization algorithm is adaptive and with less computational cost. Finally, as an illustrative example, the load balancing problem in a streaming media server cluster is formulated and addressed.

For a class of hierarchical unstructured P2P systems, a Markov Switching-Space model is introduced to describe their behavior of dynamic grouping. It is also formulated as a optimization problem based on Markov Decision Processes. And parameterized gradient algorithms are provided to optimize the performance, where two parameter reductions are mentioned. Following them are the corresponding simulations and discussions.

Recently, gradient based methods have received much attention to optimize some dynamic systems with hidden information, such as routing problems of robotic systems. In this paper, we presented a process - continuous time hidden Markov decision process (CTHMDP), which can be used to model the robotic systems. For this process, the problem of policy gradient estimation is studied. Firstly, an approximation formula to the gradient is presented, then by using the uniformization method, we introduce an algorithm, which can be considered as an extension of gradient of partially observable Markov decision process (GPOMDP) algorithm to the continue time model. Finally, the convergence and error bound of the algorithm are considered.

This paper describes the synthesis of robust and non-fragile H/sub /spl infin// state feedback controllers for a class of uncertain hybrid linear systems with time delay, Markovian jumping parameters and state multiplicative noise. Under the assumptions of complete access to the norm-bounds of the system uncertainties and controller gain variations, sufficient conditions on the existence of a robust stochastic stability and r-disturbance attenuation H/sub /spl infin// property are presented. Also, using some change of variables and Schur complements, the obtained sufficient conditions can be rewritten as a set of coupled linear matrix inequalities form, through which a robust and non-fragile controller can be constructed easily.

This paper develops an adaptive tracking control law for a class of uncertain nonlinear systems with Markovian jumping parameters. Sufficient conditions for characterizing tracking controllers of the systems are provided, and the adaptive tracking control law for the system uncertainties is also presented. Finally a numerical example is presented to show the performance of the adaptive tracking control law.

A stochastic process is a fundamental research model for discrete event dynamic systems. In this paper, we studied the problems of sensitivity analysis of the steady-state performance for a class of semi-Markov processes by using the potential theory. We gave the sensitivity formulas of the steady-state performance expressed by the potentials of systems. These formulas can be directly used in studying the problems of controlling and optimization of systems.