Search

An Automatic Voltage Regulator (AVR) system utilized to keep the terminal voltage of a synchronous generator at the desired level has received much attention among researchers. Designing an efficient and robust control scheme for the AVR system to maintain a specified voltage level is an important research area. From the control area perspective, reinforcement learning, an adaptive optimal control method, has received increasing attention in reference tracking problems. This article discusses a reinforcement learning approach to an AVR system and its experimental validation. A deep deterministic policy gradient (DDPG) agent working in continuous-time is designed offline to improve dynamic system characteristics of the AVR system besides its robustness against load disturbance, parameter uncertainties, and reference change. In the DDPG agent design process, the limits of the produced control signal are taken into account to perform a feasible simulation similar to a real-time application. The performance of the proposed learning-based controller is analyzed in three categories: transient and steady-state responses, stability analysis, and robustness analysis against parameter uncertainties, reference change, and load disturbance. A comparison with recently published papers employing Fuzzy-PID, PID-F, PI λ DND 2 N 2 , PIDD 2 ,and PID controllers in which various heuristic optimization algorithms were employed to optimally tune the controller parameters is made. Furthermore, to demonstrate that the behavior of the learning-based approach provides a stable and satisfactory performance, it is analyzed for a real synchronous generator connected to a 230 kV network using Matlab/Simulink environment. The results presented in this paper indicate that the proposed learning-based controller ensures the stability of the AVR system, significantly improves the regulating performance, and most impressively, is robust against parameter uncertainties, reference change, and load disturbance. [ABSTRACT FROM AUTHOR]

As a heuristic algorithm to solve the nonlinear optimal control problem, adaptive dynamic programming is commonly constructed from one-step temporal difference learning. Eligibility trace can effectively speed up controller learning by considering the multi-step information in reinforcement learning. However, eligibility trace will bring in additional computational consumption and increase the learning burden. This paper attempts to take advantage of eligibility trace and avoids the high learning consumption at the same time. Therefore, an event-based neural dynamic programming (λ) [ENDP(λ)] algorithm via the actor–critic framework is constructed to solve the near-optimal control problem of unknown discrete-time systems. First, the modified forward view with eligibility trace is derived, which is suitable for engineering practice. Second, based on the event-triggered mechanism, ENDP(λ) is designed to relieve the pressure of communication consumption. Then, the event-based system is proven to ensure the input-to-state stability under a suitable triggering condition. Moreover, three neural networks are given to approximate the one-step cost function, the n -step cost function, and the control law, respectively. Finally, two typical experimental simulation examples are presented to verify the effectiveness of the ENDP(λ) algorithm. [ABSTRACT FROM AUTHOR]

This paper evaluates adaptive Q-learning (AQL) and single-partition adaptive Q-learning (SPAQL), two algorithms for efficient model-free episodic reinforcement learning (RL), in two classical control problems (Pendulum and CartPole). AQL adaptively partitions the state–action space of a Markov decision process (MDP), while learning the control policy, i.e. , the mapping from states to actions. The main difference between AQL and SPAQL is that the latter learns time-invariant policies, where the mapping from states to actions does not depend explicitly on the time step. This paper also proposes the SPAQL with terminal state (SPAQL-TS), an improved version of SPAQL tailored for the design of regulators for control problems. The time-invariant policies are shown to result in a better performance than the time-variant ones in both problems studied. These algorithms are particularly fitted to RL problems where the action space is finite, as is the case with the CartPole problem. SPAQL-TS solves the OpenAI GymCartPole problem, while also displaying a higher sample efficiency than trust region policy optimization (TRPO), a standard RL algorithm for solving control tasks. Moreover, the policies learned by SPAQL are interpretable, while TRPO policies are typically encoded as neural networks, and therefore hard to interpret. Yielding interpretable policies while being sample-efficient are the major advantages of SPAQL. The code for the experiments is available at https://github.com/jaraujo98/SinglePartitionAdaptiveQLearning. • Two recent Q-learning algorithms, AQL and SPAQL, are evaluated on two classical control benchmarks. • Based on insights from control theory, a new algorithm, SPAQL-TS, is introduced. • It is shown that both SPAQL and SPAQL-TS outperform TRPO in the Cartpole problem. [ABSTRACT FROM AUTHOR]

The paper introduces an interactive machine learning mechanism to process the measurements of an uncertain, nonlinear dynamic process and hence advise an actuation strategy in real-time. For concept demonstration, a trajectory-following optimization problem of a Kinova robotic arm is solved using an integral reinforcement learning approach with guaranteed stability for slowly varying dynamics. The solution is implemented using a model-free value iteration process to solve the integral temporal difference equations of the problem. The performance of the proposed technique is benchmarked against that of another model-free high-order approach and is validated for dynamic payload and disturbances. Unlike its benchmark, the proposed adaptive strategy is capable of handling extreme process variations. This is experimentally demonstrated by introducing static and time-varying payloads close to the rated maximum payload capacity of the manipulator arm. The comparison algorithm exhibited up to a seven-fold percent overshoot compared to the proposed integral reinforcement learning solution. The robustness of the algorithm is further validated by disturbing the real-time adapted strategy gains with a white noise of a standard deviation as high as 5%. [ABSTRACT FROM AUTHOR]