Search

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles (UAVs), particularly quadrotors, have seen a surge in applications ranging from surveillance to delivery services. However, autonomous control remains a challenge due to the complexity of their dynamics and the need for real-time responsiveness. This paper addresses the significant problem of autonomous quadrotor control by introducing a novel learning method that integrates a deep neural network with advanced control techniques. Novelty of the work is distribution of different learning tasks in different layers of learning and combining the total learning system in a applicable UAV. The purpose is to enhance the quadrotor's ability to perform complex maneuvers autonomously, which is crucial for expanding its application scope. The proposed method begins with the acquisition of direction control using a fractional order proportional–integral–derivative (FOPID) controller. The learning parameters are refined through a fuzzy method, ensuring adaptability and precision. For the optimization of the FOPID parameters, we apply two stochastic algorithms: Genetic Algorithm and Particle Swarm Optimization, comparing their efficacy. Our approach successfully teaches the quadrotor to generate and control a flight path autonomously. The second stage of learning involves proximal policy optimization, enabling the execution of intricate maneuvers. The experimental setup includes a quadrotor on a test stand with three degrees of freedom, where we validate the method through rigorous simulation and physical tests. The results demonstrate that our method not only teaches the quadrotor complex maneuvers but also ensures robust and precise execution, marking a significant advancement in UAV control systems. The implications of this research extend to improving UAV reliability for critical applications, potentially transforming the landscape of autonomous aerial operations. [ABSTRACT FROM AUTHOR]

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses. [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

This paper addresses a crucial issue in the airline industry by tackling a dynamic pricing problem in the presence of patient customers, a scenario that has gained significance due to the revenue loss of airlines caused by customers' non-myopic decision-making. To effectively capture this non-myopic characteristic, we propose a Markov decision process (MDP) including a history of offered prices as a state variable. In contrast to previous studies, distributions of customers' properties are assumed to be unknown in advance for a more realistic representation of real-world scenarios. To deal with the new challenges of the problem, we propose utilizing a specific learning framework (i.e., deep exploration-based RL) that is unexplored in this domain. The numerical experiments demonstrate that its performance can be improved on the MDP we designed and show that it outperforms the benchmark algorithm. The structures of pricing policies generated from the bootstrapped deep Q-network algorithm imply that airlines should offer high and low prices alternately from the beginning of the sales period rather than increasing prices as time goes on. We also ascertain that more frequent consecutive high-priced periods can increase airlines' revenue in environments with higher customer patience levels. [ABSTRACT FROM AUTHOR]

Maintenance planning of networked multi-asset systems is a complex problem due to the inherent individual and collective asset constraints and dynamics as well as the size of the system and interdependencies among assets. Although multi-asset systems have been studied numerous times in the past decades, maintenance planning implications of the system's network characteristics have been barely analysed. Likewise, solutions that consider the network perspective suffer from scalability issues as a network-wide observability is assumed. This paper proposes a network maintenance planning approach based on the decomposition of the multi-asset network into fixed-size localised subnetworks. The overall network maintenance plan is produced by aggregating the subnetwork maintenance plans, which are computed independently via a multi-agent deep reinforcement learning (MARL) algorithm. The results are evaluated against a network-wide approach as well as the commonly-used individual approach. The paper also introduces a systematic approach to integrate the MARL resulting policy in a multi-asset agent-based model. Simulation results of several random asset networks and a large nationwide network infrastructure show that, although a network-wide approach outperforms, on average, other approaches considered, the localised subnetworks approach, provides an acceptable alternative in networks with small-world properties, without the need of a network-wide view. • A Network-wide maintenance approach outperforms individual approaches in networked assets with the topologies analysed. • In analysed networks with small-world properties, the Localised approach shows performance close to the network-wide approach. • A systematic integration of MARL and multi-asset agent-based modelling enables evaluation of maintenance approaches. [ABSTRACT FROM AUTHOR]

In recent years, consortium blockchains have attracted considerable interest from the business communities and academia. To satisfy specific application requirements, appropriately configuring or adjusting the parameters is essential but challenging for developers when building the consortium blockchain. In this paper, the parameter configuration and adjustment for the consortium blockchain is transformed as a multi-objective optimization problem, and a novel explainable deep reinforcement learning (DRL) algorithm is proposed to solve the problem. On considering that existing DRL algorithms cannot be directly used in the consortium blockchain as they suffer from lacking of explainability, a causal model for configuring and adjusting the consortium blockchain's parameters is proposed and integrated into the DRL algorithm. The causal model can be used to derive causal explanations of the DRL algorithm to increase its trustworthiness. Furthermore, the causal model-based DRL (C-DRL) algorithm can perform causal inference before taking action to eliminate unreasonable exploration and improve the DRL algorithm's performance. The experimental results demonstrate the proposed algorithm provides the consortium blockchain with adaptive parameter configuration and adjustment to achieve sustainable high performance and security. Furthermore, the proposed algorithm increases the convergence speed of the vanilla algorithm by 49.3% and is more trustworthy than the vanilla DRL algorithm. • To achieve sustainable high performance and security, we transform the parameter configuration and adjustment for the consortium blockchain as a multi-objective optimization problem and provide a formal representation of the problem. • We propose a causal model of consortium blockchain's parameters using the structural causal model (SCM) formalism. • We propose a causal model-based DRL (C-DRL) algorithm to solve the multi-objective optimization problem. The C-DRL algorithm is equipped with the causal model of consortium blockchain's parameters. • The agent can use the causal model to derive the causal explanation for its action to provide trustworthiness. Furthermore, the agent with the causal model can utilize causal inference to eliminate unreasonable exploration and improve the convergence speed of the current DRL algorithm. • Extensive experiments are conducted to validate our proposal. The simulation results show that the agent running the C-DRL algorithm has more reasonable behavior than that running the model-free DRL algorithm. Moreover, the C-DRL algorithm converges faster than the model-free DRL algorithm. [ABSTRACT FROM AUTHOR]

Industrial cyber–physical systems (ICPS) are emerging platforms for various industrial applications. For instance, remote healthcare monitoring, real-time healthcare data generation, and many other applications have been integrated into the ICPS platform. These healthcare applications encompass workflow tasks, such as processing within hospitals, laboratory tests, and insurance companies for patient payments, which necessitate a sequential flow. The external wireless, fog, and cloud services within ICPS face security issues that impact end-users' healthcare applications. Blockchain technology offers an optimal solution for ICPS-enabled applications. However, blockchain technology for the ICPS platform is still vulnerable to cyberattacks, while microservices are essential for executing applications. This paper introduces the novel "Pattern-Proof Malware Validation" (PoPMV) algorithm designed for blockchain in ICPS. It exploits a deep learning model (LSTM) with reinforcement learning techniques to receive feedback and rewards in real-time. The primary objective is to mitigate security vulnerabilities, enhance processing speed, identify both familiar and unfamiliar attacks, and optimize the functionality of ICPS. Simulations demonstrate the superiority of the proposed approach compared to current blockchain frameworks, showcasing dynamic allocation of microservices and improved security with comprehensive attack detection by 30%. [ABSTRACT FROM AUTHOR]

Astronomical seeing is the primary indicator of the observation quality of large-scale optical telescopes. Forecasting astronomical seeing is an essential and challenging task in the operation of large optical telescopes, as it can help observers more efficiently arrange observation tasks as well as providing hints for mechanism optimization of optical telescopes. In this paper, we propose an efficient data-driven astronomical seeing forecasting method by leveraging the environment monitoring data from multiple sensors deployed with optical telescopes. The basic idea is to use a large number of random yet fixed convolution kernels to extract features from the original data, and then train a simple tree-based predictor based on these features. In particular, the raw monitoring data is transformed by differential operations, enabling a more explicit representation of spatial and temporal features; then an elaborate convolution mechanism, including the convolution kernel and the pooling operator, is designed to extract a variety set of features of the input time series, which serve as the foundation for constructing the time series forecasting model. In contrast to existing deep learning models, there is no need to calculate and back-propagate the gradients during the training procedure, making the proposed method much more efficient. We collect real-world monitoring data from Large Sky Area Multi-Object Fiber Spectroscopy Telescope (LAMOST), one of the world's largest optical reflecting telescopes, and conduct extensive empirical evaluations. The experimental results demonstrate that the proposed method can achieve state-of-the-art prediction accuracy, i.e., 0.1125 in MAE, 0.0252 in MSE, and 0.0322 in MAPE, outperforming most deep-learning-based forecasting models. Moreover, it surpasses both machine learning and deep learning approaches in terms of training efficiency with a significant margin. [Display omitted] • The task of forecasting astronomical seeing for optical telescopes is investigated. • A variety of real-world monitoring data of LAMOST is collected. • A new multivariate time series forecasting method — SeeROCKET is proposed. • SeeROCKET achieves very promising performance both in accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Pairs trading is a popular classic neutral trading strategy in financial market. Deep reinforcement learning (DRL) has been widely used to improve the performance of this strategy. However, most works primarily focused on setting trading signals, but ignored selecting appropriate trading pairs. In this paper, a novel two-level reinforcement learning framework is proposed, where both pair selection and trading thresholds setting are involved. For pair selection, an Extended Option-Critic (EOC) method is utilized, which allows the agent to select trading pair on non-fixed length of time intervals. For trading thresholds setting, a three-agent Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is used for setting the opening and stop-loss thresholds as well as decide whether to trade. The simulation results in the Chinese futures market demonstrate that our proposed method achieves higher returns compared to traditional methods and popular reinforcement learning approaches. • A novel two-level reinforcement learning framework for future trading is proposed. • Extended Option-Critic method for non-fixed length of time intervals. • MADDPG method with the cooperation of three agents for trading. [ABSTRACT FROM AUTHOR]

In recent research, problems with biased datasets or domain shift have presented challenges to the practical applications of deep learning methods. In this paper, we propose a simple method using adversarial learning combined with contrastive learning and domain adaptation to solve the domain-shift problem. Because domain shift is caused by differences in the distribution of data across domains, different approaches have been proposed to resolve this by rendering the data distributions closer through adversarial training. Contrastive learning is a method of acquiring valid feature representations for downstream tasks. By combining these approaches, we aim to achieve better domain adaptation. The proposed method is simple and intuitive. By introducing a domain discriminator into SimCLR, which is a typical contrastive learning model, and training it in an adversarial manner, the feature vectors of the source and target domains are rendered closer to acquire domain-invariant features. The proposed approach facilitates high-performance pretraining without labels and demonstrates a significant improvement in accuracy in comparison to standard benchmark methods, including conventional supervised models and SimCLR. • We investigated an unsupervised domain adaptation task. • Combining adversarial and contrastive learning can address the domain-shift issue. • The proposed method is practical and implementation is very simple. [ABSTRACT FROM AUTHOR]

Home automation is seen as a potential pillar of the smart city revolution that combines smart mobility, lifestyle and ecosystem governed by intelligent sensors connected to the internet. Households can save money and be more comfortable with automated appliances. The cost of electricity and user comfort are fundamentally contradictory, so they can be presented as a dynamic multi-objective optimization problem with fluctuating priorities for the customer to use various devices at different times. For this reason, this paper proposes an advanced Intelligent Home Energy Management (IHEM) approach based on reinforcement learning to achieve home demand response (DR) efficiency. The optimal formulation of the one-hour-ahead energy consumption scheduling problem is considered a Markov Decision Process (MDP) with discrete time steps. An efficient Neural Network (NN)-based approach with a Q-learning algorithm is developed to address this problem, enabling the IHEM system to achieve better cost-effective scheduling performance. The accurate data of electricity price and energy supplied by the Photovoltaic (PV) system are analyzed in sliding periods by machine learning for uncertainty prediction. Using the newly developed approach, which has the dual objective of minimizing the electricity bill, it is possible to obtain scheduling decisions for appliances and energy storage. The results show that the proposed optimization method reduces the monthly electricity costs by 20% compared to the Integer Linear Programming (ILP)-based HEMS method. • Artificial intelligence (AI) approach to monitor household energy consumption. • An efficient deep reinforcement learning algorithm to control activity recognition in smart homes. • Design and implementation of Intelligent home energy management system. • Converge promptly and considerably reduce operating expenses adopting AI-based approach. [ABSTRACT FROM AUTHOR]

Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning architectures are showing better performance compared to the shallow or traditional models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorized into bagging, boosting, stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies based deep ensemble models. Applications of deep ensemble models in different domains are also briefly discussed. Finally, we conclude this paper with some potential future research directions. • This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. • The categorizes of deep ensemble models discussed are bagging, boosting and stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies, unsupervised, semi supervised, reinforcement learning and online/incremental, multilabel based deep ensemble models. • Application of deep ensemble models in different domains are discussed. • Finally, this paper provides an outlook towards future research directions. [ABSTRACT FROM AUTHOR]

Missions involving Unmanned Aerial Vehicle usually consist of reaching a set of regions, performing some actions in each region, and returning to a determined depot after all the regions have been successfully visited or before the fuel/battery is totally consumed. Hence, planning a route becomes an important task for many applications, especially if a team of Unmanned Aerial Vehicles is considered. From this team, coordination and cooperation are expected to optimize results of the mission. In this paper, a system for managing multiple cooperative Unmanned Aerial Vehicles is presented. This system divides the routing problem into two stages: initial planning and routing solving. Initial planning is a first step where the regions to be visited are grouped in multiple clusters according to a distance criterion, with each cluster being assigned to each of the Unmanned Aerial Vehicles. Routing solving computes the best route for every agent considering the clusters of the initial planning and a variant of the Orienteering Problem. This variant introduces the concept of shared regions, allowing an Unmanned Aerial Vehicle to visit regions from other clusters and compensating for the suboptimal region clustering of the previous stage. The Orienteering Problem with shared regions is solved using the deep learning architecture Transformer along with a deep reinforcement learning framework. This architecture is able to obtain high-quality solutions much faster than conventional optimization approaches. Extensive results and comparison with other Combinatorial Optimization algorithms, including cooperative and non-cooperative scenarios, have been performed to show the benefits of the proposed solution. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

This paper proposes a Deep Reinforcement Learning (DRL) algorithm that expands the Random Neural Network (RNN) Reinforcement Learning (RL) method to include the previous learnings entirely from previous rewards, rather than only the actual one. The Random Neural Network weighs are updated with the current reward and the previous values, including time and memory. This addition makes DRL slower to make decisions, although it also increases its performance in some experiments. Several configurations to introduce DRL, such as sampling rate and memory duration, are also proposed and analysed in this article. The proposed DRL algorithm is included in a decision process that predicts trends: upward, downward and equal market directions in addition to values. Experimental results based on market prices demonstrate that the addition of Deep Learning to the Reinforcement Learning algorithm increases its performance slightly in some experiments; however, it also increases its computational cost. In random environments such as the stock market, it is preferable to make decisions based on the previous values (short memory) rather than historical records (long memory). [ABSTRACT FROM AUTHOR]

When a complex large power system is in an emergency, the conventional discrete emergency control strategy implemented will cause excess load or derivative accidents like line power overload, thereby raising the operation risk. To overcome the problems of excessive/insufficient regulation and subsequent accidents in the above context, this paper proposes a wide area automatic generation control (WA-AGC) framework, which integrates the emergency control strategy and a performance-based frequency regulation market mechanism. According to the frequency status of the power system, WA-AGC divides the AGC into four intervals, emergency AGC (EAGC), conventional AGC (CAGC), AGC transition and optimal power flow (OPF). These four together realize a comprehensive optimization of frequency and system stability as well as economy Based on the above framework, a swarm agent exploration distributed multiple delayed deep policy gradient algorithm (SAE-MD3) is developed, which uses multiple explorers with different exploration strategies for distributed optimization. In addition, several technologies are introduced to prevent Q value overestimation and generate a more robust optimal AGC strategy. Afterward, the effectiveness and feasibility of WA-AGC are verified through the simulations of an IEEE-9 two-area system and an IEEE-118 two-area system. Compared to conventional AGC strategies, the WA-AGC algorithm reduces the constraint violation time of the power line by 92.06% and the power generation cost by 0.27% as well as improves the CPS1 index by 0.04%. • A novel AGC framework using WAMS data is proposed to substitute the-conventional AGC. • The WA-AGC framework can achieve the optimal dispatch for different objective. • A novel deep reinforcement learning algorithm with better robustness is introduced. • The WA-AGC solves the excess/lack of regulation problem and derivative accidents. • The WA-AGC considers the impact of emergency fault on the power grid. [ABSTRACT FROM AUTHOR]

The control performance of conventional analytic algorithms for double-fed induction generator-based wind turbines is a fixed feature, which needs to be optimized by optimization processes. To avoid the optimization processes and update control strategies online, this paper proposes an online control algorithm based on the quantum process, deep belief networks, and reinforcement learning for double-fed induction generator-based wind turbines. The proposed approach is named quantum deep reinforcement learning, consisting of deep belief networks, one reinforcement learning framework, and multiple subsidiary reinforcement learning parts with quantum processes. The quantum deep reinforcement learning can update the control strategy online with general initialization for dynamic systems, avoid optimal local solutions, and predict the next systemic states of double-fed induction generator-based wind turbines. The proportional–integral–derivative, fractional-order proportional–integral–derivative, active disturbance rejection controller, reinforcement learning, and the quantum deep reinforcement learning are compared under four cases, i.e., variable wind speed with step magnitude, variable wind speed with sine magnitude, voltage dropout of power grids, and both variable wind speed with random magnitude and voltage dropout of power grids. Consequently, the proposed quantum deep reinforcement learning can obtain better control performance for double-fed induction generator-based wind turbines with smaller integrated absolute error, integral squared error, and integrated time-weighted absolute error of the control error than other compared methods. • Parameters of conventional control algorithms need to be optimized offline. • An online control algorithm is proposed to avoid the optimization processes. • Quantum process and deep reinforcement learning are combined for the online method. • The method can avoid local optimal solutions and can predict next systemic states. • Higher control performance is effectively obtained for controllers of wind turbines. [ABSTRACT FROM AUTHOR]

The combinatorial optimization (CO) problems on the graph are the core and classic problems in artificial intelligence (AI) and operations research (OR). For example, the Vehicle Routing Problem (VRP) and Traveling Salesman Problem (TSP) are fascinating NP-hard problems and have important significance for the existing transportation system. Traditional methods such as heuristics methods, exact algorithms, and solution solvers can already find approximate solutions on small-scale graphs. However, they are helpless for large-scale graphs and other problems with similar structures. Moreover, traditional methods often require artificially designed heuristic functions to aid decision-making. In recent years, more and more work has focused on applying deep learning and reinforcement learning (RL) to learn heuristics, which allows us to learn the internal structure of the graph end-to-end and find the optimal path under the guidance of heuristic rules. However, most of these still need manual assistance, and the RL method used has the problems of low sampling efficiency and small searchable space. This paper proposes a novel framework (called OmegaZero) based on Alphago Zero, which does not prescribe expert experience or label data but is trained through self-play. We divide the learning into two stages: in the first stage, we employ graph attention network (GAT) and GRU to learn node representations and memory history trajectories. In the second stage, we employ Monte Carlo tree search (MCTS) and deep RL to search for the solution space and train the model. [ABSTRACT FROM AUTHOR]