Search

• A graph neural network is proposed for learning multi-satellite scheduling policy. • It analyzes the differential impact of various constraints in scheduling. • It models the multi-satellite scheduling problem as a heterogeneous graph. • It designs a variant transformer to learn various relations within graph. • It designs a two-stage model to learn a multi-satellite scheduling policy. In this paper, we investigate the multi-satellite scheduling (MSS) problem, a combinatorial optimization problem (COP) that aims to enhance satellite utilization. Numerous scholars have conducted in-depth studies on this matter. Handcrafted policies are commonly employed in various algorithms. Despite the use of graphical representations, designing policies manually becomes increasingly difficult, particularly as the number of satellites and tasks grows. Consequently, we propose a novel approach for automatically summarizing graph structures and learning policies for MSS problem. The proposed approach is developed based on graph neural network, Transformer, and sequence to sequence framework. First, MSS problem is modeled as a heterogeneous graph following various constraints. Next, the encoder, which is based on GNN and Transformer, is designed to summarize crucial information from the aforementioned graph. Finally, a two-stage decoder is proposed taking into account dynamic information in stepwise solution generation. Since obtaining optimal solutions in practical applications is challenging, we train the model using reinforcement learning. The results of the experiment demonstrate that our proposed method performs effectively in various scenarios. The quality of the solutions generated through this method transcends that of conventional construction methods. Additionally, the method generates solutions quickly, which are comparable in quality to those produced by meta-heuristic algorithms. [ABSTRACT FROM AUTHOR]

• A linearly combined form of the objective is reformulated to control the trade-off between the return and the divergence more effectively. • An improved proximal policy optimization method (i.e., alphaPPO) is proposed, with a more elaborative alpha divergence for two adjacent policies. • The effectiveness of our alphaPPO is validated using detailed experimental comparison and analysis for six benchmark environments. Proximal policy optimization (PPO) is a recent advancement in reinforcement learning, which is formulated as an unconstrained optimization problem including two terms: accumulative discount return and Kullback–Leibler (KL) divergence. Currently, there are three PPO versions: primary, adaptive, and clipping. The most widely used PPO algorithm is the clipping version, in which the KL divergence is replaced by a clipping function to measure the difference between two policies indirectly. In this paper, we revisit this primary PPO and improve it in two aspects. One is to reformulate it as a linearly combined form to control the trade-off between two terms. The other is to substitute a parametric alpha divergence for KL divergence to measure the difference of two policies more effectively. This novel PPO variant is referred to as alphaPPO in this paper. Experiments on six benchmark environments verify the effectiveness of our alphaPPO, compared with clipping and combined PPOs. [ABSTRACT FROM AUTHOR]

In advanced transportation-management systems, variable speed limits are a crucial application. Deep reinforcement learning methods have been shown to have superior performance in many applications, as they are an effective approach to learning environment dynamics for decision-making and control. However, they face two significant difficulties in traffic-control applications: reward engineering with delayed reward and brittle convergence properties with gradient descent. To address these challenges, evolutionary strategies are well suited as a class of black-box optimization techniques inspired by natural evolution. Additionally, the traditional deep reinforcement learning framework struggles to handle the delayed reward setting. This paper proposes a novel approach using covariance matrix adaptation evolution strategy (CMA-ES), a gradient-free global optimization method, to handle the task of multi-lane differential variable speed limit control. The proposed method uses a deep-learning-based method to dynamically learn optimal and distinct speed limits among lanes. The parameters of the neural network are sampled using a multivariate normal distribution, and the dependencies between the variables are represented by a covariance matrix that is optimized dynamically by CMA-ES based on the freeway's throughput. The proposed approach is tested on a freeway with simulated recurrent bottlenecks, and the experimental results show that it outperforms deep reinforcement learning-based approaches, traditional evolutionary search methods, and the no-control scenario. Our proposed method demonstrates a 23% improvement in average travel time and an average of a 4% improvement in CO, HC, and NOx emission.Furthermore, the proposed method produces explainable speed limits and has desirable generalization power. [ABSTRACT FROM AUTHOR]

Hydropower is the most prevalent source of renewable energy production worldwide. As the global demand for robust and ecologically sustainable energy production increases, developing and enhancing the current energy production processes is essential. In the past decade, machine learning has contributed significantly to various fields, and hydropower is no exception. All three horizons of hydropower models could benefit from machine learning: short-term, medium-term, and long-term. Currently, dynamic programming is used in the majority of hydropower scheduling models. In this paper, we review the present state of the hydropower scheduling problem as well as the development of machine learning as a type of optimization problem and prediction tool. To the best of our knowledge, this is the first survey article that provides a comprehensive overview of machine learning and artificial intelligence applications in the hydroelectric power industry for scheduling, optimization, and prediction. [ABSTRACT FROM AUTHOR]

To prevent the proliferation of space debris and stabilize the space environment, active debris removal (ADR) has increasingly gained public concern. Considering the complexity of space operations and the viability of ADR missions, it would be necessary to schedule the ADR process in order to remove as much debris as possible. This paper presents an active debris removal mission planning problem, devoted to generate an optimal debris removal plan to guide the mission process. According to the problem characteristics, a two-layer time-dependent traveling salesman problem(TSP) mathematical model is established, involving the debris removal sequence planning and the transfer trajectory planning. Subsequently, two main novel methods based on machine learning are proposed for the ADR mission planning problem, including a deep neural networks(DNN)-based estimation method for approximating the optimal velocity increments of perturbed multiple-impulse rendezvous and an reinforcement learning(RL)-based method for optimizing the sequence of debris removal and rendezvous time. Experimental results of different simulation scenarios have verified the effectiveness and superiority of the proposed method, indicating the good performance for solving the active debris removal mission planning problem. [ABSTRACT FROM AUTHOR]

As a promising information theory, reinforcement learning has gained much attention. This paper researches a wind-storage cooperative decision-making strategy based on dueling double deep Q-network (D3QN). Firstly, a new wind-storage cooperative model is proposed. Besides wind farms, energy storage systems, and external power grids, demand response loads are also considered, including residential price response loads and thermostatically controlled loads (TCLs). Then, a novel wind-storage cooperative decision-making mechanism is proposed, which combines the direct control of TCLs with the indirect control of residential price response loads. In addition, a kind of deep reinforcement learning algorithm called D3QN is utilized to solve the wind-storage cooperative decision-making problem. Finally, the numerical results verify the effectiveness of D3QN for optimizing the decision-making strategy of a wind-storage cooperation system. [ABSTRACT FROM AUTHOR]

In the resource-constrained embedded systems, the designing of efficient deep neural networks is a challenging process, due to diversity in the artificial intelligence applications. The quantization in deep neural networks superiorly diminishes the storage and computational time by reducing the bit-width of networks encoding. In order to highlight the problem of accuracy loss, the quantization levels are automatically discovered using Policy Feedback-based Reinforcement Learning Method (PF-RELEQ). In this paper, the Proximal Policy Optimization with Policy Feedback (PPO-PF) technique is proposed to determine the best design decisions by choosing the optimum hyper-parameters. In order to enhance the sensitivity of the value function to the change of policy and to improve the accuracy of value estimation at the early learning stage, a policy update method is devised based on the clipped discount factor. In addition, specifically the loss functions of policy satisfy the unbiased estimation of the trust region. The proposed PF-RELEQ effectively balances quality and speed compared to other deep learning methods like ResNet-1202, ResNet-32, ResNet-110, GoogLeNet and AlexNet. The experimental analysis showed that PF-RELEQ achieved 20% computational work-load reduction compared to the existing deep learning methods on ImageNet, CIFAR-10, CIFAR-100 and tomato leaf disease datasets and achieved approximately 2% of improvisation in the validation accuracy. Additionally, the PF-RELEQ needs only 0.55 Graphics Processing Unit on an NVIDIA GTX-1080Ti to develop DNNs that delivers better accuracy improvement with fewer cycle counts for image classification. [ABSTRACT FROM AUTHOR]

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput. • Develop an AI-based signal control framework to collaborate multiple intersections. • The counterfactual baseline can well assign the credit among multiple intersections. • The scheduler module helps agents better understand the entire environment. [ABSTRACT FROM AUTHOR]

This paper addresses how physical knowledge can improve machine learning in process control. A data-driven tracking control framework using physics-informed neural networks (PINNs) and deep reinforcement learning (DRL) is proposed for dynamical systems, which is particularly important when iterations or repetitions of data collection experiments are limited or even not allowed. An indirect control approach is followed to accomplish this. First, a new process model is identified using PINNs trained from a model representing the experimental data collected from the plant. Then, the DRL-based control agent utilizes the identified PINN model to deliver an offline policy. We evaluate our framework for data-driven tracking control of nonisothermal CSTR, considering measurement noise and stochastic dynamics for closed-loop control. Our approach performed comparably to an NMPC without requiring a model to predict the process dynamics. • Physical knowledge was implemented to improve machine learning. • A indirect data-driven control approach was developed. • Physics-informed neural networks and deep reinforcement learning are combined. • The developed framework was tested against nonlinear model predictive control. [ABSTRACT FROM AUTHOR]

It is well known that unmodeled dynamics and uncertainties can deteriorate the performance of classical controllers. To resolve this problem, there is growing popularity in using the capabilities of Artificial Intelligence (AI) algorithms, especially Reinforcement Learning (RL) in power systems, because it is a promising adaptive model-free control strategy that can take optimal decisions in unknown environments (dynamics). For this reason, in this paper, two state-of-the-art RL agents, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG), are used for voltage control of a DC–DC buck converter, and their performance is reported compared with other classical controllers such as Model Predictive Control (MPC) and Sliding Mode Control (SMC). The DQN agent directly controls the power switches of converters. In other words, based on the current condition of the converter, the agent decides whether or not to close the power switches. On the other hand, the DDPG agent and the other mentioned traditional controllers manipulate the duty cycle of a Pulse Width Modulation (PWM) signal to adjust the output voltage of the converter at desired setpoints. According to experimental results, both RL agents outperform the classical controllers in terms of transient response error and robustness against uncertainties. Also, with regard to computational costs and learning rate among RL-based controllers, the DQN agent can learn more from a single interaction with fewer computations because of its simpler structure and direct control of the switches of the converter. Additionally, one of the most important advantages of the RL-based controllers is that they can be applied to various configurations of DC–DC​ converters like buck, boost, and buck-boost converters, provided that it is retrained for the new environments. Finally, the number of transitions in the semiconductor switches of the converter reduces appreciably by using the DQN agent, which certainly prolongs their longevity. [ABSTRACT FROM AUTHOR]

Reinforcement learning is considered to be one of the strongest paradigms in AI domain, which can be applied to teach machines how to behave through environment interaction. Recently the concept of deep reinforcement learning (DRL) was introduced and was tested with success in games like Atari 2600 or Go, proving the capability to learn a good representation of the environment. In this moment there are few implementations of DRL in the autonomous driving field. In this paper, we present the state of the art in deep reinforcement learning paradigm highlighting the current achievements for autonomous driving vehicles. [ABSTRACT FROM AUTHOR]