Search

Generator tripping scheme (GTS) is the most commonly used scheme to prevent power systems from losing safety and stability. Usually, GTS is composed of offline predetermination and real-time scenario match. However, it is extremely time-consuming and labor-intensive for manual predetermination for a large-scale modern power system. To improve efficiency of predetermination, this paper proposes a framework of knowledge fusion-based deep reinforcement learning (KF-DRL) for intelligent predetermination of GTS. First, the Markov Decision Process (MDP) for GTS problem is formulated based on transient instability events. Then, linear action space is developed to reduce dimensionality of action space for multiple controllable generators. Especially, KF-DRL leverages domain knowledge about GTS to mask invalid actions during the decision-making process. This can enhance the efficiency and learning process. Moreover, the graph convolutional network (GCN) is introduced to the policy network for enhanced learning ability. Numerical simulation results obtained on New England power system demonstrate superiority of the proposed KF-DRL framework for GTS over the purely data-driven DRL method.

Autonomous exploration is a critical technology to realize robotic intelligence as it allows unsupervised preparation for future tasks and facilitates flexible deployment. In this paper, a novel Deep Reinforcement Learning (DRL) based autonomous exploration strategy is proposed to efficiently reduce the unknown area of the workspace and provide accurate 2D map construction for mobile robots. Different from existing human-designed exploration techniques that usually make strong assumptions about the scenarios and the tasks, we utilize a model-free method to directly learn an exploration strategy through trial-and-error interactions with complex environments. To be specific, the Generalized Voronoi Diagram (GVD) is first utilized for domain conversion to obtain a high-dimensional Topological Environmental Representation (TER). Then, the Generalized Voronoi Networks (GVN) with spatial awareness and episodic memory is designed to learn autonomous exploration policies interactively online. For complete and efficient exploration, Invalid Action Masking (IAM) is employed to reshape the configuration space of exploration tasks to cope with the explosion of action space and observation space caused by the expansion of the exploration range. Furthermore, a well-designed reward function is leveraged to guide the learning of policies. Extensive baseline tests and comparative simulations show that our strategy outperforms the state-of-the-art strategies in terms of map quality and exploration speed. Sufficient ablation studies and mobile robot experiments demonstrate the effectiveness and superiority of our strategy. [ABSTRACT FROM AUTHOR]

Abstract Autonomous exploration is a critical technology to realize robotic intelligence as it allows unsupervised preparation for future tasks and facilitates flexible deployment. In this paper, a novel Deep Reinforcement Learning (DRL) based autonomous exploration strategy is proposed to efficiently reduce the unknown area of the workspace and provide accurate 2D map construction for mobile robots. Different from existing human-designed exploration techniques that usually make strong assumptions about the scenarios and the tasks, we utilize a model-free method to directly learn an exploration strategy through trial-and-error interactions with complex environments. To be specific, the Generalized Voronoi Diagram (GVD) is first utilized for domain conversion to obtain a high-dimensional Topological Environmental Representation (TER). Then, the Generalized Voronoi Networks (GVN) with spatial awareness and episodic memory is designed to learn autonomous exploration policies interactively online. For complete and efficient exploration, Invalid Action Masking (IAM) is employed to reshape the configuration space of exploration tasks to cope with the explosion of action space and observation space caused by the expansion of the exploration range. Furthermore, a well-designed reward function is leveraged to guide the learning of policies. Extensive baseline tests and comparative simulations show that our strategy outperforms the state-of-the-art strategies in terms of map quality and exploration speed. Sufficient ablation studies and mobile robot experiments demonstrate the effectiveness and superiority of our strategy.

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym- μ RTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks. [ABSTRACT FROM AUTHOR]

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-μRTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

Spatio-temporal crowdsourcing (STC) is a typical case of complex system-of-systems (SoSs) design, wherein the primary objective is to allocate real-time tasks to suitable groups of workers. Over time, the STC allocation has gradually evolved into a dynamic matching involving three distinct entities: tasks, workers, and workplaces. Aiming at addressing the problems of poor convergence, slow response and sparse actions caused by the spatial complexity and time dynamics of the STC, this paper proposes an improved proximal policy optimization algorithm based on an invalid action masking (IAM-IPPO) for the SoSs design of the STC. Initially, the ternary dynamic matching (TDM) of tasks, workers and workplaces in the STC is described. Furthermore, the STC allocation is formulated as a Markov decision process, with the corresponding definition of state space, action space, and reward mechanism. On this basis, an invalid action masking (IAM) method is mainly introduced to update the policy-based network of proximal policy optimization (PPO), realizing sampling only from valid actions to masking invalid action selection. Subsequently, the algorithmic framework of IAM-IPPO is elaborated upon, and the model is trained to generate an effective allocation scheme. Comparative experiments are conducted on authentic datasets, aiming to assess performance indicators of the presented approach. The findings demonstrate a substantial enhancement in performance for the IAM-IPPO algorithm compared to other baselines, which is helpful in exploring excellent design schemes of the crowdsourcing SoSs, especially in dynamic large-scale cases. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL), as a promising technique, is a new approach to solve the job shop scheduling problem (JSSP). Although DRL method is effective for solving JSSP, there are still deficiencies in state representation, action space definition, and reward function design, which make it difficult for the agent to learn effective policy. In this paper, we model JSSP as a Markov decision process (MDP) and design a new state representation using the state features of bidirectional scheduling, which can not only enable the agent to capture more effective state information, improve its decision-making ability, but also effectively avoid the phenomenon of multiple optimal action selections in candidate action set. Invalid action masking (IAM) technique is employed to narrow the search space, which helps the agent avoid exploring suboptimal solutions. We evaluate the performance of the policy model on eight public test datasets: ABZ, FT, ORB, YN, SWV, LA, TA, and DMU. Extensive experimental results show that the proposed method on the whole has better optimization ability than the existing state-of-the-art models and priority dispatching rules. • A new state representation was designed by adopting the state features of bidirectional scheduling. • To narrow the search space, invalid action masking technique was employed. • To quickly identify key actions, new state feature was used to calculate reward. • The effectiveness of the proposed method was evaluated. [ABSTRACT FROM AUTHOR]