Search

Hierarchical reinforcement learning (HRL) is a promising method to extend traditional reinforcement learning to solve more complex tasks. HRL can solve the problems of long-term reward sparsity and credit assignment. However, the existing HRL methods are trained in specific environments and target tasks each time, resulting in low sample utilization. In addition, the low-level sub-policies of the agent will interfere with each other during the migration process, resulting in poor policy stability. Aiming at the issue above, this paper proposes an HRL method, Relabeling and Policy Distillation of Hierarchical Reinforcement Learning (R-PD-HRL), that integrates meta-learning, shared reward relabeling and policy distillation to accelerate the learning speed and improve the policy stability of the agent. In the training process, a reward relabeling module is introduced to act on the experience buffer. Different reward functions are used to relabel the interaction trajectory for the training of other tasks under the same task distribution. At the low-level, policy distillation technology is used to compress the sub-policies of the low-level, and the interference between the policies is reduced while ensuring the correctness of the original low-level sub-policies. Finally, according to different tasks, the high-level policy calls the low-level optimal policy to complete the decision. In both continuous and discrete state-action environments, experimental results show that compared with other methods, the improved sample utilization of this method greatly accelerates the learning speed, and the success rate is as high as 0.6. [ABSTRACT FROM AUTHOR]

On-policy deep reinforcement learning (DRL) has the inherent advantage of using multi-step interaction data for policy learning. However, on-policy DRL still faces challenges in improving the sample efficiency of policy evaluations. Therefore, we propose a multi-step on-policy DRL method assisted by off-policy policy evaluation (abbreviated as MSOAO), whichs integrates on-policy and off-policy policy evaluations and belongs to a new type of DRL method. We propose a low-pass filtering algorithm for state-values to perform off-policy policy evaluation and make it efficiently assist on-policy policy evaluation. The filtered state-values and the multi-step interaction data are used as the input of the V-trace algorithm. Then, the state-value function is learned by simultaneously approximating the target state-values obtained from the V-trace output and the action-values of the current policy. The action-value function is learned by using the one-step bootstrapping algorithm to approximate the target action-values obtained from the V-trace output. Extensive evaluation results indicate that MSOAO outperformed the performance of state-of-the-art on-policy DRL algorithms, and the simultaneous learning of the state-value function and the action-value function in MSOAO can promote each other, thus improving the learning capability of the algorithm. [ABSTRACT FROM AUTHOR]

This manuscript presents a novel control approach for Permanent Magnet Synchronous Motors (PMSMs) by integrating the widely used Direct Torque Control (DTC) method with Deep Reinforcement Learning (DRL), specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The TD3 algorithm is well-suited to address the challenges posed by high-dimensional state and action spaces in PMSM drives. The conventional Proportional–Integral (PI) controller in the outer loop of DTC is replaced with the DRL based TD3 agent to overcome the limitations of traditional PI controllers, such as model dependency and complex parameter tuning. The DRL technique allows the agent to learn optimal control policies from experience, making it suitable for complex and nonlinear control problems. Extensive simulation studies demonstrate the effectiveness of the TD3-based control in achieving accurate speed tracking under varying operating conditions. Real-time experimental validation using a TMS320F28379D digital signal processor confirms the feasibility of the proposed DRL based approach. The research offers new insights into improving the performance of PMSM drives and paves the way for future advancements in electric drive control. [ABSTRACT FROM AUTHOR]

Several variance-reduced versions of REINFORCE based on importance sampling achieve an improved O (ϵ - 3) sample complexity to find an ϵ -stationary point, under an unrealistic assumption on the variance of the importance weights. In this paper, we propose the Defensive Policy Gradient (DEF-PG) algorithm, based on defensive importance sampling, achieving the same result without any assumption on the variance of the importance weights. We also show that this is not improvable by establishing a matching Ω (ϵ - 3) lower bound, and that REINFORCE with its O (ϵ - 4) sample complexity is actually optimal under weaker assumptions on the policy class. Numerical simulations show promising results for the proposed technique compared to similar algorithms based on vanilla importance sampling. [ABSTRACT FROM AUTHOR]

Ordinary defence components like rule-based firewalls and mark based detection are not staying aware of the always expanding intricacy and frequency of cyber security dangers. The reason for this work is to explore the way that deep reinforcement learning (DRL), a subfield of artificial intelligence famous for its viability in handling testing decisionproduction situations, may be utilized to improve cyber security conventions. To mimic and balance threatening cyberattacks, we present a system that utilizes deep reinforcement learning (DRL). We propose a specialist based model that can learn and adjust ceaselessly in powerful network security situations. In light of the present status of the network and the rewards it gets for its decisions, the specialist concludes what the best game-plans are. Specifically, we utilize the policy gradient (PG)- based double deep Q-network (DDQN) model and trial on three different datasets: NSL-KDD, CIC-IDS, and AWID. Our review demonstrates the way that DRL can really further develop the detection after-effects of cyber-attacks. Utilizing the policy gradient DDQN model on different datasets, we find prominent upgrades in cyber security conventions. Specific boundary modifications upgrade the viability of our philosophy much more, displaying empowering results on different datasets. This exploration features the potential of deep reinforcement learning (DRL) as a successful instrument in the field of cyber security. Our examination progresses detection techniques and gives a versatile arrangement that can be applied to an assortment of cyber security worries by giving areas of strength for a to demonstrating and relieving cyber dangers. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Ordnance Equipment Engineering is the property of Chongqing University of Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Control Theory & Applications / Kongzhi Lilun Yu Yinyong is the property of Editorial Department of Control Theory & Applications and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

AGV conflict prevention path planning is a key factor to improve transportation cost and operation efficiency of the container terminal. This paper studies the anti-conflict path planning problem of Automated Guided Vehicle (AGV) in the horizontal transportation area of the Automated Container Terminals (ACTs). According to the characteristics of magnetic nail guided AGVs, a node network is constructed. Through the analysis of two conflict situations, namely the opposite conflict situation and same point occupation conflict situation, an integer programming model is established to obtain the shortest path. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is proposed to solve the problem, and the Gumbel-Softmax strategy is applied to discretize the scenario created by the node network. A series of numerical experiments are conducted to verify the effectiveness and the efficiency of the model and the algorithm. [ABSTRACT FROM AUTHOR]

We develop an approach for solving time‐consistent risk‐sensitive stochastic optimization problems using model‐free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time‐consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor–critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control. [ABSTRACT FROM AUTHOR]

Geostationary (GEO) satellites are employed in optical frequencies for a variety of satellite services providing wide coverage and connectivity. Multi-beam GEO high-throughput satellites offer Gbps broadband rates and, jointly with low-Earth-orbit mega-constellations, are anticipated to enable a large-scale free-space optical (FSO) network. In this paper, a power allocation methodology based on deep reinforcement learning (DRL) is proposed for optical satellite systems disregarding any channel statistics knowledge requirements. An all-FSO, multi-aperture GEO-to-ground system is considered and an ergodic capacity optimization problem for the downlink is formulated with transmitted power constraints. A power allocation algorithm was developed, aided by a deep neural network (DNN) which is fed channel state information (CSI) observations and trained in a parameterized on-policy manner through a stochastic policy gradient approach. The proposed method does not require the channels' transition models or fading distributions. To validate and test the proposed allocation scheme, experimental measurements from the European Space Agency's ARTEMIS optical satellite campaign were utilized. It is demonstrated that the predicted average capacity greatly exceeds other baseline heuristic algorithms while strongly converging to the supervised, unparameterized approach. The predicted average channel powers differ only by 0.1 W from the reference ones, while the baselines differ significantly more, about 0.1–0.5 W. [ABSTRACT FROM AUTHOR]

Actor–critic methods are leading in many challenging continuous control tasks. Advantage estimators, the most common critics in the actor–critic framework, combine state values from bootstrapping value functions and sample returns. Different combinations balance the bias introduced by state values and the variance returned by samples to reduce estimation errors. The bias and variance constantly fluctuate throughout training, leading to different optimal combinations. However, existing advantage estimators usually use fixed combinations that fail to account for the trade-off between minimizing bias and variance to find the optimal estimate. Our previous work on adaptive advantage estimation (AAE) analyzed the sources of bias and variance and offered two indicators. This paper further explores the relationship between the indicators and their optimal combination through typical numerical experiments. These analyses develop a general form of adaptive combinations of state values and sample returns to achieve low estimation errors. Empirical results on simulated robotic locomotion tasks show that our proposed estimators achieve similar or superior performance compared to previous generalized advantage estimators (GAE). [ABSTRACT FROM AUTHOR]

With decades of development, computer intelligence has now reached a really high level. Especially deep learning (DL) and reinforcement learning (RL) endow computers the perception and decision abilities. This paper aims to design a vision-based system that is able to play The Open Racing Car Simulator (TORCS) like a human player that uses images. With the DL-trained perception module, useful and low-dimensional information is extracted from first-person images. Based on that, the RL-trained module further manipulates the simulated car in the middle of the lane. The two modules are separately trained, and both DL and RL advantages are maximally utilized. Experiments on different tracks show the promising performance of the method. [ABSTRACT FROM AUTHOR]

Fault classification plays a crucial role in the industrial process monitoring domain. In the datasets collected from real-life industrial processes, the data distribution is usually imbalanced. The datasets contain a large amount of normal data (majority) and only a small amount of faulty data (minority); this phenomenon is also known as the imbalanced fault classification problem. To solve the imbalanced fault classification problem, a novel reinforcement learning (RL)-based cost-sensitive classifier (RLCC) based on policy gradient is proposed in this paper. In RLCC, a novel cost-sensitive learning strategy based on policy gradient and the actor-critic of RL is developed. The novel cost-sensitive learning strategy can adaptively learn the cost matrix and dynamically yield the sample weights. In addition, RLCC uses a newly designed reward to train the sample weight learner and classifier using an alternating iterative approach. The alternating iterative approach makes RLCC highly flexible and effective in solving the imbalanced fault classification problem. The effectiveness and practicability of the proposed RLCC method are verified through its application in a real-world dataset and an industrial process benchmark. [ABSTRACT FROM AUTHOR]

Feature matching for multimodal images is an important task in image processing. However, most methods perform image feature detection, description, and matching sequentially, resulting in a large loss, low matching accuracy, and slow performance. To tackle these challenges, we propose a detector-free method called FeMIP for feature matching of multimodal images. We design coarse matching and fine regression modules to implement accurate multimodal image feature matches in a coarse-to-fine manner. Furthermore, we add a novel data augmentation method enabling FeMIP to achieve feature matching faster and more accurately. The coarse-to-fine module automatically generates pixel-level labels on the original image, enabling FeMIP to perform pixel-level matching on data with only image-level labels. In addition, we use the principle of reinforcement learning to design a policy gradient method to improve the solution to the problem of discreteness in matching. Extensive experiments show that FeMIP has good generalization and achieves excellent matching performances. The code will be released at: https://github.com/LiaoYun0x0/FeMIP. [ABSTRACT FROM AUTHOR]

In reinforcement learning, the policy gradient algorithm often needs to model the continuous-time process as a discrete-time process through sampling. To model the problem more accurately, it improves the sampling frequency. However, the excessive sampling frequency may reduce the training efficiency. To solve this problem, this paper proposed action stable updating algorithm. This method calculated the probability of action repetition using the change of the output of the policy function, and randomly repeated or changed the action based on this probability. This paper theoretically analyzed the performance of this method. This paper evaluated the performance of this method in nine different environments and compared it with the existing methods. This method surpassed existing methods in six of these environments. The experimental results show that this method can improve the training efficiency of the policy gradient algorithm in continuous-time problems. [ABSTRACT FROM AUTHOR]

How to utilize symbolic knowledge in deep learning is an important problem. Deep neural networks are flexible and powerful, while symbolic knowledge has the virtue of interpretability and intuitiveness. It is necessary to combine the two together to inject symbolic knowledge into neural networks. We propose a novel approach to combine neural networks with logic rules. In this approach, task-specific supervised learning and policy-based reinforcement learning are performed alternately to train a neural model until convergence. The basic idea is to use supervised learning to train a deep model and use reinforcement learning to propel the deep model to meet logic rules. In the process of the policy gradient reinforcement learning, if a predicted output of a deep model meets all logical rules, the deep model is given a positive reward, otherwise, it is given a negative reward. By maximizing the expected rewards, the deep model can be gradually adjusted to meet logical constraints. We conduct experiments on the tasks of named entity recognition. The experimental results demonstrate the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Limit Orders allow buyers and sellers to set a "limit price" they are willing to accept in a trade. On the other hand, market orders allow for immediate execution at any price. Thus, market orders are susceptible to slippage, which is the additional cost incurred due to the unfavorable execution of a trade order. As a result, limit orders are often preferred, since they protect traders from excessive slippage costs due to larger than expected price fluctuations. Despite the price guarantees of limit orders, they are more complex compared to market orders. Orders with overly optimistic limit prices might never be executed, which increases the risk of employing limit orders in Machine Learning (ML)-based trading systems. Indeed, the current ML literature for trading almost exclusively relies on market orders. To overcome this limitation, a Deep Reinforcement Learning (DRL) approach is proposed to model trading agents that use limit orders. The proposed method (a) uses a framework that employs a continuous probability distribution to model limit prices, while (b) provides the ability to place market orders when the risk of no execution is more significant than the cost of slippage. Extensive experiments are conducted with multiple currency pairs, using hourly price intervals, validating the effectiveness of the proposed method and paving the way for introducing limit order modeling in DRL-based trading. [ABSTRACT FROM AUTHOR]

In this article, we formulate the standard mixture learning problem as a Markov Decision Process (MDP). We theoretically show that the objective value of the MDP is equivalent to the log-likelihood of the observed data with a slightly different parameter space constrained by the policy. Different from some classic mixture learning methods such as Expectation–Maximization (EM) algorithm, the proposed reinforced algorithm requires no distribution assumptions and can handle the non-convex clustered data by constructing a model-free reward to evaluate the mixture assignment based on the spectral graph theory and Linear Discriminant Analysis (LDA). Extensive experiments on both synthetic and real examples demonstrate that the proposed method is comparable with the EM algorithm when the Gaussian mixture assumption is satisfied, and significantly outperforms it and other clustering methods in most scenarios when the model is misspecified. A Python implementation of our proposed method is available at https://github.com/leyuanheart/Reinforced-Mixture-Learning. [ABSTRACT FROM AUTHOR]

We consider a policy gradient algorithm applied to a finite-arm bandit problem with Bernoulli rewards. We allow learning rates to depend on the current state of the algorithm rather than using a deterministic time-decreasing learning rate. The state of the algorithm forms a Markov chain on the probability simplex. We apply Foster–Lyapunov techniques to analyze the stability of this Markov chain. We prove that, if learning rates are well-chosen, then the policy gradient algorithm is a transient Markov chain, and the state of the chain converges on the optimal arm with logarithmic or polylogarithmic regret. [ABSTRACT FROM AUTHOR]

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym- μ RTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks. [ABSTRACT FROM AUTHOR]

In this paper, we present a new policy gradient (PG) method, namely, the block policy mirror descent (BPMD) method, for solving a class of regularized reinforcement learning (RL) problems with (strongly) convex regularizers. Compared to the traditional PG methods with a batch update rule, which visits and updates the policy for every state, the BPMD method has cheap per-iteration computation via a partial update rule that performs the policy update on a sampled state. Despite the nonconvex nature of the problem and a partial update rule, we provide a unified analysis for several sampling schemes and show that BPMD achieves fast linear convergence to the global optimality. In particular, uniform sampling leads to worst-case total computational complexity comparable to batch PG methods. A necessary and sufficient condition for convergence with on-policy sampling is also identified. With a hybrid sampling scheme, we further show that BPMD enjoys potential instance-dependent acceleration, leading to improved dependence on the state space and consequently outperforming batch PG methods. We then extend BPMD methods to the stochastic setting by utilizing stochastic first-order information constructed from samples. With a generative model, O(|S||A|κ) (resp., O(|S||A|κ²)) sample complexities are established for the strongly convex (resp., non-strongly convex) regularizers, where κdenotes the target accuracy. To the best of our knowledge, this is the first time that block coordinate descent methods have been developed and analyzed for policy optimization in reinforcement learning, which provides a new perspective on solving large-scale RL problems. [ABSTRACT FROM AUTHOR]

Credit assignment is a crucial issue in multi-agent tasks employing a centralized training and decentralized execution paradigm. While value decomposition has demonstrated strong performance in Q-learning-based approaches and certain Actor–Critic variants, it remains challenging to achieve efficient credit assignment in multi-agent tasks using policy gradient methods due to decomposable value limitations. This paper introduces Predictive Contribution Measurement, an explicit credit assignment method that compares prediction errors among agents and allocates surrogate rewards based on their relevance to global state transitions, with a theoretical guarantee. With multi-agent proximal policy optimization (MAPPO) as a training backend, we propose Predictive Contribution MAPPO (PC-MAPPO). Our experiments demonstrate that PC-MAPPO, with a 10% warm-up phase, outperforms MAPPO, QMIX, and Weighted QMIX on StarCraft multi-agent challenge tasks, particularly in maps requiring heightened cooperation to defeat enemies, such as the map corridor. Employing a pre-trained predictor, PC-MAPPO achieves significantly improved performance on all tested super-hard maps. In parallel training scenarios, PC-MAPPO exhibits superior data efficiency and achieves state-of-the-art performance compared to other methods. [ABSTRACT FROM AUTHOR]

With the pervasiveness of dynamic task allocation in sharing economy applications, online bipartite graph matching has attracted more and more research attention. In sharing economy applications, crowdsourcing platforms need to allocate tasks to workers dynamically. Previous studies have low allocation utility. To increase the allocation utility of the Spatio-temporal crowdsourcing system, this paper proposes a dynamic delay bipartite matching(DDBM) problem, and designs Value Based Task Allocation(VBTA) and Policy Gradient Based Task Allocation(PGTA) frameworks respectively. According to the current state, VBTA and PGTA could enhance the allocation utility by selecting appropriate thresholds. The convergence of the algorithm is proved. Extensive experimental results on two real datasets demonstrate that the proposed algorithms are superior to the existing algorithms in effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

Abstract Rational drug design often starts from specific scaffolds to which side chains/substituents are added or modified due to the large drug-like chemical space available to search for novel drug-like molecules. With the rapid growth of deep learning in drug discovery, a variety of effective approaches have been developed for de novo drug design. In previous work we proposed a method named DrugEx, which can be applied in polypharmacology based on multi-objective deep reinforcement learning. However, the previous version is trained under fixed objectives and does not allow users to input any prior information (i.e. a desired scaffold). In order to improve the general applicability, we updated DrugEx to design drug molecules based on scaffolds which consist of multiple fragments provided by users. Here, a Transformer model was employed to generate molecular structures. The Transformer is a multi-head self-attention deep learning model containing an encoder to receive scaffolds as input and a decoder to generate molecules as output. In order to deal with the graph representation of molecules a novel positional encoding for each atom and bond based on an adjacency matrix was proposed, extending the architecture of the Transformer. The graph Transformer model contains growing and connecting procedures for molecule generation starting from a given scaffold based on fragments. Moreover, the generator was trained under a reinforcement learning framework to increase the number of desired ligands. As a proof of concept, the method was applied to design ligands for the adenosine A2A receptor (A2AAR) and compared with SMILES-based methods. The results show that 100% of the generated molecules are valid and most of them had a high predicted affinity value towards A2AAR with given scaffolds.

Abstract Background The initial policy response to the COVID-19 pandemic has differed widely across countries. Such variability in government interventions has made it difficult for policymakers and health research systems to compare what has happened and the effectiveness of interventions across nations. Timely information and analysis are crucial to addressing the lag between the pandemic and government responses to implement targeted interventions to alleviate the impact of the pandemic. Methods To examine the effect government interventions and technological responses have on epidemiological and economic outcomes, this policy paper proposes a conceptual framework that provides a qualitative taxonomy of government policy directives implemented in the immediate aftermath of a pandemic announcement and before vaccines are implementable. This framework assigns a gradient indicating the intensity and extent of the policy measures and applies the gradient to four countries that share similar institutional features but different COVID-19 experiences: Italy, New Zealand, the United Kingdom and the United States of America. Results Using the categorisation framework allows qualitative information to be presented, and more specifically the gradient can show the dynamic impact of policy interventions on specific outcomes. We have observed that the policy categorisation described here can be used by decision-makers to examine the impacts of major viral outbreaks such as SARS-CoV-2 on health and economic outcomes over time. The framework allows for a visualisation of the frequency and comparison of dominant policies and provides a conceptual tool to assess how dominant interventions (and innovations) affect different sets of health and non-health related outcomes during the response phase to the pandemic. Conclusions Policymakers and health researchers should converge toward an optimal set of policy interventions to minimize the costs of the pandemic (i.e., health and economic), and facilitate coordination across governance levels before effective vaccines are produced. The proposed framework provides a useful tool to direct health research system resources and build a policy benchmark for future viral outbreaks where vaccines are not readily available.

Reinforcement learning(RL) is an artificial intelligence algorithm with the advantages of clear calculation logic and easy expansion of the model.Through interacting with the environment and maximizing value functions on the premise of obtaining little or no prior information, RL can optimize the performance of strategies and effectively reduce the complexity caused by physical models .The RL algorithm based on strategy gradient has been successfully applied in many fields such as intelligent image recognition, robot control and path planning for automatic driving.However, the highly sampling-dependent characteristics of RL determine that the training process needs a large number of samples to converge, and the accuracy of decision making is easily affected by slight interference that does not match with the simulation environment.Especially when RL is applied to the control field, it is difficult to prove the stability of the algorithm because the convergence of the algorithm cannot be guaranteed.Considering that swarm intelligence algorithm can solve complex problems through group cooperation and has the characteristics of self-organization and strong stability, it is an effective way to be used for improving the stability of RL model.The pigeon-inspired optimization algorithm in swarm intelligence was combined to improve RL based on strategy gradient.A RL algorithm based on pigeon-inspired optimization was proposed to solve the strategy gradient in order to maximize long-term future rewards.Adaptive function of pigeon-inspired optimization algorithm and RL were combined to estimate the advantages and disadvantages of strategies, avoid solving into an infinite loop, and improve the stability of the algorithm.A nonlinear two-wheel inverted pendulum robot control system was selected for simulation verification.The simulation results show that the RL algorithm based on pigeon-inspired optimization can improve the robustness of the system, reduce the computational cost, and reduce the algorithm’s dependence on the sample database.

A novel policy gradient (PG) adaptive dynamic programming method is developed to deal with nonlinear discrete-time zero-sum games with unknown dynamics. To facilitate the implementation, a policy iteration algorithm is established to approximate the iterative Q-function, as well as the control and disturbance policies via three neural network (NN) approximators, respectively. Then, the iterative Q-function is exploited to update the control and disturbance policies via PG method. To stabilize the training process and improve the data usage efficiency, the experience replay technique is applied to train the weight vectors of the three NNs by using mini-batch empirical data from replay memory. Furthermore, the convergence in terms of the iterative Q-function is proved. Simulation results of two numerical examples are provided to show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

To find the optimal policy quickly for reinforcement learning problems, policy gradient (PG) method is very effective, it parameters the policy and updates policy parameter directly. Besides, momentum methods are commonly employed to improve convergence performance in the training of centralized deep networks, which can accelerate training rate by changing the descending direction of gradients. However, decentralized variants with momentum of PG are rarely investigated. For this reason, we propose a Decentralized Policy Gradient algorithm with Momentum called DPGM for solving multi-task reinforcement learning problems. Moreover, this article makes theoretical analysis on the convergence performance of DPGM rigorously, it can reach the rate of O(1/T), where T denotes the number of iterations. This rate can match the state of the art of decentralized PG methods. Furthermore, we provide experimental verification on decentralized reinforcement learning environment to support the theoretical result. [ABSTRACT FROM AUTHOR]

In recent years, a new attempt has been made to capture label co-occurrence by applying the sequence-to-sequence (Seq2Seq) model to multi-label text classification (MLTC). However, existing approaches frequently ignore the semantic information contained in the labels themselves. Besides, the Seq2Seq model is susceptible to the negative impact of label sequence order. Furthermore, it has been demonstrated that the traditional attention mechanism underperforms in MLTC. Therefore, we propose a novel Seq2Seq model with a different label semantic attention mechanism (S2S-LSAM), which generates fused information containing label and text information through the interaction of label semantics and text features in the label semantic attention mechanism. With the fused information, our model can select the text features that are most relevant to the labels more effectively. A combination of the cross-entropy loss function and the policy gradient-based loss function is employed to reduce the label sequence order effect. The experiments show that our model outperforms the baseline models. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) methods for energy saving and greening have recently appeared in the field of autonomous driving. In inter-vehicle communication (IVC), a feasible and increasingly popular research direction of RL is to obtain the optimal action decision of agents in a special environment. This paper presents the application of reinforcement learning in the vehicle communication simulation framework (Veins). In this research, we explore the application of reinforcement learning algorithms in a green cooperative adaptive cruise control (CACC) platoon. Our aim is to train member vehicles to react appropriately in the event of a severe collision involving the leading vehicle. We seek to reduce collision damage and optimize energy consumption by encouraging behavior that conforms to the platoon's environmentally friendly aim. Our study provides insight into the potential benefits of using reinforcement learning algorithms to improve the safety and efficiency of CACC platoons while promoting sustainable transportation. The policy gradient algorithm used in this paper has good convergence in the calculation of the minimum energy consumption problem and the optimal solution of vehicle behavior. In terms of energy consumption metrics, the policy gradient algorithm is used first in the IVC field for training the proposed platoon problem. It is a feasible training decision-planning algorithm for solving the minimization of energy consumption caused by decision making in platoon avoidance behavior. [ABSTRACT FROM AUTHOR]

While the world is working quietly to repair the damage caused by COVID-19's widespread transmission, the monkeypox virus threatens to become a global pandemic. There are several nations that report new monkeypox cases daily, despite the virus being less deadly and contagious than COVID-19. Monkeypox disease may be detected using artificial intelligence techniques. This paper suggests two strategies for improving monkeypox image classification precision. Based on reinforcement learning and parameter optimization for multi-layer neural networks, the suggested approaches are based on feature extraction and classification: the Q-learning algorithm determines the rate at which an act occurs in a particular state; Malneural networks are binary hybrid algorithms that improve the parameters of neural networks. The algorithms are evaluated using an openly available dataset. In order to analyze the proposed optimization feature selection for monkeypox classification, interpretation criteria were utilized. In order to evaluate the efficiency, significance, and robustness of the suggested algorithms, a series of numerical tests were conducted. There were 95% precision, 95% recall, and 96% f1 scores for monkeypox disease. As compared to traditional learning methods, this method has a higher accuracy value. The overall macro average was around 0.95, and the overall weighted average was around 0.96. When compared to the benchmark algorithms, DDQN, Policy Gradient, and Actor–Critic, the Malneural network had the highest accuracy (around 0.985). In comparison with traditional methods, the proposed methods were found to be more effective. Clinicians can use this proposal to treat monkeypox patients and administration agencies can use it to observe the origin and current status of the disease. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning (DRL) algorithm is a popular policy search method and has been successfully applied to a series of challenging control tasks. However, DRL is difficult to be applied to large-scale practical problems due to its difficulty in dealing with reward sparseness, lack of effective exploration and fragile convergence sensitive to hyperparameters. Particle Swarm Optimization (PSO) is an evolutionary optimization method, which uses the cumulative rewards of the entire episode as the fitness value and is insensitive to the environment with sparse rewards. Moreover, this method also has population-based diversification exploration and stable convergence, but the sample efficiency is low. In this paper, PSO and DRL based on policy gradient are combined. DRL trains the policies with the lowest cumulative rewards in the population through a variety of data provided by the PSO population, and every time the policies with improved cumulative rewards after training is inserted into the PSO population to enhance the information exchange between DRL and PSO population. This algorithm, called PSO-RL, can improve the sample efficiency of PSO and improve the performance and stability of DRL algorithm. Experiments on the challenging continuous control task of the PyBullet module show that PSO-RL performs better than both DRL and the evolutionary reinforcement learning algorithm. [ABSTRACT FROM AUTHOR]

Wargames are essential simulators for various war scenarios. However, the increasing pace of warfare has rendered traditional wargame decision-making methods inadequate. To address this challenge, wargame-assisted decision-making methods that leverage artificial intelligence techniques, notably reinforcement learning, have emerged as a promising solution. The current wargame environment is beset by a large decision space and sparse rewards, presenting obstacles to optimizing decision-making methods. To overcome these hurdles, a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) based wargame decision-making method is presented. The Partially Observable Markov Decision Process (POMDP), joint action-value function, and the Gumbel-Softmax estimator are applied to optimize MADDPG in order to adapt to the wargame environment. Furthermore, a wargame decision-making method based on the improved MADDPG algorithm is proposed. Using supervised learning in the proposed approach, the training efficiency is improved and the space for manipulation before the reinforcement learning phase is reduced. In addition, a policy gradient estimator is incorporated to reduce the action space and to obtain the global optimal solution. Furthermore, an additional reward function is designed to address the sparse reward problem. The experimental results demonstrate that our proposed wargame decision-making method outperforms the pre-optimization algorithm and other algorithms based on the AC framework in the wargame environment. Our approach offers a promising solution to the challenging problem of decision-making in wargame scenarios, particularly given the increasing speed and complexity of modern warfare. [ABSTRACT FROM AUTHOR]

Control augmentation systems (CAS) are conventionally built with classical controllers which have the following drawbacks: dependence on domain specific knowledge for tuning and limited self-learning capability. Consequently, these drawbacks lead to sub-optimal aircraft stability and performance when exposed to time varying disturbances. Hence, to curb the stated problems; this paper proposes the development of a deep reinforcement learning (DRL) pitch-rate CAS (qCAS), aimed at guaranteeing adaptive stability, pitch-rate control tracking and disturbance rejection across the longitudinal dynamics of an aircraft. This stated aim was actualized by developing a CAS with a deep deterministic policy gradient (DDPG) agent. Subsequently, this proposed method was compared with two classical qCAS methods (a developed PID-aCAS and a benchmark PIqCAS obtained from literature). The results show that the developed DDPG-qCAS method outperformed the classical methods in peak overshoot, reference command tracking and disturbance rejection as well as mean absolute error (MSE) and mean steady state error (MSSE). Hence, it can be inferred that it is important to apply artificially intelligent controllers to the flight control systems of aircraft in order to achieve superior time response, control command tracking accuracy and disturbance rejection. [ABSTRACT FROM AUTHOR]

Abstract Initial residual stress is the main reason causing machining deformation of the workpiece, which has been deemed as one of the most important aspects of machining quality issues. The inference of the distribution of initial residual stress inside the blank has significant meaning for machining deformation control. Due to the principle error of existing residual stress detection methods, there are still challenges in practical applications. Aiming at the detection problem of the initial residual stress field, an initial residual stress inference method by incorporating monitoring data and mechanism model is proposed in this paper. Monitoring data during machining process is used to represent the macroscopic characterization of the unbalanced residual stress, and the finite element numerical model is used as the mechanism model so as to solve the problem that the analytic mechanism model is difficult to establish; the policy gradient approach is introduced to solve the gradient descent problem of the combination of learning model and mechanism model. Finally, the initial residual stress field is obtained through iterative calculation based on the fusing method of monitoring data and mechanism model. Verification results show that the proposed inference method of initial residual stress field can accurately and effectively reflect the machining deformation in the actual machining process.

Rational drug design often starts from specific scaffolds to which side chains/substituents are added or modified due to the large drug-like chemical space available to search for novel drug-like molecules. With the rapid growth of deep learning in drug discovery, a variety of effective approaches have been developed for de novo drug design. In previous work we proposed a method named DrugEx, which can be applied in polypharmacology based on multi-objective deep reinforcement learning. However, the previous version is trained under fixed objectives and does not allow users to input any prior information (i.e. a desired scaffold). In order to improve the general applicability, we updated DrugEx to design drug molecules based on scaffolds which consist of multiple fragments provided by users. Here, a Transformer model was employed to generate molecular structures. The Transformer is a multi-head self-attention deep learning model containing an encoder to receive scaffolds as input and a decoder to generate molecules as output. In order to deal with the graph representation of molecules a novel positional encoding for each atom and bond based on an adjacency matrix was proposed, extending the architecture of the Transformer. The graph Transformer model contains growing and connecting procedures for molecule generation starting from a given scaffold based on fragments. Moreover, the generator was trained under a reinforcement learning framework to increase the number of desired ligands. As a proof of concept, the method was applied to design ligands for the adenosine A2A receptor (A2AAR) and compared with SMILES-based methods. The results show that 100% of the generated molecules are valid and most of them had a high predicted affinity value towards A2AAR with given scaffolds. [ABSTRACT FROM AUTHOR]