Search

Against the backdrop of global warming, the carbon trading market is considered as an effective means of emission reduction. With more and more companies and individuals participating in carbon markets for trading, it is of great theoretical and practical significance to help them automatically identify carbon trading investment opportunities and achieve intelligent carbon trading decisions. Based on the characteristics of the carbon trading market, we propose a novel deep reinforcement learning (DRL) trading strategy - Deep Recurrent Q-Network (DRQN). The experimental results show that the carbon allowance trading model based on the DRQN algorithm can provide optimal trading strategies and adapt to market changes. Specifically, the annualized returns for the DRQN algorithm strategy in the Guangdong (GD) and Hubei (HB) carbon markets are 15.43% and 34.75%, respectively, significantly outperforming other strategies. To better meet the needs of the actual implementation scenarios of the model, we analyze the impacts of discount factors and trading costs. The research results indicate that discount factors can provide participants with clearer expectations. In both carbon markets (GD and HB), there exists an optimal discount factor value of 0.4, as both excessively small or large values can have adverse effects on trading. Simultaneously, the government can ensure the fairness of carbon trading by regulating the costs of carbon trading to limit the speculative behavior of participants., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:Haiying Liu reports financial support was provided by National Social Science Fund of China. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

As a special type of multi-objective combinatorial optimization problems (MOCOPs), the multi-objective traveling salesman problem (MOTSP) plays an important role in practical fields such as transportation and robot control. However, due to the complexity of its solution space and the conflicts between different objectives, it is difficult to obtain satisfactory solutions in a short time. This paper proposes an end-to-end algorithm framework for solving MOTSP based on deep reinforcement learning (DRL). By decomposing strategies, solving MOTSP is transformed into solving multiple single-objective optimization subproblems. Through linear transformation, the features of the MOTSP are combined with the weights of the objective function. Subsequently, a modified graph pointer network (GPN) model is used to solve the decomposed subproblems. Compared with the previous DRL model, the proposed algorithm can solve all the subproblems using only one model without adding weight information as input features. Furthermore, our algorithm can output a corresponding solution for each weight, which increases the diversity of solutions. In order to verify the performance of our proposed algorithm, it is compared with four classical evolutionary algorithms and two DRL algorithms on several MOTSP instances. The comparison shows that our proposed algorithm outperforms the compared algorithms both in terms of training time and the quality of the resulting solutions., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

This study introduces a novel approach to adaptive traffic signal control (ATSC) by leveraging multi-objective deep reinforcement learning (DRL) techniques. The proposed scheme aims to optimize control strategies at intersections while concurrently addressing the objectives of safety, efficiency, and decarbonization. Traditional ATSC schemes primarily emphasize traffic efficiency and often lack the ability to adapt to real-time dynamic traffic conditions. To overcome these limitations, the study proposes a DRL-based ATSC algorithm that integrates the Dueling Double Deep Q Network (D3QN) framework. The performance of the proposed algorithm is evaluated through a simulated intersection in Changsha, China. Specifically, the proposed ATSC algorithm outperforms both traditional ATSC and ATSC with efficiency optimization only algorithms by achieving more than a 16% reduction in traffic conflicts and a 4% reduction in carbon emissions. In terms of traffic efficiency, waiting time reduces by 18% compared to traditional ATSC, but slightly increases (0.64%) compared to DRL-based ATSC algorithm that integrates D3QN framework. This small increase indicates a trade-off between efficiency and other objectives such as safety and decarbonization. Moreover, the proposed scheme demonstrates superior performance specifically in highly traffic-demand scenarios in terms of all three objectives. The findings of this study contribute to the advancement of traffic control systems by providing a practical and effective solution for optimizing signal control strategies in real-world traffic scenarios., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023. Published by Elsevier Ltd.)

The Centralized Training and Decentralized Execution (CTDE) paradigm, where a centralized critic is allowed to access global information during the training phase while maintaining the learned policies executed with only local information in a decentralized way, has achieved great progress in recent years. Despite the progress, CTDE may suffer from the issue of Centralized-Decentralized Mismatch (CDM): the suboptimality of one agent's policy can exacerbate policy learning of other agents through the centralized joint critic. In contrast to centralized learning, the cooperative model that most closely resembles the way humans cooperate in nature is fully decentralized, i.e. Independent Learning (IL). However, there are still two issues that need to be addressed before agents coordinate through IL: (1) how agents are aware of the presence of other agents, and (2) how to coordinate with other agents to improve joint policy under IL. In this paper, we propose an inference-based coordinated MARL method: Deep Motor System (DMS). DMS first presents the idea of individual intention inference where agents are allowed to disentangle other agents from their environment. Secondly, causal inference was introduced to enhance coordination by reasoning each agent's effect on others' behavior. The proposed model was extensively experimented on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that the proposed method outperforms independent learning algorithms and the coordination behavior among agents can be learned even without the CTDE paradigm compared to the state-of-the-art baselines including IPPO and HAPPO., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

Background: Scattering photons can seriously contaminate cone-beam CT (CBCT) image quality with severe artifacts and substantial degradation of CT value accuracy, which is a major concern limiting the widespread application of CBCT in the medical field. The scatter kernel deconvolution (SKD) method commonly used in clinic requires a Monte Carlo (MC) simulation to determine numerous quality-related kernel parameters, and it cannot realize intelligent scatter kernel parameter optimization, causing limited accuracy of scatter estimation., Purpose: Aiming at improving the scatter estimation accuracy of the SKD algorithm, an intelligent scatter correction framework integrating the SKD with deep reinforcement learning (DRL) scheme is proposed., Methods: Our method firstly builds a scatter kernel model to iteratively convolve with raw projections, and then the deep Q-network of the DRL scheme is introduced to intelligently interact with the scatter kernel to achieve a projection adaptive parameter optimization. The potential of the proposed framework is demonstrated on CBCT head and pelvis simulation data and experimental CBCT measurement data. Furthermore, we have implemented the U-net based scatter estimation approach for comparison., Results: The simulation study demonstrates that the mean absolute percentage error (MAPE) of the proposed method is less than 9.72% and the peak signal-to-noise ratio (PSNR) is higher than 23.90 dB, while for the conventional SKD algorithm, the minimum MAPE is 17.92% and the maximum PSNR is 19.32 dB. In the measurement study, we adopt a hardware-based beam stop array algorithm to obtain the scatter-free projections as a comparison baseline, and our method can achieve superior performance with MAPE < 17.79% and PSNR > 16.34 dB., Conclusions: In this paper, we propose an intelligent scatter correction framework that integrates the physical scatter kernel model with DRL algorithm, which has the potential to improve the accuracy of the clinical scatter correction method to obtain better CBCT imaging quality., (© 2023 American Association of Physicists in Medicine.)

Background: In recent years, deep reinforcement learning (RL) has been applied to various medical tasks and produced encouraging results., Objective: In this paper, we demonstrate the feasibility of deep RL for denoising simulated deep-silicon photon-counting CT (PCCT) data in both full and interior scan modes. PCCT offers higher spatial and spectral resolution than conventional CT, requiring advanced denoising methods to suppress noise increase., Methods: In this work, we apply a dueling double deep Q network (DDDQN) to denoise PCCT data for maximum contrast-to-noise ratio (CNR) and a multi-agent approach to handle data non-stationarity., Results: Using our method, we obtained significant image quality improvement for single-channel scans and consistent improvement for all three channels of multichannel scans. For the single-channel interior scans, the PSNR (dB) and SSIM increased from 33.4078 and 0.9165 to 37.4167 and 0.9790 respectively. For the multichannel interior scans, the channel-wise PSNR (dB) increased from 31.2348, 30.7114, and 30.4667 to 31.6182, 30.9783, and 30.8427 respectively. Similarly, the SSIM improved from 0.9415, 0.9445, and 0.9336 to 0.9504, 0.9493, and 0.0326 respectively., Conclusions: Our results show that the RL approach improves image quality effectively, efficiently, and consistently across multiple spectral channels and has great potential in clinical applications.

Objective. In computer-assisted minimally invasive surgery, the intraoperative x-ray image is enhanced by overlapping it with a preoperative CT volume to improve visualization of vital anatomical structures. Therefore, accurate and robust 3D/2D registration of CT volume and x-ray image is highly desired in clinical practices. However, previous registration methods were prone to initial misalignments and struggled with local minima, leading to issues of low accuracy and vulnerability. Approach. To improve registration performance, we propose a novel CT/x-ray image registration agent (CT2X-IRA) within a task-driven deep reinforcement learning framework, which contains three key strategies: (1) a multi-scale-stride learning mechanism provides multi-scale feature representation and flexible action step size, establishing fast and globally optimal convergence of the registration task. (2) A domain adaptation module reduces the domain gap between the x-ray image and digitally reconstructed radiograph projected from the CT volume, decreasing the sensitivity and uncertainty of the similarity measurement. (3) A weighted reward function facilitates CT2X-IRA in searching for the optimal transformation parameters, improving the estimation accuracy of out-of-plane transformation parameters under large initial misalignments. Main results. We evaluate the proposed CT2X-IRA on both the public and private clinical datasets, achieving target registration errors of 2.13 mm and 2.33 mm with the computation time of 1.5 s and 1.1 s, respectively, showing an accurate and fast workflow for CT/x-ray image rigid registration. Significance. The proposed CT2X-IRA obtains the accurate and robust 3D/2D registration of CT and x-ray images, suggesting its potential significance in clinical applications., (© 2023 Institute of Physics and Engineering in Medicine.)

Combinatorial optimization problems such as Traveling Salesman Problem (TSP) have a wide range of real-world applications in transportation, logistics, manufacturing. It has always been a difficult problem to solve large-scale TSP problems quickly because of memory usage limitations. Recent research shows that the Transformer model is a promising approach. However, the Transformer has several severe problems that prevent it from quickly solving TSP combinatorial optimization problems, such as quadratic time complexity, especially quadratic space complexity, and the inherent limitations of the encoder and decoder itself. To address these issues, we developed a memory-efficient Transformer-based network model for TSP combinatorial optimization problems, termed Tspformer, with two distinctive characteristics: (1) a sampled scaled dot-product attention mechanism with O(Llog(L)) (L is the length of input sequences) time and space complexity, which is the most different between our work and other works. (2) due to the reduced space complexity, GPU/CPU memory usage is significantly reduced. Extensive experiments demonstrate that Tspformer significantly outperforms existing methods and provides a new solution to the TSP combinatorial optimization problems. Our Pytorch code will be publicly available on GitHub https://github.com/yhnju/tspFormer., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

Autonomous underwater vehicles (AUVs)-assisted mobile data collection in underwater wireless sensor networks (UWSNs) has received significant attention because of their mobility and flexibility. To satisfy the increasing demand of diverse application requirements for underwater data collection, such as time-sensitive data freshness, emergency event security as well as energy efficiency, in this paper, we propose a novel multi-modal AUV-assisted data collection scheme which integrates both acoustic and optical technologies and takes advantage of their complementary strengths in terms of communication distance and data rate. In this scheme, we consider the age of information (AoI) of the data packet, node transmission energy as well as energy consumption of the AUV movement, and we make a trade-off between them to retrieve data in a timely and reliable manner. To optimize these, we leverage a deep reinforcement learning (DRL) approach to find the optimal motion trajectory of AUV by selecting the suitable communication options. In addition to that, we also design an optimal angle steering algorithm for AUV navigation under different communication scenarios to reduce energy consumption further. We conduct extensive simulations to verify the effectiveness of the proposed scheme, and the results show that the proposed scheme can significantly reduce the weighted sum of AoI as well as energy consumption.

Airborne radars are susceptible to a large number of clutter, noise and variable jamming signals in the real environment, especially when faced with active main lobe jamming, as the waveform shortcut technology in the traditional regime can no longer meet the actual battlefield radar anti-jamming requirements. Therefore, it is necessary to study anti-main-lobe jamming techniques for airborne radars in complex environments to improve their battlefield survivability. In this paper, we propose an airborne radar waveform design method based on a deep reinforcement learning (DRL) algorithm under clutter and jamming conditions, after previous research on reinforcement-learning (RL)-based airborne radar anti-jamming waveform design methods that have improved the anti-jamming performance of airborne radars. The method uses a Markov decision process (MDP) to describe the complex operating environment of airborne radars, calculates the value of the radar anti-jamming waveform strategy under various jamming states using deep neural networks and designs the optimal anti-jamming waveform strategy for airborne radars based on the duelling double deep Q network (D3QN) algorithm. In addition, the method uses an iterative transformation method (ITM) to generate the time domain signals of the optimal waveform strategy. Simulation results show that the airborne radar waveform designed based on the deep reinforcement learning algorithm proposed in this paper improves the signal-to-jamming plus noise ratio (SJNR) by 2.08 dB and 3.03 dB, and target detection probability by 26.79% and 44.25%, respectively, compared with the waveform designed based on the reinforcement learning algorithm and the conventional linear frequency modulation (LFM) signal at a radar transmit power of 5 W. The airborne radar waveform design method proposed in this paper helps airborne radars to enhance anti-jamming performance in complex environments while further improving target detection performance.

Space situational awareness (SSA) is becoming increasingly challenging with the proliferation of resident space objects (RSOs), ranging from CubeSats to mega-constellations. Sensors within the United States Space Surveillance Network are tasked to repeatedly detect, characterize, and track these RSOs to retain custody and estimate their attitude. The majority of these sensors consist of ground-based sensors with a narrow field of view and must be slew at a finite rate from one RSO to another during observations. This results in a complex combinatorial problem that poses a major obstacle to the SSA sensor tasking problem. In this work, we successfully applied deep reinforcement learning (DRL) to overcome the curse of dimensionality and optimally task a ground-based sensor. We trained several DRL agents using proximal policy optimization and population-based training in a simulated SSA environment. The DRL agents outperformed myopic policies in both objective metrics of RSOs' state uncertainties and the number of unique RSOs observed over a 90-min observation window. The agents' robustness to changes in RSO orbital regimes, observation window length, observer's location, and sensor properties are also examined. The robustness of the DRL agents allows them to be applied to any arbitrary locations and scenarios.

In this paper, a cooperative search method for multiple UAVs is proposed to solve the problem of low efficiency of multi-UAV task execution by using a cooperative game with incomplete information. To improve search efficiency, CBBA (Consensus-Based Bundle Algorithm) is applied to designate the tasks area for each UAV. Then, Independent Deep Reinforcement Learning (IDRL) is used to solve Nash equilibrium to improve UAVs' collaborations. The proposed reward function is smartly developed to guide UAVs to fly along the path with higher reward value while avoiding the collisions between UAVs during flights. Finally, extensive experiments are carried out to compare our proposed method with other algorithms. Simulation results show that the proposed method can obtain more rewards in the same period of time as other algorithms.

Deep Reinforcement Learning (DRL) is increasingly applied in cyber-physical systems for automation tasks. It is important to record the developing trends in DRL's applications to help researchers overcome common problems using common solutions. This survey investigates trends seen within two applied settings: motor control tasks, and resource allocation tasks. The common problems include intractability of the action space, or state space, as well as hurdles associated with the prohibitive cost of training systems from scratch in the real-world. Real-world training data is sparse and difficult to derive and training in real-world can damage real-world learning systems. Researchers have provided a set of common as well as unique solutions. Tackling the problem of intractability, researchers have succeeded in guiding network training with handcrafted reward functions, auxiliary learning, and by simplifying the state or action spaces before performing transfer learning to more complex systems. Many state-of-the-art algorithms reformulate problems to use multi-agent or hierarchical learning to reduce the intractability of the state or action spaces for a single agent. Common solutions to the prohibitive cost of training include using benchmarks and simulations. This requires a shared feature space common to both simulation and the real world; without that you introduce what is known as the reality gap problem. This is the first survey, to our knowledge, that studies DRL as it is applied in the real world at this scope. It is our hope that the common solutions surveyed become common practice., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2022 Elsevier Ltd. All rights reserved.)

In an attempt to overcome the limitations of reward-driven representation learning in vision-based reinforcement learning (RL), an unsupervised learning framework referred to as the visual pretraining via contrastive predictive model (VPCPM) is proposed to learn the representations detached from the policy learning. Our method enables the convolutional encoder to perceive the underlying dynamics through a pair of forward and inverse models under the supervision of the contrastive loss, thus resulting in better representations. In experiments with a diverse set of vision control tasks, by initializing the encoders with VPCPM, the performance of state-of-the-art vision-based RL algorithms is significantly boosted, with 44% and 10% improvement for RAD and DrQ at 100 steps, respectively. In comparison to the prior unsupervised methods, the performance of VPCPM matches or outperforms all the baselines. We further demonstrate that the learned representations successfully generalize to the new tasks that share a similar observation and action space.

Effective exploration is one of the critical factors affecting performance in deep reinforcement learning. Agents acquire data to learn the optimal policy through exploration, and if it is not guaranteed, the data quality deteriorates, which leads to performance degradation. This study investigates the effect of initial entropy, which significantly influences exploration, especially in the early learning stage. The results of this study on tasks with discrete action space show that (1) low initial entropy increases the probability of learning failure, (2) the distributions of initial entropy for various tasks are biased towards low values that inhibit exploration, and (3) the initial entropy for discrete action space varies with both the initial weight and task, making it hard to control. We then devise a simple yet powerful learning strategy to deal with these limitations, namely, entropy-aware model initialization. The proposed algorithm aims to provide a model with high initial entropy to a deep reinforcement learning algorithm for effective exploration. Our experiments showed that the devised learning strategy significantly reduces learning failures and enhances performance, stability, and learning speed.

In order to solve the problem that the scheduling scheme cannot be updated in real time due to the dynamic change of node demand in material emergency dispatching, this article proposes a dynamic attention model based on improved gated recurrent unit. The dynamic codec framework is used to track the change of node demand to update the node information. The improved gated recurrent unit is embedded between codecs to improve the representation ability of the model. By weighted combination of the node information of the previous time, the current time and the initial time, a more representative node embedding is obtained. The results show that compared with the elitism-based immigrants ant colony optimization algorithm, the solution quality of the proposed model was improved by 27.89, 27.94, 28.09 and 28.12% when the problem scale is 10, 20, 50 and 100, respectively, which can effectively deal with the instability caused by the change of node demand, so as to minimize the cost of material distribution.

Meta-reinforcement learning (meta-RL) algorithms enable agents to adapt quickly to tasks from few samples in dynamic environments. Such a feat is achieved through dynamic representations in an agent's policy network (obtained via reasoning about task context, model parameter updates, or both). However, obtaining rich dynamic representations for fast adaptation beyond simple benchmark problems is challenging due to the burden placed on the policy network to accommodate different policies. This paper addresses the challenge by introducing neuromodulation as a modular component to augment a standard policy network that regulates neuronal activities in order to produce efficient dynamic representations for task adaptation. The proposed extension to the policy network is evaluated across multiple discrete and continuous control environments of increasing complexity. To prove the generality and benefits of the extension in meta-RL, the neuromodulated network was applied to two state-of-the-art meta-RL algorithms (CAVIA and PEARL). The result demonstrates that meta-RL augmented with neuromodulation produces significantly better result and richer dynamic representations in comparison to the baselines., Competing Interests: Declaration of Competing Interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Praveen K. Pilly and Nicholas A. Ketz are affiliated with HRL Laboratories LLC, both having pending patent applications in lifelong learning. Other authors declare no conflict of interest. Funding. This material is based upon work supported by the United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750-18-C0103. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA)., (Copyright © 2022 The Authors. Published by Elsevier Ltd.. All rights reserved.)

Underwater mapping with mobile robots has a wide range of applications, and good models are lacking for key parts of the problem, such as sensor behavior. The specific focus here is the huge environmental problem of underwater litter, in the context of the Horizon 2020 SeaClear project, where a team of robots is being developed to map and collect such litter. No reinforcement-learning solution to underwater mapping has been proposed thus far, even though the framework is well suited for robot control in unknown settings. As a key contribution, this paper therefore makes a first attempt to apply deep reinforcement learning (DRL) to this problem by exploiting two state-of-the-art algorithms and making a number of mapping-specific improvements. Since DRL often requires millions of samples to work, a fast simulator is required, and another key contribution is to develop such a simulator from scratch for mapping seafloor objects with an underwater vehicle possessing a sonar-like sensor. Extensive numerical experiments on a range of algorithm variants show that the best DRL method collects litter significantly faster than a baseline lawn mower trajectory.

Network Slicing and Deep Reinforcement Learning (DRL) are vital enablers for achieving 5G and 6G networks. A 5G/6G network can comprise various network slices from unique or multiple tenants. Network providers need to perform intelligent and efficient resource management to offer slices that meet the quality of service and quality of experience requirements of 5G/6G use cases. Resource management is far from being a straightforward task. This task demands complex and dynamic mechanisms to control admission and allocate, schedule, and orchestrate resources. Intelligent and effective resource management needs to predict the services' demand coming from tenants (each tenant with multiple network slice requests) and achieve autonomous behavior of slices. This paper identifies the relevant phases for resource management in network slicing and analyzes approaches using reinforcement learning (RL) and DRL algorithms for realizing each phase autonomously. We analyze the approaches according to the optimization objective, the network focus (core, radio access, edge, and end-to-end network), the space of states, the space of actions, the algorithms, the structure of deep neural networks, the exploration-exploitation method, and the use cases (or vertical applications). We also provide research directions related to RL/DRL-based network slice resource management.

Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles' (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs' maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal.

Deep reinforcement learning (DRL) augments the reinforcement learning framework, which learns a sequence of actions that maximizes the expected reward, with the representative power of deep neural networks. Recent works have demonstrated the great potential of DRL in medicine and healthcare. This paper presents a literature review of DRL in medical imaging. We start with a comprehensive tutorial of DRL, including the latest model-free and model-based algorithms. We then cover existing DRL applications for medical imaging, which are roughly divided into three main categories: (i) parametric medical image analysis tasks including landmark detection, object/lesion detection, registration, and view plane localization; (ii) solving optimization tasks including hyperparameter tuning, selecting augmentation strategies, and neural architecture search; and (iii) miscellaneous applications including surgical gesture segmentation, personalized mobile health intervention, and computational model personalization. The paper concludes with discussions of future perspectives., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2021 Elsevier B.V. All rights reserved.)

This paper proposes a new robust update rule of target network for deep reinforcement learning (DRL), to replace the conventional update rule, given as an exponential moving average. The target network is for smoothly generating the reference signals for a main network in DRL, thereby reducing learning variance. The problem with its conventional update rule is the fact that all the parameters are smoothly copied with the same speed from the main network, even when some of them are trying to update toward the wrong directions. This behavior increases the risk of generating the wrong reference signals. Although slowing down the overall update speed is a naive way to mitigate wrong updates, it would decrease learning speed. To robustly update the parameters while keeping learning speed, a t-soft update method, which is inspired by Student-t distribution, is derived with reference to the analogy between the exponential moving average and the normal distribution. Through the analysis of the derived t-soft update, we show that it takes over the properties of the Student-t distribution. Specifically, with a heavy-tailed property of the Student-t distribution, the t-soft update automatically excludes extreme updates that differ from past experiences. In addition, when the updates are similar to the past experiences, it can mitigate the learning delay by increasing the amount of updates. In PyBullet robotics simulations for DRL, an online actor-critic algorithm with the t-soft update outperformed the conventional methods in terms of the obtained return and/or its variance. From the training process by the t-soft update, we found that the t-soft update is globally consistent with the standard soft update, and the update rates are locally adjusted for acceleration or suppression., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2020 Elsevier Ltd. All rights reserved.)

In many reinforcement learning tasks, the goal is to learn a policy to manipulate an agent, whose design is fixed, to maximize some notion of cumulative reward. The design of the agent's physical structure is rarely optimized for the task at hand. In this work, we explore the possibility of learning a version of the agent's design that is better suited for its task, jointly with the policy. We propose an alteration to the popular OpenAI Gym framework, where we parameterize parts of an environment, and allow an agent to jointly learn to modify these environment parameters along with its policy. We demonstrate that an agent can learn a better structure of its body that is not only better suited for the task, but also facilitates policy learning. Joint learning of policy and structure may even uncover design principles that are useful for assisted-design applications.

Intelligent game adversarial algorithms not only make full use of the portrayal accuracy of the game model, but also solve the equilibrium solution through the powerful computational ability of neural network and the trial-and-error mechanism of reinforcement learning, which makes the intelligent game adversarial algorithms achieve good results in many fields. Through the multi-intelligence body game learning, multi-intelligence body game reinforcement learning and multi-intelligence body game deep reinforcement learning three levels of intelligent game confrontation algorithm is systematically sorted out, and the corresponding mapping with the intelligence field of work, demonstrates the feasibility and necessity of intelligent game confrontation algorithm in the field of intelligence, and finally gives the specific application of the intelligent game confrontation algorithm in the field of intelligence and the effective measures of the follow-up to improve the quality and efficiency. Finally, it gives the specific application of intelligent game confrontation algorithm in the field of intelligence, as well as the effective measures to improve the quality and efficiency. [ABSTRACT FROM AUTHOR]

The application of deep reinforcement learning (DRL) techniques in intelligent transportation systems garners significant attention. In this field, reward function design is a crucial factor for DRL performance. Current research predominantly relies on a trial-and-error approach for designing reward functions, lacking mathematical support and necessitating extensive empirical experimentation. Our research uses vehicle velocity control as a case study, build training and test sets, and develop a DRL framework for speed control. This framework examines both single-objective and multi-objective optimization in reward function designs. In single-objective optimization, we introduce "expected optimal velocity" as an optimization objective and analyze how different reward functions affect performance, providing a mathematical perspective on optimizing reward functions. In multi-objective optimization, we propose a reward function design paradigm and validate its effectiveness. Our findings offer a versatile framework and theoretical guidance for developing and optimizing reward functions in DRL, particularly for intelligent transportation systems. [ABSTRACT FROM AUTHOR]

This study introduces a Deep Reinforcement Learning approach (DRL-MD) aimed at optimizing the deployment of mitigations to minimize redundancy while ensuring effective defense against cyberattacks. DRL-MD initially enhances ATT &CK (Adversarial Tactics, Techniques, and Common Knowledge) to underscore the formal relationships between attacks and defenses. Over the enhanced ATT &CK, DRL-MD then operates in two phases: (1) Estimating Node Importance: DRL-MD proposes a model to estimate the importance of deployed nodes in the network, prioritizing mitigation deployment locations for better evaluation of mitigation effectiveness; and (2) Optimizing Mitigation Deployment: A Soft Actor-Critic algorithm finds the optimal mitigation deployment policy through multi-objective optimization of the importance of deployed nodes, the effectiveness of mitigations in preventing cyberattacks, vulnerability repair, and deployment cost. A case study with DRL-MD against the state-of-the-art counterparts has been performed considering the WannaCry threat, and results indicate that: (1) DRL-MD performs the best with 6.4–11% decrease in deployment cost; and (2) DRL-MD can significantly reduce redundancy in mitigation deployments, which partially benefits from the enhanced ATT &CK model. Overall, a comprehensive solution of mitigation deployment has been fostered to significantly lower the redundancy with more effective defenses against cyberattacks sustained. [ABSTRACT FROM AUTHOR]

In the pursuit of advancing robotic navigation, a notable enhancement to the distributed distributional deterministic policy gradients (D4PG) algorithm is presented. Cutting-edge methodologies, including the implicit quantile network (IQN) critic, prioritized experience replay (PER), and N-step Bootstrapping, are integrated to achieve significantly improved learning efficiency and performance in autonomous systems. The enhanced algorithm is implemented using PyTorch and systematically evaluated within the robo-gym environment, demonstrating substantial advancements over traditional deep reinforcement learning methods. The results highlight the augmented D4PG framework's superior capability in navigating complex control challenges and suggest promising avenues for further exploration in advanced robotics and autonomous systems. [ABSTRACT FROM AUTHOR]

In view of the problems such as low exploration efficiency, slow convergence speed or even non - convergence of deep reinforcement learning algorithm in the motion planning task of robot arm under the requirement of high dimensional state space and high precision, this study introduces asynchronous advantage mechanism based on SAC (Soft Actor - Critic) algorithm, and proposes an AA - SAC(Asynchronous Advantage Soft Actor - Critic) algorithm integrating asynchronous advantage. This algorithm replaces the original V network with a Qtarget network, which effectively reduces the variance of the Q network. The n independent processes can be trained in parallel, which improves the training efficiency. The study also divides the experience playback pool of the AA - SAC algorithm into two parts, store and sample high - quality empirical data separately to improve the utilization of effective empirical data. The simulation results show that AA - SAC algorithm has the best performance in convergence speed, success rate and stability. Compared with the SAC algorithm, the convergence time of AA - SAC algorithm is 3 000 rounds earlier. After convergence, the success rate of AA - SAC algorithm reaches 96%, which is 6% higher than SAC algorithm and 26% higher than DDPG(Deep Deterministic Policy Gradient) algorithm. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Shanghai Jiao Tong University (1006-2467) is the property of Journal of Shanghai Jiao Tong University Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

On-policy deep reinforcement learning (DRL) has the inherent advantage of using multi-step interaction data for policy learning. However, on-policy DRL still faces challenges in improving the sample efficiency of policy evaluations. Therefore, we propose a multi-step on-policy DRL method assisted by off-policy policy evaluation (abbreviated as MSOAO), whichs integrates on-policy and off-policy policy evaluations and belongs to a new type of DRL method. We propose a low-pass filtering algorithm for state-values to perform off-policy policy evaluation and make it efficiently assist on-policy policy evaluation. The filtered state-values and the multi-step interaction data are used as the input of the V-trace algorithm. Then, the state-value function is learned by simultaneously approximating the target state-values obtained from the V-trace output and the action-values of the current policy. The action-value function is learned by using the one-step bootstrapping algorithm to approximate the target action-values obtained from the V-trace output. Extensive evaluation results indicate that MSOAO outperformed the performance of state-of-the-art on-policy DRL algorithms, and the simultaneous learning of the state-value function and the action-value function in MSOAO can promote each other, thus improving the learning capability of the algorithm. [ABSTRACT FROM AUTHOR]

It is gradually becoming popular to use multiple cellular-connected UAVs as inspectors to fulfill automatic surveillance and monitoring. However, in actual situations, UAVs should respond to several service requests from different hotspots, whilst the requests usually present randomness in the arrival time, data amount, and the concurrency. This paper proposes a dynamic dual-UAV response policy for multi-hotspot services based on single-agent deep Q-learning, where the UAVs controlled by a ground base station can be dispatched automatically to hotspots and then send videos back. First, this issue is formulated as an optimization problem, whose goal is to maximize the number of successfully served requests with the constraints of both the UAV's energy limit and request waiting time. Second, a reward function based on service completion is designed to overcome the potential challenges posed by the delay reward. Finally, a simulation was conducted, comparing the conventional time priority algorithm and distance priority algorithm, respectively, to the proposed algorithm. The results illustrate that the proposed algorithm can achieve one more response than the others under different service densities, with the lowest failure number and appropriate average waiting time. This method can give a technical solution for the joint communication-and-control problem of multiple UAVs within complex situations. [ABSTRACT FROM AUTHOR]

This paper presents the development of a teaching approach for Reinforcement Learning (RL) for students at the Faculty of Electrical Engineering, University of Ljubljana. The approach is designed to introduce students to the basic concepts, approaches, and algorithms of RL through examples and experiments in both simulation environments and on a real robot. The approach includes practical programs written in Python and presents various RL algorithms. The Q‐learning algorithm is introduced and a deep Q network is implemented to introduce the use of neural networks in deep RL. The software is user‐friendly and allows easy modification of learning parameters, reward functions, and algorithms. The approach was tested successfully on a Franka Emika Panda robot, where the robot manipulator learned to move to a randomly generated target position, shoot a real ball into the goal, and push various objects into target position. The goal of the presented teaching approach is to serve as a study aid for future generations of students of robotics to help them better understand the basic concepts of RL and apply them to a wide variety of problems. [ABSTRACT FROM AUTHOR]

Distribution network reconfiguration (DNR) is used by utilities to enhance power system performance in various ways, such as reducing line losses. Conventional DNR algorithms rely on accurate values of network parameters and lack scalability and optimality. To tackle these issues, a new data-driven algorithm based on reinforcement learning is developed for DNR in this paper. The proposed algorithm comprises two main parts. The first part, named action-space sampling, aims at analyzing the network structure, finding all feasible reconfiguration actions, and reducing the size of the action space to only the most optimal actions. In the second part, deep Q-learning (DQN) and dueling DQN methods are used to train an agent to take the best switching actions according to the switch states and loads of the system. The results show that both DQN and dueling DQN are effective in reducing system losses through grid reconfiguration. The proposed methods have faster execution time compared to the conventional methods and are more scalable. [ABSTRACT FROM AUTHOR]

Treatment planning for chronic diseases is a critical task in medical artificial intelligence, particularly in traditional Chinese medicine (TCM). However, generating optimized sequential treatment strategies for patients with chronic diseases in different clinical encounters remains a challenging issue that requires further exploration. In this study, we proposed a TCM herbal prescription planning framework based on deep reinforcement learning for chronic disease treatment (PrescDRL). PrescDRL is a sequential herbal prescription optimization model that focuses on long-term effectiveness rather than achieving maximum reward at every step, thereby ensuring better patient outcomes. We constructed a high-quality benchmark dataset for sequential diagnosis and treatment of diabetes and evaluated PrescDRL against this benchmark. Our results showed that PrescDRL achieved a higher curative effect, with the single-step reward improving by 117% and 153% compared to doctors. Furthermore, PrescDRL outperformed the benchmark in prescription prediction, with precision improving by 40.5% and recall improving by 63%. Overall, our study demonstrates the potential of using artificial intelligence to improve clinical intelligent diagnosis and treatment in TCM. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) has made significant strides in addressing tasks with high-dimensional continuous action spaces. However, the field still faces the challenges of low sample utilization and insufficient exploration-exploitation balance, limiting the generalizability of algorithms across different environments. To effectively improve sample utilization, optimize the exploration-exploitation balance, and achieve higher rewards in tasks, this paper designs the selective priority timing entropy (SPTE) algorithm. Subsequently, selective prioritized experience replay (SPER) is proposed, which employs frequent replay of multiframe memories to enhance sample utilization and improve the stability of policy updates. Additionally, the temporal advantage with decay (TAD) method introduces a decay factor to help adjust the weights of the variance and bias, thereby reducing estimation errors. The reward mechanism is augmented with multientropy (ME) for entropy-regularized training, achieving a balance between information exploration and exploitation. Finally, experimental testing on the challenging Arcade platform demonstrated that the SPTE algorithm surpasses the average testing level of human players by 104.936%. Furthermore, compared to other algorithms, SPTE achieves an average score increase of over 32.75%, and it consistently outperforms the compared methods in more than 60% of tasks, indicating its strong adaptability and robustness. The implementation process of SPTE [ABSTRACT FROM AUTHOR]

Human Activity Recognition (HAR) is a rapidly evolving field with the potential to revolutionise how we monitor and understand human behaviour. This survey paper provides a comprehensive overview of the state-of-the-art in HAR, specifically focusing on recent techniques such as multimodal techniques, Deep Reinforcement Learning and large language models. It explores the diverse range of human activities and the sensor technologies employed for data collection. It then reviews novel algorithms used for Human Activity Recognition with emphasis on multimodality, Deep Reinforcement Learning and large language models. It gives an overview of multimodal datasets with physiological data. It also delves into the applications of HAR in healthcare. Additionally, the survey discusses the challenges and future directions in this exciting field, highlighting the need for continued research and development to fully realise the potential of HAR in various real-world applications. [ABSTRACT FROM AUTHOR]

Reinforcement learning algorithms show significant variations in performance across different environments. Optimization for reinforcement learning thus becomes the major research task since the instability and unpredictability of the reinforcement learning algorithms have consistently hindered their generalization capabilities. In this study, we address this issue by optimizing the algorithm itself rather than environment-specific optimizations. We start by tackling the uncertainty caused by the mutual influence of original action interferences, aiming to enhance the overall performance. The Phasic Parallel-Network Policy (PPP), which is a deep reinforcement learning framework. It diverges from the traditional policy actor-critic method by grouping the action space based on action correlations. The PPP incorporates parallel network structures and combines network optimization strategies. With the assistance of the value network, the training process is divided into different specific stages, namely the Extra-group Policy Phase and the Inter-group Optimization Phase. PPP breaks through the traditional unit learning structure. The experimental results indicate that it not only optimizes training effectiveness but also reduces training steps, enhances sample efficiency, and significantly improves stability and generalization. [ABSTRACT FROM AUTHOR]

This paper investigates the orbital pursuit strategy for spacecraft swarms, where both pursuers and evaders possess limited delta-v maneuvering capabilities. Given that swarms can consist of numerous members, ranging from dozens to thousands, the swarm-to-swarm problem has significant coupling effects that cannot be solved by merely superimposing one-to-one or few-to-few problems. Otherwise, one would cause a dimension explosion in decision-making space. To avoid this issue, the swarm orbital pursuit problem is divided into two coupled problems: swarm target allocation and orbital maneuver. First, considering the relative orbit state between pursuers and evaders and estimated orbital maneuver costs of pursuit, the swarm target allocation problem is formulated into three coupled sub-problems, i.e., target allocation order, target allocation for sub-swarms, and target allocation for individuals. These sub-problems are solved by the presented double-layer contract net protocol algorithm to maximize task completion efficiency. Then, the orbital maneuver problem is decomposed into two coupled problems to maximize pursuit benefits, i.e., long-range rendezvous, and close-distance game maneuvers. For the long-range rendezvous, the optimal Lambert algorithm is developed to efficiently obtain the discrete optimal pursuit time and corresponding orbital transfer maneuver. To address close-distance games more proficiently, a hybrid orbital maneuver algorithm, combining the proximal policy optimization and the Lambert algorithm is proposed. Finally, comparison simulations verified the effectiveness and superiority of the proposed method. • Satellite swarm pursuit problem is transformed to the coupled problem with swarm target allocation and orbital maneuvers. • Swarm target allocation is formulated as a coupled problem with target order, allocations for subgroups and individuals. • An improved DLCNP algorithm is proposed to pursue a reasonable target allocation solution for higher task efficiency. • Optimal Lambert algorithm is developed to obtain the optimal maneuver strategyfor long-range rendezvous. • A hybrid algorithm combining PPO and the Lambert algorithm is presented to get closed-loop control for close-distance games. [ABSTRACT FROM AUTHOR]

UAV technology is advancing rapidly, and variable-sweep wing UAVs are increasingly valuable because they can adapt to different flight conditions. However, conventional control methods often struggle with managing continuous action spaces and responding to dynamic environments, making them inadequate for complex multi-UAV cooperative formation control tasks. To address these challenges, this study presents an innovative framework that integrates dynamic modeling with morphing control, optimized by the multi-agent deep deterministic policy gradient for two-sweep control (MADDPG–VSC) algorithm. This approach enables real-time sweep angle adjustments based on current flight states, significantly enhancing aerodynamic efficiency and overall UAV performance. The precise motion state model for wing morphing developed in this study underpins the MADDPG–VSC algorithm's implementation. The algorithm not only optimizes multi-UAV formation control efficiency but also improves obstacle avoidance, attitude stability, and decision-making speed. Extensive simulations and real-world experiments consistently demonstrate that the proposed algorithm outperforms contemporary methods in multiple aspects, underscoring its practical applicability in complex aerial systems. This study advances control technologies for morphing-wing UAV formation and offers new insights into multi-agent cooperative control, with substantial potential for real-world applications. [ABSTRACT FROM AUTHOR]

When multiple objects are positioned close together or stacked, pre-grasp operations such as pushing objects can be used to create space for the grasp, thereby improving the grasping success rate. This study develops a model based on a deep Q-learning network architecture and introduces a fully convolutional network to accurately identify pixels in the workspace image that correspond to target locations for exploration. In addition, this study incorporates image masking to limit the exploration area of the robotic arm, ensuring that the agent consistently explores regions containing objects. This approach effectively addresses the sparse reward problem and improves the convergence rate of the model. Experimental results from both simulated and real-world environments show that the proposed method accelerates the learning of effective grasping strategies. When image masking is applied, the success rate in the grasping task reaches 80% after 600 iterations. The time required to reach 80% success rate is 25% shorter when image masking is used compared to when it is not used. The main finding of this study is the direct integration of image masking technique with a deep reinforcement learning (DRL) algorithm, which offers significant advancement in robotic arm control. Furthermore, this study shows that image masking technique can substantially reduce training time and improve the object grasping success rate. This innovation enables the robotic arm to better adapt to scenarios that conventional DRL methods cannot handle, thereby improving training efficiency and performance in complex and dynamic industrial applications. [ABSTRACT FROM AUTHOR]

The growing interest in the flexibility and operational capabilities of soft manipulators in confined spaces emphasizes the need for precise modeling and accurate motion control. Conventional control methods encounter difficulties in modeling and involve intricate computations. This work introduces a novel deep reinforcement learning (DRL) control algorithm based on neural network modeling. Using the Whale Optimization Algorithm, an approximate dynamic model for the soft manipulator is established. The twin delayed deterministic policy gradient is employed for DRL control. Domain randomization is applied during pretraining in a simulated environment. The algorithm addresses issues related to dependency on measurement data quality and redundant mappings, outperforming other methods by 8–15 mm in control accuracy. The trained DRL controller achieves precise trajectory tracking within the soft manipulator's task space, enabling successful grasping tasks in various complex environments, including pipelines and other narrow spaces. Experimental results confirm the autonomy of our controller in performing these tasks without human intervention. [ABSTRACT FROM AUTHOR]

The threat to naval platforms from missile systems is increasing due to recent advancements in radar seeker technology, which have significantly enhanced the accuracy and effectiveness of missile targeting. In scenarios where a naval platform with limited maneuverability faces salvo attacks, the importance of an effective defense strategy becomes crucial to ensuring the protection of the platform. In this study, we present a multi-agent reinforcement learning-based decoy deployment approach that employs six decoys to increase the survival likelihood of a naval platform against salvo missile strikes. Our approach entails separating the decoys into two clusters, each consisting of three decoys. Subsequently, every cluster is allocated to a related missile threat. This is accomplished by training the decoys with the multi-agent deep reinforcement learning algorithm. To compare the proposed approach across different algorithms, we use two distinct algorithms to train the decoys; multi-agent deep deterministic policy gradient (MADDPG) and multi-agent twin-delayed deep deterministic policy gradient (MATD3). Following training, the decoys learn to form groups and establish effective formation configurations within each group to ensure optimal coordination. We assess the proposed decoy deployment strategy using parameters including decoy deployment angle and maximum decoy speed. Our findings indicate that decoys positioned on the same side outperform those positioned on different sides relative to the target platform. In general, MATD3 performs slightly better than MADDPG. Decoys trained with MATD3 succeed in more successful formation configurations than those trained with the MADDPG method, which accounts for this enhancement. [ABSTRACT FROM AUTHOR]

In human error–based learning, the size and direction of a scalar error (i.e., the "directed error") are used to update future actions. Modern deep reinforcement learning (RL) methods perform a similar operation but in terms of scalar rewards. Despite this similarity, the relationship between action updates of deep RL and human error–based learning has yet to be investigated. Here, we systematically compare the three major families of deep RL algorithms to human error–based learning. We show that all three deep RL approaches are qualitatively different from human error–based learning, as assessed by a mirror-reversal perturbation experiment. To bridge this gap, we developed an alternative deep RL algorithm inspired by human error–based learning, model-based deterministic policy gradients (MB-DPG). We showed that MB-DPG captures human error–based learning under mirror-reversal and rotational perturbations and that MB-DPG learns faster than canonical model-free algorithms on complex arm-based reaching tasks, while being more robust to (forward) model misspecification than model-based RL. [ABSTRACT FROM AUTHOR]

This paper tackles the intricate challenge of autonomous navigation for Unmanned Aerial Vehicles (UAVs) through dynamically changing environments. We focus on a sophisticated Deep Reinforcement Learning (DRL) approach using the Soft Actor-Critic (SAC) algorithm, optimized for UAV path planning within a continuous action space. This methodology leverages environmental image data to enhance the precision of flight maneuvers and effective obstacle avoidance. Our approach, validated through extensive simulations in Gazebo and field tests, demonstrates the algorithm’s efficacy in enabling UAVs to adeptly navigate obstacles using depth maps. The study further explores the robustness of the SAC algorithm by comparing it with traditional DRL methods, emphasizing its superior performance in real-world applications. This research contributes significantly to advancing UAV technology, particularly in autonomous motion planning, by integrating cutting-edge machine learning techniques. The video link is: https://www.youtube.com/watch?v=Nd_aMzejNXY. [ABSTRACT FROM AUTHOR]

In this paper, a UAV swarm engagement scenario is considered, where the defender swarm tries to intercept the attacker swarm cooperatively to prevent it from entering into the target area. Different from the previous attack–defense strategy on deep reinforcement learning, we remove the assumption that the swarms have perfect knowledge of situation information of both sides and the environment. On this basis, a double experience pool strategic interaction Q-learning (DEP-SIQ) swarm attack and defense algorithm is proposed, which makes the dimension of the network input reduced and the UAV with the same task use the same network. The algorithm sets up different experience pools for the attacker and the defender, respectively. During training, both sides take samples from their own experience pool to train their own network. The estimation reward of the UAV is decomposed into the sum of the interaction reward values with other friendly UAVs, which is effectively suitable for large-scale swarm. Simulation experiments shows the feasibility of the proposed algorithm. Compare with other algorithms, the proposed DEP-SIQ algorithm has a faster learning efficiency, higher win rate and better applicability. [ABSTRACT FROM AUTHOR]

This study focuses on unspecified dynamic seru scheduling problems with resource constraints (UDSS-R) in seru production system (SPS). A mixed integer linear programming model is formulated to minimize the makespan, which is solved sequentially from both allocation and scheduling perspectives by a strip-packing constructive algorithm (SPCA) with deep reinforcement learning (DRL). The training samples are trained by the DRL model, and the reward values obtained are calculated by SPCA to train the network so that the agent can find a better solution. The output of DRL is the scheduling order of jobs in serus, while the solution of UDSS-R is solved by SPCA. Finally, a set of test instances are generated to conduct computational experiments with different instance scales for the DRL-SPCA, and the results confirm the effectiveness of proposed DRL-SPCA in solving UDSS-R with more outstanding performance in terms of solution quality and efficiency, across three data scales (10 serus × 100 jobs, 20 serus × 250 jobs, and 30 serus × 400 jobs), compared with GA and SAA, the Avg. RPD of DRL-SPCA decreased by 9.93% and 7.56%, 13.36% and 10.72%, and 9.09% and 7.08%, respectively. In addition, the Avg. CPU time was reduced by 29.53% and 27.93%, 57.48% and 57.04%, and 61.73% and 61.76%, respectively. [ABSTRACT FROM AUTHOR]

In the deep reinforcement learning method, the intrinsic curiosity model (ICM) guides the agent to obtain the opportunity to learn unknown strategies in the sparse reward environment, but the curiosity reward is a state difference value, which will make the agent pay too much attention to the exploration of new states, then could be the problem of blind exploration arises. To solve the above problem, this paper proposed an intrinsic curiosity model algorithm based on knowledge distillation (KD-ICM). Firstly, it introduced the method of knowledge distillation to make the agent acquire more abundant environmental information and strategy knowledge in a short time and accelerate the learning process. Secondly, by pre-training teachers' neural network model to guide the forward network to obtain a forward network model with higher accuracy and performance, reduced the blind exploration of agents. It designed two different simulation experiments on the Unity simulation platform for comparison. The experiments show that in the complex simulation task environment, the average reward of KDICM algorithm is 136% higher than that of ICM, and the optimal action probability is 13.47% higher than that of ICM. Both the exploration performance of the agent and the exploration quality can be improved, and it verifies the feasibility of the algorithm. [ABSTRACT FROM AUTHOR]

We present a novel ramp metering algorithm that incorporates multi-agent deep reinforcement learning (DRL) techniques, which utilizes monitoring data from loop detectors. Our proposed approach employed a multi-agent DRL framework to generate optimized ramp metering schedules for each ramp meter in real-time, enhancing the operational efficiency of urban freeways with less investment. To simplify the implementation and training of the algorithm, we developed a simulation platform based on SUMO microscopic traffic simulator. We conducted a series of simulation experiments, including local and coordinated ramp metering scenarios with various traffic demands profiles. The simulation results indicate that the proposed DRL-based algorithm outperforms the state-of-the-practice ramp metering methods, considering a comprehensive evaluation index encompassing mainstream speed at the bottleneck and queue length on ramp. Additionally, the method exhibits robustness, scalability, and the potential for further improvement through online learning during implementation. [ABSTRACT FROM AUTHOR]

For over forty years, TCP has been the main protocol for transporting data on the Internet. To improve congestion control algorithms (CCAs), delay bounding algorithms such as Vegas, FAST, BBR, PCC, and Copa have been developed. However, despite being designed to ensure fairness between data flows, these CCAs can still lead to unfairness and, in some cases, even cause data flow starvation in WiFi networks under certain conditions. We propose a new CCA switching solution that works with existing TCP and WiFi standards. This solution is offline and uses Deep Reinforcement Learning (DRL) trained on features such as noncongestive delay variations to predict and prevent extreme unfairness and starvation. Our DRL-driven approach allows for dynamic and efficient CCA switching. We have tested our design preliminarily in realistic datasets, ensuring that they support both fairness and efficiency over WiFi networks, which requires further investigation and extensive evaluation before online deployment. [ABSTRACT FROM AUTHOR]

This paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS. [ABSTRACT FROM AUTHOR]